Code Editing in Copilot Arena

As an initial set of models, we selected 6 of the best models across multiple model providers that include both open, code-specific, and commercial models. To ensure a fair comparison between models, we do the following…

• We randomize whether models appear at the left or right panel along with which models are paired for each battle.
• We show both completions at the same time. This means that a faster model completion needs to wait for the slower model.
• We set the same max number of output tokens, input tokens, top-p, and temperature (unless specified by the model provider).

^ Table 1. Elo ratings and median latency of six popular models based on over 3K votes. We color rows based on tiers determined by confidence intervals. Each model has at least 1K votes.

Table 1 presents the current code completion leaderboard and stratifies them into tiers. Here are our main takeaways:

• The clear winner is Claude 3.5 Sonnet in terms of Arena Score.
• While some of the models in the middle fluctuate, we have generally observed that Llama-3.1-405b is the least preferred by users by far of the models we evaluate, losing a majority of the time to all other models (see Figure 2).

• We inspect whether response token length per model is correlated with preferences. Interestingly, we tend to see that people tend to prefer shorter responses. This is the opposite effect compared to what has been observed in prior work where people tend to prefer longer responses. This may however be correlated with model quality.

We follow the same leaderboard computation as the latest version of Chatbot Arena, which is based on learning Bradley-Terry coefficients that minimize loss when predicting whether one model will beat the other. Please check out this blog post for a more in-depth description.

For general information about how people use Copilot Arena, check out the first blogpost. Here, we will focus on code edit usage.

How long are the prompts that people write? We find that the median prompt length is 34 characters and mean is 139 characters. Most are fairly short and thus depend on the context that is highlighted. In comparison to the chat messages sent by users in Chatbot Arena, user prompts for inline edits tend to be much shorter. The model must instead mostly focus on test the model’s ability to infer user goals based on the context (e.g., the highlighted code snippet).

What context lengths are we looking at? We look at the distribution of code-to-edit token lengths, as computed by the number of highlighted tokens. The median is 138 tokens and the mean is 647 tokens. While there are some outliers, this indicates that most people are highlighting targeted portions of code for edits as this is much shorter than the full file length which is typically closer to 4.5k tokens on average.

What kind of edits are people trying to make? We find users write prompts for code edits in multiple languages, predominantly English but also in Russian, Chinese, and Spanish. Users typically write prompts using informal language and the prompts are typically directed towards addressing a specific goal. The distribution can be found in Figure 5.

The main categories include:

1. Resolve errors
3. Optimize code
5. Write code or build on existing code
7. Code translation
9. Test code
11. Styling and formatting
13. Documentation and explanation

We’re still actively collecting votes for code edits and will continue with deeper analysis in the future. We’re also looking into evaluating methods other than code completions and code edits.

In general, we are always looking to improve Copilot Arena. Ping us to get involved!