Chatbot Arena Categories

Our newest category, creative writing, evaluates a model’s ability to craft original, imaginative, and emotionally resonant content. Unlike straightforward writing tasks, creative writing prompts involve originality, artistic expression, and often, a different kind of thinking compared to more technical prompts. For instance, while “write a thank-you email” wouldn’t qualify as creative writing, “write a humorously sarcastic thank-you email” would. The category encompasses traditional creative formats (stories, poems, lyrics) and extends to areas like joke creation, meme explanation, and responses to philosophical questions that can’t be answered through simple fact-checking.

We note that this category is in a sense, orthogonal, to our previously introduced categories, such as Math and Coding. As such, we find interesting differences in rankings when looking at creative writing. Studying categories like creativity is important— the ranking signal on this type of prompts are innately dependent on human preference, and are difficult to obtain from traditional close-ended benchmarks without human-in-the-loop.

So, how is the creative writing category created? It all starts with the definition. A category can be thought of as a grouping of prompts based on the common skills required to respond effectively. Categories can be well defined (e.g. prompts longer than 500 tokens) or more ambiguous. One prompt could also be classified with multiple categories. Consider the “creative” category— how can we even define prompts that require creativity? In the following sections, we will walk you through how we construct a category in Chatbot Arena.

First, we define each category with a checklist. If enough items in the checklist are satisfied, the prompt is labeled as an element of that category. For “Creative Writing” that checklist looks like this:

What qualifies as creative writing?

• Prompts requiring originality and imagination
• Tasks involving emotional or artistic expression
• Requests for unique perspectives or interpretative responses
• Writing that goes beyond factual reporting or analysis

We include example creative writing prompts in the Appendix link. Additionally we also detail the definitions of other categories, including Math, Hard Prompt, Instruction-Following, Code, and various languages in the Appendix link.

Defining categories is one thing—deploying them at scale is another. With over 2 million prompts, category classification cannot be done by hand. While some categories (eg. Language, Code) can use heuristic algorithms for fast classification, others (eg. Creative Writing, Hard Prompt) cannot. For categories like Creative Writing, we build a fast and robust category classification system using an LLM. We prompt the LLM with the category checklist, and judge whether the prompt satisfies the checklist.

Of course, it is crucial to verify whether the classifications made by the LLM judge are within the desired specifications. To do this we check the classification performance on a wide sample of Chatbot Arena prompts. As a whole, we call this auto-evaluation pipeline Label Bench, which we define in detail below.

The Label Bench framework is as follows:

1. Initial Prompt Design: An prototype system prompt is carefully crafted based on a determined definition of the category. Extensive manual testing should be conducted to ensure the system prompt is high-quality.
2. Ground Truth Labeling: Random sample 2000-5000 battles from Chatbot Arena. Employing one or multiple strong LLMs (eg. Claude 3.5 Sonnet) to annotate ground truth labels for each battle. A manual check can be conducted to ensure reasonable accuracy.
3. Optimization: Employ a smaller, open-source model for classification. Remove any COT in the system prompt for additional inference speed up. Verification: Evaluate the smaller classifier against the ground truth labels on the sample battles. We typically iterate until reaching a high precision and recall.
4. Deploy: The best classifier and optimal system prompt are chosen to label the entire Chatbot Arena dataset.

For example, when developing our new creative writing category using the Label Bench framework, we found that with no CoT, Llama-3.1-70B-Instruct classified the prompts with 96.1% accuracy (precision: 66.7%, recall: 96.6%) with respect to high-quality labels generated by GPT-4o-mini with CoT. We also cross check this classifier against labels generated by Claude 3.5 Sonnet with COT. After ensuring its performance, we deployed this scalable LLM for the final labeling of our 2 million battle prompts.

After data labeling, we can compute the Chatbot Arena leaderboard only on the subset of Arena battles labeled as belonging to the category in order to produce a category-specific leaderboard.

With the groundwork of category creation completed, we can now delve into the meaningful results: statistical trends, categorical correlations, and insights into models’ strengths and weaknesses.

Are people asking harder questions as LLMs become better? We found the answer is yes! The percentage of Hard Prompts, Instruction-Following, Coding, and Math prompts has been increasing since Chatbot Arena’s launch back in April 2023.

The data also reveals some interesting seasonal variations in prompt complexity. For example, we observe an increase in the percentage of hard prompts during the school year, when students may be asking difficult homework-related questions.

One way to assess the relationship between categories is to compare their leaderboards. Similar categories may have similar leaderboards. In this subsection we compare the leaderboards of different categories via their Kendall rank correlations.

Figure 3 reports the Kendall correlations. First, technical leaderboards (Coding, Math, and Hard Prompt) tend to be more correlated with one another compared with Creative Writing. Secondly, consistent with the fact that it is a mixture of them all, the overall leaderboard is moderately correlated with all categories. The highest-correlation category with the overall leaderboard is Instruction-Following. Finally, it is worth mentioning that all these correlations are relatively high; the differences between categories are on the smaller side, although not insignificant.

Above Figure 4 reports the Kendall correlation between different languages. Latin based languages exhibit higher correlations with one another. Chinese is most highly correlated with Japanese. Korean exhibits weaker correlations with the other languages.

Now that we’ve examined the broad relationships between categories, let’s dive deeper into how individual models perform across different tasks. Understanding these performance patterns is crucial for both users and developers, as it reveals the specialized strengths and potential limitations of each model. By analyzing win-rates across categories and comparing models with similar overall performance, we can uncover nuanced insights that might be obscured in the overall leaderboard rankings.

In Figure 5, we observed o1 Mini’s win-rate conditioned on specific categories, providing a visual map of its strengths and weaknesses. In math and hard prompts, o1 Mini performs significantly better, indicated by the blue half of the decision tree. On the other hand, o1 Mini struggles on creative writing when compared to other models.

Unlike o1-mini, Gemini-1.5-Pro-002 demonstrates exceptional performance on creative writing compared to other top models, as illustrated in Figure 6. However, its performance on coding tasks remains a notable weakness.

Our analysis of Yi-Lightning, a recent state-of-the-art Chinese LLM, also shows interesting performance patterns. As shown in Figure 7, Yi-Lightning excels at handling complex prompts but shows relatively weaker performance in following basic instructions compared to other top models. Notably, when instruction-following tasks incorporate creative elements, Yi-Lightning’s performance improves.

Want to see another category on the leaderboard? Contribute! Our project is open-source, and we’re excited to invite contributions from the community. Whether you have ideas for new categories (not just text, even for vision models), suggestions for refining existing ones, or even want to improve our classifiers, there’s plenty of room for collaboration.

Code for the classifiers: repo link

Label Bench Dataset: huggingface link

Battle examples for categories: huggingface link

🚀 Join Us in Making Chatbot Arena Even Better!

We’d love your help in enhancing Chatbot Arena! Whether it’s tuning and improving our classifiers, validating results, or even suggesting new categories we haven’t explored yet, every contribution makes a difference.

Together, we can shape the future of LLM evaluation and create tools that truly serve the community. Let’s push the boundaries and make Chatbot Arena the best it can be!

We thank Fireworks AI for their fast open model endpoint supports.

We now go into detail on other categories and the checklists we used to define them.

Hard Prompt category features user-submitted prompts from the Arena that are specifically designed to be more complex, demanding, and rigorous. Carefully curated, these prompts test the capabilities of the latest language models, providing valuable insights into their strengths and weaknesses in tackling challenging tasks.

What are the important criteria for a prompt to be “Hard”?

• Specificity: Being specific, well-defined, and without ambiguity.
• Domain Knowledge: Tests the AI’s knowledge and understanding in a specific domain or set of domains.
• Complexity: Has multiple levels of reasoning, components, or variables.
• Problem-Solving: Requires active problem-solving: analyzing and clearly defining the problem and systematically devising and implementing a solution.
• Creativity: Involves a level of creativity when approaching the problem.
• Technical Accuracy: Requires a high degree of technical accuracy, correctness and precision.
• Real-world Application: Relates to real-world applications.

For more information, refer to the released blog dedicated to the Hard Prompt category link.

Mathematics evaluation focuses on a model’s ability to actively apply mathematical reasoning and problem-solving skills. Contrary to prompts that merely seek explanations of mathematical concepts, these prompts require direct mathematical computation or logical deduction.

Mathematical Prompts:

• Requires active application of mathematical concepts
• Involves some sorts of numerical calculations or algebraic manipulations or geometrical reasoning
• Contains clear, well-defined, and objective problems
• Tests one or multiple mathematical competencies

For example, “Explain what a derivative is” wouldn’t qualify as a math prompt, but “Find the derivative of f(x) = x³ + 2x² - 5x + 3” would. For more examples, see Appendix link.

This category evaluates a model’s ability to precisely follow given instructions, particularly focusing on multi-step tasks and specific requirements. Similar to the Hard Prompt category’s emphasis on specificity and technical accuracy, instruction-following prompts zoom in on assessing the AI’s capability to execute directions with precision and completeness. The category focuses specifically on the model’s ability to understand and execute detailed instructions rather than on domain knowledge or problem-solving capabilities.

Instruction-following prompts include:

• Clear, actionable instructions from the user
• Specific formatting or structural requirements for the response
• Unique or challenging aspects that would test the AI’s ability to follow directions precisely

For example, while “write about dogs” wouldn’t qualify as an instruction-following prompt, “Write a 3-paragraph description of dogs, using exactly 5 adjectives per paragraph, and conclude with a bulleted list of 3 care tips” would. For more examples, see link.

The coding category evaluates a model’s capability to understand, generate, and debug code across different programming languages and paradigms. Unlike the previous categories, we employ a heuristic algorithm to classify coding prompts.

Our Coding Classifier looks for:

• Code blocks in prompt and responses
• Programming language names and keywords
• Code commands and other miscellaneous items

While traditional coding benchmarks target specific tasks (such as code editing or algorithm implementation), Chatbot Arena’s coding category encompasses all code-related activities. Taking advantage of its open-ended nature, we evaluate models on complex, multi-layered software engineering tasks. For example, a prompt like “Help me build a professional website using TypeScript” assesses LLMs not only on code generation but also on broader software development decision-making capabilities. For more examples, see Appendix link.

Each language category assesses a model’s proficiency in responding to prompts within the language. Many Chatbot Arena users engage with LLMs in languages beyond English, testing models on their versatility, linguistic precision, and cultural context in many languages.

We tag each prompt with a language tag using Polyglot.

Answer style indeed has a strong effect on models’ performance on Chatbot Arena leaderboard. Certain model could be favored because it includes a lot of details when responding or uses heavy markdowns to make its responses appear nicer to human voters. To address this, we develop Style Control as a way to separate the effect of answer style from the content, so you can observe both effects individually.

We dedicated an entire blog for style control: please check it out here link.

We also include a number of other categories, many of which are self-explanatory. As a few examples, we include

• Exclude Refusal: Excludes all battles where one or more LLM refused to respond. We currently identify refusals via simple keyword matching (eg. “I can’t assist”).
• Multi-turn: Conversations with multiple turns.
• Long queries: Queries longer than 500 tokens (approximately 10% of all prompts).
• Short queries: Queries shorter than 5 tokens. These queries tend to be trivial (e.g. “Hello”).

Below we present strengths and weaknesses decision trees for GPT-4o-mini and Claude-3.5-Sonnet.