Arena-Rank: Open Sourcing the Leaderboard Methodology

We have several more advanced example notebooks in the examples folder of the repo, covering techniques such as the style-controlled leaderboard on LMArena, analysis of voter patterns on the PRISM alignment dataset, and analysis of sports and video game competitions using the general Bradley-Terry methodology on professional basketball seasons and Super Smash Bros. tournaments.

In today’s release, Arena-Rank implements the Bradley-Terry model and an extension for handling contextual features, which we use for the style-controlled leaderboards. We’ve disentangled the upstream data pipeline logic from the leaderboard calculation to allow for faster experimentation and iteration on pure leaderboard-related experimentation and easier extensibility to more model variants and applications.

We’ve also decoupled data preprocessing from the model optimization by adopting a pattern of datasets classes and model classes, where a dataset can be preprocessed once and then have many different ranking models fit on it, allowing for efficient hyperparameter sweeping and computation of many leaderboard variants at the same time.

We’ve opted to use the JAX package as the computational backend. The just-in-time compilation and efficient automatic differentiation enables a significant speedup over our previous NumPy/SciPy implementation, and there is still more room for improvement due to JAX's support for scaling on hardware accelerators like GPUs and TPUs. Together with other computational improvements, such as the use of closed-form confidence intervals instead of the bootstrap, the overall speedup compared to the previous FastChat version is more than 30x.

The Arena-Rank package is, of course, built with AI evaluation in mind, but we’ve also intentionally developed it to be general purpose and easy to use for calculating rankings for any type of competition data. Our repo includes examples of using Arena-Rank to compute leaderboards for sports and for competitive video games.

With today’s release, we are proud to take this step to prioritize openness and empower the AI evaluation and rating systems communities. But it doesn’t end here. We are committed to maintaining and improving this package both as our own methodologies evolve and as we get feedback from users and researchers who try it out.

As part of our broader commitment to transparent evaluation and open science, we’re looking forward to building out our framework for more regular leaderboard and data releases to build a fruitful ecosystem of open and reproducible AI evaluation.

• Check out the code on GitHub: https://github.com/lmarena/arena-rank
• Swing by to ask questions and request features in our discord: discord.gg/LMArena