Arena’s Factuality Ranking Lifts GPT-5.5, but Meta Needs a Version Check

• Arena’s non-default factuality view combines 75% human preference with 25% automated claim checking.
• A launch account put GPT-5.5 at No. 7 after a 13-place rise and Meta’s Muse Spark near No. 20 after a 13-place fall.
• A July 16 snapshot complicates the Meta comparison: muse-spark-1.1 ranked No. 9, while the older muse-spark ranked No. 21.
• The system evaluates objective, web-verifiable claims—not every kind of hallucination, usefulness or domain risk.

Arena has added automated fact-checking to an optional version of its influential model leaderboard. The immediate reshuffle favors OpenAI, but the more consequential result is methodological: a model’s apparent standing can change with the objective, model version and price comparison a user chooses.

The launch reshuffle is real but easy to misread

The new composite score assigns factuality a 25% weight and human preference the remaining 75%. An account of the launch table said GPT-5.5 climbed 13 places to No. 7, Claude Fable 5 moved to No. 2 and Meta’s Muse Spark dropped 13 places to No. 20.

That report supports a dated launch comparison, not a permanent ordering or a finding about Meta models as a group. Arena’s own methodology and provider analysis says muse-spark-1.1 loses score as factuality receives more weight. Yet it also says Meta made a large factuality improvement from muse-spark to muse-spark-1.1, even though the newer model remained behind other leading closed and open model labs on Arena’s pure factuality measure.

The retained Text leaderboard snapshot, dated July 16, makes the version problem concrete. It lists muse-spark-1.1 at No. 9 and the older muse-spark at No. 21. The older model’s position is close to the launch account’s No. 20; the newer version should not be treated as the same result.

GPT-5.5’s adjusted rank is a favorable quality signal for OpenAI. It is not by itself a procurement verdict. Its displayed uncertainty band overlaps that of the preliminary Muse Spark 1.1 score, while the newer Meta model’s listed token prices are substantially lower. The ranking does not resolve workload-specific accuracy, latency or total deployment cost.

What the 25% factuality signal actually measures

Arena randomly samples real head-to-head battles, extracts atomic claims from both answers and keeps claims it judges objective and reasonably verifiable on the web. Search agents assign truth probabilities, which Arena says it calibrates against verified annotations. The system averages those probabilities for each response and converts the gap between the two sides into a soft factuality outcome.

Arena then fits one composite Bradley-Terry rating from human votes and factuality outcomes. Its product guide says the factuality view is a non-default filter in Text and Search Arena. The methodology can support other weights, but users cannot currently move the 25% setting.

The evidence base is broad: Arena says it labeled more than 2 million claims, including more than 1.3 million from roughly 130,000 Text battles and more than 700,000 from about 40,000 Search battles. At least one checkable claim appeared in 76% of Text battles and 88% of Search battles. Models averaged five claims per Text response and nearly 10 per Search response, with marginal true-claim rates of 87% and 89%, respectively. A July 16 news digest also recorded the launch and the use of more than 2 million extracted claims; it did not independently validate the labels.

Scale does not erase scope. The score excludes subjective assertions and mistakes that cannot be checked against public web sources. Arena’s guide distinguishes factuality from broader hallucinations, such as a model falsely saying it ran tests within the conversation. The published aggregate label does not by itself show which sources the agents retrieved, how authoritative they were or whether the verifier interpreted them correctly.

The treatment of abstention is another limit. If only one answer makes verifiable claims and those claims are truthful in aggregate, the factuality component records a tie. If those claims are false, the answering model loses. A battle in which neither response makes a checkable claim is omitted from the factuality loss but remains in the human-preference calculation. Arena’s design therefore relies on human votes to penalize an evasive but error-free answer.

That tradeoff is why Arena did not make factuality the whole score. Its guide warns that a pedantic verifier can prefer a short answer that is entirely true to a much more useful answer containing a small error. Arena also found differences by subject: mathematical responses were largely factual, while legal and government prompts were the least factual category. A general rank should not be read as proof of reliability in a specific high-stakes domain.

Changing the objective redistributes the advantage

Arena describes human preference and factuality as only weakly correlated. In its analysis, most OpenAI models rise as factuality gets more weight, while Anthropic models tend to fall or remain flat. It says OpenAI is the only provider whose flagship models steadily improved on its pure factuality score over the period studied, although recent SpaceXAI models also improved.

The provider story contains several reversals. Arena says Google’s older Gemini 2.5 series remained its most factual Gemini generation. It found the Grok 4.1 series had regressed from Grok 4-0709 before later Grok 4.20, 4.3 and 4.5 versions improved. Most open models declined as factuality received more weight, but Mistral Medium 3.5 and Tencent’s Hunyuan HY3 Preview were exceptions. Those are Arena’s findings from its own pipeline, not independent model audits.

Factuality is also not Arena’s first correction to raw preference voting. Its 2024 style-control analysis adjusted for response length and Markdown features and found noticeable rank changes. Length was the dominant measured style factor. Arena called that study observational and acknowledged that unobserved factors—such as a relationship between longer explanations and substantive quality—could remain.

The two projects point to the same constraint. Human votes can reward presentation as well as substance; automated corrections introduce their own definitions, omissions and calibration choices. No single adjusted table removes that judgment.

Arena now sells the evaluation signal it controls

The methodology carries more commercial weight than it did when Arena began as a UC Berkeley research project in 2023. A June 29 business report said the project incorporated in April 2025 and was co-founded by CEO Anastasios Angelopoulos, CTO Wei-Lin Chiang and UC Berkeley professor and Databricks co-founder Ion Stoica.

The same report said Arena reached $100 million in annualized run-rate revenue eight months after launching its commercial service. Angelopoulos clarified that customers pay based on consumption, meaning the revenue is not recurring in the traditional sense. Arena had reported a $30 million annualized rate in January, when it announced a $150 million Series A at a $1.7 billion post-money valuation; the company has raised $250 million in total.

Arena’s public leaderboards remain free. Its paid evaluation service sells model labs and enterprises deeper performance analytics derived from community activity. Angelopoulos said Arena competes for post-training budgets with human-labeling companies such as Mercor, Surge and Scale AI, even though the report described Arena as lacking a direct leaderboard rival after crowdsourced model-selection startup Yupp shut down.

That business model does not show that Arena’s rankings are biased. It does concentrate influence: the company defines a prominent public score, attracts users with access to new models and sells more detailed analysis of the resulting evaluation data. Labs have reason to improve against whichever traits the public leaderboard exposes, while buyers have reason to ask how closely those traits match their own workloads.

What evidence should come before a buying decision

The next question is not whether facts should matter. It is whether the new composite predicts accuracy and usefulness outside Arena closely enough to justify a model choice.

That requires independent checks of the claim extractor, the sources retrieved by search agents, calibration error by domain and the sensitivity of ranks to factuality weights other than 25%. It also requires repeated snapshots using exact model versions, because a launch-day move can quickly become ambiguous when a lab replaces or updates a model.

Stability deserves separate testing. A May preprint on Bradley-Terry leaderboards found that targeted modifications affecting less than 1% of comparisons could change the top-ranked model, reduce overall rank agreement and alter confidence intervals across Chatbot Arena and six other pairwise datasets. The study did not test Arena’s new composite factuality ranking, so it cannot establish that these particular positions are fragile. It shows why ordinal rank alone is insufficient evidence.

For Arena, the next consequential choice is whether to expose alternative weights and enough audit material for outsiders to reproduce the score. For model buyers, the next step is narrower: compare exact versions on their own prompts, error costs and deployment economics. Until those checks exist, the defensible conclusion is that factuality changes Arena’s ordering—not that one provider has become universally more reliable than another.