Google and Artificial Analysis launch MKA visual-work benchmark | Magica
Google and Artificial Analysis launch MKA to test AI agents on visual work
Editorial Team
••📖9 min read
Google and Artificial Analysis launched a 200-task test of AI agents working with visually dense professional files. Google's study finds close leading scores and sharply different API bills, as its new Carbon model reaches customers alongside the Argon alternative.
MKA tests 200 professional workflows involving visually dense files and finished work products.
Google's study finds no statistically significant difference among the top three models on the harder criteria.
The evaluation set stays private; ten separate showcase tasks are available for public inspection.
Google and AI benchmarking operator Artificial Analysis launched the Multimodal Knowledge Agent benchmark on November 10 to test whether autonomous AI agents can turn blueprints, schematics, scans and charts into professional work products. Google's four-model study found no statistically significant quality difference among the three highest-scoring models on the harder checks, but estimated API costs ranged from $5.63 to $35.28 per task for those three.
The launch accompanies the availability of Gemini 4 Carbon, Google's model for especially demanding coding, scientific reasoning and knowledge work, through the Gemini API in Google AI Studio. Google has a direct commercial stake in the new yardstick: its AI research organization, Google DeepMind, developed a test on which its own model competes. Artificial Analysis supplies the shared agent software and runs the private evaluation independently, according to Google's benchmark announcement and the paper.
Carbon was previewed late last month. Google says in its Carbon launch announcement that Gemini 4 Argon, its alternative model for coding and knowledge work, remains the preferred choice for most such tasks; Carbon uses more tokens to pursue harder problems. That positioning makes Argon a material alternative for buyers interpreting Carbon's MKA results.
Google's illustration of company-reported MKA task examples shows the progression from input files to reference deliverables and grading checks. Source: Google. Source: Google.
Alibaba’s Qwen4-Max preview gives developers a fixed model to test through November 25. API access requires Token Plan, while claimed coding gains remain unquantified in the launch materials.
MKA covers 83 occupations across ten equally represented sectors, from construction and engineering to finance and healthcare. Each sector contributes 20 tasks. Authors had at least five years of professional experience, and a second expert with the same job profile reviewed each task, the technical paper says.
Agents receive between two and 45 input files per task. They must combine information across formats; 176 of the 200 tasks require a standalone file, such as a spreadsheet, presentation, document or report. The remaining tasks accept a text response. Google's announcement describes the grading as weighted 75% toward factual accuracy and 25% toward visual design.
One public architecture task makes the requirement concrete. The agent must compare five floor plans with sections and elevations, identify mismatches and items that cannot be determined, and create an audit presentation of no more than ten slides supported by drawing crops. Artificial Analysis publishes that task and sample model submissions on its evaluation page.
Professional deliverables themselves are an established benchmark format. Artificial Analysis already runs GDPval-AA on OpenAI's workplace-task dataset and operates its own AA-Briefcase knowledge-work evaluation. That existing evaluation work and its Stirrup agent framework explain the partner's role here. MKA is a standalone evaluation, separate from its composite Intelligence Index, according to its methods documentation.
Google's narrower claim is that MKA makes visual inspection consequential. In the paper's experiment, removing Carbon's image-viewing tool lowered its score on MKA's harder criteria by 5.94 percentage points and increased output-token use by 40%. The model retained code execution, optical character recognition and image-processing libraries, allowing it to attempt workarounds. The same intervention produced no statistically significant change in overall quality on the public GDPval subset tested by the researchers.
That supports a distinction between the task sets under this setup. It does not establish how much every other model depends on vision: the researchers ran the experiment only on Carbon. They also found that coding could recover some visual information from clean schematics, while photographs proved much harder.
Company-reported mean API costs in the four-model, 200-task study, using list prices with prompt caching. These figures exclude a complete deployment budget. Source: Google DeepMind; chart: Magica. Source: Google DeepMind technical paper.
Close quality scores, very different API bills
Every model in the paper used the same Stirrup agent framework, maintained by Artificial Analysis, with common instructions, tools, environment and budgets. An agent could take up to 250 turns and eight hours per task in a Linux sandbox with document-processing and OCR software. It used code execution, image viewing and web search; desktop operation through mouse clicks was outside the test.
The headline metric, Climb-WSR, measures the weighted share of harder rubric checks satisfied within each task, averaged equally across tasks. Those checks are selected using a fixed panel of earlier models; requirements the whole panel already passes reliably are tracked separately as regression checks. A 57% score therefore does not mean 57% of jobs, or even whole tasks, were completed correctly. The paper does not report an all-checks-passed task rate, arguing that a single missed minor requirement would make that measure too brittle.
The paper reports these results from one run per model on each of the 200 tasks:
Model
Harder-criteria score, Climb-WSR
Overall rubric score
Mean runtime per task
Estimated API cost per task
Claude Fable 5.2
56.56%
73.26%
83.2 minutes
$35.28
Gemini 4 Carbon
54.34%
71.89%
29.2 minutes
$5.63
Claude Opus 5.5
53.01%
70.98%
33.8 minutes
$10.29
GPT-6 Astra
50.24%
69.34%
18.5 minutes
$6.03
The top three are statistically indistinguishable on Climb-WSR in the paper's paired task-level analysis. Fable's 2.22-percentage-point lead over Carbon has a 95% confidence interval running from −0.94 to +5.49 points. Carbon and Fable each significantly outperform Astra on that metric.
API costs diverge despite the close leading scores. Dividing the paper's $35.28 Fable bill by Carbon's $5.63 gives a roughly 6.3-fold difference. Fable also took nearly three times as long. Astra was faster and used fewer tokens than Carbon, but cost slightly more because of its per-token pricing. Quality, waiting time and dollar savings are different comparisons.
These figures use provider list prices and prompt-cache pricing at the time of the study. Between 93.7% and 97.7% of cumulative input tokens were served from cache, making that assumption material. Artificial Analysis says its published MKA costs cover calls to the model being tested and exclude judge calls; they are not a complete operating budget for a deployed workflow.
Nor is Carbon the cheapest model across the broader leaderboard. Artificial Analysis's evaluation page lists 21 models and names GPT-6 Luna as the lowest-cost option at $0.18 per task. The paper's favorable Carbon comparison covers its four-model frontier sample.
A separate launch report quotes Google claiming Carbon ranks first on MKA while using three times fewer tokens than the next-best model. That quoted ranking conflicts with Google's direct Carbon launch announcement, its dedicated benchmark announcement, the paper and the leaderboard, which place Carbon second by score. The sources do not explain the discrepancy. The paper's token comparison is also more specific: Fable used 5.74 times Carbon's total tokens and 2.93 times its output tokens.
The launch report lists introductory Carbon pricing of $2 per million input tokens and $10 per million output tokens through November 30, rising afterward to $4 and $20 respectively. It also says Carbon is reaching AI Ultra subscribers in the Gemini app, while Argon remains available to AI Pro subscribers and above. Access tier and future token prices matter alongside the paper's current API comparison; the measured bill is not a fixed deployment price.
Independent execution, developer-defined criteria
The separation of roles provides checks on Google's involvement. Artificial Analysis runs the private suite, and the grader uses OpenAI's GPT-6 Luna rather than a Google model. The grader operates in its own sandbox, rendering submitted documents and inspecting file contents against expert-written criteria.
It does not receive the original input files. Instead, the rubrics must contain enough ground truth to judge the deliverable without solving the task again. That makes rubric construction an important part of the measurement, alongside model performance. The authors also caution that internet access may let contestants retrieve some information online instead of extracting it visually.
The paper reports 96.2% agreement between the automated grader and domain experts in a study covering 50 tasks across nine sectors. That study focused on higher-weight criteria and adjusted the selection to maintain a balanced mix of passing and failing checks. The authors caution that the agreement rate may not transfer exactly to lower-weight formatting requirements.
Against the human majority verdict, the grader's reported error rate was 3.8%. The direction matters: the paper reports that it passed 7.1% of human-failed checks, while failing 2.1% of human-passed checks. Those conditional rates describe grading errors in the study, not errors in deployed professional work.
Other design choices also matter. The panel defining the harder criteria includes two earlier Gemini models, so the split reflects that panel's existing capabilities. The paper also credits Gemini Argon and Carbon with assisting benchmark design, results analysis and manuscript writing. Those disclosures distinguish independent execution from independent authorship of the test.
An input image from Google's public MKA engineering showcase, separate from the private evaluation. Company-reported benchmark material. Source: Google DeepMind; Apache 2.0. Source: Google dataset on Hugging Face.
What buyers can verify next
The full evaluation stays private to reduce the risk of tasks entering training data or being retrieved through search. The public release instead contains ten disjoint showcase tasks, with prompts, inputs and grading rubrics under an Apache 2.0 license. These let developers inspect the format and test their own agents, while leaving the full 200-task score dependent on the operator's evaluation.
The unresolved question for employers is how these rubric scores translate into dependable work and total cost. Each main-study task had one agent run and one grading pass; the reported confidence intervals capture differences across sampled tasks, not variation between repeated runs. Google's announcement describes projects taking professionals three to eight hours, but the paper's task-authoring description includes assembling files, solving the task and writing its rubric within that window. It is not a measured labor-saving result.
Repeated trials on an organization's own files, checked by its own specialists, would test the remaining issue: whether a low API bill still buys an acceptable result after review and correction. MKA supplies a more demanding visual-work test; the published scores leave that deployment decision open.
Railway estimates stolen customer keys generated about $740,000 in unauthorized AI spending after an October intrusion. Secrets from up to about 100,000 workspaces could have been decrypted, and the company says it will cover losses providers decline to refund.
Anthropic expects retail investors worldwide to receive up to 20% of its IPO, potentially $12.9 billion at the proposed upper price. Access through six brokerage channels depends on demand and broker rules, while a separate technology-company investment would help finance vast infrastructure commitments and founders and a special trust retain control.
Anthropic's seven founders plan equal IPO share sales worth about $250 million each at the proposed midpoint. Their sales make up 2.9% of the public offering, while the company seeks much larger financing under a structure that limits public investors' voting influence.
Nvidia’s independent directors retained Wachtell Lipton partner Joon H. Kim for an investigation into reports of a foreign intelligence effort targeting co-founder and CEO Jensen Huang. The board previously expressed confidence in Huang; the appointment provides no findings or timetable.
RRE Ventures led the funding for Graphon’s enterprise memory infrastructure. The startup is pitching connected, persistent context across different agents, but benchmark gains leave deployment costs and commercial scale unresolved.
Goldman Sachs, a lead bank on Anthropic’s IPO, reportedly forecasts $685 billion in annual revenue by 2030. The Claude developer has substantial cash reserves and rapidly growing sales, but rising compute costs, competitive pricing and delivery risks complicate the enterprise growth case.
Memories Upload pays contributors for accepted first-person video and sells annotated data to robotics developers. Its million-hour collection claim leaves open how much footage improves robot performance, while contributors surrender ownership without royalties.
OpenAI has acquired AI travel concierge Odessia and is bringing its founder, former Sonder chief Francis Davidson, into its travel effort. The standalone service has closed, leaving its future role alongside ChatGPT’s existing travel partners unclear.
Anthropic says expected advances from future models have triggered its research-acceleration threshold. An outside assessment finds faster progress but does not certify compliance with that threshold, while the voluntary policy leaves consequential decisions about safeguards and continued development with the company.
OpenAI’s free business-profile beta adds branded agents for eligible U.S. ecommerce and online service firms. The agents can also serve as paid-ad destinations, while OpenAI controls distribution and businesses bear responsibility for outputs.
Teleport has made Beams generally available with task-specific infrastructure permissions and a kill switch. Its access-security platform supplies an enterprise foothold, but established competitors and custom pricing for key controls complicate the buying decision.
New York will connect more than 80 researchers with Empire AI computing resources to evaluate advanced AI systems and advise state oversight, expanding a center first announced in January.
Anthropic’s amended IPO filing raises its Microsoft spending commitment while disclosing an exit right from a $34.9 billion Lambda agreement. Google’s larger commitment was already disclosed earlier, and expensive short-term computing continues to squeeze margins.
The cash deal would bring Oura’s existing AI and physician care partner under its ownership. The $159 annual integration already connects members to doctors, but geographic limits, extra service costs and unproven clinical benefits qualify the expansion.
Reddit is testing AI-organized review pages for roughly 10,000 products. The pages link back to users’ conversations, but lack purchase verification—a safeguard Amazon says it uses for its own AI review highlights.
The renamed MegaETH Foundation is funding its own products after an earlier shift away from its accelerator. Its December inference market must show how it improves on existing services that already route requests by price and reliability.
Cambridge startup Tendrils Compute has raised a Benchmark-led seed round to build processors based on interaction nets. Its backers see an opening in branching workloads, including AI agents, but theoretical parallelism has yet to establish a commercial hardware advantage.
Google is widening paid access to Gemini 4 Carbon for complex coding, business and scientific work. Gemini API token rates double after November 30, while benchmark results show that its cost advantage depends on the task and reasoning setting.
MiniMax has released downloadable weights for H3-Image, a model combining image generation and editing at native 16-megapixel resolution. Local deployment offers an alternative to its paid API, but hardware demands and a restrictive community license qualify the promise of open access.
