Bespoke Labs MAC Bench tests the value of AI agent teams | Magica
Bespoke Labs benchmark finds AI agent teams buy speed at a steep cost
Editorial Team
••📖8 min read
Bespoke Labs’ MAC Bench tests agent teams against solo agents and task-specific alternatives without coordination. Eight-agent teams improved average scores, but communication added no statistically significant overall benefit and model calls cost 8.8 times as much as for one agent.
Eight-agent teams averaged 0.42, against 0.35 for a single agent and 0.44 for the no-coordination baseline.
Messaging’s overall effect was not statistically significant; Opus 5.5 was the only tested model to benefit from it.
Teams cost 8.8 times as much in model calls, although they often reached a solo agent’s score sooner.
Bespoke Labs, a startup building data and reinforcement-learning environments for improving AI agents after their initial training, released Multi Agent Coordination Bench on November 18. Its central finding is a limit on the case for agent teams: adding workers improved results, but communication produced no statistically significant overall benefit against task-specific alternatives.
The distinction matters for developers deciding whether to pay for multiple agents to tackle one job. A higher team score can reflect more attempts or more workers rather than successful collaboration. Bespoke’s benchmark report tries to separate those effects across twelve simulated tasks, including designing a Mars sample-return mission, controlling air traffic and restoring a power grid. An agent here is a language-model-based system that can use tools and act on a task.
Bespoke brings relevant experience and a commercial stake. In its July financing announcement, the company announced $40 million in seed and Series A financing and described two years of data-curation research and reinforcement-learning environment development. It also identified itself as a core contributor to Terminal-Bench, an agentic coding benchmark. Wing VC led the Series A; 8VC led the $8.25 million seed round. The financing supports a broader post-training business, rather than being identified as funding for MAC Bench alone.
The company also lists the OpenThoughts reasoning dataset and GEPA, an optimizer for prompts and policies, among its projects. Its infrastructure work spans environment creation, sandboxed execution and agent optimization. MAC Bench addresses the agent-reliability problem around which Bespoke is building that business.
What the comparison actually measures
The company tested Anthropic’s Opus 5.5, OpenAI’s GPT-6.1 Sol and Google’s Gemini 4 Argon, each in its developer’s harness—the software that runs the agent and connects it to tools. Every run lasted four hours, regardless of team size. Eight agents therefore received eight times the agent-hours, rather than an equal compute budget.
Each model tackled every task alone, in an eight-agent team with communication, and in an eight-agent setting without coordination. Agents received the same instruction and no assigned roles. A hidden deterministic grader scored outcomes between zero and one; settings were usually repeated three times, with reported 95% confidence intervals reflecting variation between runs.
Amazon is in advanced talks to buy Decart, whose software helps AI models run efficiently across chips. The reported deal could strengthen AWS’s Trainium business, with conditional payments and five-year founder commitments making execution central to its value.
Editorial Team
The no-coordination comparison needs care. Nine tasks are classified as shared: agents start with the same information and capabilities. For most of those, the baseline is the best result from eight independent solo attempts, rather than eight silent agents working in one world. Cave Rescue uses the best of four for Opus 5.5.
The other three tasks distribute information or control among agents. There, the baseline generally disables communication within the team. Two exceptions complicate the division: shared-task Dubai ATC uses a silent team, while distributed-task Power Outage uses independent solo runs because agents could still exchange information through a shared repository with messaging disabled.
MAC Bench therefore compares collaboration with different ways of deploying agents. In many shared tasks, the relevant alternative is running independent attempts and selecting the best result; the aggregate comparison is not a uniform experiment in switching off a chat channel.
Across the three models and twelve tasks, the communicating team averaged 0.42, compared with 0.35 for one agent and 0.44 without coordination. Using unrounded results, Bespoke reports a coordination effect of minus 0.018, with a 95% interval from minus 0.038 to plus 0.003. That interval includes zero: the evidence supports no statistically significant overall benefit, rather than proof that messaging always harms performance.
The accompanying analysis notes another limit: GPT-6.2 and newer Gemini 4 models had already shipped. The results cover the three tested model-and-harness combinations, rather than every current model.
Bespoke Labs’ Holdout-game environment in its company-reported benchmark. Each agent controls one survivor with a limited view of the surroundings. Source: Bespoke Labs.
Where communication helps—and where workers collide
Model choice changed the result. Opus 5.5 and GPT-6.1 Sol performed about equally alone, but their eight-agent teams scored 0.53 and 0.41 respectively. Only Opus 5.5 benefited from messaging in the reported comparison. Because models ran in different developer harnesses, these results describe model-and-harness combinations, rather than isolating model capability alone.
Task structure also mattered. In Holdout-game, each agent controlled a survivor with a limited view of a continuously running zombie-survival simulation. A solo agent averaged 0.13, a silent team of eight 0.37, and a communicating team 0.42. Bespoke reports an unrounded coordination gain of 0.052. Here, teammates held information that others could not directly observe, giving communication a concrete purpose.
Mars Return illustrates the competing alternative. Across the three models, a communicating team averaged 0.26, above one agent’s 0.21 but below the best-of-eight solo result of 0.38. More agents could also make a shared project worse: on that task, Opus 5.5 scored 0.57 with four agents and 0.33 with eight. The agents had to integrate vehicle designs and flight software while sharing simulated budgets and test facilities. Additional workers also meant additional changes that could invalidate teammates’ work.
Communication itself consumed resources: message-board activity accounted for 22% of tool calls even in teams of two, according to Bespoke. More messages per agent were not reliably associated with higher scores.
Bespoke’s classifiers detected a teammate changing course within five minutes for 34% of 1,087 sampled findings. The report cautions that a finding without detected uptake can still confirm a plan or require no action.
Duplicated work appeared in 64% of 139 analyzed team runs, but the authors found no resulting damage in those labeled episodes. Most observed damage instead came from ignored messages and components that failed to fit together. In one Holdout-game run, five Opus 5.5 agents agreed that an honest stranger was a thief, and the team killed her after overlooking corroborating information. Agreement could spread a mistake as well as useful knowledge.
These are approximate judgments: Claude Sonnet 5.5 classified the logs and Claude Opus 5.5 checked the episodes.
Company-reported Bespoke Labs results: eight-agent model-call costs were 8.8 times the solo cost. In 30 of 36 task-model pairs, teams reached the solo agent’s final score within four hours, a median of 2.2 times sooner. Source: Bespoke Labs.
The price of a shorter wait
At list model prices, an eight-agent team cost 8.8 times as much as one agent, according to Bespoke’s cost analysis. That multiple is a geometric mean of within-pair cost ratios across 36 task-model combinations; it measures model calls, rather than a complete deployment bill. A single agent delivered the highest score per dollar in every pairing, according to Bespoke.
The stronger economic argument for teams was speed. In 30 of the 36 pairings, eight agents reached the solo agent’s final score within four hours, doing so a median of 2.2 times sooner. The timing estimate came from scores at five checkpoints, with linear interpolation between them. It measures time to a comparable score, rather than time to complete the entire task.
The reported median time to that score across the twelve tasks was 84 minutes for Opus 5.5, 156 for GPT-6.1 Sol and 109 for Gemini 4 Argon. An eightfold speedup from a four-hour run would mean 30 minutes.
For a buyer, that presents a choice between waiting longer and spending more to reach a comparable result sooner. It does not establish that a larger team is cost-effective for a particular production workload, where the value of time and the cost of errors may differ substantially from a simulator’s score.
Bespoke distinguishes MAC Bench from short multi-agent tests such as CooperBench and TeamBench, and from long single-agent tests such as ProgramBench and FrontierCS 2. Its claimed contribution is combining tasks lasting hours with solo and no-coordination baselines, rather than inventing agent-team evaluation.
The broader warning has precedent. An earlier research paper, revised in April 2026, evaluated 260 configurations across six benchmarks, five architectures and three model families while standardizing tools, prompts and compute. It found that coordination’s value depended on task structure, with overhead in tool-heavy work and more error propagation in architectures without centralized verification. Its different tasks and budget controls prevent a direct numerical comparison with MAC Bench, but narrow any claim that coordination costs are a newly discovered problem.
What a reproducible result would settle
These remain the company’s own evaluations. A report on the release notes that they have not been independently replicated. Twelve simulations and three model-and-harness combinations cannot establish a universal ranking of agent architectures or predict every workplace outcome.
Bespoke says in the benchmark report that it plans to release the harness, evaluate more models and larger or mixed-model teams, and add tasks with conflicting goals. The present benchmark covers collaboration toward a common goal; competition between agents remains outside its measured scope.
Releasing the harness would let others test whether better role assignment, information handling or integration changes the economics. The useful next result is a reproducible improvement over a credible no-coordination alternative, with both the extra spending and the time saved accounted for.
The University of California is expanding ChatGPT and Codex access across 10 campuses, with extra charges for advanced models and an unresolved dispute over faculty authority.
Ray-Ban’s US store cuts the Gen 2 Wayfarer by 25% to $284.25. Buyers still face separate prescription-lens prices, a cheaper Gen 1 option and newer glasses outside Meta’s frame sale.
Ubisoft has made the team behind its Teammates experiment a dedicated subsidiary targeting AI characters with memory and outside game creators. Earlier playtests show both progress and limitations, while commercial plans remain undisclosed amid group cost cuts and refinancing.
Neuralink has shown a trial participant controlling GTA VI through intended movements. Earlier gaming use and company-reported trial results put the demonstration in context, while its reliability across users remains unmeasured in the reports.
SoftBank founder Masayoshi Son reportedly explored financing a $1 trillion acquisition of memory-chip maker SK hynix, but the company says the deal is no longer under active consideration. The target brings surging earnings, existing shareholders and customer commitments that complicate any simple equation between ownership and access to AI-memory supply.
Nvidia reported $115.8 billion in quarterly revenue, but its shares fell 14% Friday amid a wider technology selloff, a leadership transition and a reported accounting investigation. Strong cash generation offers a counterweight to concerns about the durability of AI spending.
The Nasdaq 100 fell 5.6% on November 20 and closed 19.8% below its record after an intraday bear-market breach. Nvidia’s revenue surge and Anthropic’s huge IPO did not halt the rout; infrastructure bills and resilience elsewhere in the market complicate the picture.
OpenAI reports sharply lower reasoning-extraction rates in internal GPT-6.2 tests and backs sanctions against illicit foreign distillation. Its own research and rival Anthropic’s experience show why protecting hidden reasoning leaves capability transfer and customer costs unresolved.
Alberta proposes 600-metre setbacks for large data centres and expanded utility oversight, but exemptions and rules still to be written will determine how much protection communities receive.
President Donald Trump backed Commerce Secretary Howard Lutnick as lawmakers scrutinize alleged delays in restricting a suspected route for AI chips to China. A new congressional inquiry also questions Nvidia’s influence on an existing investigation.
Amazon’s month-old Echo Time is $50 off for Black Friday. Its touchscreen, home hub and optional sleep-aware alarm cost $90 more than the reported Echo Spot price, while Alexa+ access and evidence of better waking remain key buying questions.
Merck will pair its experimental data with Isomorphic Labs’ AI to seek small-molecule cancer drug candidates. The deal follows Isomorphic’s $5 billion financing, but its payments, target and development timetable remain undisclosed.
Thinking Machines Lab released a roughly 20GB open-weights model that reuses its middle layers for reasoning. It brings the Inkling family to local hardware, with competitive coding results but workload-dependent costs and preliminary evidence on oversight.
Apple’s first HomePod mini update since 2020 adds Mac audio and four-speaker surround, but Siri AI remains unavailable, core home-hub features carry over, and reviews disagree on playback reliability.
Treasury Secretary Scott Bessent says an unnamed Anthropic founder will announce a large gift to children's investment accounts within weeks, but the amount, recipients and connection to existing philanthropic pledges remain undisclosed.
Google says air cooling will limit water use at its Visakhapatnam AI hub, but environmental proceedings and residents’ accounts raise broader questions about land, electricity, public incentives and the evidence behind promised benefits.
Nvidia slipped behind Alphabet to fourth in U.S. market value during November 20 trading. The reversal follows a steep selloff despite strong reported revenue growth, with a permanent CEO still to be chosen.
Google will close the Nest app on January 25, 2027 and retire Assistant on home speakers and displays in August. Device transfers are automatic, but continued voice access requires accepting Gemini; basic assistance remains free.
Amazon says Cairn agents exploited a MONAI Label zero-day to take over a university server. Other attacks failed, and maintainers list no patched package release.