Berkeley’s robot-coding agents show gains, but physical tests still decide

• Coding agents can assemble and revise robot programs using existing skills and simulated feedback.
• Ambi Robotics reports one successful warehouse update from three candidates; two reduced capacity in production tests.
• Benchmarks show gains under controlled variation, alongside limits in speed, complete-sequence reliability and simulation accuracy.

Researchers at the University of California, Berkeley and NVIDIA are releasing tools that let AI agents write, test and repair robot-control programs in simulation before engineers deploy them. The work, being presented at this week’s Conference on Robot Learning (CoRL), targets repetitive industrial tasks with changing objects and positions. An early warehouse application shows the opportunity—and the importance of testing candidates on physical machines.

The potential saving is in task-specific development: agents can connect reusable perception, planning and control skills, then revise the program when simulated attempts fail. That could reduce the engineering and demonstration collection needed for a new behavior. It still depends on existing skills, calibrated equipment, useful physical observations and a simulator that captures the relevant failure.

GaP constructs robot programs from existing skills, refines them in simulation and executes them with feedback. Credit: GaP research team. Source: GaP project.

Existing skills make automated programming possible

Ken Goldberg, a UC Berkeley engineering professor and co-founder of warehouse robotics supplier Ambi Robotics, is presenting the approach, according to the university’s November 9 announcement. His four-decade robotics career includes skepticism about how quickly data-driven robots could acquire general skills: last year, he warned of a large gap between available robot experience and the internet-scale data used by language models. He now argues that coding agents offer a practical bridge between classical robot programming and learned policies.

Ambi supplies both deployment experience and a commercial stake in that argument. Founded in 2018, the company introduced its AmbiOS operating system publicly in 2021 with simulation-to-reality deep learning based on Dex-Net technology. Simulation and learned components were already part of its business; the new element is using coding agents to improve how robot skills are coordinated.

Language models writing robot code also have precedent. The Code as Policies paper, first submitted in September 2022, demonstrated generating robot policy code from natural-language commands and composing calls to existing control interfaces.

The newer Graph-as-Policy framework, or GaP, was first submitted in July and revised in November. It uses an orchestration agent to divide a task among specialized coding agents. They assemble a directed computation graph: connected steps for sensing, planning, movement and recovery, with interfaces checked before execution. Parallel simulation rehearsals localize failures so agents can revise the graph’s connections or parameters.

GaP’s Modular Open Robot Skill Library initially contains 51 skills, covering perception, grasp and motion planning, vision and verification, among other functions. Its target, called variational automation, assumes a known workcell, robot and sensors, with bounded variation in object geometry and position. Reusing object models, calibrated sensors and those skills is central to its efficiency proposition. This is a more constrained problem than operating a general-purpose robot in unfamiliar surroundings.

Company-reported annual capacity estimates from three production-test candidates, assuming 12 operating hours a day, six days a week. These are projections, not a measured year of output. Graphic: Magica; data: Ambi Robotics. Source: Ambi Robotics.

One warehouse fix survived the production test

Ambi’s September 17 case study describes a GaP-based agent harness inside AmbiOS. Its AmbiSort system picks and scans packages, then uses a gantry to sort them into bags. Packages near the opening’s size limit sometimes failed to fit because sensor uncertainty and deformation led to a poor placement orientation, forcing a rejection and retry.

The company says agents using Anthropic’s Claude models, including Sonnet 4.6 and Opus 4.8, examined 500 production events from the preceding week. Half were selected for the known failure mode; the other half were randomly selected to check for regressions. Given existing skills, production records and a simulator for replaying events, the agents generated and evaluated three candidate solutions within 10 hours.

Ambi then compared each candidate with baseline software in customer-warehouse A/B tests. After 10 days, the company reported:

These are annualized capacity estimates, not a measured year of output. Ambi reports that H3 increased actual test throughput by 4.2 packages an hour; its annual projection assumes operation for 12 hours a day, six days a week. The successful program changed how packages were routed between placement routines, balancing the orientation needed to fit the bag against the risk of dangling material catching on the platform.

Ambi says the update runs on 30% of its U.S. fleet, including systems serving a Fortune 50 package shipper. For background scale, a September 1 podcast summary republished by the company described more than 100 deployed robots and over 150 million packages handled. That broader deployment figure does not establish the number receiving this particular update.

The case study does not disclose baseline packages per hour, so the percentage throughput gain cannot be calculated. Nor does it report agent-computation costs or the number of robots in each test. Its estimate that hand-tuning would have taken several weeks is a company judgment, rather than a timed comparison with a parallel engineering team.

The company’s reported time saving concerns candidate development; physical tests distinguished the helpful change from two harmful ones. The harness also constrained edits to preserve safety checks and error handling. Ten hours of automated development did not remove the subsequent validation work.

Task success and sustained reliability are different measures

GaP’s researchers evaluated eight benchmarks, four simulated and four physical. Their results show strong improvements in some controlled settings. On physical grocery fulfillment, GaP completed 25 of 25 trials against eight of 25 for the TipTop baseline. Average single pick-and-place completion time was 67 seconds for GaP against 95 seconds for TipTop. The authors nevertheless identify industrial cycle time as an area requiring faster perception and motion planning.

For popcorn making, 10 rounds of simulated rehearsal raised reported success from 33% initially to 94% in simulation. The revised program then succeeded in 18 of 20 physical trials. That is promising transfer, with a small physical sample.

Cable insertion demonstrates why individual-action success is insufficient. In ascending port order, GaP completed 32 of 35 insertions but only two of five full sequences. A customer relying on an uninterrupted sequence needs the second measure as well as the first.

Expert programming remains a competitive reference. In simulated bimanual crate washing, GaP succeeded in 143 of 150 trials, compared with 148 of 150 for a hand-engineered graph. A three-hour continuous test recorded 18.33 successful cycles an hour for GaP against 19.33 for the expert program. Automated authoring approached that reference without exceeding it on either measure.

The researchers also tested combinations with vision-language-action policies, which map visual and language inputs to robot actions. GaP positioned the wrist camera before handing control to a learned policy, improving results under several larger variations. The combination did not improve every setting: both tested learned policies lost success on the original LIBERO benchmark after this staging step. The evidence favors testing the combination against each task’s conditions, rather than assuming a universal gain.

The GaP benchmarks focus on quasi-static manipulation. The project identifies deformable objects, dynamics and moving targets as further work; its results do not establish reliability across unrestricted robot operation.

A simulated sponge-wiping task in RoboSimStudio, which exposes physical parameters for adjustment against observed behavior. Credit: RoboSimStudio research team. Source: RoboSimStudio project.

The simulator must reproduce the failure

RoboSimStudio, a related toolkit, addresses another dependency: accurately modeling contact, deformation, stiffness and friction. It exposes solver choices, contact models and physical parameters so humans and coding agents can revise a scene against observed behavior.

The project demonstration starts with a smartphone video of sponge wiping and a short prompt, then builds a simulation in about an hour. Rigid bodies run in Newton, an open physics engine, coupled with a deformable sponge model. Humans and agents inspect and tune the result before using varied simulated conditions to improve a GaP program.

In the university’s account, Goldberg said one physical wiping attempt pushed over a metal bar because its mass had not been reliably inferred. The failure provided evidence the agent could use to revise its simulation and control program. Automated coding still needs trustworthy observations of the physical world.

NVIDIA’s role extends across competing and complementary approaches. Its CoRL announcement presents GaP alongside Isaac Lab, its open robot-learning framework, and GR00T 2, an open world action model trained on robot and human data. NVIDIA says agents can call GR00T 2’s skills through an interface to sequence actions and recover from errors. Agent-written control programs and learned models can therefore be components of the same system.

VABAR tool-sorting examples compare separated objects at Level 1 with overlapping objects at Level 2. Each column is independently seeded. Credit: VABAR research team. Source: VABAR project.

What the next tests need to establish

The Variational Automation Benchmark for Agentic Robotics, or VABAR, supplies six simulated task classes: tool sorting, tool hanging, gear meshing, syringe packing, cable insertion and cable routing. Built on Newton, it provides standardized variations, difficulty tiers, success metrics and 33 reusable skills with human-written reference solutions.

NVIDIA also presents DexBench, developed by RLWRLD with NVIDIA, as another industrial evaluation option, covering 15 tasks across four scenes. VABAR is part of a broader effort to measure robot performance under variation.

Its current comparison tests eight coding models on five of those task classes, excluding tool sorting. After two rounds of execution feedback, the best tested agent’s overall held-out per-object success was 77%, against 82% for the human-written reference, averaged over five runs. Results varied substantially by task; this is a simulation comparison of object-level outcomes, not a production uptime measure. The two available difficulty levels form a validation set, with a harder test set still promised.

A related controlled simulation study across nine manipulation tasks finds that a frequent bottleneck is interpreting feedback into the right program change, rather than simply having enough information. More rehearsal alone does not establish that an agent will diagnose the problem correctly.

For industrial buyers, the next evidence needs to connect these measures: longer physical runs, complete-sequence reliability, throughput against existing production software, and total development cost including simulation setup, agent computation, human review and deployment testing. Ambi’s trial shows why those tests matter: the same automated development process produced both a useful update and candidates that made operations worse.