WHEN SOURCE CODE HELPS
From no successes
to 57%.
On SweepIntoDrawer, access to source code raises Claude Code’s mean success from 0% to 57%. The agent discovers it can use the gripper itself to sweep.
01 / THE BIG PICTURE
One program.
Many instances.
Can a general-purpose coding agent discover the physical reasoning needed for task and motion planning? We let agents investigate a simulator, develop a policy, and test that frozen program on unseen instances.
28simulated environments
700synthesized programs
70,000held-out evaluation episodes
0LLM calls at test time
02 / THE METHOD
Room to experiment.
Code that generalizes.
The agent tests its assumptions, uses what it observes to improve its program, then freezes the code for unseen instances. Follow one real Shelf run from an inaccurate arm model to a reusable policy.
GIVEN TO THE AGENT
- Task description
- Simulator:
reset+step - $20 synthesis budget
No simulator source or hand-designed planning abstractions.
-
01 DURING SYNTHESIS
Test the model
Floor probe · 3×The prediction does not match reality.
The arm stalls against the floor, even though the agent’s initial model predicts clearance. It needs to measure the robot’s actual geometry.
38.9 mmInitial position RMSE
on calibration observationsNEXTCollect positions to correct the model → -
02 DURING SYNTHESIS
Fit & refine
Collect block positions · 8×Use the held block as a measuring tool.
Observed block positions let the agent fit six parameters for the robot mount and grasp offsets. It uses the calibrated model for inverse kinematics and refines its policy.
1.8 mmPosition RMSE after calibration
on the same observationsNEXTFreeze the resulting program → -
03 HELD-OUT EVALUATION
Freeze & test
Same program, new instances · 8×New instances. The same frozen code.
Evaluate the program on 100 unseen instances with different configurations. The coding agent is no longer involved: the program computes the actions itself.
100Held-out test instances
0 LLM calls at test timeOUTPUTA reusable programmatic policy ✓
WHAT CHANGED BETWEEN STAGES 1 AND 2?
A model that matches
what the robot actually does.
The agent compares where its model predicts the held block will be with where the simulator says it is. Fitting the geometry brings those positions much closer together.
POSITION RMSE0–40 mm
Before calibration38.9 mm
After calibration1.8 mm
Learn the geometry → use it to choose actions.
The calibrated model helps the policy solve inverse kinematics. These errors describe the calibration observations; held-out task success is evaluated separately in stage 3.
03 / THE LEADERBOARD
Put every method
to the test.
Five runs. One hundred held-out instances per environment. Explore the results across both geometric and dynamic tasks.
Download the results ↓Loading the paper’s results…
| Rank | Method / backend | Environment access | Coverage |
|---|
Equal-environment averages of the rounded means in Tables I–II. The source-access reference is listed separately. Missing planner results are excluded from its average.
Read the tables ↗More coding agents on the way.
04 / A CLOSER LOOK
Different worlds.
Different challenges.
Explore all 28 environments. Compare every method, including the range across synthesis runs, alongside an example rollout.
EXAMPLE ROLLOUT
Bars show mean success; brackets give min–max across the five runs. Example videos illustrate behavior and do not represent an average run.
From no successes
FAST PROGRAMS AT TEST TIME
Decisions in
4.0 milliseconds.
Policy computation per action. Equal-environment means on 44 matched seeds across 13 environments where both settings achieve 100% success. Table III.
05 / THE GALLERY
We didn’t tell it
to do that.
Unexpected tool use, clever workarounds, and recovery from failure. A few moments worth watching.
EXPLORE THE RESEARCH