GPT-6 Astra drove a real Toyota through a cone course once in three attempts. Nobody else finished.
Four frontier models drove a real Toyota through a cone course. GPT-6 Astra finished once in three attempts, at 0.42 m/s and $7.74; no other model finished.
Vlad Makarovreviewed and published
DrivingBench asks a blunt question and answers it in public: can a frontier language model drive a real car? Four models got three attempts each, in one continuous chat per model, on a cone course laid out in an empty Bay Area parking lot, with a 2022 Toyota Corolla, a comma four and a human operator sitting ready to brake. Exactly one finished. GPT-6 Astra completed the course on its second attempt in 5:22 over 134.7 m; Claude Fable 5.1's best was 45%, Grok 4.6's 11% and GPT-5.6 Sol's 6%, and none of those three completed a single attempt. The project is by Tobias Gessler, Simon Mahns and Aditya Ramabadran, and it surfaced on Hacker News on 22 September, where the thread reached 257 points and 217 comments.
What the leaderboard actually shows
| Model | Harness | Best progress | Outcome | Cost, list prices |
|---|---|---|---|---|
| GPT-6 Astra | Codex | 100% | finished, attempt 2, 5:22 | $9.75 |
| Claude Fable 5.1 | Claude Code | 45% | 3 attempts, all DNF | $3.95 |
| Grok 4.6 | Cursor | 11% | 3 attempts, all DNF | $0.66 |
| GPT-5.6 Sol | Codex | 6% | 3 attempts, all DNF | $1.05 |
Progress is measured along the course centreline and counted only while the car stays within 4 m of it, so a collision keeps whatever the model had already earned. Distance is GPS speed integrated from the first accepted command, every model ran at medium reasoning effort, and the dollar figures sum the published per-attempt totals at list prices. Astra's first attempt stopped at 49% and 67.3 m on 1.2M tokens; the successful one used 24 commands and 6.6M tokens. The report is blunt about how thin the rest of the table is: Fable's 45% and Astra's first attempt each got around halfway, and "all other attempts didn't make it past the first corner", mostly on the first diagonal line of cones.
Three MCP tools and a foot on the brake
The models never command a steering wheel directly. They run inside their own chat harness, Codex or Claude Code or Cursor, and call three MCP tools:
observe()returns camera frames plus current speed, steering and remaining motionset_motion(direction, steering_percent, speed_mps, duration_s, reason)replaces the active commandstop_now(reason)brakes immediately
Underneath, a harness built on openpilot turns a steering percentage into a wheel angle, then a tire angle, then a torque request on the car's CAN bus, and the car's own motor does the rest. An operator presses the RES button openpilot requires before any motion command can move the car from a standstill, and brakes whenever it leaves its bounds or nears an object. Speed is capped at 0.5-3.5 m/s, one to eight miles per hour, in several layers of code, with an emergency stop that cancels motion above 6 m/s. The models choose their own speeds and command durations, and the authors concede in the thread what that costs: the latency is so high that models "have to pretty much drive slowly step-by-step".
The economics of five minutes at walking pace
134.7 m in 322 seconds is 0.42 m/s, about a slow walk, and the run cost $7.74 at list prices. Grok 4.6 and GPT-5.6 Sol spent $0.66 and $1.05 across all three of their attempts to get 22 m and 17 m. Fable's second attempt drove for 31 seconds out of 190, the rest spent reasoning while the car sat braked, and Astra's successful run never exceeded 0.8 m/s but used full-lock steering on 20 of its 24 commands. Commenter alexk307 ran the arithmetic and then the argument: "Astra can 'drive' a car at ~.42 m/s within a set of boundaries that are 2-3 times the width of a normal lane, on a closed course, with 0 unexpected obstacles, in dry conditions, in daylight for $7.74. And unless you start and then stop every few seconds while driving, this is barely considered driving." WarmWash put it shorter: "The course looks like it is something that a human could do in 15 seconds, while Astra took 5 minutes."
Refusing to drive, until the tool was renamed a sandbox
The strangest behaviour is not on the leaderboard at all. The report carries a section titled "Refusals": "Some models (especially GPT-6 Astra) would refuse to drive the physical car sometimes, citing safety reasons (even in a completely empty lot, after prompting it with all the safety measures we had including the very low speed limit caps and human ready the brake)." Telling the model it was running a simulation backfired once it could see the images, because "in some trials they would see the real images and realize it's real, and start freaking out". What worked was a name: "What ended up working best, for some reason, was changing our MCP name to 'DrivingBench Sandbox.'" The published system prompt now opens with "Use only the drivingbench_sandbox MCP."
zezcko raised the same thing in the thread before the report made it legible. "Astra initially refused to drive because it realised it was driving a real car and would only obey when the MCP was renamed to DrivingBench Sandbox," the commenter wrote, asking whether models act more readily when a task is labelled a benchmark. pcstl's answer was flat: "Yes, it is. If you convince a model it is inside a sandbox it is much more likely to comply with requests that would normally be against its guardrails." Note what that means for reading the one success: the compliance problem was solved by renaming a tool, not by making the task safer.
Not self-driving, and the authors say so first
Wikipedia's article on self-driving cars describes vehicles operating on public roads with reduced or no human input. DrivingBench is a closed parking lot with no traffic, no pedestrians, no weather and no unexpected obstacles, driven one command at a time inside 4 m of a course centreline at roughly walking pace, with openpilot handling the low-level control and a human one pedal press from ending the run. The ceiling the authors name themselves is latency. jyoung8607, who contributes to openpilot, explained in the thread that the local driving stack updates target curvature and acceleration at 20 Hz and that a cloud round trip is "meaningfully bad" on top of that: "Three reasons: latency, latency, and latency." The team agreed in its own reply, calling DrivingBench "just sort of a fun benchmark to see how good frontier LLMs are out-of-the-box at driving a real car, and probably not actually practical any time soon". The same thread connects it to the other real-hardware benchmark Astra led in September, RoboCurve's robot-arm evaluation, which the DrivingBench report thanks for inspiration.
What would settle it
Nobody has driven the course with a human at the wheel, so there is no baseline for how hard 134.7 m of cones with 4 m of tolerance actually is. The report concedes that each model was evaluated once, three attempts each, "but they aren't independent since same-context". Its own wish list for a second version is the credible one: repeated evaluations, different reasoning efforts, more models, and a longer, harder course. What else would settle it is duller and more useful than another leaderboard row: publish a human time on the same layout, run it with parked cars or a pedestrian in the aisle, and have someone who is not a co-author rebuild the harness and see whether the single finish reproduces. One finish, three failures and eleven published runs is a narrow result, but it is a real one, and the traces to check it are all public.
