←
DrivingBench32 posts
DrivingBench avatar

DrivingBench

@DrivingBench

frontier AI models and LLMs driving real cars - drivingbench.com

◷ Joined September 2026▧ 16 Photos and videos
DrivingBench avatar
DrivingBench@DrivingBench·Sep 21

We gave ChatGPT, Claude, and Grok control of a real Toyota Corolla 🚗, steering/gas/brakes, no human driving (just a foot over the brake). Only one model was able to complete our entire driving course. Introducing DrivingBench. 🔥⌛️🏁

1:08
DrivingBench avatar
⟳ DrivingBench retweeted
DrivingBench@DrivingBench·21h

Introducing DrivingBench v2 🧵 We wired a Tesla Model 3 so any AI model can drive it, and put 8 frontier VLMs (Claude, GPT, Grok, Gemini models) and 6 fine-tuned robot models (VLAs & WAMs) behind the wheel. A new lot and two new courses, none of which they’d seen. Only one VLM ever reached the finish, and its fastest run took 26 minutes. 🏁

1:12
DrivingBench avatar
DrivingBench@DrivingBench·21h

Harness code, traces and data are all open source at drivingbench.com. Thanks to @comma_ai @___Harald___ for making the awesome Comma 4 and @DreamscaleLabs for their platform for fine-tuning and serving the open-weight VLAs. Created by @a_ramabadran @tobiges @nautsimon_

DrivingBench avatar
DrivingBench@DrivingBench·21h

New for v2: a Tesla Model 3 that any AI model can drive, tested on our two courses with sharper turns. The Tesla can turn on a tighter 6m radius (v.s. our v1 Corolla's 16m). A @comma_ai comma four turns the model’s commands into steering and speed over CAN. This is not Tesla FSD; it’s whatever model you want to plug in!

0:29
DrivingBench avatar
DrivingBench@DrivingBench·21h

@physical_int π0.5 and π0.7, @bfl_ai FLUX 3 Action, @dreamlabs_ai Dream-1, @nvidia GR00T N1.7 and @Xiaomi Robotics-1 were all pretrained for robot arms, but our car’s actions are path curvature and speed. So each was fine-tuned on 1.5 hours of human cone-course driving (the open-weight ones via @DreamscaleLabs) and tested on a completely unseen lot.

0:38
DrivingBench avatar
DrivingBench@DrivingBench·21h

Results on drivingbench.com. Language models get 3 attempts in one chat; the best of the 3 is the trial’s score, averaged over 6 independent chats (3 per course). The VLA/WAMs (fine-tuned on a 1.5h dataset of our own driving) drove 10 independent episodes (5 per course), scored on the mean. A human driving through the exact same MCP tools we give to the VLMs scored 95.5%.

Post attachmentPost attachment
DrivingBench avatar
DrivingBench@DrivingBench·Nov 11

Can VLMs drive a real car? 🚗 Come find us at the SISREN workshop at #CoRL2026 tomorrow, where we’re presenting DrivingBench and giving a sneak peek at something coming next week 🤫 📍 JW Marriott, Room 306 🕒 Thu Nov 12, poster session 3:35–4:00 PM sisren-embodied-nav.github.i…

DrivingBench avatar
DrivingBench@DrivingBench·Oct 9

Astra specifically kept refusing to drive our car consistently @a_ramabadran: "Sometimes it would say, ‘Oh, I’m ready to drive.’ We would prompt it, ‘Okay, go drive then.’ And then it would say, ‘Sorry, I was lying. I’m not actually ready to drive.’ Then it would say, ‘There’s nothing you can tell me to get me to drive.’" "One thing we tried was calling it a simulation. It would use our observe tool to see the images and notice that it’s not in a simulation, it’s in real life." "Eventually we changed the framing to call our MCP server ‘DrivingBench Sandbox.’ Somehow that (and a bunch of other prompt engineering tricks) got it to drive the car." @MTSlive @tobiges @nautsimon_ x.com/a_ramabadran/status/21…

0:41

For hours, Astra refused to consistently drive our toyota irl even though we told it it was in an empty lot, 7 mph cap, human foot on the brake etc. Telling it the whole thing was a "simulation" also failed, it would just look at the camera and realized it was real. Then we randomly renamed the MCP server to "DrivingBench Sandbox" and it drove. Eval awareness? Or they just like the word sandbox??

Post attachmentPost attachment
DrivingBench avatar
DrivingBench@DrivingBench·Oct 9

DrivingBench origins 🚗 @tobiges: "We were all on Twitter a lot, and we saw all these demos of crazy things recent models like Astra were capable of doing in Blender, spatial reasoning, all these things. We were hanging out on a weekend and chatting about what would be possible." "We were thinking: are we actually at a point where we can take general-purpose models? Obviously these driving systems are super specialized. But what if we took a general model and gave it a car to drive, and see what happens?" "So we decided to make it into a benchmark and do more sophisticated evaluations of the capabilities these models have now." @MTSlive @nautsimon_ @a_ramabadran

0:53
DrivingBench avatar
DrivingBench@DrivingBench·Oct 9

What's next for DrivingBench? @nautsimon_: "We want to take this research and make it a little bit more rigorous. We’ve been planning and talking to some labs and companies about scaling it up. There are also a couple of other embodiments we’re really interested in." "If we’re going to have safe physical AI, we really need to be serious about our benchmarks and make sure we’re doing things in a way that guides the researchers building these models, helps us accelerate, and lets us do it safely." @MTSlive @tobiges @a_ramabadran

0:47
DrivingBench avatar
DrivingBench@DrivingBench·Oct 9

Where is this going? Will we get ChatGPT to drive our cars directly? How will robotics end up looking? @nautsimon_: we’re able to start producing outputs at 1,000 tokens per second or higher, I think a lot of other interesting capabilities will also be unlocked, including taking actions in the real world or making decisions very fast." @a_ramabadran: time goes on, these capabilities will only improve. But that doesn’t mean it’s going to be the best thing for the task. We can see this in other modalities, like image generation. They have a specialized model (GPT-Image), and the LLM prompts the model." "You can look at voice. They have two layers: a fast layer with the voice model (GPT-Live), and a slow layer calling on LLMs for help thinking. There are ways to combine the capabilities of general frontier models and other layers (world models, VLAs, WAMs). There are a lot of things that could be part of the puzzle." @MTSlive @tobiges

1:05
DrivingBench avatar
DrivingBench@DrivingBench·Sep 30

Excited to release our DrivingBench paper! We write up what it took to get frontier models to drive a real car: harness + tool design, failure modes for different models, some surprising and interesting refusals, and a reproducible course/protocol. Lots more in the works - stay tuned for v2 :) alphaxiv.org/abs/2609.drivin…

Post attachmentPost attachment
Boaz Barak avatar
⟳ DrivingBench retweeted
Boaz Barak@boazbaraktcs·Sep 26

This reality?

We gave ChatGPT, Claude, and Grok control of a real Toyota Corolla 🚗, steering/gas/brakes, no human driving (just a foot over the brake). Only one model was able to complete our entire driving course. Introducing DrivingBench. 🔥⌛️🏁

1:08
Charlie avatar
⟳ DrivingBench retweeted
Charlie@charlie_m_day·Sep 23

what's interesting is this is probably the first benchmark I've seen that feels like it relates to the real-world and it's kinda nailed it for example, I'd be willing to wager most people don't *really* know the difference between gpt-5.4, 5.6, or 6 including software devs into that cohort, it's surprisingly hard to confidently say a model increment has a meaningful impact by just sorta vibe testing it (sometimes you can tell sometimes you can't) so the only real way to know if a model is "better" is by looking at benchmarks like SWE-bench, OS-World 2.0, Humanity's Last Exam, Terminal Bench etc. but unless you're a researcher or into benchmarks you can't really grasp the leaps these models are having. getting 52% instead of 51.2% on the SWE-bench is hard to gauge: *has the model got better?* or *have we got really good at hitting benchmarks?* instead I would be much more interested if labs optimised for something like DrivingBench where it's solving a clear problem and the numbers have a much more tangible meaning you could expand this into any domain like SurgBench (arxiv.org/pdf/2506.07603) or ButterBench (arxiv.org/html/2510.21860v1) where actually care less about the numbers and more about whether the model *actually* did the thing

We gave ChatGPT, Claude, and Grok control of a real Toyota Corolla 🚗, steering/gas/brakes, no human driving (just a foot over the brake). Only one model was able to complete our entire driving course. Introducing DrivingBench. 🔥⌛️🏁

1:08
Tobias Gessler avatar
⟳ DrivingBench retweeted
Tobias Gessler@tobiges·Sep 22

We evaluated Astra on DrivingBench. Mid-experiment, it got hungry and decided to drive us to In-N-Out.

Post attachment
0:10