New preprint alert 🚨
Can LLM agents develop video games?
We release GameDevBench, the first benchmark evaluating agentic game development in a game engine, Godot.
We also present two simple multimodal feedback mechanisms that lead to immediate performance gains.
/🧵
check out @iamwaynechi 's new work studying interaction-as-a-Judge, where computer-use agents directly interact with games and applications to improve evaluation. this is an intuitive idea that works quite well, and a nice collaboration with @arena
Game and application development is getting really good. At the same time computer-use agents are finally starting to work. I don't think this is a coincidence.
Improved computer-use enables the ability to interact then judge applications which then improves generation as well.
Check out my new work Interaction-as-a-Judge for a study on this paradigm!
Game and application development is getting really good. At the same time computer-use agents are finally starting to work. I don't think this is a coincidence.
Improved computer-use enables the ability to interact then judge applications which then improves generation as well.
Check out my new work Interaction-as-a-Judge for a study on this paradigm!
Agents are demonstrating increasingly strong capabilities in recreating games and applications. One likely reason is their growing ability to directly interact with these applications.
We studied Interaction-as-a-Judge, a growing evaluation paradigm where computer-use agents directly interact with games and applications to improve evaluation.
Interaction-as-a-Judge improves:
- alignment with user preferences.
- a model’s ability to detect errors in games.
- downstream generation. We see improvements in inference-time scaling and the ability for interaction-only reinforcement learning.
It also scales with computer-use capabilities; the better the computer-use model and method, the better the judgments.
We’re also open-sourcing two agent trace datasets, ArenaWebdevPreferences and ArenaWebdevCorrectness, to help support future research. Dive into the thread to learn more.
An arena FPS with portals - Fable 5.2 is the first model that passes my bar for seriously fun and beautiful games.
We have had a few fun evenings playtesting this against eachother at the office
Foundation models are trained in stages: pretraining, midtraining, and post-training. But is this separation fundamental?
We introduce unified training: a single algorithm for the full pipeline. This gives
⚙️ Simpler infrastructure
🔬 A framework for better science
(1/n)
Video2Game
Claude can't design very original little game on his own. So I let H3 Max on @fal generate the gameplay video first, and Claude turns it into a real game you can play in the browser: every image, 3D model and sound generated on fal, written in @threejs only.
Two models (Opus 5.5 and Sol 6.1) have finally scored over 70.0% on GameDevBench!
It's really amazing to see how quickly models have improved on game development. At the same time it's so clear that there's still so much more to improve on!
Last weekend I had an idea: a multiplayer world where you build anything just by describing it
I invited a few friends to play, and we lost hours to it
This weekend, you and your friends can get lost in Allcraft too!
These Anthropic safeguards have gone beyond just an annoyance. The platform is barely usable.
Been doing a full force switch to codex simply because Fable is growing more unusable by the day.
Game development remains one of the most-requested, and most-challenging, categories on Arena.
@iamwaynechi, PhD candidate at Carnegie Mellon University and research intern at Arena, just walked us through GameDevBench: a benchmark built from real tutorials that turns game development into verifiable, deterministic tasks.
How do top frontier models perform on tasks a human beginner could complete in under an hour, and is the biggest bottleneck coding or multimodal understanding?
Check out Wayne’s full talk to hear results and learn more about GameDevBench: youtube.com/watch?v=yoHcvrmE…
Week 6 of building a fishing game entirely with AI
What started as an experiment is now a playable game you can actually download.
The creator has zero game dev experience. Claude handles all the coding in Godot, while Claude and now GPT-6 Astra help create the 3D assets. Characters are made with Tripo3D.
This week added new islands, docking areas, fishing progression, improved lighting, redesigned buildings and an almost finished starting zone.
No one-shot prompting. Just 6 weeks of building, testing, fixing and seeing how far a fully AI-made game can go.
Note that this is still clearly far from saturation! And if you work on game dev it's obvious that models are great at prototyping but still fail on many small details.
More work to be done!