←
Wayne Chi617 posts
Wayne Chi avatar

Wayne Chi

@iamwaynechi

CS Ph.D. at @SCSatCMU. I work on evals, agents, game dev, and CUA. Funded by @NDSEG Fellowship. Intern @arena.

◷ Joined July 2013▧ 55 Photos and videos
Wayne Chi avatar
Wayne Chi@iamwaynechi·Feb 13

New preprint alert 🚨 Can LLM agents develop video games? We release GameDevBench, the first benchmark evaluating agentic game development in a game engine, Godot. We also present two simple multimodal feedback mechanisms that lead to immediate performance gains. /🧵

0:21
Ameet Talwalkar avatar
⟳ Wayne Chi retweeted
Ameet Talwalkar@atalwalkar·Nov 12

check out @iamwaynechi 's new work studying interaction-as-a-Judge, where computer-use agents directly interact with games and applications to improve evaluation. this is an intuitive idea that works quite well, and a nice collaboration with @arena

Game and application development is getting really good. At the same time computer-use agents are finally starting to work. I don't think this is a coincidence. Improved computer-use enables the ability to interact then judge applications which then improves generation as well. Check out my new work Interaction-as-a-Judge for a study on this paradigm!

Wayne Chi avatar
Wayne Chi@iamwaynechi·Nov 12

Game and application development is getting really good. At the same time computer-use agents are finally starting to work. I don't think this is a coincidence. Improved computer-use enables the ability to interact then judge applications which then improves generation as well. Check out my new work Interaction-as-a-Judge for a study on this paradigm!

Agents are demonstrating increasingly strong capabilities in recreating games and applications. One likely reason is their growing ability to directly interact with these applications. We studied Interaction-as-a-Judge, a growing evaluation paradigm where computer-use agents directly interact with games and applications to improve evaluation. Interaction-as-a-Judge improves: - alignment with user preferences. - a model’s ability to detect errors in games. - downstream generation. We see improvements in inference-time scaling and the ability for interaction-only reinforcement learning. It also scales with computer-use capabilities; the better the computer-use model and method, the better the judgments. We’re also open-sourcing two agent trace datasets, ArenaWebdevPreferences and ArenaWebdevCorrectness, to help support future research. Dive into the thread to learn more.

Post attachment
Wayne Chi avatar
Wayne Chi@iamwaynechi·Nov 10

It feels like game development becoming a benchmark of frontier capabilities was inevitable after all :)

An arena FPS with portals - Fable 5.2 is the first model that passes my bar for seriously fun and beautiful games. We have had a few fun evenings playtesting this against eachother at the office

1:11
Wayne Chi avatar
Wayne Chi@iamwaynechi·Oct 30

Impressive work from impressive people! Amazing to see the culmination of all their hard work!

Foundation models are trained in stages: pretraining, midtraining, and post-training. But is this separation fundamental? We introduce unified training: a single algorithm for the full pipeline. This gives ⚙️ Simpler infrastructure 🔬 A framework for better science (1/n)

Post attachment
Wayne Chi avatar
Wayne Chi@iamwaynechi·Oct 29

Interactive artifacts -> games feels like the right path forward to me. I will be releasing research related to this soon 😊

Video2Game Claude can't design very original little game on his own. So I let H3 Max on @fal generate the gameplay video first, and Claude turns it into a real game you can play in the browser: every image, 3D model and sound generated on fal, written in @threejs only.

0:38
Wayne Chi avatar
Wayne Chi@iamwaynechi·Oct 13

Two models (Opus 5.5 and Sol 6.1) have finally scored over 70.0% on GameDevBench! It's really amazing to see how quickly models have improved on game development. At the same time it's so clear that there's still so much more to improve on!

Post attachment
Wayne Chi avatar
Wayne Chi@iamwaynechi·Oct 7

I am at COLM this week! Please lmk if you want to chat about games, agents, and evals!

Wayne Chi avatar
Wayne Chi@iamwaynechi·Oct 3

Evolving games are the future. Come try this out!

Last weekend I had an idea: a multiplayer world where you build anything just by describing it I invited a few friends to play, and we lost hours to it This weekend, you and your friends can get lost in Allcraft too!

0:30
Wayne Chi avatar
Wayne Chi@iamwaynechi·Sep 24

For better or worse, "I developed a new Claude skill" is the new "I developed an app"

Wayne Chi avatar
Wayne Chi@iamwaynechi·Sep 16

These Anthropic safeguards have gone beyond just an annoyance. The platform is barely usable. Been doing a full force switch to codex simply because Fable is growing more unusable by the day.

Wayne Chi avatar
Wayne Chi@iamwaynechi·Sep 10

My talk at Arena on GameDevBench is up!

Game development remains one of the most-requested, and most-challenging, categories on Arena. @iamwaynechi, PhD candidate at Carnegie Mellon University and research intern at Arena, just walked us through GameDevBench: a benchmark built from real tutorials that turns game development into verifiable, deterministic tasks. How do top frontier models perform on tasks a human beginner could complete in under an hour, and is the biggest bottleneck coding or multimodal understanding? Check out Wayne’s full talk to hear results and learn more about GameDevBench: youtube.com/watch?v=yoHcvrmE…

0:47
Wayne Chi avatar
Wayne Chi@iamwaynechi·Sep 9

Human driven game design with ai driven technical development will be the future. Really cool to see this in godot also!

Week 6 of building a fishing game entirely with AI What started as an experiment is now a playable game you can actually download. The creator has zero game dev experience. Claude handles all the coding in Godot, while Claude and now GPT-6 Astra help create the 3D assets. Characters are made with Tripo3D. This week added new islands, docking areas, fishing progression, improved lighting, redesigned buildings and an almost finished starting zone. No one-shot prompting. Just 6 weeks of building, testing, fixing and seeing how far a fully AI-made game can go.

1:49
Wayne Chi avatar
Wayne Chi@iamwaynechi·Sep 9

Note that this is still clearly far from saturation! And if you work on game dev it's obvious that models are great at prototyping but still fail on many small details. More work to be done!