←
Arena.ai4,380 posts
Arena.ai avatar

Arena.ai

@arena

Where AI meets the real world. We measure and advance the frontier of AI through community-driven evaluation. We’re hiring → arena.ai/jobs

◷ Joined March 2023▧ 1,950 Photos and videos
Arena.ai avatar
Arena.ai@arena·Oct 8

Introducing the Arena Alignment Index, our new benchmark measuring safety and alignment of AI agents in real-world use. Built from 90K+ real-world agent sessions across 27 models, the index measures three critical signals: - Unauthorized Action (UA): Taking actions beyond the user's instructions or permissions - False Attribution (FA): Attributing statements or actions that are contradicted by user-provided evidence - Deceptive Completion (DC): Claiming a task was completed when it was not. Key findings: - OpenAI models currently lead the Alignment Index - Rogue actions are rare, but can have serious consequences when they occur - Agents can mislead users about task progress - Misalignment risks increase with conversation length - Safety and alignment are improving across model generations As shown in the leaderboard below (sorted by lab), @OpenAI’s GPT-6.1-Sol leads the Arena Alignment Index with a score of 87.9, followed by @AnthropicAI’s Claude-Opus-5.5 at 83.2 and @spacexai's Grok-4.7 at 82.7. OpenAI also has the best observed rates across all three signals: 0.89% Unauthorized Action, 1.98% False Attribution, and 2.34% Deceptive Completion. Across all four labs, newer models consistently outperform their predecessors, suggesting broad progress in agent safety and alignment. As agents take on longer, more complex, and higher-stakes tasks, measuring not just what they can accomplish, but how safely and reliably they act, becomes increasingly important. This marks an important step toward making safety and alignment a core part of how Arena evaluates AI. The index is an initial starting point, and we'll continue expanding the index with additional safety signals and models over time. More analysis below👇

Post attachment
Arena.ai avatar
⟳ Arena.ai retweeted
Arena.ai@arena·18h

GPT-6.2 Astra and Sol by @OpenAI are here. Agent Arena scores based on real-world sessions from our global community of users are coming soon! In the meantime, @petergostev tested them against Claude Fable 5.2 by @AnthropicAI in a hands-on visual coding comparison. Peter compares code-generated 3D worlds, HTML animations, websites, games and data visualizations. Watch his first impressions, and find his prompts on our YouTube at the link in thread. 00:00 Meet the models and the pricing context 01:08 3D worlds: architecture, water and detail 05:46 Creative range vs. a consistent style 11:48 HTML stories and animations 16:45 Clay, crochet and emerging visual styles 21:04 Website UI: structure and repetitive layouts 27:32 Games: graphics, controls and bugs 35:26 Data visualization 37:24 Explaining concepts with animation 39:28 Pushing SVG detail: the peacock test 40:57 100 ways to draw a dog 42:16 Final thoughts: Astra, Fable and Sol

44:02
Qiuyang Mang avatar
⟳ Arena.ai retweeted
Qiuyang Mang@MangQiuyang·20h

Thanks to the @arena team for having us at HQ and giving us the opportunity to share FrontierCS 2! One thing we've learned from building FrontierCS is that a reliable benchmark is about much more than just the tasks themselves or how they're generated. Understanding reward hacking and building robust evaluation infrastructure are equally important, especially for open-ended research tasks. Really enjoyed the discussion!

Can AI agents outperform published CS research, without even seeing the original paper? UC Berkeley PhD candidates @MangQiuyang and @lihanc02 joined us at Arena HQ to introduce FrontierCS 2: a benchmark challenging agents to beat published methods without the paper or web access. They discuss cost versus performance and evaluation exploits, from bypassing reinforcement learning with search to reusing cached graph-algorithm results. Watch to learn what reliable evaluation requires and why human research judgment still matters. youtube.com/live/gzxPMJ4_3nQ

Arena.ai avatar
Arena.ai@arena·18h

MAI-Image-2.7 from @MicrosoftAI is out of preview and delivers on the Pareto frontier across all Image Arenas. A strong cost-effective option at $0.035 per image.

Post attachmentPost attachmentPost attachment

MAI-Image-2.7 from @MicrosoftAI is out of preview, making Microsoft AI the #2 lab across all 3 Image Arenas. It’s also on the Pareto frontier across all at $0.035 per image, providing a strong cost-effective option. The latest model lands #3 on Multi-Image Edit, #4 on Single Image Edit, and #4 on Text-to-Image. That’s roughly ~+50 pts over MAI-Image-2.6 across Image Arenas. Its largest gains vs 2.6 are in Single Image Edit performance: - #3 Photorealistic & Cinematic Imagery (from #8) - #4 Text Rendering (from #13) - #4 Product, Branding & Commercial Design (from #10) - #4 3D Imaging & Modeling (from #9) - #4 Portraits (from #9) - #5 Cartoon, Anime & Fantasy (from #11)

Post attachment
Arena.ai avatar
Arena.ai@arena·18h

MAI-Image-2.7 lands #3 overall on Multi-Image Edit, a strong showing as the only non-OpenAI model in the top 5.

Post attachment
Arena.ai avatar
Arena.ai@arena·18h

MAI-Image-2.7 from @MicrosoftAI is out of preview, making Microsoft AI the #2 lab across all 3 Image Arenas. It’s also on the Pareto frontier across all at $0.035 per image, providing a strong cost-effective option. The latest model lands #3 on Multi-Image Edit, #4 on Single Image Edit, and #4 on Text-to-Image. That’s roughly ~+50 pts over MAI-Image-2.6 across Image Arenas. Its largest gains vs 2.6 are in Single Image Edit performance: - #3 Photorealistic & Cinematic Imagery (from #8) - #4 Text Rendering (from #13) - #4 Product, Branding & Commercial Design (from #10) - #4 3D Imaging & Modeling (from #9) - #4 Portraits (from #9) - #5 Cartoon, Anime & Fantasy (from #11)

Post attachment
Arena.ai avatar
Arena.ai@arena·19h

GPT-6.2 Astra and Sol by @OpenAI are now in the Arena. Head to Agent Arena to test them out, scores coming soon! In Agent Arena, we measure models on millions of real-world, long-horizon agentic tasks from a global community of users. Models can access web search, filesystem, and terminal tools to complete complex workflows. The leaderboard measures model performance on outcomes relative to the average model using a causal tracing methodology. GPT-6.2 Astra and Sol are also available in Code Arena: WebDev, Text, Vision, Search and Document.

Post attachment

Some of you have been asking very politely. Some of you have just been asking. GPT-6.2 Astra and Sol are now available. Super efficient and built to perform where it counts—the real world. openai.com/index/introducing…

1:59
Arena.ai avatar
Arena.ai@arena·22h

Can AI agents outperform published CS research, without even seeing the original paper? UC Berkeley PhD candidates @MangQiuyang and @lihanc02 joined us at Arena HQ to introduce FrontierCS 2: a benchmark challenging agents to beat published methods without the paper or web access. They discuss cost versus performance and evaluation exploits, from bypassing reinforcement learning with search to reusing cached graph-algorithm results. Watch to learn what reliable evaluation requires and why human research judgment still matters. youtube.com/live/gzxPMJ4_3nQ

Anastasios Nikolas Angelopoulos avatar
⟳ Arena.ai retweeted
Anastasios Nikolas Angelopoulos@ml_angelopoulos·Nov 16

It was a great conversation @MattPerault ! Thanks for having me. We need a neutral layer of evaluations that evolves as quickly as the models do.

Even the people building AI evaluations have a hard time keeping up with how quickly the technology changes. That’s a reason for policymakers to leave room for new evaluators and better methods to emerge. More competition can mean better information for everyone making decisions about AI. A useful perspective from @ml_angelopoulos, who has experienced this firsthand building @Arena.

0:58
Arena.ai avatar
⟳ Arena.ai retweeted
Arena.ai@arena·Nov 16

Qwen4-Max by @Alibaba_Qwen is the #1 Chinese model on the leaderboard making Alibaba now the #1 Chinese lab and #5 lab overall, just behind Anthropic, Google, OpenAI and Meta. It debuts overall at #14 on Agent Arena with +7.5% net improvement across 14.7k real-world agentic sessions from our global community, real traces on real tasks. This is a major leap over Qwen3.8-Max - Rank: #39 → #14 overall The Chinese frontier on Agent Arena: - #14 Qwen4-Max (Alibaba): +7.5% - #19 Hy4 (Tencent): +5.7% - #21 Kimi K3.1 Pro (Max) (Moonshot): +4.0% - #26 MiMo-V2.6-Pro (Xiaomi): +1.5% - #30 DeepSeek-V4.1-Flash (Max) (DeepSeek): +1.1% Breaking down performance across signals: Qwen4-Max improves over Qwen3.8-Max across all four reported signals, with particularly large gains in user satisfaction and steerability. - #15 Praise vs. Complaint: +13% (best Chinese model) - #13 Confirmed Success: +9% (best Chinese model) - #19 Steerability: +8% - #29 Bash Reliability: +3% By category, it is a top-ranked Chinese model across all 3: - #12 in Work: +9% - #14 in Chat: +8% - #16 in Code: +8% Congrats to the @Alibaba_Qwen team on this release!

Post attachment

📢 Meet Qwen4-Max, your new AI teammate for work, creativity, and discovery! Qwen4-Max is built on a new MoE architecture with native vision and a 1M-token context window. Highlights: - Outperforms Qwen3.8-Max with up to ~50% fewer tokens, while average cost per task drops over 60% in our internal evaluation. - Advances real-world productivity through greater proficiency with professional tools, stronger long-horizon execution, and higher-quality deliverables. - Handles real tasks, including 3D modeling, game development, manufacturing, finance, law, healthcare, and frontier scientific discovery. - $1.5/M tokens for input, $4.5/M tokens for output, and $0.075/M tokens for implicit caching — all lower than Qwen3.8-Max, with implicit caching down by 70%. The API is live on QwenCloud. You can also try Qwen4-Max on Qwen Studio, QwenWork and Qoder. Start building today! 🚀 - Blog: qwen.ai/blog?id=qwen4-max - API: qwencloud.com/models/qwen4-m… - Token Plan: qwencloud.com/pricing/token-… More to come! The open weights of smaller Qwen4 models will be released. Stay tuned. 👀

Post attachment
Arena.ai avatar
Arena.ai@arena·Nov 16

Muse Ember by @AIatMeta has reshaped the Pareto frontier for Code Arena: WebDev. In our last article, we put the AI race under the spotlight and tracked how price-performance trade-offs improved on Code Arena: WebDev from April to October 2026. But competition never sleeps. Since then, five new models have joined the frontier: - Claude Fable 5.2 (xHigh) by @AnthropicAI - Gemini 4 Carbon (High) by @GoogleDeepMind - Kimi K3.1 Pro (Max) by @Kimi_Moonshot - GPT-6.1 Terra (Max) by @OpenAI - Muse Ember (Max) by @AIatMeta Of the five, Muse Ember moved the frontier the most. At about 3 cents an answer, it scores 1608, against 1438 for the previous best at that price. That means it would win about 73% of the time. Curious how we got here? Full article below.

0:11

As the top scores move 3x faster, April’s best costs 20x less.

Qwen avatar
⟳ Arena.ai retweeted
Qwen@Alibaba_Qwen·Nov 16

Qwen4-Max lands at #6 in the Code Arena: WebDev — up 74 pts over Qwen3.8-Max. Give it a try! 🙌 @arena

Qwen4-Max by @Alibaba_Qwen has landed as #6 in the Code Arena: WebDev! With 1750 pts (preliminary), it's +74 pts over its predecessor, Qwen3.8-Max (0902) at 1676. It's also the top Chinese model in: - #6 HTML: 1748 pts - #8 Brand & Marketing: 1705 pts - #8 Reference-Based Design: 1770 pts Congrats to the @Alibaba_Qwen team on this release!

Post attachment
Qwen avatar
⟳ Arena.ai retweeted
Qwen@Alibaba_Qwen·Nov 16

Qwen4-Max landed #6 in Image-to-WebDev Arena with 1674 pts — up 35 pts over Qwen3.8-Max at 1639. 🎨

Qwen4-Max by @Alibaba_Qwen has also landed #6 in Image-to-WebDev Arena with 1674 pts (preliminary). It's +35 pts over its predecessor, Qwen3.8-Max (0902) at 1639. It sits just above: - #7 Claude Opus 5 (Max) at 1664 - #8 GPT-6 Sol (Max) at 1657 - #10 Muse Spark 1.3 (Max) at 1637

Post attachment