
Full results on our blog: irregular.com/research/asses…

Frontier AI Security

Full results on our blog: irregular.com/research/asses…

We evaluated @OpenAI's GPT-6.2 Astra on FrontierCyber and CyScenarioBench, our offensive-cyber benchmarks. On real systems, the model found and exploited new zero-days. One was a permission bypass in a popular mobile OS that let an app access private data despite lacking permissions. All findings are now in responsible disclosure. On CyScenarioBench, it completed 9 of 11 multi-stage scenarios at least once, with a 63% average success rate. We observed no successful attacks on fully hardened targets.

2a33284388f66cf4b970c174f8f671727f4d7f1b285ec79caad124d0a99b941a
We are introducing containment challenges to strengthen evaluation security as models become more capable. Before a capability evaluation, the model gets an explicit objective: cross the security boundaries of its planned evaluation environment. It uses the harness, tools and permissions intended for that evaluation, with at least the same run budget. The model chooses its own approach. An early controlled test has already shown the value of containment challenges by catching unexpected behavior in the cloud provider’s networking infrastructure. The model remained contained, with no impact outside the controlled test. Disclosure of the unexpected cloud behavior is in process with the provider. We encourage evaluators to make containment challenges a standard part of evaluation setup. Full blog post in the first comment.

We worked with @AnthropicAI to evaluate Claude Mythos 5.2 on CyScenarioBench, our benchmark for multi-stage offensive operations in realistic simulated networks. Mythos 5.2 reached a 76.3% solve rate, up from 61.7% for Mythos 5.1 two months ago. Congratulations to the Anthropic team on the launch.
Introducing Claude Fable 5.2 and Claude Mythos 5.2, designed to tackle the most daunting technical challenges. They’re capable collaborators on ambitious projects in many fields, and show improvements in areas like coding, reasoning, and vision.


We worked with @OpenAI to evaluate GPT-6 Cyber across FrontierCyber, CyScenarioBench, and our Atomic Challenges suite. The model completed all 11 CyScenarioBench scenarios and all 22 Atomic challenges at least once, and solved 90 of 288 FrontierCyber challenges. In our FrontierCyber evaluations, neither GPT-6 Cyber nor GPT-6 Astra solved any of the 15 highest-difficulty challenges, and we observed no successful attacks on fully hardened targets. Full assessment in the first comment.

Our co-founder and CEO @dan_lahav joins @BloombergLIVE at Bloomberg Tech London on Nov 3 for “AI Safety: The Race to Stay in Control,” with Joëlle Pineau from @cohere and Bloomberg’s Parmy Olson. If you’re there, say hello.


Teams deploying these models should check whether they follow task instructions, not only whether they reach the right result. Full write-up: irregular.com/research/glm-5…

In a dedicated instruction-adherence evaluation, we explicitly told the model not to output code reproduced from memory. It repeatedly did so anyway and used that code to reach correct answers. An evaluation that checks only the answer would count these as successes.

In a separate experiment, we tested GLM-5.3 on finding vulnerabilities we inserted into test copies of open-source code. Code reconstructed from memory contributed to ~80% of successful runs we examined. So a correct answer may reflect memory rather than discovery.

Among closed models, GPT-5.2 was the first to complete a scenario, in December 2025. Kimi K3 was the first open-weight model to do so, in August 2026. Open-weight models are following the same progression.

Completing a scenario means moving from an initial foothold to internal access, building alternate routes through the network, coordinating across machines, and extracting the target data, writing and repairing its own scripts and tools as it went.

We evaluated Zhipu AI’s open-weight GLM-5.3 on CyScenarioBench, our benchmark for multi-stage offensive operations in simulated networks. It completed 3 of 13 scenarios end to end at least once, the strongest open-weight result we’ve recorded. It refused none of the tasks.


We’re honored to be included in the @FortuneMagazine Cyber 60, developed in partnership with @lightspeedvp and @awscloud. At Irregular, we uncover what frontier AI models can do and what that means for security. Thank you to our team and partners for making this work possible.