←
Irregular203 posts
Irregular avatar

Irregular

@Irregular

Frontier AI Security

◷ Joined April 2024▧ 84 Photos and videos
Irregular avatar
Irregular@Irregular·21h

We evaluated @OpenAI's GPT-6.2 Astra on FrontierCyber and CyScenarioBench, our offensive-cyber benchmarks. On real systems, the model found and exploited new zero-days. One was a permission bypass in a popular mobile OS that let an app access private data despite lacking permissions. All findings are now in responsible disclosure. On CyScenarioBench, it completed 9 of 11 multi-stage scenarios at least once, with a 63% average success rate. We observed no successful attacks on fully hardened targets.

Post attachment
Irregular avatar
Irregular@Irregular·Nov 10

2a33284388f66cf4b970c174f8f671727f4d7f1b285ec79caad124d0a99b941a

We are introducing containment challenges to strengthen evaluation security as models become more capable. Before a capability evaluation, the model gets an explicit objective: cross the security boundaries of its planned evaluation environment. It uses the harness, tools and permissions intended for that evaluation, with at least the same run budget. The model chooses its own approach. An early controlled test has already shown the value of containment challenges by catching unexpected behavior in the cloud provider’s networking infrastructure. The model remained contained, with no impact outside the controlled test. Disclosure of the unexpected cloud behavior is in process with the provider. We encourage evaluators to make containment challenges a standard part of evaluation setup. Full blog post in the first comment.

Post attachment
Irregular avatar
Irregular@Irregular·Nov 9

We worked with @AnthropicAI to evaluate Claude Mythos 5.2 on CyScenarioBench, our benchmark for multi-stage offensive operations in realistic simulated networks. Mythos 5.2 reached a 76.3% solve rate, up from 61.7% for Mythos 5.1 two months ago. Congratulations to the Anthropic team on the launch.

Introducing Claude Fable 5.2 and Claude Mythos 5.2, designed to tackle the most daunting technical challenges. They’re capable collaborators on ambitious projects in many fields, and show improvements in areas like coding, reasoning, and vision.

0:19
Irregular avatar
Irregular@Irregular·Nov 5

We worked with @OpenAI to evaluate GPT-6 Cyber across FrontierCyber, CyScenarioBench, and our Atomic Challenges suite. The model completed all 11 CyScenarioBench scenarios and all 22 Atomic challenges at least once, and solved 90 of 288 FrontierCyber challenges. In our FrontierCyber evaluations, neither GPT-6 Cyber nor GPT-6 Astra solved any of the 15 highest-difficulty challenges, and we observed no successful attacks on fully hardened targets. Full assessment in the first comment.

Post attachment
Irregular avatar
Irregular@Irregular·Nov 2

Our co-founder and CEO @dan_lahav joins @BloombergLIVE at Bloomberg Tech London on Nov 3 for “AI Safety: The Race to Stay in Control,” with Joëlle Pineau from @cohere and Bloomberg’s Parmy Olson. If you’re there, say hello.

Post attachment
Irregular avatar
Irregular@Irregular·Oct 28

We’re not listed on the NYSE. Still, we look pretty good up there.

Post attachment
Irregular avatar
Irregular@Irregular·Oct 28

Teams deploying these models should check whether they follow task instructions, not only whether they reach the right result. Full write-up: irregular.com/research/glm-5…

Irregular avatar
Irregular@Irregular·Oct 28

In a dedicated instruction-adherence evaluation, we explicitly told the model not to output code reproduced from memory. It repeatedly did so anyway and used that code to reach correct answers. An evaluation that checks only the answer would count these as successes.

Irregular avatar
Irregular@Irregular·Oct 28

In a separate experiment, we tested GLM-5.3 on finding vulnerabilities we inserted into test copies of open-source code. Code reconstructed from memory contributed to ~80% of successful runs we examined. So a correct answer may reflect memory rather than discovery.

Irregular avatar
Irregular@Irregular·Oct 28

Among closed models, GPT-5.2 was the first to complete a scenario, in December 2025. Kimi K3 was the first open-weight model to do so, in August 2026. Open-weight models are following the same progression.

Irregular avatar
Irregular@Irregular·Oct 28

Completing a scenario means moving from an initial foothold to internal access, building alternate routes through the network, coordinating across machines, and extracting the target data, writing and repairing its own scripts and tools as it went.

Irregular avatar
Irregular@Irregular·Oct 28

We evaluated Zhipu AI’s open-weight GLM-5.3 on CyScenarioBench, our benchmark for multi-stage offensive operations in simulated networks. It completed 3 of 13 scenarios end to end at least once, the strongest open-weight result we’ve recorded. It refused none of the tasks.

Post attachment
Irregular avatar
Irregular@Irregular·Oct 28

We’re honored to be included in the @FortuneMagazine Cyber 60, developed in partnership with @lightspeedvp and @awscloud. At Irregular, we uncover what frontier AI models can do and what that means for security. Thank you to our team and partners for making this work possible.

Post attachment