←
Logan Graham1,433 posts
Logan Graham avatar

Logan Graham

@logangraham

Head of the Frontier Red Team @anthropicai. 🌎 Make things radically good.

◷ Joined June 2009▧ 110 Photos and videos
Logan Graham avatar
Logan Graham@logangraham·Nov 9

from using Mythos/Fable 5.2 internally, this is one of those “big model smell” models — has a je ne sais quoi at writing, 3d modeling, making videos, 1-shotting interactive tools, and robotics. Also the best for cyberdefense.

Fable 5.2 shows substantially improved skills in controlling physical robots, even on tasks it has never explicitly been trained to do. On RoboDojo, a benchmark with a simulated two-arm robot, it outperformed other models and most AI systems built to control robots.

0:23
Logan Graham avatar
Logan Graham@logangraham·Nov 3

my (brilliant) fiancée is starting to study AI from a moral psychology lens -- and is thinking about better evals for real-world decisionmaking. If you want to build more/better model evals on topics like this / using good experimental designs, follow/ping!

Today, we’re releasing a dataset of >2k LLM conversations with real voters about the 2026 Midterms. We wanted to know how the ~millions of American who use AI models to talk about elections actually prompt them. Our hope is to use this data to build better evals for LLMs, to prepare for a world where most voters use AI to think through election decisions. We observe that: - People mostly want basic information - Multi-turn chats follow predictable patterns - Models differ widely in whether they’ll recommend candidates

Post attachment
Alexandra Barr avatar
⟳ Logan Graham retweeted
Alexandra Barr@BarrAlexandra·Oct 27

i feel so excited for econ majors these days — there’s so much alpha in knowing the discipline well and applying it to ai alignment seems important to study - game theory - collective action problems - invisible incentives to better understand - swarm behavior - revealed preferences - how the invisible hand guides agent behavior more than individual alignment what are ppl reading in the overlap of these two domains?

Logan Graham avatar
Logan Graham@logangraham·Oct 26

I think this story is important because it tells us about the future of cybersecurity (and of course what we can do better at Anthropic). But some of the Washington Post story / reaction on X is wrong. A few corrections: - it didn’t take 12 hours. we saw several days of attempts. After these attempts, they exploited two 3rd party vulnerabilities, one they reported and the other we identified independently (and cut off access) - nothing came close to our most sensitive systems, like model weights or Claude conversations - this wasn't an ordinary bug bounty finding – this involved exploiting 3rd parties and accessing data. While this is unusual and out of scope of our safe harbor policy, we’re grateful these issues were found From my researcher POV, the event is important because I'm concerned this is what the future will look like. Adversaries will use whatever models/tools they can. They’ll target everything with agents 24/7. They’ll hit your dependencies. I think models ~0.5-1 generations behind the current frontier can do this. And these are the dumbest models your adversaries will ever use! So, what to do? You probably want to red team yourself all the time. Current models are *very* creative. I also think we should redesign software / systems so that you’re fundamentally less exposed. (We put out some research recently on formally verifying software and reducing attack surfaces; my mental model is basically “secure code is cheap to write now, so why not rewrite everything”)

EXCLUSIVE: In less than 12 hours, a group of ex-DOGE cyber researchers were able to penetrate Anthropic’s systems using AI agents — including Slack channels where the most sensitive models are discussed. This is just weeks after Anthropic whistleblower Joseph Coxon charged the company w/launching “a hubristic gamble” from a “private company’s Slack.” Read our story here (gift link): wapo.st/4smNlsC

Logan Graham avatar
Logan Graham@logangraham·Oct 23

We are going to: + research how to make software 1-2 orders of magnitude more secure + ship these tools as fast as possible to defenders.

Logan Graham avatar
Logan Graham@logangraham·Oct 23

We should also change the way we *design* software from the start. @chrisrohlf has found we can automate reducing the attack surface of major OSS libraries:

Reducing the attack surface of deployed services and apps is a fundamental component of software that is secure by design. Earlier this year we tasked Claude with autonomously mapping and reducing the attack surface of 25 large and widely used OSS codebases. This blog post covers some of our methodology and a preview of the data we collected. This research resulted in a significant reduction in OpenSSH pre-authentication attack surface (now fixed in 10.6). And over 500 findings where reachability could be reduced and configurations hardened to ensure security controls weren’t silently disabled. anthropic.com/research/attac…

Logan Graham avatar
Logan Graham@logangraham·Oct 23

step 1: find and fix all the bugs step 2: rewrite the foundations of software to not have bugs in the first place Proud of this work from @keenlooks and @theoremlabs and team. This is a glimpse of the direction we're going.

New on our Frontier Red Team blog: using Claude to formally verify software at large scale. Formal verification can prove that code is free of certain bugs, but historically it has only been feasible for small codebases (or required a lot of expert time). With AI models, however, this is beginning to change. In collaboration with @theoremlabs, we used Claude to prove security properties in over one hundred real-world open-source libraries over the span of a few weeks. Read our research in full here: anthropic.com/research/forma…

Logan Graham avatar
Logan Graham@logangraham·Oct 19

Some updates on one of our initiatives to support OSS from the team behind it:

11 days ago, our team @AnthropicAI launched OSS Scanner to give open source projects free, periodic vulnerability scans from our strongest models, inspired by OSS-Fuzz. Some numbers so far: • ~650 projects applied • 114 accepted • 77 critical and 803 high model-rated findings reported (we haven’t scanned all projects yet) How our reports ended up flagged as malware, and other learnings/updates below 🧵

Logan Graham avatar
Logan Graham@logangraham·Oct 16

Going live to talk about worms (and cyber / autonomy / agents and whatever) in a bit here:

MTS is LIVE - here's what we're talking about this morning: - Gemini 4 Argon beats Minecraft - China instilling new export controls on AI and other tech - Taiwan planning $4 billion chip facility in Texas Followed by some great interviews: - @scottaar Professor at UT Austin talking about his recent writing and the rapid rate of AI capabilities growth in mathematics - @logangraham Frontier Red Team lead at Anthropic talking about self-replicating AI and autonomous worms - @RyanFedasiuk and @Satvik_Pen of AEI talking about their recent writing on AI chip restrictions and the race for chips - @SamoBurja Founder of Bismarck Analysis talking about live v. dead players in AI - @founderengineer co-founder of abliteration ai talking about building unrestricted ai - Ahmad Osman founder @OsmanticAI talking about open models & the hardware needed to run it - Suddu Head of Data and AI @Plaid talking about the company's latest breakthroughs in AI - @BernAceituno co-founder of StackAI (acq. by Asana) discussing what it's like building capable enterprise agents - @efarrellgregory Fellow at the Foundation for American Innovation talking about Sinomodernity - Aden Barton head of evals @mercor talking about the company's latest report on the US economy - Masumi Reynders co-founder and COO of Majestic Labs talking about the "memory wall" and what Majestic is working on

Logan Graham avatar
Logan Graham@logangraham·Oct 14

Daniel's very good thoughts on AI worms: - we look for very early glimpses to prepare early... historically, we worked on bio/cyber/autonomy in 2022-2025 when people claimed they'd never be risks! - there are different threat models, which one will happen is - everything is complicated; similar open models are a major boon for cyber defenders

Some reflections/responses to some commentary I’ve seen on the autonomous worms work: - how “scary” is this / to what extent is the point of this work to “be scary”: I think this is the wrong frame, in the same way that “you’re just fearmongering about mold!” isn’t really a coherent response to the observation that dampness will cause overgrowth of mold. We perform this research to better understand what is coming, and to help motivate cyber defensive efforts. It’s true that IoT devices have been vulnerable for a very long time, but we expect this extant vulnerability to get a lot weirder exactly _because_ of the prevalence of digital intelligences. - why would small models “want” to do this: they might not! But humans absolutely will make models do this, and many are already trying! A better guiding principle here is something more like: if a model has a capability, you should expect to see that capability exhibited in the wild. Different threat models can help you reason about the source and motive of that capability (be it humans or models), but there are enough bad actors (and probably soon enough autonomous agents) that you can safely bet on bad behavior happening. - implications for open weight: while it’s certainly true this particular threat vector wouldn’t be as severe in a world that didn’t have highly capable, small open weight models, it’s simultaneously true that open weight models have been an enormous boon for cyber defenders. The world is complicated! Positive and negative externalities can exist simultaneously! We try pretty hard to reflect this nuance in our work!

Logan Graham avatar
Logan Graham@logangraham·Oct 13

From Daniel, FRT mentor and lead on the worms research:

The world is filled with computers, and many of those computers are pretty poorly secured. Models are also becoming _really_ capable cyber agents. Combined this with the observation that frontier model performance tends to be recapitulated in significantly smaller models—models that can fit on consumer GPU hardware—within about 18 months, and you arrive at a pretty clear prediction: small, capable models will exist, they’ll be able to gobble up vulnerable compute if they want to, and they’ll be able to do this within a few months if trends continue.

Logan Graham avatar
Logan Graham@logangraham·Oct 12

Exciting work from @willzhou97 (Fellow, now full time!) and @bucketofkets on the Frontier Red Team. One of the cool parts of the job is seeing a future world early. We think this one is ~pretty underpriced. "We should aim to harden our systems to make it difficult for such replicators to spread, but also prepare to deal with a world where they are here to stay."

Anthropic's Frontier Red Team believes small AI models may now be months away from being capable of self replicating without access to a datacenter. Quoting from the blog: 'The capabilities that make AI models generally powerful coding agents also generalize to cyber tasks, including autonomous replication. Like coding agents in 2024, we’re seeing signs of life for AI worms. We don’t yet have a full rogue replicator, but agents are becoming increasingly capable of completing the individual steps in the replication chain (and passing on their adaptations to future copies of themselves). At the same time, increasing amounts of inference compute are now available both inside and outside of datacenters (see Appendix D). It’s uncertain exactly what form autonomous AI replication will take. It could be a slow, distributed “digital slime mold,” living on relatively insecure compute. It could also be a powerful cyber model that breaches datacenters at a high pace, or a model that scams people for compute credits. As a prediction that goes beyond our experiments, we expect a future awash with agents running on all types of hardware, including some that aren’t under human control. While it’s true that IoT devices have been vulnerable for a long time, lightweight agents may be much more willing than human attackers to put in the effort to compromise large fractions of the world’s IoT compute. The rise of slow replicators would also be a bellwether for more aggressive, faster ones that can breach more hardened machines or be effective API key thieves. We should aim to harden our systems to make it difficult for such replicators to spread, but also prepare to deal with a world where they are here to stay.'

Post attachment
Logan Graham avatar
Logan Graham@logangraham·Oct 8

Through The Mission (launching today), we'll release research, tools, and resources to help secure the world. Two programs: 1/ OSS Scanner, inspired by OSS Fuzz, for maintainers to receive more bug reports/pocs/fixes faster and 2/ the Critical Infrastructure Defense Program, where we will work with critical infrastructure providers to figure out what works then scale them up. Backed by our own research and engineers. Think of the Mission as the ~complement to our model access. We recently expanded cyber access this wk via our new CVP. But we need to make it really easy to find & fix everything, but also rewrite software/systems, especially for critical infrastructure. Our sense is we've got a ~few years to transform security. That's going to take require a fast feedback loop that brings AGI cybersecurity research into the world. We've built a very cool team, led by @benhawkes, and then we'll scale our work as we learn what works.

We’re launching the Anthropic Cyber Mission, a new effort to secure critical infrastructure and open-source software. anthropic.com/news/anthropic…

Logan Graham avatar
Logan Graham@logangraham·Sep 29

When I was in government, I liked to think about it as a massive information processing machine, but basically all the wires were broken and sometimes the cogs and wheels all hated each other. Obviously you just want to ask the machine a question and get an answer. You don't want to have to think about the wires and cogs and wheels! Or which ones have how much power! And that's for someone *inside* government. Then imagine being on the outside. It was ~2020 and GPT-3 had just come out. I ran around telling people this was how we were all going to do government from now on. I simulated some war games and some generals' eyes went wide. I would put quotes from GPT-3 in notes to the Prime Minister. I called up the digital service and they ~sort of laughed at the time, but kindly went along with it when I told them to form a team and work on it with me. I mostly left feeling like we should just scrap it all and rebuild a country around post-2020 technology. I'm fairly confident that the governments that do that will win.