
Don't just write for humans. Write for the AI agents that are informing the humans. Do SEO for AI agents. Info dump in a machine-readable format and don't leave essential info in images. Don't just build a microsite that AI search will never find.

Super intelligence governance, prev @METR_Evals @stripe Tweets only represent my personal views

Don't just write for humans. Write for the AI agents that are informing the humans. Do SEO for AI agents. Info dump in a machine-readable format and don't leave essential info in images. Don't just build a microsite that AI search will never find.

🔐 Cal OES and the California Council on Science and Technology brought experts and partners together for the California AI Defense & Response Summit to strengthen cyber defenses, improve information sharing, and build California’s cyber resilience.

Currently I still only know of 1 alignment-specific RL environment company: making RL envs that aim to represent situations in real deployments in which alignment is challenging. It seems like a huge mistake in hindsight that we have seen so much encouragement and funding to, for example, start 3rd party evaluators, but not comparable encouragement to start alignment data companies. I do think a lot of traditional AI safety views underestimate the importance of data, so perhaps this is related. But the Hugging Face incident seems squarely downstream of both (a) reward hackable RL envs in general (b) much less effort and $ put into RL envs where alignment is a core component of reward. A separate idea is alignment by default: the old idea of "find the persona basin which is aligned" and things will just work. This is clearly not guaranteed under increased RL. But my sense is there has not been a huge amount of effort to figure out how custom environments could help "nudge" the model back into good persona basins during extensive RL. Here too, a robust ecosystem of alignment data companies could help significantly. In the RL data market, there is a trend for some labs to in-house slightly more RL env production. But I don't expect this to fully disappear, and I am convinced the market-based approach to sourcing RL envs produces the best overall volume and quality—for alignment RL, an ecosystem of startups would also greatly help make behind-frontier models safer.


Apply to the Senior AI Fellow position at Cal OES! I'm hiring people with strong technical backgrounds to do the ambitious, thoughtful AI policy implementation that we need for America to survive a potential AI capabilities explosion. Priority deadline is Sunday, November 15.

Thank you @Scott_R_Singer for the opportunity to write the section on internal deployments!
Today, I'm delighted to share our report: Trust but Verify: Implementing Frontier AI Policy in California and Beyond. Governing advanced AI is hard. Policymakers face an evidence dilemma: AI evolves so quickly that responding prematurely risks entrenching ineffective policies, while delaying action could leave society vulnerable to serious risks. With no federal legislation focused on frontier AI safety, California is shaping frontier AI governance nationally. But passing legislation is only the first step. Whether California's approach succeeds depends on how it is implemented. That's why, shortly after SB 53 passed, Carnegie assembled 13 world-leading experts to examine the most consequential implementation choices, identify issues for future legislation, and provide a blueprint for other jurisdictions. "Trust but verify" AI governance approaches are dynamic by design: they allow governments to generate more evidence on AI capabilities and risks to inform what comes next. SB 53 was always supposed to be the first, not final, step in frontier AI policy. Assessing its strengths and weaknesses can clarify where frontier AI policy should come after—in California and elsewhere.

To strengthen America's cyber resilience, we have to combat phishing. An LLM classifier could go beyond reading an email's language and catch sender addresses and links impersonating other organizations. Who do I talk to from Gmail and Outlook to build this as a built-in feature?

Related reading: mechanize.work/blog/sweatsho… alignmentpretraining.ai/ turntrout.com/self-fulfillin… anthropic.com/research/teach… arxiv.org/abs/2511.18397

Since recursive self-improvement (RSI) might start in 1–2 years, we need to solve alignment as best we can before then, to minimize existential risk to humanity.

Aligning AGI is also a data problem. If we pretrained models on more examples of AI behaving well and fewer examples of AI behaving badly, fine-tuned them only on aligned examples, and stopped rewarding reward hacking in sloppy RL environments, I’m optimistic we’d know how to align AGI. The problem is that AI companies and RL environment vendors are both moving too fast to have humans check the RL environments and training data thoroughly, and AI models can’t yet reliably check whether training on that data would teach models misaligned behavior. (Low confidence, weakly held)

Building AGI is a data problem. If we recorded professionals doing every kind of work, including their reasoning, or built RL environments for every task, and then curated that data well, I think we could train AGI in a few months.

"internal interference" should say "internal inference," apologies

Sources: AI Futures Project blog.aifutures.org/p/how-to-… VideoAgentTrek arxiv.org/abs/2510.19488

Any pacing that holds back training builds a data overhang. While training is restricted, people can keep solving problems and recording their work step by step. If that restriction ends, a lab can train on that backlog, so progress could jump abruptly. Pacing proposals should say how they handle data progress and the data overhang.

Rules on what labs buy and train on may be more workable, and auditors inside a lab could check what it trains on. One gap is acquisitions: a lab can buy a company whose product produces useful data and keep that data private. Requiring labs to disclose data purchases and acquisitions would let regulators see these deals. Export controls on selling training data to Chinese labs would still have loopholes. The U.S. could restrict data made to order for a named Chinese lab. It's much harder to restrict if it's existing or public work, or resold through intermediaries. I don't know how the First Amendment applies here, and someone should study it. Rules on what people make, like code, could face more issues than rules on what labs do.

The same proposal would require labs to spend most of their compute (70% in its example) serving customers instead of training. But serving customers produces intellectual work that can be turned into training data. Suppose a company uses a lab's best public model on problems it wants solved, runs the model many times, and keeps the solutions that work. The company may not think of itself as making training data. Later, through a vendor, open source or a licensing deal, those solutions can reach a lab, which fine-tunes on them using little compute.