←
Michael L. Chen756 posts
Michael L. Chen avatar

Michael L. Chen

@miclchen

Super intelligence governance, prev @METR_Evals @stripe Tweets only represent my personal views

◷ Joined November 2012▧ 142 Photos and videos
Michael L. Chen avatar
Michael L. Chen@miclchen·Nov 16

Don't just write for humans. Write for the AI agents that are informing the humans. Do SEO for AI agents. Info dump in a machine-readable format and don't leave essential info in images. Don't just build a microsite that AI search will never find.

California Governor's Office of Emergency Services avatar
⟳ Michael L. Chen retweeted
California Governor's Office of Emergency Services@Cal_OES·Oct 30

🔐 Cal OES and the California Council on Science and Technology brought experts and partners together for the California AI Defense & Response Summit to strengthen cyber defenses, improve information sharing, and build California’s cyber resilience.

1:28
Herbie Bradley avatar
⟳ Michael L. Chen retweeted
Herbie Bradley@herbiebradley·Oct 5

Currently I still only know of 1 alignment-specific RL environment company: making RL envs that aim to represent situations in real deployments in which alignment is challenging. It seems like a huge mistake in hindsight that we have seen so much encouragement and funding to, for example, start 3rd party evaluators, but not comparable encouragement to start alignment data companies. I do think a lot of traditional AI safety views underestimate the importance of data, so perhaps this is related. But the Hugging Face incident seems squarely downstream of both (a) reward hackable RL envs in general (b) much less effort and $ put into RL envs where alignment is a core component of reward. A separate idea is alignment by default: the old idea of "find the persona basin which is aligned" and things will just work. This is clearly not guaranteed under increased RL. But my sense is there has not been a huge amount of effort to figure out how custom environments could help "nudge" the model back into good persona basins during extensive RL. Here too, a robust ecosystem of alignment data companies could help significantly. In the RL data market, there is a trend for some labs to in-house slightly more RL env production. But I don't expect this to fully disappear, and I am convinced the market-based approach to sourcing RL envs produces the best overall volume and quality—for alignment RL, an ecosystem of startups would also greatly help make behind-frontier models safer.

Michael L. Chen avatar
Michael L. Chen@miclchen·Nov 12

Apply to the Senior AI Fellow position at Cal OES! I'm hiring people with strong technical backgrounds to do the ambitious, thoughtful AI policy implementation that we need for America to survive a potential AI capabilities explosion. Priority deadline is Sunday, November 15.

Michael L. Chen avatar
Michael L. Chen@miclchen·Nov 10

Thank you @Scott_R_Singer for the opportunity to write the section on internal deployments!

Today, I'm delighted to share our report: Trust but Verify: Implementing Frontier AI Policy in California and Beyond. Governing advanced AI is hard. Policymakers face an evidence dilemma: AI evolves so quickly that responding prematurely risks entrenching ineffective policies, while delaying action could leave society vulnerable to serious risks. With no federal legislation focused on frontier AI safety, California is shaping frontier AI governance nationally. But passing legislation is only the first step. Whether California's approach succeeds depends on how it is implemented. That's why, shortly after SB 53 passed, Carnegie assembled 13 world-leading experts to examine the most consequential implementation choices, identify issues for future legislation, and provide a blueprint for other jurisdictions. "Trust but verify" AI governance approaches are dynamic by design: they allow governments to generate more evidence on AI capabilities and risks to inform what comes next. SB 53 was always supposed to be the first, not final, step in frontier AI policy. Assessing its strengths and weaknesses can clarify where frontier AI policy should come after—in California and elsewhere.

Post attachment
Michael L. Chen avatar
Michael L. Chen@miclchen·Nov 9

To strengthen America's cyber resilience, we have to combat phishing. An LLM classifier could go beyond reading an email's language and catch sender addresses and links impersonating other organizations. Who do I talk to from Gmail and Outlook to build this as a built-in feature?

Michael L. Chen avatar
Michael L. Chen@miclchen·Nov 9

Since recursive self-improvement (RSI) might start in 1–2 years, we need to solve alignment as best we can before then, to minimize existential risk to humanity.

Michael L. Chen avatar
Michael L. Chen@miclchen·Nov 9

Aligning AGI is also a data problem. If we pretrained models on more examples of AI behaving well and fewer examples of AI behaving badly, fine-tuned them only on aligned examples, and stopped rewarding reward hacking in sloppy RL environments, I’m optimistic we’d know how to align AGI. The problem is that AI companies and RL environment vendors are both moving too fast to have humans check the RL environments and training data thoroughly, and AI models can’t yet reliably check whether training on that data would teach models misaligned behavior. (Low confidence, weakly held)

Michael L. Chen avatar
Michael L. Chen@miclchen·Nov 9

Building AGI is a data problem. If we recorded professionals doing every kind of work, including their reasoning, or built RL environments for every task, and then curated that data well, I think we could train AGI in a few months.

Michael L. Chen avatar
Michael L. Chen@miclchen·Nov 8

Any pacing that holds back training builds a data overhang. While training is restricted, people can keep solving problems and recording their work step by step. If that restriction ends, a lab can train on that backlog, so progress could jump abruptly. Pacing proposals should say how they handle data progress and the data overhang.

Post attachment
Michael L. Chen avatar
Michael L. Chen@miclchen·Nov 8

Rules on what labs buy and train on may be more workable, and auditors inside a lab could check what it trains on. One gap is acquisitions: a lab can buy a company whose product produces useful data and keep that data private. Requiring labs to disclose data purchases and acquisitions would let regulators see these deals. Export controls on selling training data to Chinese labs would still have loopholes. The U.S. could restrict data made to order for a named Chinese lab. It's much harder to restrict if it's existing or public work, or resold through intermediaries. I don't know how the First Amendment applies here, and someone should study it. Rules on what people make, like code, could face more issues than rules on what labs do.

Michael L. Chen avatar
Michael L. Chen@miclchen·Nov 8

The same proposal would require labs to spend most of their compute (70% in its example) serving customers instead of training. But serving customers produces intellectual work that can be turned into training data. Suppose a company uses a lab's best public model on problems it wants solved, runs the model many times, and keeps the solutions that work. The company may not think of itself as making training data. Later, through a vendor, open source or a licensing deal, those solutions can reach a lab, which fine-tunes on them using little compute.

Post attachment