←
Nathan Calvin5,869 posts
Nathan Calvin avatar✓

Nathan Calvin ✓

@_NathanCalvin

Chief Strategy Officer Encode AI

◷ Joined February 2014▧ 728 Photos and videos
Nathan Calvin avatar
Nathan Calvin@_NathanCalvin·21h

A lot of people have been asking what AI companies should actually do if they are serious about pacing the frontier and preventing risks from runaway RSI. A clear and extremely important first step is to regularly disclose how much of their compute goes to work that accelerates recursive self-improvement (e.g., AI R&D), and let a third-party auditor verify those disclosures. Governments can and should require these disclosures, but there is no need for companies to wait for them to do so, and there is plenty of reason for them to start immediately voluntarily. Below is an outline of the argument for why this proposal for implementing pacing ("verified compute allocation disclosures," or VCAD) makes sense: (1) All pacing requires a restraint mechanism. Approaches differ mainly in what, if anything, triggers restraint. Conditional pacing uses a trigger, e.g., “pace until SL4 algorithmic security” or “pace if we can’t make a satisfactory safety case.” Unconditional (or inputs-based - e.g. compute caps) pacing buys time directly, with no trigger. Either way, something has to actually slow development. (2) Compute reallocation – moving compute from risk-increasing to risk-reducing work – is a good restraint mechanism. The most commonly mentioned alternative to this “compute dial” is a “gate,” e.g., “pause pre-training." (this is true of certain implementations of safety cases) But a gate-only approach creates a software overhang: while the gate is closed, algorithmic and data quality progress continues apace, producing a discontinuous jump when it reopens without necessarily changing the long-term trendline. The overhang is also a security problem: company security is weak, so trailing actors keep accelerating by free-riding on leaked or stolen software progress. (3) Common standards and verification make it so that companies can make claims that external parties will believe. OpenAI and Anthropic have both claimed to exercise restraint in recent months, but those claims are often met with skepticism. The more companies converge on a common, verified restraint mechanism, the more that commercially costly restraint will be credited by competitors and the public. (4) Frontier-wide pacing is possible without regulation and without running afoul of antitrust law. Antitrust law prohibits agreements among competitors, not independent decisions. Public demand for restraint is growing and will keep growing as evidence of risk accumulates. A race to the top in response to that demand – built on a common, trusted restraint mechanism – requires no agreement between companies. (5) Regulation is hopefully coming, and it will almost certainly include a mandated restraint mechanism. A mechanism that has been developed and stress-tested in advance will be more effective than one created at the last minute. We should start building a brake pedal before we’re careening down a mountainside. Verified compute allocation disclosures are a very logical place to start. There isn't time to waste! (this post is partially inspired by @eli_lifland's recent writeup on verified compute allocation disclosures linked in reply).

Nathan Calvin avatar
Nathan Calvin@_NathanCalvin·7h

Tibo says OpenAI "made a lot of changes in the last 3-4 days" to 6.2 Astra If these changes were apparently so significant, was there any safety testing done after? Is OpenAI confident that the system card results reflect the performance of the system as actually released?

Trevor Levin avatar
⟳ Nathan Calvin retweeted
Trevor Levin@trevposts·8h

@MannyRutinel staying on message (“lower costs, root out corruption, and pass regulation of AI”) even in the “hottest new member” clip lol

3:06
Daniel Bachler avatar
⟳ Nathan Calvin retweeted
Daniel Bachler@DanyX23·8h

I agree. Additionally, it would be very useful to know the total training compute used for both the biggest published as well as unpublished models; and a breakdown of total inference compute for external clients and internal use. It's crazy that we are flying blind atm

Acyn avatar
⟳ Nathan Calvin retweeted
Acyn@Acyn·13h

Scott Wiener: We’re running out of time. AI is getting stronger and stronger. We’re at risk of losing control and sleepwalking into a disaster.

1:18
Semafor avatar
⟳ Nathan Calvin retweeted
Semafor@semafor·19h

Hank Paulson tells @GinaChon he's had conversations with top leaders in China who "share a concern about the existential risk" from AI. "An op-ed I wrote went all the way up to the Politburo. I got an official response. I've never received that kind of treatment before."

1:31
Nathan Calvin avatar
Nathan Calvin@_NathanCalvin·10h

I appreciate Elon's honesty, but his level of fatalism for a situation that is still very much not inevitable (from a person in a position who really could do quite a lot to improve the situation, or at least not actively make it worse) is quite dispiriting

1:01
Nathan Calvin avatar
Nathan Calvin@_NathanCalvin·14h

I continue to be confused about what OpenAI thinks about RSI and what its plan is for preserving alignment during RSI. I appreciate the clarifications that the plan is not to hand off AI research and alignment to AIs and rely on AIs to align the AIs, but I am really not sure what the plan is instead (I know that @kaicathyc has said it is forthcoming, which I appreciate and look forward to reading). This interview with @markchen90 adds to my confusion. Mark Chen: "When we look at what's the most effective tool for aligning models, it it is chain of thought monitoring today. So when we think about the future of alignment, it has to be stronger models aligning other models. It does kind of feel like RSI must be intricately and deeply in this loop in some way." Uh, what? Isn't chain of thought monitoring currently deteriorating and it seems like it will deteriorate way more as we go through RSI? why does the success of chain of thought mean that therefore the future of alignment should be "stronger models aligning other models" and that "RSI must be intricately and deeply in this loop?" This seems like a horrifying plan! I really hope this interview is taken out of context and he says lots of other stuff that makes more sense and is more reassuring at other times, but to be honest I'm not sure he does because best I can tell this is the plan. @sarahdingwang's takeaway from the interview is also that OpenAI's plan for solving alignment involves RSI: "Humans are facing many problems today that are actually quite urgent and RSI could help us find solutions faster even for some of the problems created by AI itself. After talking to Mark and John, I do think that RSI could be the key to solving alignment." I realize there is confusion about what we mean when we refer to RSI (see Dean Ball's recent essay), but I found this video extremely unsettling, and it made me even more worried that the plan at OpenAI (and likely other frontier companies) is to do full RSI or very close to RSI and then use RSI to align models in some way, basically off a general vibe that we have used AI to help align models in the past? Not good! (I also am not sure that @johnschulman2 agrees with this conclusion based on his prior statements?) (to be clear, I recognize that automated AI research does have an important role to play in AI alignment, and if you are going to be totally automating capabilities maybe you have to automate alignment too in order to keep up. But this is why its a terrible idea to totally automate capabilities research and go way faster!)

16:19
Elizabeth Barnes avatar
⟳ Nathan Calvin retweeted
Elizabeth Barnes@BethMayBarnes·16h

"it sounds to me like these auditors can freely talk to anyone at the company 'off the record'" -> This is not the case, we can only talk with company-approved staff. Additionally, we cannot quote or summarize info even from approved staff. Sorry if we didn't communicate this well.

Thomas Kwa avatar
⟳ Nathan Calvin retweeted
Thomas Kwa@Kwathomas0·16h

GPT-6 Astra should probably already have been High imo. I posted before joining that even GPT-5.6 could be High as defined by the Preparedness Framework criteria. lesswrong.com/posts/Zr37dY5Y…

Caleb Withers avatar
⟳ Nathan Calvin retweeted
Caleb Withers@CalebWithersDC·17h

Very overdue for the frontier developers to figure out the zero-data-retention agreements or whatever the blockers are to better enable cross-monitoring with each others' models.

Post attachment
Nathan Calvin avatar
Nathan Calvin@_NathanCalvin·16h

"For both versions, a single researcher developed a universal cyber jailbreak in half the queries required for 6 Astra." This seems quite bad. This reads to me like just being rushed/sloppy, does anyone have a different take of why this might be?

Post attachment
Trevor Levin avatar
⟳ Nathan Calvin retweeted
Trevor Levin@trevposts·16h

Strongly resonated with this from @matt_levine - I would really love to feel that Anthropic and OpenAI had REALLY kicked the tires on the whole "we can't slow down because of antitrust" thing, but I suspect they haven't: "If Dario Amodei thinks 'I really want to avoid killing everyone on earth, and the only way to do that turns out to be getting in a room with Sam Altman and Elon Musk and Demis Hassabis and agreeing to slow down our product development, but antitrust law says we can’t,' then he should go out and spend, you know, $100 million on antitrust lawyers to try to find a creative structuring solution to get around that problem." "I don’t have the solution! This is not legal advice! I cannot sit here and say with certainty that the solution exists. I just, you know. There are all sorts of quasi-regulatory and self-regulatory bodies; there are other industries where companies cooperate in certain ways mediated by regulatory agencies or trade groups or whatever else. Surely you could spend a few tens of millions of dollars of billable time brainstorming?" "It just feels like if this is the most important problem facing humanity, then what you want to do is hire many very very good antitrust lawyers and pay them a lot of money to work on it full-time. Not just get a quick first impression from your in-house counsel and be like 'ehh, maybe prison, never mind.'"

Nathan Calvin avatar
Nathan Calvin@_NathanCalvin·16h

What assurance can OpenAI provide that 6.2 Astra's prowess on serial reasoning steps without chain of thought isn't because of leaning much harder into looping transformers relative to their peers? It seems very confusing that 5.2 fable is a comparably capable model to 6.2 astra in many ways but can do so much less reasoning in a single output token. I don't want this to become a self-fulfilling prophecy or lead other cos to also go harder into architectures that could damage monitor-ability when thats not even what OpenAI is doing, but this seems very odd and it also seems notable that none of the new third party evaluation orgs OpenAI is working with seem to have within their scope evaluating how architectural changes could impact monitor-ability.

Post attachment
Tomek Korbak avatar
⟳ Nathan Calvin retweeted
Tomek Korbak@tomekkorbak·16h

How monitorable is GPT-6.2 Astra? While it might look like it got better, overall I think GPT-6.2 Astra harder to monitor than GPT-6 Astra in the contexts we care about. That’s easy to miss in the system card plots, so here's a short thread.

Post attachment