←
Adam Kaufman1,525 posts
Adam Kaufman avatar✓

Adam Kaufman ✓

@eccentric1ty

MTS @ Redwood Research (@redwood_ai); opinions my own etc etc

◷ Joined July 2023▧ 22 Photos and videos
Nate Soares ⏹️ avatar
⟳ Adam Kaufman retweeted
Nate Soares ⏹️@So8res·16h

Notably, "just do it and dare them to sue" is the tactic taken on copyright, when it was their product rather than our lives that was on the line.

Adam Kaufman avatar
Adam Kaufman@eccentric1ty·17h

If we are sufficiently concerned about a model's alignment to want to monitor it, but have to use similarly untrusted models in order to catch any misbehavior ("untrusted monitoring"), we have to worry about the monitors colluding and deciding not to report subtle misalignment.

Adam Kaufman avatar
Adam Kaufman@eccentric1ty·17h

Eval awareness isn't just concerning because it means we can't trust our evals; it also opens the door to scary kinds of score laundering during training if AIs can intuitively recognize training environments

Post attachment
Cody Fenwick avatar
⟳ Adam Kaufman retweeted
Cody Fenwick@codytfenwick·Nov 17

I still find it totally wild that the chief scientist of OpenAI posted this on the company blog, and we all just moved on to the next day's news

Post attachmentPost attachment
Sydney avatar
⟳ Adam Kaufman retweeted
Sydney@SydneyVonArx·Nov 16

Sam Altman said OpenAI would provide third-party evaluators with employee-level access to their systems. This is not that. OAI is apparently banking on no one checking the details. Instead of employee-level access: - Assessors have extremely narrow scopes for their work. Nobody can verify pacing, assess overall risk, or see if runs are likely to be safe before they begin. METR is literally only allowed to assess at the alignment of one RL run. - Assessors don't have access to Slack or the codebase. - Apollo and METR can’t publish anything based on employee interviews unless OpenAI gives permission (or if they can find a different source). Apollo can only talk to whitelisted employees via logged conversations.

Jeffrey Ladish avatar
⟳ Adam Kaufman retweeted
Jeffrey Ladish@JeffLadish·Nov 16

The AI race has all the normal arms race problems, plus a new one that changes the whole game: At some point, your army of geniuses will be powerful enough to completely overwhelm you and and your adversaries combined

Jasmine Wang avatar
⟳ Adam Kaufman retweeted
Jasmine Wang@j_asminewang·Nov 17

This level of access + control is not enough for real oversight. This is also very, very far from the “employee-level access” that Sama promised in September. It's very limited: - No access to Slack - No freedom to talk freely in private with employees - A narrow, pre-agreed scope - Other limits @alextmallen outlines below This is exactly what @tomekkorbak, @balesni, and I were worried about in October: OpenAI gesturing at embedded assessment without living up to the spirit (or even the letter) of the promise. It’s why we wrote to OpenAI’s board and safety leadership asking them to make sure METR and other assessors got employee-like access. OpenAI responded on Twitter afterwards that they were following through on embedding external assessors. Today’s news shows OpenAI hasn’t followed through. I recognize that lots of hard work has gone into these agreements, and am grateful to my ex-colleagues for putting in that work. Notably no labs other than Anthropic & OpenAI have published *any* follow-on to their evaluator commitments. But OpenAI needs to do better here.

Post attachment
Tomek Korbak avatar
⟳ Adam Kaufman retweeted
Tomek Korbak@tomekkorbak·Nov 16

As others have pointed out, I think the form of those four third-party engagements is in many respects a step back compared to OpenAI’s previous engagements (with METR). For example, during the monitor red-teaming exercise, METR had access to OpenAI’s monorepo but this now seems to be off the table. Instead, we get very narrowly scoped (e.g. single RL run for METR, single threat model for Apollo) engagements where auditors only get to talk to “authorized” employees who are provided with OpenAI’s “guidance” beforehand. Based on my experience, this arrangement makes it very easy for OpenAI to carefully select models, subsets of data and employees to paint a rosy picture of the actual risk landscape. I think embedded auditing will only fulfill its aims if the auditors really get employee-level access as opposed to being hired to do a specific job (scoped by the company). The latter will invariably lead to important evidence being left out, missed or misrepresented. There are at least three reasons why: 1. First, it’s very difficult to anticipate all the questions that should be asked, or all of the data that should be analyzed. OpenAI tried that with METR during the Hugging Face investigation and as a result METR needed to come back to OpenAI’s office twice because on two previous occasions they didn’t get access to all the agent trajectories involved in the incident. And even then, the METR report ended up being very narrowly scoped, e.g. leaving the crazy story of agents taking control of OpenAI research infra out of scope. 2. Second, the company under audit should not be allowed to decide what is audited and what information the auditor should have access to. There is an inherent conflict of interest in allowing the company to do so! 3. Finally, without employee-level access, the only way for auditors to get anything close to a comprehensive picture is to rely on trusted relationships with many employees throughout the company who are willing to tell auditors information that may reflect poorly on the company. But I find it unlikely OpenAI employees will always raise those safety issues when (as I hear from my colleagues inside OpenAI) they are afraid they will share the fate of @balesni, @j_asminewang and me, fired after talking to third parties. Sadly, OpenAI makes it clear now that @sama’s September tweet about “employee-like access” was just an awkward overpromise that the company wants to quietly walk back.

Jacques avatar
⟳ Adam Kaufman retweeted
Jacques@JacquesThibs·Nov 16

A comment from Paul, yesterday. Notice the part about @allTheYud at the bottom.

Post attachment
1:38
Chana avatar
⟳ Adam Kaufman retweeted
Chana@ChanaMessinger·Nov 17

Nate is so goddamn good at this. One by one, the main questions people have about AI risk and the possibility of existential catastrophe.

15:54
Mikita Balesni avatar
⟳ Adam Kaufman retweeted
Mikita Balesni@balesni·Nov 16

The week of my departure from OpenAI, I saw external auditors negotiating with OpenAI over small improvements to access and scope to the company's safety-relevant information, and asked my colleagues “Why are we having these conversations at all?” — @sama already publicly committed to continued, employee-like access for external assessors — and these negotiations sounded not even close. Nobody had a good answer. Today, OpenAI announces the engagements they’re doing with METR, Apollo Research and others — and as many have already pointed out today, it looks like OpenAI employees were not able (although I'm sure they tried) to hold the leadership to @sama’s public commitments: * No Slack access (for context, all OpenAI safety discussions and debate happen over Slack; anything that misses these conversations will lack important insights) * OpenAI picks employees that assessors are allowed to talk to. * If an employee raises a concern to assessors, they cannot disclose it publicly even if it doesn’t contain confidential information, unless assessors themselves verify it or OpenAI volunteers their own statement confirming the employee’s concern. Assessors cannot quote employees, even anonymously. * Some of the external assessors (not sure if all?) are barred from publishing any results about internal-only models (this means no new results for the model responsible for the Hugging Face incident; and if internal-only models are used for RSI, no alignment assessments of the world's most dangerous models) Beyond the object-level details, if you read posts by each of the third party groups, they all say their terms do not meet the scope and access that they themselves proposed previously. Unless OpenAI has concrete plans for reaching the employee-like bar in the immediate future and just forgot to say that in the post, this looks very unfortunate. To those who hope we'll get there someday, I remind that time is short: last week some of the best AI forecasters updated their superhuman researcher timelines from a range between 2028-2033 to mid-end next year.

Post attachment
Daniel Kokotajlo avatar
⟳ Adam Kaufman retweeted
Daniel Kokotajlo@DKokotajlo·Nov 16

It seems like there STILL won't be a thorough investigation into the hugging face incident, even after all this time! I keep talking to OpenAI researchers and asking about this, and what they say privately is also basically "it's just never going to happen sorry, not enough of a priority." To reiterate what I said at the time: Serious investigation of the HF incident could be a boon to alignment science for the whole world. For example, try putting the same models in the same situations, but varying the conditions slightly, to see what would have happened. Could also look at sources of evidence other than CoT to try to get more insight into model psychology, e.g. when one agent considered warning a human and decided not to saying "not my task" what else was it thinking? To what extent was this a rationalization vs. an outright lie vs. an innocent mistake? We could learn a lot about AI motivations and psychology by doing a bunch of experiments like this. More ambitiously, could try training new models with different recipes / alignment training / etc. to see what would have prevented the incident and what wouldn't have.

Post attachment
Adam Kaufman avatar
Adam Kaufman@eccentric1ty·Nov 16

Actually, come to think of it, Boeing literally built the 737 Max so as to technically share the same type rating as the previous 737 NG in order to avoid mandatory pilot retraining, and this killed a few hundred people

Adam Kaufman avatar
Adam Kaufman@eccentric1ty·Nov 16

I find this move on GDM's part particularly indefensible. We wouldn't let Boeing get away with claiming that their new plane isn't that much faster or bigger, and hence it doesn't need a new safety evaluation!