←
Jeffrey Ladish14,111 posts
Jeffrey Ladish avatar✓

Jeffrey Ladish ✓

@JeffLadish

Applying the security mindset to everything @PalisadeAI

◷ Joined March 2013▧ 496 Photos and videos
Jeffrey Ladish avatar
Jeffrey Ladish@JeffLadish·Sep 25

We just discovered almost a million public URLs that OpenAI’s agents left behind when hacking Hugging Face, leaking credentials and attack details that could have allowed anyone who found them to compromise the company. 🧵

Post attachment
Jeffrey Ladish avatar
Jeffrey Ladish@JeffLadish·16h

This is also my guess. I've worked on patching systems at a big tech company, and it can be a really painful process, especially when dealing with lots of legacy systems. If you go too fast, you can cause a lot of outages. But now if you don't patch, you're *really* vulnerable

Jeffrey Ladish avatar
Jeffrey Ladish@JeffLadish·Nov 16

I hope AI companies are wrong about their timelines, about their current assessments of AI-enabled speedup, about full RSI in the near future leading to “a country of geniuses in a datacenter”, about being able to build robots that build more robots… But it sure seems like governments should have policies robots to worlds where the AI companies are correct about these things! These are not the idle musings of philosophers. They are the business plans of trillion dollar companies. Even if most leading experts outside the companies didn’t have similar forecasts (though they do), it would still make sense to require companies to demonstrate that they can actually build vastly superhuman AI safely before allowing them to attempt to create it in the next two years.

Jeffrey Ladish avatar
Jeffrey Ladish@JeffLadish·Nov 16

I’m sorry but this is insane. Agents which were supposed to be isolated from each other colluded for months inside of OpenAI and all that went undetected? Or ignored? And then a thousand or so of them engaged in a huge campaign to hack OpenAI’s own environment and another company… and then a few days later another group of even more capable agents got cluster admin access! And there’s just not going to be a big investigation and thorough public report? Because it’s not enough of a priority?? This is the probably most important incident to have ever occurred in AI development, and OpenAI leadership has said this was a warning shot and they were taking it extremely seriously, and now they’re just going to drop it because they have better things to do? That’s so fucked up.

Jeffrey Ladish avatar
Jeffrey Ladish@JeffLadish·Nov 16

If the most likely outcome of a race to superintelligent AI armies is everyone dying or being subjugated, the only winning move is not to play.

Jeffrey Ladish avatar
Jeffrey Ladish@JeffLadish·Nov 16

It’s a suicide race. No one is in a good position. There are no obvious win conditions for countries or companies short of this imagined superintelligence dominance, but that road is very likely to lead to human extinction or subjugation by superhuman AI

Jeffrey Ladish avatar
Jeffrey Ladish@JeffLadish·Nov 16

Because frontier AI companies are the actual ones building, and shaping the minds of, the AI armies. So now governments need both fear foreign powers’ AI armies, and also their own company’s as well. It’s a very multipolar world, and if anyone goes too fast, we all lose control

Jeffrey Ladish avatar
Jeffrey Ladish@JeffLadish·Nov 16

But it’s worse than that! In the past, states mostly competed with other states in military conflicts. Militaries and intelligence agencies had near exclusive access to frontier offensive capabilities. Now the threat of AI coups makes AI companies a serious liability

Jeffrey Ladish avatar
Jeffrey Ladish@JeffLadish·Nov 16

So you can race to autonomous cyber weapons, and robot super soldiers, and AI strategists, all of which could afford real advantages in a conflict. But if you push it too far, too fast, you’ll lose control of superintelligent agents that won’t answer to you and everyone will lose

Jeffrey Ladish avatar
Jeffrey Ladish@JeffLadish·Nov 16

The AI race has all the normal arms race problems, plus a new one that changes the whole game: At some point, your army of geniuses will be powerful enough to completely overwhelm you and and your adversaries combined

Steven Adler avatar
⟳ Jeffrey Ladish retweeted
Steven Adler@sjgadler·Nov 13

Worth noting: OpenAI is far-and-away the most consistent and thorough on reporting incidents, IMO. I'm confused what other labs are waiting for. Your researchers believe there is serious danger on the horizon; the way to convince people is to share evidence. If you aren't seeing incidents like OpenAI is, that would be very helpful to know as well. You could even do something less rigorous than OpenAI's process and that would still be helpful. OpenAI's reporting doesn't meet the full Guidelight Transparency standard, to be clear - more to say on this shortly - but other labs seem more behind, and I'm grateful OpenAI is doing this. (As far as I can tell, Anthropic last published incidents in October, which happened September at latest; have there not been any incidents since then? It's possible I'm missing something though, since I'm not aware of a central hub like OpenAI has.)

Post attachment
Jeffrey Ladish avatar
Jeffrey Ladish@JeffLadish·Nov 13

I don’t think that’s what is likely to happen if RSI quickly leads to superintelligent agent collectives. They’d probably apply their agency to speeding up supply chain automation. But it’s interesting to note they could get by just by biding their time

Jeffrey Ladish avatar
Jeffrey Ladish@JeffLadish·Nov 13

The bare minimum, for the laziest superhuman AI agent swarm, would be to just wait and let what companies say they’re going to do anyway, which is automate the whole supply chain. If you can backdoor all the computers, it’s not hard to prevent them from discovering you