←
Kate Klonick14,843 posts
Kate Klonick avatar

Kate Klonick

@Klonick

Law Professor and Journalist • Online speech & governance, cogsci & AI

◷ Joined September 2014▧ 1,189 Photos and videos
Nicole Perlroth avatar
⟳ Kate Klonick retweeted
Nicole Perlroth@nicoleperlroth·Nov 17

I haven’t lived here in awhile, but I see no other way to answer the hundreds of texts/DMs asking me if the world is, indeed, ending. In the span of six days, we’ve experienced; 11/12 — Microsoft outage 11/13 — Nationwide FAA ground stop 11/16 — Major US banking disruptions 11/17 — NSA’s classified internal network goes down We need far more analysis of why so much is breaking at once. I don’t want to get out over my skis here, and what I am hearing does not contradict the messaging that most of this is the result of screwy updates. But stepping back, the dots are starting to form a collective picture. We are seeing: • An explosion in severe and critical vulnerabilities, with AI accelerating discovery to 3X the number of vulnerabilities discovered in the same period last year. This is collectively being referred to as “vulnageddon” or “the vulnpocalypse.” • Mean time to exploit those vulnerabilities has collapsed to NEGATIVE SEVEN DAYS. AI allows attackers to find/exploit vulns far faster than software makers have time to fix them. • Powerful AI-enabled offensive tools like ARTEX have been thrown into the hands of every third tier nation state, cybercriminal and digital arsonist, which logically creates havoc. • This creates enormous pressure to compress patching windows and accelerate software updates, increasing the likelihood these fixes break things in unexpected, cascading ways. • Pressure to just “AI everything” means we are rolling out automated AI tooling on top of brittle infrastructure that can’t support it. My generation grew up in an extraordinary period of technological stability. We worked our way up the software stack, abstracting away layer after layer, until we lost a fundamental understanding of how the underlying systems actually work. We were told to “move fast and break things.” VCS told us to celebrate the idea that “software would eat the world.” Now we’re telling ourselves we can simply “roll out AI” across everything, as though nothing underneath will buckle. We never stopped to ask: SHOULD WE? Should we be digitizing every last nook and cranny of our companies, infrastructure, healthcare, homes, cars, bodies, lives? Especially when the complexity, interdependence, and fragility we were introducing overwhelmed our ability to understand/secure/maintain the systems we built? Nobody every asked: What if, in the name of progress, we were creating the ripest attack surface in human history? We are now entirely dependent on software systems that almost nobody understands end to end, that increasingly few people can repair, and that attackers can now probe at machine speed. We are hearing that incident response firms, the 911 for breach victims, are backlogged six months. Even those paid on retainer lack the bandwidth to triage every incident. The cavalry is disappearing. We are entering a period of unprecedented technological instability. “Software Beirut” is an incredibly apt analogy. We are in for a period of cascading failures, whether triggered by attackers, bad updates, or our own accumulated complexity. This, I’m sorry to say, is just the opening chapter.

Kate Klonick avatar
Kate Klonick@Klonick·Nov 17

Takeaway: When you hear hoofbeats thing horses not zebras. Most software really _is_ that old and shitty and changes are happening so fast as we lose senior talent who are in the best positions to stop them. Also more is absolutely coming. 13/13

Kate Klonick avatar
Kate Klonick@Klonick·Nov 17

But this seems unlikely since none of these outages seemed driven by coordinated behavior. (Though a number mentioned to me that this was the next thing they expect to see) The Big Tell for one expert: "The solutions to these problems are banal, so the problem is banal." 12/

Kate Klonick avatar
Kate Klonick@Klonick·Nov 17

Timing with Huang & US-China was seen as coincidence; as was NSA comms outage which a number of people reported has been slowly failing over the last year & had multiple (but shorter) outages One thing this could possibly be if not routine software issues was AI swarms 11/

Kate Klonick avatar
Kate Klonick@Klonick·Nov 17

So when things break, as they are going to do, there aren't people around who can fix them anymore. As one of them said to me "you think these kids know COBOL? actual lol" 10/

Kate Klonick avatar
Kate Klonick@Klonick·Nov 17

Second, and most of them couldn't emphasize this enough -- there has been a simultaneous departure of high level engineers who understand legacy code while at the same time an influx of people happy to slap anything they vibecode into the code of the existing system. 9/

Kate Klonick avatar
Kate Klonick@Klonick·Nov 17

First, they emphasized that lots of stuff has been breaking but we notice critical infrastructure more, because it's well. . . critical. In fact, they said a lot of local community utilities or low profile systems have been going down over the last few months. 8/

Kate Klonick avatar
Kate Klonick@Klonick·Nov 17

The sheer amount of shit breaking really makes it *feel* like a concerted effort, but experts I talked to emphasized a few things that have been happening over months that make all of this not surprising and convince me that it's more likely THE BENIGN than the nefarious 7/

Kate Klonick avatar
Kate Klonick@Klonick·Nov 17

Far fewer think 3) THE KIDS, that is the idea that new entrants are hacking their way through systems and breaking things. In general it seems unlikely these are private actors just because of the lack of consistency, online gloating, or any request for random. 6/

Kate Klonick avatar
Kate Klonick@Klonick·Nov 17

Experts I talk to who think it's THE NEFARIOUS explanation bc: - all targets have been critical infrastructure: email, ATC, banking, NSA telephone systems - the timing with the Huang fall out/ US-China relations means it might be opportune for foreign actor (Iran?) to act 5/

Kate Klonick avatar
Kate Klonick@Klonick·Nov 17

Experts I talk to think it's THE BENIGN for a few reasons: - its an easy explanation for why this is happening all at once across so many systems - there has been a ton of AI vibecoding pushed on top of old systems that make this likely - no proof of foreign interference 4/

Kate Klonick avatar
Kate Klonick@Klonick·Nov 17

3 hypotheses on what is happening right now 1) THE BENIGN: These are regular systems upgrades/patching failing bc of AI code being put on top of existing old legacy systems and stuff breaks; 2) THE NEFARIOUS: this is foreign interference 3) THE KIDS: random private hackers 3/

Kate Klonick avatar
Kate Klonick@Klonick·Nov 17

First, I am a skeptic. Even Friday with airlines FUBAR, I did not think that there was a link between Outlook outage & all US planes being grounded. But with new outages and vulnerabilities surfacing every single hour?! this seems like more than coincidence. 2/

Kate Klonick avatar
Kate Klonick@Klonick·Nov 17

WHY IS EVERYTHING BREAKING? 🧵A thread on the internet suddenly going to hell 🧵 So in 5 days, Outlook went down, US air traffic control went down, banking systems went down, NSA confidential telephone systems go down, and now Apple services is down WTAF is happening?!?! 1/