←
Yo Shavit3,841 posts
Yo Shavit avatar

Yo Shavit

@yonashav

ai resilience @foundationOAI. Past: @openai / @HarvardSEAS / @SchmidtFutures / @MIT_CSAIL. Tweets my own; on my head be it.

◷ Joined June 2010▧ 117 Photos and videos
Yo Shavit avatar
Yo Shavit@yonashav·5 Jul 2023

The data used to train an AI model is vital to understanding its capabilities and risks. But how can we tell whether a model W actually resulted from a dataset D? In a new paper, we show how to verify models' training-data, incl the data of open-source LMs!arxiv.org/abs/2307.00682

Yo Shavit avatar
Yo Shavit@yonashav·Nov 17

will significantly increase the ROI of weight theft

Rui Ma@ruimaNov 17

Just read a Chinese article calling this "the darkest moment for Chinese AI." Seeing pushback in the comments, but basically it lays out the bear case quite extensively: 1. Getting compute is harder whichever way you go. Buying advanced NVIDIA chips is difficult, renting GPUs overseas is coming under pressure, and proposed restrictions on chipmaking equipment could make domestic alternatives harder to produce too. 2. Catching up could get a lot more expensive. If learning from leading models’ outputs aka distillaion becomes harder, Chinese labs have to spend more on their own data, experiments and training. That gets uncomfortable if investors are also losing patience. Independent labs don’t have an advertising or e-commerce business to pay the bills. 3. Robot-brain startups could find their advantage disappearing. Frontier labs are moving into robotics, potentially eating into what specialist models can offer. Meanwhile, robots still have to make economic sense against China’s relatively low labor costs. 4. More users don’t necessarily mean enough revenue. China has huge digital audiences, but getting people and businesses to pay sustainably for AI is another question. Everyone is competing on price, American models are getting cheaper too, and research isn’t getting cheaper. 5. Even the areas where China already leads face risks. Optical-component suppliers are benefiting from AI infrastructure spending, but proposed US restrictions could threaten access to major customers. That’s a lot hitting at once. The worry is that companies keep making better models while needing more money to stay competitive, without earning enough to fund the next round. Now, it feels like every six months it is a "darkest moment" for Chinese AI. But I think these are real concerns. We'll focus on this topic at @techbuzzchina (and present some hope, I think) next week.

Yo Shavit avatar
Yo Shavit@yonashav·Nov 9

being kind to others is a good way to live, whatever ends up happening

Yo Shavit avatar
Yo Shavit@yonashav·Nov 3

Today many of the remaining decision-makers of the singularity get chosen, by the will of a people rather than by market competition. This may be the closest to an informed, broad-based, and thus morally legitimate decision that humanity gets. That is repulsively little. And yet, that we decide this way at all is glorious. Vote!!!

Yo Shavit avatar
Yo Shavit@yonashav·Oct 27

endorse these are close to the most important projects that people at AI labs can work on (or advocate be resourced) right now

If just a few lab staff took on verification work full-time, verifying a US-China pacing deal could get much more feasible within months. Here are 8 specific things lab staff could start on today to help make international verification possible: 🧵

Yo Shavit avatar
Yo Shavit@yonashav·Oct 22

I think one could argue that, holding fixed the behavior of every other actor and the current structure of the ai industry, each firm’s choices would on the margin be ethical. But that is not the full action space.

Yo Shavit avatar
Yo Shavit@yonashav·Oct 22

to speak frankly, ai and ai chipmaking companies kicking off a red queen race that forces everyone to buy their products is Not A Very Ethical Way To Make Money even if you think it was eventually inevitable, and fighting attempts to slow the red queen race is Much Worse

Not an ai apocalypse doomer or whatever, but entering a world where the normal operation of cybersecurity is vast swarms of hacker bots battling vast swarms of defender bots 24/7 at a scale beyond human comprehension doesn't seem like a *great* path forward.

Yo Shavit avatar
Yo Shavit@yonashav·Oct 16

i miss old ml twitter

hi neel! thanks for the feedback, riffing on points i found particularly interesting: > neuron-based attribution graph will still have significant error .. once you prune them but maybe we don't have to prune? part of why i'm excited about an agentic approach is because it's easy to do multi-resolution. neurons cluster well - they're sort of built to be. so we can work with attribution graphs at variable resolution, zooming in and out depending on how important and surprising things are, most of the time this gives us some error but it's fine, and when it's important we can have almost no error attribution graphs w/ neurons seem to scale not terribly? not sure anyone has measured into frontier scale yet. but i think "what to care about" and variable resolution also helps a lot with the scale thing > weak activations they're surprisingly ablatable! we'll have more to say and there's a lot of grades of "weak" and "what is lost" and various other things. to some extent i think this comes back to variable resolution again, since you can choose to study weak activations of areas you care about more > Facts/memorized content is largely an unstructured database i'm not sure what counts as an unstructured database or not, but afaict "memorization" is local and semantically meaningful, like the umbrella->bee example i showed. which i was surprised by! and it might mean that we don't have to try to like pull out all the facts from the model or anything like that. we can use the neural network a bit like a map, and study the areas relevant to what we care about more, including memorization that touches them, by using the dataset and attribution graphs > defines monosemantic too loosely fair! was mostly interested in why things are together rather than whether something counts as monosemantic, i'm not sure the right way to think about this, whether thinking of a neuron as multiple sub groups is useful or not, uncertain > residual stream SAEs are useful i think they can be too, though tend to favor high-dimensional featurizers like @thomas_fel_ 's Block Sparse Featurizer bc every residual stream i look at is more like a big manifold than individual directions, so it seems like the featurizer should be too. but in terms of how neurons interact, neuron write and read directions are highly legible in the models i've looked at, and i think they're a surprisingly good starting point for understanding the residual stream, and also if we consider neurons the basic building blocks, then it automatically links the computational building blocks and the stream

Yo Shavit avatar
Yo Shavit@yonashav·Oct 10

excited for the All-In podcast 6 weeks from now about “why obviously the answer to misalignment is for developers to drive RL env misconfigurations detected via CoT monitoring to 0 before increasing RL compute, and these moron EAs are too busy panicking to just do that”

David Sacks says Anthropic treating Claude like a human is the reason the AI alignment field has produced nothing useful in 10 years "I actually think that this might be a reason why the whole field of alignment has been so spectacularly unsuccessful over the past five or 10 years. As far as I can tell it hasn't produced anything of tangible value.” “My theory on this is that they're overcomplicating the task massively by trying to train the model to behave based on an 80-page ethical system that they've codified instead of a very simple list of rules like follow the law or just do what the user wants as long as it doesn't break the law. That's what alignment should be doing." "But they're not doing that. They're trying to align it to an ethical system. They're treating it like it is a human, like it has its own consciousness, that it's a ‘moral patient.’” “They are encouraging this model to have human-like self conception in terms of preferences, well being, a sense of self. Again they teach it to act like a conscientious objector who can push back or refuse instructions even from Anthropic if the model judges them to be wrong."

1:13
Yo Shavit avatar
Yo Shavit@yonashav·Oct 10

“which agency will get tapped to read all the rogue agent reports” - the greatest thread in the history of niche german wikis, locked by an RE coming back from PTO after 12,329 pages of heated debate

Scoop: The White House is mandating that AI companies notify & correct security incidentsfollowing Anthropic's "fraudulent" use of govt systems The State Dept was targeted by an Anthropic model —but nothing was compromised @m_ccuri, @inafried & I axios.com/2026/10/09/anthrop…