Given a mountain of slop proof that checks out, it’s not just that humans can’t understand it. Also the AIs may have trouble building on it until the proof is simplified.
Now apply to AI companies trying to delegate alignment to not-yet-superintelligent machines during RSI...
Fascinating observation from the researchers behind the world record for the fastest matrix multiplication algorithm:
AI struggles to understand its own mathematical reasoning, but becomes much better at building on its ideas when humans simplify them.
Norbert Wiener, a forefather of control theory and arguably the first alignment researcher, warned in 1960 that we had better be sure “the purpose put into the machine is the purpose which we really desire.” New post: patterning as control (in the Toy Model of Superposition).
A bit of advertising: I recently completed revisions for a tutorial paper titled "Geometric Functional Inequalities for Markov Processes: An Invitation via the Langevin Diffusion", which can now be found online at arxiv.org/abs/2611.07785.
Why are many orgs aggressively hiring/scaling up, yet it's extremely hard to find a full time position?
Why are there calls to expand the number of people doing research in AI safety, yet our academic institutions are crumbling under the weight of the sheer number of papers and outputs?
Why are we exhorted to do research slowly and deeply, yet sped along on crunch time, with the hope that scale will clarify our confusions?
In this new blogpost (done in my personal time), I argue that these are rooted in a tension between a frame of "war mode" and "civilisation mode". This tension, in my view, creates uneasiness and despair, as well as makes it difficult to see clearly.
Check it out on LW: lesswrong.com/posts/7e38jwHh…
or on Substack: davidafrica.substack.com/p/n…
New paper: we generate Lean-grounded synthetic data to train verifiers that can find mistakes in natural language math proofs.
Broadly, I hope we can use Lean and formal-to-informal generalization as part of the solution to supervising superhuman models in the future.
I've tweeted before about how architecture improvements usually improve the constant factor in loss scaling laws, keeping the same exponent. The scaling ladders in the Ripple-65B release from @RicursiveAI are a nice illustration of this at scale! 🧵
Hi! this is Rohan Anil, I develop training algorithms among other things - my work trains various frontier models.
At CoreAuto, I was fascinated by the problem of training on compute islands, eventually in space training.
I hope you all enjoy this write up!🪐🛰️
coreauto.com/hitchhikers-gui…
"Assessing AI Consciousness" is now available as a book: hutter1.net/ai/bayesconsc.ht… It develops a Bayesian framework separating theoretical credence from empirical evidence, providing a precise, scalable method for calibrated assessments (with online tool and code): ai-cognition.org/cacophony-t… Happy Halloween.
I’m really excited to read this! I’m a bit biased because I think @BuchananBen is a gem of a human and I really enjoyed working with him back at Georgetown/CSET. I also want to get to the bottom of something that’s been bugging me since I read @kevinroose's 'The AGI Chronicles', and I hope 'The Bitter Struggle' covers it.
There are a few passages in 'The AGI Chronicles' that suggest Dario Amodei and Jack Clark played a role in shaping how the Commerce Dept thought about export controls (am attaching the pages here so folks know what I’m referring to).
In a nutshell, Roose suggests that Amodei and Clark sent a memo to Buchanan on export controls, and then Buchanan drafted his own proposal on export controls, which eventually became policy. The causation is fuzzy, and I think Roose is probably trying to avoid over-claiming how much influence Amodei and Clark had.
It matters a lot how policies get developed, and I think it would be great to hear more accounts from folks who were in the room about what went down. (Bonus if @jackclarkSF wants to write a book...)
I led AI policy in the Biden administration, and my colleague @tantumscollins was Director for Technology on the National Security Council. Our book on AI and national security -- THE BITTER STRUGGLE -- comes out in 11 days. @tylercowen said it “will become the standard source” on the history of the China chip bans and Biden-era AI policy. @crmiller1 said it is “essential reading for understanding how AI will impact geopolitics and how the great powers are shaping AI.”
We wrote this book because the decisions shaping AI -- and the future for all of us -- are too important to only be debated behind closed doors. If you're planning on reading it, pre-ordering is one of the best ways to help us bring this conversation to a broader audience.
penguinrandomhouse.com/books…
This is so awesome and exciting!!! I'm really grateful to @sethlazar and the others involved. It's hard to create new infrastructure in academia (especially academic philosophy), and many hours of labour go into building and maintaining journals. Thank you for this!
Announcing the Call for Papers for Oxford Studies in Philosophy of AI and Computing!
Rapid AI progress is raising *so many* philosophical questions; but philosophers aren't as involved in the conversation as they should be, for many different reasons.
One is that, aside from some work in logic and philosophy of mind, philosophy of AI and computing has often sat on the margins of the discipline. One reason for *that* is that so many philosophers working on AI have had to do so basically on their own, figuring things out for themselves as they go along, without a real community around them.
OSPAIC is going to help fix this! We're going to have an annual conference and volume. The conference (in DC! where much of the future of AI will be decided) is especially important as we think about the nature of intellectual life in a world where AI systems can one-shot a superficially-plausible philosophy paper. OSPAIC sees philosophical progress as being about more than just the accumulation of true claims. The conference is a big part of our vision for how we contribute to philosophical progress.
And our aim is also to help philosophers be part of the broader public conversation! To that end we're also going to be launching, soon, a subcode on arXiv for philosophical foundations of AI (cs.pfai), and encouraging everyone who submits to OSPAIC to also upload their draft to arXiv (at present, use keyword PFAI).
We're also partnering with @cosmos_inst, who will be sponsoring an annual prize for the best paper on philosophy and AI, as well as helping fund the first round of the conference and Open Access publication for the volume. And with Cosmos we'll be highlighting some of the most interesting submissions to cs.pfai as they come in.
I'm the executive editor; Jeff Howard (UCL) and Rachael Sterken (University of Hong Kong) are associate editors; Ned Howells-Whitaker (JHU and Pittsburgh) is the assistant editor. We're grateful for the support of a wonderful advisory editorial board, including brilliant philosophers and computer scientists from academia, industry, and civil society.
The deadline for submissions is 11.59pm Feb 14, 2027, anywhere on earth. All details are at ospaic.org.
I agree with Geoffrey that even bringing AI up to the standards of more rigorous industries might not be enough. His examples are from engineering, but I'd say the same about governance.
Many proposals for more serious AI governance take finance as their model, calling for independent audits, bank-style supervision or an AI version of FINRA. I worked in AI in finance for a few years, and I understand the appeal. These approaches are certainly better than the status quo at AI companies. But we shouldn't treat them as sufficient.
The financial sector has had decades to build these controls, and they still fail. We've seen big failures, and there are many, many small ones that never reach the headlines, so the public record makes these mechanisms look more reliable than they are.
The risks in finance are also better understood and more predictable than the risks posed by AI systems that are new and change quickly, let alone systems that are, as Geoffrey says, good at being adversarial. These governance mechanisms would keep us safer than we currently are, but probably not safe.
Lots of conversation around AI risks is about the gap between the unserious of AI labs' methods and how more rigorous industries work: airplanes, nuclear power, AI hardware. It would help a ton to lift AI to that level, but it is unclear even that level would be enough. 🧵
Lots of conversation around AI risks is about the gap between the unserious of AI labs' methods and how more rigorous industries work: airplanes, nuclear power, AI hardware. It would help a ton to lift AI to that level, but it is unclear even that level would be enough. 🧵
I am happy to announce Resolution's theoretical computer science (TCS) leadership!
1. Omer Reingold is joining as TCS Technical Lead, alongside his work at Stanford. ❤️
2. Tal Rabin is joining as Board Advisor, and helping with team building and leadership within TCS. ❤️