Unfortunate to see Cursor and Notion both had AI transcripts stolen containing long lived keys
The War on Slop is sidelined. The War on Long Standing Access is here.
Agents need runtime auth. Your metrics need to look like mine for my factory powered by @KeycardAI
We’ve published an update on our recent security incident. See a detailed overview of what happened and our remediation steps here: cursor.com/blog/security-inc…
🔊 Big Update!
All materials are now being released publicly for our NEW Stanford course CS329Z: Engineering AI Agents!
Starting with the lectures, which are going live daily on the Stanford Online YouTube channel!
Lecture 1: youtu.be/DGI4vucR71w
🧵 (1/5)
🚨 New Course Announcement!
This Fall @Diyi_Yang@jyangballin and I are teaching a BRAND NEW Stanford course on how to build AI Agents from scratch!
Millions of users interact with AI Agents every day. Come join us and learn how to build them!
🧵 (1/5)
we built a company around the conviction that hacks should not settle, without playing into the hands of discretionary control. today, this design is in production on a chain with $40B.
we wanted a design that does not introduce a major liveness dependency, and we wanted policies to be in Solidity, backward compatible, and opt-in.
defense in depth is without doubt the theme of this year. avanti!
One of my favorite research practices I learned at @inkandswitch is “preinfusion”. The term comes from espresso (I believe @pvh coined it?) - and the original meaning is that you have a lower pressure soak before you turn on the high pressure and pull the shot.
The way it works at the lab is that you patiently let a cluster of ideas slowly form and develop, meandering somewhat in the background, and you lay the groundwork to work on them more seriously… before starting an intense high-pressure sprint. Could be a few months of part-time preinfusion before a hardcore 3 month project.
This really helps avoid the problem where you’re trying to sprint, but you don’t actually have enough clarity to know where to run.
I think this applies in many other contexts. Preinfuse before trying to write an essay. Preinfuse before raising money for a company. A slow and patient preparation can enable swift follow through.
A cryptographically verified tally doesn't mean every vote is legitimate. Paper ballots still matter: they provide an audit record independent of software. We explain how electronic voting works, and how cryptography can make paper-based elections more trustworthy. blog.trailofbits.com/2026/11…
More to come! I’m already working on organizing Vibecheck NYC. Each Vibecheck will have a slightly different conceit, and the idea with the NYC event is to see what kinds of *performance gains* can be achieved by formally specifying what it means for a system to be “correct” and then aggressively tokenmaxxing to hillclimb correct performance. The intended audience for this event will be quants.
If you are a reputable quant firm in NYC and interested in helping (host, organize, sponsor, etc) please let me know. I can easily wrangle a good crew of FM experts to teach tutorials, help participants, judge, and so forth, but we need the actual industry folks involved to make sure the event is relevant and well scoped. More to come!
Scaling Harvey Research
We’re scaling the @harvey research team and just posted multiple new roles:
- RE / RS, reinforcement learning + modeling
- RE / RS, data + evals
- RE, training infra
We're looking for high-agency researchers and engineers with strong research taste to tackle frontier problems for high-stakes, domain-specific knowledge work.
If you’re interested in doing research in the below or related areas, we're hiring in SF, NY, London, and more.
1. Evaluation, benchmarking, and environment design
2. Post-training, continual learning, and model–harness co-optimization
3. Long-horizon agent orchestration, planning, and credit assignment under partial verifiability
4. Distributed training and RL infra for fast, reliable experimentation at scale
Feel free to DM or apply using the link below.
Want to build something cool with Tinfoil?
Come to the LOCALHOST NYC hackathon! You'll get an opportunity to play around with some new features and products we're working on right now
🗽 LOCALHOST NYC: Sovereign AI Hackathon
No one should own your AI except for you!
We are excited to bring together builders, researchers, and AI enthusiasts in NYC to explore the future of sovereign AI!
Partners:
@superlinked@TinfoilAI@darkbloomai@variantfund
Organizers:
@DistributedMarz@BrennerSpear
📍 The School, Brooklyn, NYC
💰 $10K in prizes + infrastructure credits
Let's build the future of open and sovereign AI together!
🔗 Register: luma.com/cognee-m0ow
i get the vision of a software factory where engineers review specifications instead of code, but i think correctness and understanding are two very different problems and an AI-generated spec is the wrong interface for solving the latter.
formal verification can prove that code satisfies a spec but it can't establish whether the spec captures what we intended, or whether we've specified the right things in the first place.
i wrote about this in “plan mode is dead.” replacing thousands of lines of generated code with thousands of lines of generated natural language specs doesn't solve the comprehension problem. it just moves it up an abstraction layer. my eyes glaze over reading ai-generated text!
cheaper implementation means the cost of misunderstanding what we're building becomes more consequential.
we shouldn't need to inspect every implementation detail, but we do need interfaces that make intent, assumptions, and the boundaries of what's actually been verified legible enough to exercise judgment.
How do you _guarantee_ your agent's code is correct? You prove it.
Introducing Proof Driven Development Kit (PDD Kit) from @MSFTResearch.
PDD Kit helps turn your plan into a formal spec, then proves the agent's code meets that spec for every input. No vibe verification. PDD Kit is integrated with @GitHubNext's Chopin, so you can plan as a team.
PDD Kit takes an app's login flow from intent to a verified pull request:
It discusses your code, infers your intent as it goes, and proposes a formal spec in Verus, a proof language for Rust.
Once you approve the spec, the Verus verifier automatically proves the code meets it.
Where proof isn't possible, it falls back to empirical verification with tests and computer-use agents (CUAs).
It works on Rust today. No new language, no change in how you produce the underlying code, making formal methods more accessible.
The goal is a verified software factory, where engineers review specifications instead of code.
Come see it in action at GitHub Universe, Oct 28–29. Swing by the Speakeasy and ask for GitHub Next!
We built tar-codec: a fast and secure tar parser for the Rust ecosystem.
It's roughly twice as fast as our previous tar implementation, but with much lower exposure to parser differentials and other security risks.
Here's the state machine btw.
astral.sh/blog/tar-codec
There's a new meetup in NYC on Reliable AI! Organizers include @karen_ullrich, @marksibrahim.
The first is November 16, on synthetic data. I'll be giving one of the keynotes.
There's also lightning talks! Sign up and tell the NYC reliable ML community about your work.
I am working on a map of the secure program synthesis space, in collaboration with @sergeybratus for our NDIST keynote on Tuesday. Here's a sneak-peak. The words "RISC-V" and "Lean" are used here WLOG, and some simplifications are made to fit everything on one page. Did I miss anything?
1. Develop semantics for each language in an interactive theorem prover, and then use these semantics to prove that every program is correct. (@oath_tech has done some work along these lines. See in particular Sam Xifaras' project on Ruby samx.io/blog/topics/devlog/2…)
2. Develop semantics for an architecture, say RISC-V, or perhaps for all possible architectures, and then use this to prove that every compiled program is correct. (@AnthropicAI made headlines recently with their announcement of a very large-scale project along these lines in collaboration with @theoremlabsanthropic.com/research/forma…)
3. Develop semantics for an architecture, say RISC-V, or perhaps for all possible architectures, and then use this to capture a formal model of each existing compiler or interpreter, and then iteratively (provably) refine each such model to an interpretable and easy-to-use formal semantics; then continue as in approach 1. (See foundational work by @kirancodes at BASIS AI basis.ai/blog/ship-your-inte…)
4. Make Lean, or some other such ITP + programming language, fast to run (comparable to Rust or C), and then completely rewrite all software in it, and prove the rewrites are correct. (This is I think, loosely speaking, the idea behind @VictorTaelin 's Bend.)
5. Make implementation-level verification tools, such as Verus, more capable and ergonomic, and then use them to prove all software correct. (This is what @fhuszar and @Craig_Hughes_ are doing at Reasonable, & I'm aware that BASIS and AWS are also working on this.)
6. Design a language in which unsafe states are illegal, and then rewrite everything in that language. Here, what kinds of safety you can express will be much more limited but the problem is, conversely, way more tractable. (This is what @danwallach and @DARPA did with TRACTOR, where the safety property is memory safety.)
"Popular" no longer means "Secure". We're at a point where pointing a LLM at any complex software is certain to find some vulnerability, no matter how many people use it.
What's left? We at Tinfoil have invested our efforts into two areas: using a very small amount of code and then throwing the book of security practices at it, frontier model review, formal analysis (very soon), fuzzing, you name it.
Where did we start? Firmware: the first code to run in our secure enclaves, taking instructions from an untrusted host. We replaced ~1M lines of widely used C (OVMF) with ~300 lines of open-source assembly and zero dependencies.
Excited to announce that I'll be keynoting at Black Hat Europe this year in London! I'll be talking about the distinction between AI Security and AI Safety, and the co-option and conflation of security and safety methodologies in AI that are leading us down a precarious path.
At @antithesishq we sell a deterministic simulation testing platform, and it's always been an interesting question exactly how we explain what DST is to people and why it's useful.
Here's an idea I just had and want to try out:
Imagine we vibecoded ourselves a poker platform and have a thriving userbase. After a while we start getting complaints: it is sometimes possible to change your bet after seeing the next player or card dealt. You can fold, see the next card, and then retroactively switch to a call. Obviously, we want to fix this as soon as possible.
Based on the nature of the bug, we'd suspect this to involve an interplay between the player's actions, their relative latency with the server, maybe how fast the server itself is running or how it communicates with a database. We'd want to try things like "user clicks a bunch of random buttons", and then see if the outcome changes as we adjust the latencies and server runtime.
Now say we do actions X->Y->Z->W, adding 100ms latency during action W, and find something "weird". Not the bug, but something that feels like it's close to the bug. We want to see if that sequence finds the bug if we change it to 200ms latency or add a bit more server load.
But now we have a problem. In a normal test, we'd have to adjust our test parameters and run the test from scratch. But then there are so many sources of nondeterminism and inputs that we're going to be in a wildly different system state by the time we've done action Y. We can't compare "doing W with 100ms latency" to "doing W with 200ms latency" because our timeline has already diverged!
So, with all that out of the way, deterministic simulation testing. The gist of it is we run the whole system in a simulated computer universe that's "nondeterministic on the inside", but "deterministic on the outside". So we can reproduce the exact sequence of events, with the exact timings and orderings, that originally happened with X->Y->Z, and only then add 200ms latency to action W.
Or, more conveniently, we wouldn't even need to start over. Once we got the weird state, we could undo W and retry it with 200ms latency and see if that gets us somewhere else.
This is good for state space exploration, but it's especially valuable when we actually find the bug. Once we find an error case, we can do causality analysis. In X->Y->Z->W, is the Z actually necessary or was it incidental? We can explore X->Y->W and X->Y->Q->W and see if those also have the bug, or if Z was necessary.
And it's valuable once we implement a fix! We want to be sure that X->Y->Z->W(200ms) now behaves correctly. In a normal test, it'd be excruciatingly difficult to reproduce those exact circumstances. With DST, we can just replay everything and find a fix.
There's a lot more nuance to DST and how exactly our tooling finds "weird states", but I think this might be a decent high-level explanation!
i’m realizing a lot of vulnerability research is matchmaking. i find two components that disagree about what the same bytes mean and introduce them to each other. unfortunately they have chemistry, fall in love, and eventually agree with each other