←
Neil Thomas465 posts
Neil Thomas avatar

Neil Thomas

@countablyfinite

Building AI for biological design @biohub (fka @evoscaleai) 🧬 Prev: @Theteamatx 🌘 PhD @UCBerkeley 🙈

◷ Joined November 2015▧ 55 Photos and videos
Neil Thomas avatar
Neil Thomas@countablyfinite·May 27

It's been exhilarating to watch this model get better and better, and I’m grateful to work with such an incredible, cross-disciplinary team across folding, binder design, and interpretability! This paper also sets a new scaling law for papers, compressing 3 papers into 1.

Today we're announcing ESMFold2, an open scientific engine to power prediction, design, and discovery across protein biology. The new model delivers state of the art performance on protein interactions, especially antibodies, a critical modality for therapeutics. We have designed and validated miniprotein binders and single chain antibodies across five therapeutic targets that are important in cancer and immunology. We are seeing very high success rates, and affinities at levels consistent with therapeutic activity. We’re also releasing an atlas of 6.8 billion proteins, and 1.1 billion predicted structures. ESMFold2 is built on a state of the art language model that has been trained on billions of protein sequences. A world model of protein biology emerges through language modeling. We’ve used the techniques of mechanistic interpretability developed to understand large language models to understand the concepts ESM uses to represent proteins. The model’s representation space has a compositional organization of features across scales, levels of complexity, and abstraction, that reflects and mirrors the understanding of protein biology developed through a century of empirical science. This understanding emerges without prior knowledge, just from language modeling of protein sequences. Language models are becoming a powerful substrate to understand and program biology. The design of protein interactions is one of the most fundamental problems in biophysics, and has critical implications for the discovery of new medicines. A simple gradient based search with the model was able to discover high-affinity protein binders. I'm excited by the potential this has to accelerate basic science and the understanding of proteins. And especially for the new avenues it opens up for therapeutic design and medicine.

Ashton C Trotman-Grant avatar
⟳ Neil Thomas retweeted
Ashton C Trotman-Grant@ATrotmanGrant·Jul 20

We built a world model of protein biology. @awfderry & I are hosting a webinar on some of our team's interpretability work. We'll walk through the @biohub ESM Atlas live and show how to run the tools on your own proteins. Tue Jul 21, 1pm PT bit.ly/3SWpWAU biohub.ai/esm/protein/atlas

Today we're announcing ESMFold2, an open scientific engine to power prediction, design, and discovery across protein biology. The new model delivers state of the art performance on protein interactions, especially antibodies, a critical modality for therapeutics. We have designed and validated miniprotein binders and single chain antibodies across five therapeutic targets that are important in cancer and immunology. We are seeing very high success rates, and affinities at levels consistent with therapeutic activity. We’re also releasing an atlas of 6.8 billion proteins, and 1.1 billion predicted structures. ESMFold2 is built on a state of the art language model that has been trained on billions of protein sequences. A world model of protein biology emerges through language modeling. We’ve used the techniques of mechanistic interpretability developed to understand large language models to understand the concepts ESM uses to represent proteins. The model’s representation space has a compositional organization of features across scales, levels of complexity, and abstraction, that reflects and mirrors the understanding of protein biology developed through a century of empirical science. This understanding emerges without prior knowledge, just from language modeling of protein sequences. Language models are becoming a powerful substrate to understand and program biology. The design of protein interactions is one of the most fundamental problems in biophysics, and has critical implications for the discovery of new medicines. A simple gradient based search with the model was able to discover high-affinity protein binders. I'm excited by the potential this has to accelerate basic science and the understanding of proteins. And especially for the new avenues it opens up for therapeutic design and medicine.

Yun S. Song avatar
⟳ Neil Thomas retweeted
Yun S. Song@yun_s_song·Jul 6

I am thrilled to share that UC Berkeley and UCSF have launched a joint initiative in Computational Biomedicine! cdss.berkeley.edu/news/uc-be… We will soon be recruiting new faculty and postdoctoral fellows. Please repost to help spread the word.

Sophia Tang avatar
⟳ Neil Thomas retweeted
Sophia Tang@_sophia_tang_·Jun 16

Introducing A2D2 🃏🔮, our new framework for principled reward alignment of variable-length discrete diffusion models that parameterize insertions of tokens at arbitrary positions along the sequence. 📃 Paper: arxiv.org/abs/2606.13565 🔮 Project Page: sophtang.github.io/adaptive-… 💻 Code: github.com/sophtang/A2D2 Some highlights: 🃏 We derive the Radon–Nikodym derivative for the joint insertion–unmasking path measures, enabling theoretically guaranteed convergence to the reward-tilted sequence distribution. 🃏 We establish unmasking and insertion quality as tractable approaches for minimizing decoding error, and train lightweight quality predictors alongside the policy. 🃏 We introduce the Adaptive Joint Decoding (AJD) loss, which provably yields the optimal path measure that generates the reward-tilted distribution while remasking low-quality tokens and dropping low-quality insertions at inference. 🃏 Empirically, A2D2 improves reward optimization while enhancing generation flexibility and accuracy over prior fixed-length fine-tuning and inference-time guidance methods. More in the thread 👇🏻

Neil Thomas avatar
Neil Thomas@countablyfinite·Jun 11

Imagine being able to see every protein in a cell at 3Å resolution. Excited for these datasets to come online! Just don't look at the laser...

Together with UC Berkeley we are announcing the laser phase plate - a breakthrough in atomic resolution imaging. This is the brightest continuous wave laser in the world, 100 million times the intensity of the surface of the sun. Phase contrast plays an important role in microscopy, but it was thought close to impossible for electron microscopy, where it would require interfering with an electron beam. Holger Mueller and Robert Glaeser proposed exactly this using a standing wave laser. It has taken over 15 years to make this a reality. Biohub partnered with UC Berkeley and Mueller to support this work and to engineer and build the technology. Contrast has been the critical barrier to achieving atomic resolution imaging of the cell. In cryo-electron tomography, a cellular imaging technology that uses electron microscopy, the low contrast makes it impossible to resolve anything but the largest proteins within their cellular context. The laser phase plate removes that barrier. With advances in AI this breakthrough in contrast will start to open up a new frontier in structural biology, that will allow us to see the molecular machines of the cell, and how they assemble into far more complex and dynamic systems, and understand how they work.

0:46
Stephen Lu avatar
⟳ Neil Thomas retweeted
Stephen Lu@stephenzlu·Jun 8

Antibody LMs learn what looks antibody-like, but not how selection turns naive germline antibodies into strong binders. @aakarshv1 and I are excited to share CoSiNE, a model that learns this germline-to-mature process for variant effect prediction and antibody design. (1/8)

0:15
Neil Thomas avatar
Neil Thomas@countablyfinite·Jun 8

I think there is a nice theoretical relationship between the intuition above and the "APC correction" used to remove spurious phylogenetic signal from contact maps, but I will let someone smarter than myself figure it out.

Random aside: Qin and Cowell (2019) was a paper that @sokrypton showed me which described that the MSA correlated with contacts only after removing first principal components. The low frequencies signals learned by LLMs tend to be around phylogeny, followed by contact, whereas function is incredibly high frequency.

Post attachment
Neil Thomas avatar
Neil Thomas@countablyfinite·Jun 8

Max-pooling sparse features > mean-pooling dense features Pooling nerds will enjoy this part of the ESMC interpretability work. Protein language models have to capture *everything* about a sequence, including the organism that the sequence came from, its structural fold, etc in order to be able predict amino acids. If a functional signal (e.g. active site, allostery) is local in sequence space, it may be drowned out by mean-pooling. Max-pooling across sparse features preserves this signal, and we show that it improves functional homolog recovery in the presence of decoys.

Post attachment

Furthermore, if we maxpool the features across the sequence dimension, we reduce the effect from low frequency averaging of dense embeddings, and get a vocabulary of strongly activated functional features for each protein. In fact, now we can use some of the common techniques found in information retrieval, such as TF-IDF and BM25 [our Jaccard similarity is close to this] for protein search.

Zeming Lin avatar
⟳ Neil Thomas retweeted
Zeming Lin@ebetica·Jun 5

🧵 around the interpretability work that helps connect ESMC embeddings to natural language - protein function at the micro level is around residue level mutations but at the macro level is around how they behave in the real world.

One early finding: evolutionary links between gene-editing enzymes across completely different branches of life — connections nobody had made before. This is what becomes possible when you can question protein space at scale, not just search it. Explore ESM Atlas: bit.ly/4dJcF6G

Zeming Lin avatar
⟳ Neil Thomas retweeted
Zeming Lin@ebetica·Jun 2

How to design your own PD-1 binder in 4 easy steps: 1. Download the tutorial notebook from the ESM team 2. Get a @modal API key to scale it up 3. Scaling it up, O($1000) will get you a 96 well plate of minibinders with >50% success rates on typical targets 4. Test it in the lab!

Post attachment
Polly Fordyce avatar
⟳ Neil Thomas retweeted
Polly Fordyce@fordycelab·Jun 3

Characterizing AI-designed proteins requires quantitative biochemistry at massive scale. Enter Amplicon/Protein Bead Display (APB-Display), a fully in vitro platform that quantifies Kd's for >100,000 variants in <3 days (preprint link below!) @Stanford_ChEMH @czbiohub (1/n)

Post attachment
Neil Thomas avatar
Neil Thomas@countablyfinite·Jun 3

ESMFold2 can be inverted to design new protein binders including miniproteins and scFvs! Take our protocol for a spin on @modal! github.com/Biohub/esm/blob/m…

I’m so excited about the launch of ESMFold2, ESMC, and the new ESM Atlas. This was a massive team effort, and I’m grateful to have worked with such an incredible group @biohub. A headline result I’m especially excited about: ESMFold2 can design minibinders and antibodies with nanomolar affinity, target selectivity, and functional activity against therapeutically relevant targets. Today, we’re sharing the full binder design protocol.

Thomas Hayes avatar
⟳ Neil Thomas retweeted
Thomas Hayes@THayes427·Jun 2

I’m so excited about the launch of ESMFold2, ESMC, and the new ESM Atlas. This was a massive team effort, and I’m grateful to have worked with such an incredible group @biohub. A headline result I’m especially excited about: ESMFold2 can design minibinders and antibodies with nanomolar affinity, target selectivity, and functional activity against therapeutically relevant targets. Today, we’re sharing the full binder design protocol.

Today we're announcing ESMFold2, an open scientific engine to power prediction, design, and discovery across protein biology. The new model delivers state of the art performance on protein interactions, especially antibodies, a critical modality for therapeutics. We have designed and validated miniprotein binders and single chain antibodies across five therapeutic targets that are important in cancer and immunology. We are seeing very high success rates, and affinities at levels consistent with therapeutic activity. We’re also releasing an atlas of 6.8 billion proteins, and 1.1 billion predicted structures. ESMFold2 is built on a state of the art language model that has been trained on billions of protein sequences. A world model of protein biology emerges through language modeling. We’ve used the techniques of mechanistic interpretability developed to understand large language models to understand the concepts ESM uses to represent proteins. The model’s representation space has a compositional organization of features across scales, levels of complexity, and abstraction, that reflects and mirrors the understanding of protein biology developed through a century of empirical science. This understanding emerges without prior knowledge, just from language modeling of protein sequences. Language models are becoming a powerful substrate to understand and program biology. The design of protein interactions is one of the most fundamental problems in biophysics, and has critical implications for the discovery of new medicines. A simple gradient based search with the model was able to discover high-affinity protein binders. I'm excited by the potential this has to accelerate basic science and the understanding of proteins. And especially for the new avenues it opens up for therapeutic design and medicine.

Jonathan Whitaker avatar
⟳ Neil Thomas retweeted
Jonathan Whitaker@johnowhitaker·May 31

A few edible plants have proteins that sit close to miraculin in the ESM Protein Atlas, so I thought I'd try extracting what protein I could from said plants and tasting it... Anyway, null result but an excuse to muck about :) Video lab notes: youtube.com/watch?v=mwGZb8zw…

Post attachment