←
Evan Miller1,140 posts
Evan Miller avatar

Evan Miller

@EvMill

Statistically inclined software developer, occasional blogger about math + stats stuff. Working @AnthropicAI

◷ Joined May 2009▧ 63 Photos and videos
Klaviyo avatar
⟳ Evan Miller retweeted
Klaviyo@klaviyo·11 Feb 2025

🚀 New on the Klaviyo Data Science Podcast: @EvMill joins us to discuss his paper, Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. AI metrics are everywhere—but how much uncertainty is behind them? Understanding variability matters. Listen now: bit.ly/3CV0CmX #AI #DataScience

Post attachment
Anthropic avatar
⟳ Evan Miller retweeted
Anthropic@AnthropicAI·2 Dec 2024

We’re starting a Fellows program to help engineers and researchers transition into doing frontier AI safety research full-time. Beginning in March 2025, we'll provide funding, compute, and research mentorship to 10–15 Fellows with strong coding and technical backgrounds.

Post attachment
Marius Hobbhahn avatar
⟳ Evan Miller retweeted
Marius Hobbhahn@MariusHobbhahn·14 Nov 2024

This paper on the statistics of evals is great (and seems to be flying under the radar): arxiv.org/abs/2411.00640v1 The author basically shows all the relevant statistical tools needed for evals, e.g. how to do compute the right error bars, how to compare model performance, and how to do power analysis. Back when @jeremy_scheurer and I wrote the "We need a Science of Evals" post (apolloresearch.ai/blog/we-ne…) this paper is exactly the kind of thing we had in mind and more.

Evan Miller avatar
Evan Miller@EvMill·5 Aug 2024

I think I've finally cracked quantiles… A/B testing medians, instead of means, usually requires an expensive bootstrap. But we can use a likelihood-ratio test (Wilks' theorem) instead. This reduces the quantile problem to a few simple formulas. Read on! arxiv.org/abs/2401.10233

Nat Friedman avatar
⟳ Evan Miller retweeted
Nat Friedman@natfriedman·5 Feb 2024

Ten months ago, we launched the Vesuvius Challenge to solve the ancient problem of the Herculaneum Papyri, a library of scrolls that were flash-fried by the eruption of Mount Vesuvius in 79 AD. Today we are overjoyed to announce that our crazy project has succeeded. After 2000 years, we can finally read the scrolls: This image was produced by @Youssef_M_Nader, @LukeFarritor, and @JuliSchillij, who have now won the Vesuvius Challenge Grand Prize of $700,000. Congratulations!! These fifteen columns come from the very end of the first scroll we have been able to read and contain new text from the ancient world that has never been seen before. The author – probably Epicurean philosopher Philodemus – writes here about music, food, and how to enjoy life's pleasures. In the closing section, he throws shade at unnamed ideological adversaries – perhaps the stoics? – who "have nothing to say about pleasure, either in general or in particular." This year, the Vesuvius Challenge continues. The text that we revealed so far represents just 5% of one scroll. In 2024, our goal is to from reading a few passages of text to entire scrolls, and we're announcing a new $100,000 grand prize for the first team that is able to read at least 90% of all four scrolls that we have scanned. The scrolls stored in Naples that remain to be read represent more than 16 megabytes of ancient text. But the villa where the scrolls were found was only partially excavated, and scholars tell us that there may be thousands more scrolls underground. Our hope is that the success of the Vesuvius Challenge catalyzes the excavation of the villa, that the main library is discovered, and that whatever we find there rewrites history and inspires all of us. It's been a great joy to work on this strange and amazing project. Thanks to Brent Seales for laying the foundation for this work over so many years, thanks to the friends and Twitter users whose donations powered our effort, and thanks to the many contestants whose contributions have made the Vesuvius Challenge successful! Read more in our announcement: scrollprize.org/grandprize

Post attachment
Beidi Chen avatar
⟳ Evan Miller retweeted
Beidi Chen@BeidiChen·2 Oct 2023

The blog about Softmax+1 plays a very important role when we were trying to identify the root cause of the sink @Guangxuan_Xiao can comment more!

Georgi Gerganov avatar
⟳ Evan Miller retweeted
Georgi Gerganov@ggerganov·2 Oct 2023

Have a few thoughts about this approach But most importantly, I'm happy to see @EvMill's idea on softmax1 recognized - to my very basic and intuitive understanding of LLMs, it made enough sense to warrant further analysis arxiv.org/abs/2309.17453

Evan Miller avatar
Evan Miller@EvMill·25 Sep 2023

👀

Unlike with clipped softmax, to achieve an exact zero in the output using softmax1 for a (partial) no-update, the input requires to be -infinity. However, after @EvMill blog post we experimented with softmax1 and found it in practice competitive with our proposed approaches.

Astatide avatar
⟳ Evan Miller retweeted
Astatide@Astatide42·23 Sep 2023

Results of my latest nerdsnipe from @TetraspaceWest! The plot below shows the predicted shape of the water flow, with a model taking into account gravity and surface tension. It looks just like the real thing! Conclusion: yep, it's surface tension details below 😁

Post attachment

This tweet is unavailable

Thomas Capelle avatar
⟳ Evan Miller retweeted
Thomas Capelle@capetorch·14 Sep 2023

Following @EvMill great blog post on encountered issues on the GPT-like models training that appear to be related to the SoftMax function, I wrote this small piece mostly to understand what was going on. wandb.me/tinyllama