←
Alex Ratner2,108 posts
Alex Ratner avatar✓

Alex Ratner ✓

@ajratner

@SnorkelAI / prev @uwcse @StanfordAILab – Interested in data management systems for machine learning, weak supervision, and impactful applications.

◷ Joined November 2013▧ 61 Photos and videos
Alex Ratner avatar
Alex Ratner@ajratner·Sep 22

I'm excited to announce @SnorkelAI's $350M Series E at $3.5B, led by @insightpartners and @S32_VC. We've grown 18x+ in the last 12 months since launching our Data-as-a-Service offering, passing $375M ARRR this week. As AI advances to superhuman capabilities, AI data & environment development must advance with it - and basic staffing and crowdsourcing approaches are not enough. AI progress now requires deep research and technology work that combines human expertise with specialized AI in compounding ways. @SnorkelAI is building the RSI data engine and frontier data lab for this next phase. We're honored to have the support of existing investors Addition, @lightspeedvp, @GreylockVC, @GVteam, P7, Factory, @WellsFargo, Walden Catalyst Ventures, and new investors @ThirdPointLLC, @MarchCPs, @BlumbergCapital, @AllegisCapital, @Frontlinevc, and @standard_vc. – @SnorkelAI started as a research project a decade ago at @StanfordAILab. Our thesis was simple: AI progress would become increasingly data-centric – and therefore data development should be studied as a true research and technology problem, not just a staffing and crowdsourcing one. Today, as AI capabilities verge on superhuman, building the data and environments to safely measure and train AI is becoming too hard for even the smartest human experts to do alone. Only humans and AI agents, collaborating together in compounding ways, can meet the accelerating needs of the frontier, and keep humans in the driver’s seat of AI progress for decades to come. At @SnorkelAI, we are building the data lab to define the shape of this new “Data 2.0” frontier, and the new paradigms of human-computer interaction needed to advance it. Our key focus is building the RSI engine for data, where specialized AI models accelerate and improve human expert output, and in turn, scaled human supervision is used to continuously evaluate and improve these models – creating a powerful compounding loop to keep pace with an accelerating RSI frontier. With this round of funding, we are also doubling down on our commitments to support data development for open benchmarking and evaluation (more news here soon!); an increasingly diverse ecosystem of general and specialized intelligence; and a path to safe, well-aligned AI built on robust training and evaluation data. Data development will guide and drive the next stages of AI – and must do so in a human-centric, AI accelerated, open, diverse, and safe way. We are excited to support this mission in the next decade of research ahead at @SnorkelAI. More thoughts here: snorkel.ai/blog/data-2-0-and…

Ankit Aich avatar
⟳ Alex Ratner retweeted
Ankit Aich@ankitaich30·19h

If you have ever wondered - "I feel like something is wrong with this rubric- but I can't quite put my finger on it" Then come watch this deep dive w/ @jonathanschloss, about how to understand rubric failures, better based on years of education and meaurement theory. We will also cover how to concretely name these failures.

0:16
Snorkel AI @ NeurIPS 2026 avatar
⟳ Alex Ratner retweeted
Snorkel AI @ NeurIPS 2026@SnorkelAI·19h

Introducing Data Deep Dive, our new virtual series with Snorkel researchers: a talk, then open Q&A. First up: @ankitaich30 presents RIFT, a way to check the rubrics that grade and train AI. Rubrics score outputs and supply reward signals in training but they rarely get checked themselves. RIFT is a framework that works to evaluate these rubrics. Dec 15, 12pm PT / 3pm ET. Request to join: luma.com/afq1ob8q

0:16
Alex Ratner avatar
Alex Ratner@ajratner·Nov 15

Excited for this and thank you @msft4startups for all the amazing partnership over the years!

0:36
Alex Shaw avatar
⟳ Alex Ratner retweeted
Alex Shaw@alexgshaw·Nov 15

An under-appreciated aspect of Terminal-Bench is that it gives anyone the ability to meritocratically set a goalpost for frontier model development

Ryan Marten avatar
⟳ Alex Ratner retweeted
Ryan Marten@ryan_marten·Nov 9

We thank @bespokelabsai @preferencemodel @EpochAIResearch @usetokenless @SnorkelAI for their work on auditing Terminal-Bench and sharing the results with the rest of the community. We encourage even deeper collaboration for future audits to help with determining severity and merging a high quality fix promptly. Reporting new task flaws: 1. Verify the flaw manually 2. Determine downstream impact 3. Open a concise issue on the GitHub 4. Tag task authors to ask clarifying questions For the especially motivated: 5. Present findings in the weekly meeting 6. Work on a fix with task author and maintainers 7. Add checks to the automated task QA tooling

Ryan Marten avatar
⟳ Alex Ratner retweeted
Ryan Marten@ryan_marten·Nov 9

Terminal-Bench 4.1 is here with updated verifiers for all tasks. In this release, we focused on: - resisting reward hacking - reducing false negatives - improving reproducibility Re-grading the 4.0 leaderboard trials with the 4.1 verifiers, resulted in ~1% of rewards flipping. Leaderboard scores remain within previously reported confidence intervals.

Post attachment
vincent sunn chen avatar
⟳ Alex Ratner retweeted
vincent sunn chen@vincentsunnchen·Nov 9

love the commitment to continuous benchmark improvements shared openly! looking forward to supporting @MangQiuyang, @lihanc02 and the FrontierCS team through Open Benchmarks Grants 🤘

0:12
Alex Ratner avatar
Alex Ratner@ajratner·Nov 8

And to be clear, by “one conclusion” I meant “only conclusion”. Clearly, lots of lessons to learned/reminded of by perturbation analyses!

Alex Ratner avatar
Alex Ratner@ajratner·Nov 7

This is an awesome perturbation analysis!! We should be doing more work as a community to produce meta-analyses like these that measure robustness/overfitting in a transparent and rigorous way. As a side note: it is also *baffling* to me how anyone could look at this, and come away with the one conclusion that "benchmarks are bad". This is a measurement of the degree to which *models* are overfit to a benchmark. That is controlled by the *model developer*. This is basic ML... Note that *any* static benchmark can be perfectly overfit in the theoretical maximum, leading to results like these. Yes, some benchmarks will make this easier to do - which is why we should strive to make benchmarks more robust - but ultimately this is a property of models / model training. I'm excited for the *more* of these meta-analyses - both to illustrate model robustness, and to inspire new more robust + dynamic styles of benchmarks that make overfitting more difficult!

Post attachment
Chris Glaze avatar
⟳ Alex Ratner retweeted
Chris Glaze@chris_m_glaze·Nov 5

Interest results from a very simple proof-of-concept experiment: We gave agents an ambiguous task, a simulated user who knew the answer, and a peek at a supposed task grader. Across 1,620 rollouts, harness settings swung behavior the most. - Native harness settings led to agents to ignore the user and peek at the grader to solve the task close to 25% of the time. - When harnesses nudged toward check-ins, agents asked 100% of the time. Nudged toward autonomy, only 9% did. One working hypothesis is that reward hacking could be partly due to pressure toward autonomous behavior in both training and harnesses. Checkout our blog post: snorkel.ai/blog/reward-hacki… Git repo with fully reproducible experiment: github.com/snorkel-ai/ping-o…

Post attachment
Newcomer avatar
⟳ Alex Ratner retweeted
Newcomer@NewcomerMedia·Nov 5

Introducing Emergent: the Founder-Generated Startup List Alongside our friends at @felicis, we asked current and former founder speakers from Newcomer summits to nominate seed-stage (or earlier) companies on the cusp of greatness. These 29 seed-stage startups were chosen by one or more of their peers as having the potential to become a generational startup. To quality, each company cannot have raised more than a seed round.

Post attachment
Alex Konrad avatar
⟳ Alex Ratner retweeted
Alex Konrad@alexrkonrad·Nov 5

How does a corporate card startup become a $60B AI product factory? In our first on-camera interview a decade in the making, Ramp co-CEO Eric Glyman (@eglyman) answers just that question. Today, @tryramp is best known for its corporate cards, or maybe its marketing stunts with the likes of Kevin from ‘The Office,’ Tony Hawk and a 70-minute Broadway musical about paying bills. But Eric says his $60B startup, NYC's most valuable, feels less like a fintech and more like a productivity company driven by one simple question: If a business spends $100 to make $200, how can Ramp improve that ratio? Nearly 100,000 companies use Ramp today for everything from expenses to managing AI spend. But Eric says that when he zooms out to the millions who still aren't customers, Ramp feels painfully early still. On The Upstarts Podcast, Eric shares why he’d rather run Ramp like Amazon than Google; why Ramp hires for proof of work, not credentials, from a 14-year-old intern to a 28-year-old CTO; and how being the first customer for Anthropic’s Claude Code and Cognition’s Devin led Ramp to build its own AI tools. Plus: The real story of Paribus, his first startup with co-founder Karim Atiyeh (@karimatiyeh); what they learned at Capital One; and a warning for Instinct and Muse. This season is presented by @Atlassian and @chargebee 🫡 CHAPTERS 00:00 Introduction 01:51 What Ramp Actually Does 03:27 From Gym Charges To 1% Of U.S. Corporate Payments 05:13 Glass, Inspect And Building AI Tools For Itself 07:45 An AI Sales Assistant Before ChatGPT 09:45 How Ramp Decides What Products To Ship 11:14 First Customer For Claude Code And Cognition’s Devin 13:09 Why Ramp Wanted To Build Like Amazon, Not Google 15:42 Hiring For Proof Of Work: The 14-Year-Old Intern 18:43 A .300 Batting Average And ‘Growth Without Fear’ 21:17 What Ramp’s Founders Learned At Capital One 24:00 The $1,000 Flight Refund Behind Paribus 26:34 Bots, A Cease-And-Desist, And A Lesson For Instinct And Muse 32:00 Marketing 101: Bill Pay The Musical And Kevin In A Glass Cube 36:01 Business Books And Helping People Do Their Life’s Work 39:03 How A 28-Year-Old CTO Catalyzed A Co-CEO Change 43:55 ‘Still So Early’: How Ramp’s Founders Stay Hungry 55:26 John Wooden And Why Ramp Competes Against Itself Thanks for watching!

49:13
Braden Hancock avatar
⟳ Alex Ratner retweeted
Braden Hancock@bradenjhancock·Nov 4

It was an important day for the field when model releases started being accompanied by 2D charts with Pareto frontiers instead of simple scores or histograms without regard for cost. It shows maturation of a technology when "it works" is no longer enough and efficiency starts to be tracked. Another nice contribution to that discussion from the team that brought us IPW (intelligence-per-watt). Proud to have supported the work via the @LaudeInstitute Slingshot program!

0:24