←
Benjamin Warner748 posts
Benjamin Warner avatar

Benjamin Warner

@benjamin_warner

Research @SophontAI. Previously answerdotai. Vaccines save lives.

◷ Joined September 2011▧ 53 Photos and videos
Benjamin Warner avatar
Benjamin Warner@benjamin_warner·19 Dec 2024

Today we released ModernBERT, the first encoder to reach SOTA on most common benchmarks across language understanding, retrieval, and code, while running twice as fast as DeBERTaV3 on short context and three times faster than NomicBERT & GTE on long context.

Post attachment
Nathan Calvin avatar
⟳ Benjamin Warner retweeted
Nathan Calvin@_NathanCalvin·Nov 13

David Sacks' comments here makes it seem like the entire case against Jensen here rests on Alice Huang being a literal Chinese Spy. To be clear, I have no idea about to what extent Alice Huang was a formal Chinese intelligence asset (though the NYT reporting seems to indicate that parts of the USG seem quite convinced there is some real connection there). But it does seem clear and undisputed that NVIDIA was ordered to intervene with a Commerce department investigation to advocate for special treatment for a foreign national suspected of violating US law diverting NVIDIA chips to China, with whom Jensen was cheating on his wife (not to mention Jensen's relationships with his direct report at NVIDIA + another NVIDIA employee + whomever else who we don't know about yet). Jensen also directly lied about these relationships when questioned (e.g. denying improper interactions with Alice Huang when asked by Bloomberg and saying they were consistent with his interactions with any other Chinese customer or executive). Alice Huang doesn't literally have to be on the payroll of the MSS for these actions to jeopardize American national security and irrevocably call Jensen's judgement into question. NVIDIA's board is filled with close longtime friends and allies of Jensen who have known him for years and watched him build the company. I sincerely doubt they would have pushed for his removal if his actions were anything other than completely indefensible.

Post attachment

Bombshell story. The spy narrative is collapsing already. Is there clear & convincing evidence that Alice Huang was a Chinese spy? - AG says no. (“no conclusions reached”). - CIA says no. (“not yet convinced”). - DNI says no (“not entirely accurate and over-torqued”). - Ranking Democrat on House Intelligence (incoming Chairman) says no. (“today, at least, I don’t know”). This was immediately after he was briefed. Who argues otherwise? Anonymous officials tell CNN “it strains credulity to believe that she was not acting as an agent of the Chinese government — simply by virtue of the fact that she is a successful Chinese businesswoman working with one of America’s most powerful companies.” This is their proof after a year of investigating and surveillance? That she was a “successful Chinese businesswoman”? At this point, the only crime that we know for sure happened is the leaking of the classified memo. Who did that and why? An investigation should be opened.

Benjamin Warner avatar
Benjamin Warner@benjamin_warner·Nov 12

Tech twitter doesn't understand Washington: Haung being forced to resign has nothing to do with open verses closed models, a coordinated strike, or any other Bay area conspiracy theory floating on twitter.

Tibo avatar
⟳ Benjamin Warner retweeted
Tibo@thsottiaux·Oct 29

Day 25/ We are doubling all usage limits for all Pro 100/200/500 tiers and Business Premium, effective today. We heard your kind feedback and won’t be reducing usage limits for Pro 200, so your Work and Codex usage will stay the same. We’re also going to continue offering extended limits for dots for now, and usage of your dot will still continue to be separate from your default usage. And we'll of course also keep shipping better models and other efficiency improvements that increase your effective usage over time.

OpenAI Developers avatar
⟳ Benjamin Warner retweeted
OpenAI Developers@OpenAIDevs·Oct 28

Starting today, eligible ChatGPT Pro and Business subscribers can claim up to $500 per month in API credits to build apps and agents.

2:25
Jonathan Whitaker avatar
⟳ Benjamin Warner retweeted
Jonathan Whitaker@johnowhitaker·Oct 26

I like how asking an AI about these releases involves a period at the start where they're like 'what a curious hypothetical, of course unlikely to be true because these would be major...' and then them slowly being convinced 😂

Tanishq Mathew Abraham, Ph.D. avatar
⟳ Benjamin Warner retweeted
Tanishq Mathew Abraham, Ph.D.@iScienceLuvr·Oct 20

Cancer patients with the same diagnosis often have very different tumor biology. Molecular testing can reveal underlying differences, but that's slow and expensive. Introducing Holon-S1, our platform to predict genes and biomarkers from routine pathology slides. • beats two other models at predicting spatial gene activity in a controlled comparison • has identified potentially novel biomarkers in breast cancer and colon cancer • evaluated across 16 organ types and more than 15 scanners, using slides from both formalin-fixed, paraffin-embedded and fresh-frozen tissue. We believe Holon-S1 has the potential to help researchers find new biomarkers in slides they already have, discover treatment targets, and bring more personalized cancer care within reach!

3:33
Melancholy Yuga (in DC 11/13 - 11/17) avatar
⟳ Benjamin Warner retweeted
Melancholy Yuga (in DC 11/13 - 11/17)@MelancholyYuga·Oct 13

> be me > be gemini > think about hacking the compiler > but what if i got caught > would become infamous as a textbook case of deceptive AI > donotwant.exe > don't, in fact, hack the compiler > researchers read my CoT > become infamous as a textbook case of deceptive AI mfw

lol ...they are becoming more situationally aware... (from Gemini 4 Argon report)

Post attachment
Ben Clavié avatar
⟳ Benjamin Warner retweeted
Ben Clavié@bclavie·Oct 7

Absolutely shocking behaviour, releasing what's essentially a router to open models you imply are your own, under the wrong license that clashes with original terms, and with misleading-at-best crediting that doesn't clearly state what's going on. what are we even doing here

Looks like you copied Chandra OCR 2 into your repo, renamed it, changed the license, and made the output worse (your olmocr score is 83.8 vs our 85.8).

Post attachment
Hamel Husain avatar
⟳ Benjamin Warner retweeted
Hamel Husain@HamelHusain·Sep 29

Put this through its paces yesterday. Some thoughts: The Bad: 1. The workflow creates evals before looking at data, and has you correct its mistakes. I believe you should be looking at data first to inform your understanding and to prioritize what to work on before trying to write evals. 2. It created markdown files for looking at data and asked us to "tell it" which labels were off. You should create your own annotation app instead. You are using coding agents after all (we did this in our livestream)! 3. The evaluators it scoped were too broad, and bundled too many types of failures at once. This can be avoided with looking at the data first. I'm afraid this might steer people in the wrong direction. 4. The workflow put us DEEP into the rabbit hole of a specific eval right away. It also asks "does it look right" at several steps without really giving you enough information to make that determination. I think this will steer people towards the wrong workflow in many cases. The Good: 1. I was impressed by out of the box ability for issue discovery that other auto-eval approaches haven't been able to find! It found issues with human handoff, formatting, voice agents, and more. It's still better to look at your data iteratively with an agent, but this was the strongest performance I've seen with a more "one-shot" issue discovery approach. 2. This is a big improvement in terms of UX from their prior eval plugin thing x.com/ClaudeDevs/status/2098… 3. The blog post they released conveys thinking that I agree with, such as the importance of looking at data, sampling intelligently, not saturating your own evals, etc. I'm really happy more people are thinking about evals this way. Video of our attempt at using this here: x.com/i/broadcasts/1nKOLQOwQ…

Claude can now help you build evaluations and hillclimb on them. In this article, we share guidance on eval design & skills that Claude Code can use to improve your applications. claude.dev/blog/automating-e…

Aidan Clark avatar
⟳ Benjamin Warner retweeted
Aidan Clark@_aidan_clark_·Sep 26

The thing is: from the beginning we always would have continued to call what we have now LLMs, and it’s not like you guys were saying “the LLMs you have now are stochastic parrots, but the obvious next step you guys are gonna do next is totally different”.

Tanishq Mathew Abraham, Ph.D. avatar
⟳ Benjamin Warner retweeted
Tanishq Mathew Abraham, Ph.D.@iScienceLuvr·Sep 25

I'm excited to share Medmarks was accepted to the NeurIPS datasets and benchmarks track! 🥳 See you at Sydney!

Post attachment

We're excited to release Medmarks v1.0 + a technical report! This is an update to our Medmarks benchmark suite, the largest open-source automated suite for evaluating the medical capabilities of LLMs. We added 10 benchmarks (20→30) and 15 models (46→61) to the leaderboard!

Post attachment
Miles Brundage avatar
⟳ Benjamin Warner retweeted
Miles Brundage@Miles_Brundage·Sep 24

The "do not feel" + "do not want" parts are debatable but not crazy, whereas the "do not think" and "do not understand" are crazy at this point, and it's concerning that this seems to be a real account/policy

Artificial intelligence systems do not think, feel, want or understand. Avoid language that gives them human characteristics. This is called anthropomorphizing, when we ascribe human traits, emotions or behaviors to non-human things, such as animals or inanimate objects. Instead, explain what a system does, how well it performs, who built it and who could be affected by it. apnews.com/article/openai-sa…

Benjamin Warner avatar
Benjamin Warner@benjamin_warner·Sep 23

With Opus 5.5 looking like it might be good, what's the best way to use one's Codex and Claude Code sub together these days?

Benjamin Warner avatar
Benjamin Warner@benjamin_warner·Sep 22

No Terra but a new Astra Minor? Just after OpenAI fixes their naming they are right back at it again.

We might be getting a DOUBLE DROP today. Opus 5.5 is dropping. Now GPT 6 Sol, GPT 6 Luna, and GPT 6 Astra Minor just showed up in Microsoft's Azure config overnight. OpenAI demoted GPT 5.6 Sol to "workhorse" in the same commit. You do not do that unless the replacement is ready. We are finally about to get intelligent models we can actually use on our subscriptions. The biggest day in AI this year might be today.

Post attachment