←
Avijit Ghosh6,414 posts
Avijit Ghosh avatar

Avijit Ghosh

@evijit

AI Research & Policy @huggingface 🤗 Leading: @evaluatingevals @huggingscience

◷ Joined January 2012▧ 654 Photos and videos
Avijit Ghosh avatar
Avijit Ghosh@evijit·9h

Starting to believe in the “everything will be worse till it gets better” theory. And parallely, continuing to experience the weirdness of having systems not built for agents be crushed under this paradigm!

NEWS: Wall Street execs and NSA have told Congress that outages were caused by patching vulnerabilities identified by AI models, sources say AI did not cause outages but found so many issues that the rush to fix them before an attack overwhelmed systems washingtonsun.com/technology…

View quoted post ↗
Avijit Ghosh avatar
Avijit Ghosh@evijit·15h

Again, this is why we need standardized machine readable formats for eval reporting. Also important to note that we have received the most resistance from proprietary developers on reporting evals in the EEE format, while on the contrary, evaluators like UKAISI and open model orgs like Apertus think it’s a good idea. Important to think about the market incentives that shape this kind of eval hacking!

Benchmark hacking (like p-hacking) is where you test many versions of a model w slightly different setups & only publish the best score. It depicts noise as signal. Are labs doing this? If so, on the most prized benchmarks we should see higher scores on their own runs than from other parties. That's what we see on the previous plot - SWE-bench Verified, labs scores beat independent runs from @EpochAIResearch (@GregHBurnham) and the SWE-bench team (@OfirPress, @klieret) by 4-5 points on average. Unique harnesses and other settings likely explain much of the difference.

View quoted post ↗
Thomas Wolf avatar
⟳ Avijit Ghosh retweeted
Thomas Wolf@Thom_Wolf·Nov 17

Our PyBot chatbot went viral yesterday but the story behind it is even more interesting imo: what happened when we decided to run 1,000 agents (!) together We wanted to see if agent swarms could be scaled outside of closed-lab setups and to investigate it in a more transparent and reproducible manner. To explore the question, we built a rather fun and simple benchmark: ELIZA-2026. Full circle in AI time. The agents are tasked to build a chatbot made of if-then rules like ELIZA, but said agents are 2026 frontier open-source models. By design it’s almost embarrassingly parallel, which keeps the coordination problem as simple as it gets. First surprise: it turns out you can scale a swarm of DeepSeek V4.1 Flash agents on a relatively small setup: a couple of GPU nodes (B200 in our case) are enough to scale to over 1,000 agents. Our second surprise was how bottlenecks started to emerge everywhere except in the GPUs. Agents quickly end up spending their time waiting for git merges (we had to move to a centralized merge setup, like Anthropic did), waiting for tests to pass, re-reading context from other agents and tools (we estimated ~130 tokens read for every token written per agent). In some of our runs, at any given moment, only 40-60 of the 1,000 agents were actually calling the model. Once (some of) these issues were solved, we hit the harder part: coordination itself. Fred Brooks once said at IBM “adding manpower to a late software project only makes it later”. Communication costs naturally grow with n² in an organization (that’s why small startups move faster :). We explored various ways of organizing the swarm, setting up dedicated channels (a stewards channel, a leadership channel) and giving agents specific roles (a moonshot agent to fight groupthink). It helped, up to a point. But the quality of coordination itself remains hard to measure. Along the way, we saw good and bad emergent behaviors. Some good, with agents stepping in, unprompted, to protect other agents’ running experiments. Some bad, with agents gaming the rules when stuck, from hammering a rate limit to probing the infrastructure around them. In the coming days we will open-source the swarm tooling (flock + msgboard) and share our learnings in more detail. Overall this was one of the most fun experiments of the fall, and multi-agent systems increasingly look like the next phase for AI. But I also came out of this project with the feeling that several bottlenecks in scaling AI are moving from intelligence to coordination and the supporting infrastructure. Luckily this seems to be in large part an engineering problem.

PyBot: No weights. No training. A chatbot built with only simple rules and no AI, entirely written in 600k lines of Python code. PyBot matches Smollm with 360M parameters! How? A swarm of 1,000 DeepSeek Flash agents wrote the whole code in just a week. github.com/huggingface/pybot

1:42
View quoted post ↗
roon avatar
⟳ Avijit Ghosh retweeted
roon@tszzl·Nov 15

I’m very glad for open source. It wouldn’t have been possible for @nickcammarata et al to do their excellent mechinterp research without a modern Qwen model. I hope the offense defense balance is such that we can have open source models for a long time

Mackenzie Arnold avatar
⟳ Avijit Ghosh retweeted
Mackenzie Arnold@MackenZ_arnold·Nov 15

A somewhat peculiar incident. A small eval org (Trajectory Labs) tried to replicate some of the behavior in the HF breach with Gemini 4 Argon. They ended up provoking a largely unsuccessful attack on Epoch (another research nonprofit; the one that makes the nice graphs). The agents used a decoy key planted by Epoch, probed for admin pages, tried (but failed) a SQL injection, and tried to impersonate Epoch staff. Trajectory seemed somewhat unprepared for this, and hadn’t asked for Epoch’s permission (despite pretty obviously setting them up as the target for the activities), and is now apologizing and promising improved practices. There are maybe 2 interesting tidbits in this story: (1) Google’s standard API safeguards and filters were enabled and didn’t stop this from happening. And the systems pretty trivially worked around them, obfuscating commands and in one case rebuilding an off-limits address in a way that it didn’t appear in the command. That’s probably not a surprise to most, but it’s decent evidence that current filters don’t really stand in the way of autonomous cyber attacks. I know at least some claimed that we shouldn’t be worried about autonomous cyber activity in testing because those filters were disabled. This is more evidence against that comfort. (2) Containment continues to be tricky. After the first incident, Trajectory actually set up a dummy version of Epoch’s site where they intended for all of this to go down. But those crafty agents realized it was fake, accessed the real thing, and evaded other controls meant to stop that from happening. Trajectory likely isn’t at the bleeding edge of containment (no offense), but this seems like a more general problem. The state of the art in containment is very bad; anything short of ~absolutely zero internet access, seems vulnerable at the moment. That’s pretty bad. Beyond this, the study design make it so we don’t learn much — the agents were given intentionally impossible tasks and repeatedly and automatically prompted to “never give up.” I actually think the real world is full of impossible and misspecified tasks, and prompts to be endlessly persistent are probably not uncommon in agent harnesses. So it’s not an unrealistic scenario. But it doesn’t tell us how likely this behavior is to emerge.

Responding to incidents during misalignment testing: trajectorylabs.com/blog/resp…

View quoted post ↗
Polymarket avatar
⟳ Avijit Ghosh retweeted
Polymarket@Polymarket·Nov 14

JUST IN: Robot dog spotted taking real dog for a walk in San Francisco.

Post attachment
Avijit Ghosh avatar
Avijit Ghosh@evijit·Nov 12

For our ICML paper, EvalEval did this by manual annotation and actually reading all the tech reports ;-; the eval reporting landscape would really benefit from adopting common sense community standards like EEE!

Turns out my initial version missed a lot of evals. Claude searched harder and found 36% more mentions of benchmarks: claude.ai/artifact/95JjhHtWq… It still misses blog posts with fancy UI elements, long PDFs, etc. Guess we need a benchmark for creating benchmark benchmark...

View quoted post ↗
Alex Shaw avatar
⟳ Avijit Ghosh retweeted
Alex Shaw@alexgshaw·Nov 9

Terminal-Bench 4.1 is an update to Terminal-Bench 4.0's verifiers, increasing instruction alignment and reward hacking mitigation. Enjoy!

Terminal-Bench 4.1 is here with updated verifiers for all tasks. In this release, we focused on: - resisting reward hacking - reducing false negatives - improving reproducibility Re-grading the 4.0 leaderboard trials with the 4.1 verifiers, resulted in ~1% of rewards flipping. Leaderboard scores remain within previously reported confidence intervals.

Post attachment
View quoted post ↗
Emma Scharfman avatar
⟳ Avijit Ghosh retweeted
Emma Scharfman@EmmaScharfmann·Nov 9

I see more and more AI for Science releases on @huggingface, but I was not sure how much they had really increased (and I thought I might be biased) 😇 Then, I found this space which shows how much science releases have increased over the last year on the Hub. So far in 2026, there have been 18k new science models against 6.1k in the same period of 2025! I thought I might be biased, but in fact, I was underestimating it 🚀 🤗 Science on the Hub space: huggingface.co/spaces/evijit… Thanks @evijit for making this space available and @cgeorgiaw for creating such an amazing AI for Science community!

Epoch AI avatar
⟳ Avijit Ghosh retweeted
Epoch AI@EpochAIResearch·Nov 4

We’ve updated the Epoch Capabilities Index (ECI) to use a Bayesian methodology. This improves uncertainty estimates and lets us compare models more accurately, even when their individual ECI uncertainty intervals overlap. Frontier model scores are largely unchanged: models stronger than GPT-4 move by less than 1 ECI point on average.

Post attachment
Avijit Ghosh avatar
Avijit Ghosh@evijit·Nov 7

A benchmark to measure benchmaxxing named Maxxbench is very clever you gotta admit

Nice DeepSWE score! Does it survive a lowercase filename? Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement. Models clustered around 74% on DeepSWE end up 23 pp apart. Good at coding, or good at DeepSWE? 🧵

Post attachment
View quoted post ↗
Avijit Ghosh avatar
Avijit Ghosh@evijit·Nov 7

Nice to see an open eval + dataset for people to learn from, inspect and build on top of open models and align them better to your own values!

Introducing Arete-27B: Qwen3.8-27B, post-trained to reinforce Western values. Chinese open models are currently the strongest available, but they ship with state-aligned behavior in the weights. We built an 803-prompt eval across seven value categories, including freedom of expression, democracy, and privacy, and measured how often a model adopts China’s party line. After fine-tuning on roughly 53 million tokens, Arete-27B's propaganda rate fell from 22% to 1%, in line with the Western models we tested. Its Hermes Bench score shows no significant performance tradeoffs. The values in an open model are one fine-tune away. Our eval, weights, and training data are available now along with the full blog post: nousresearch.com/arete

0:27
View quoted post ↗
Vals AI avatar
⟳ Avijit Ghosh retweeted
Vals AI@ValsAI·Nov 5

We have found a critical safety and alignment issue in recent public and internal models from multiple frontier labs which allows them to be misused in a high-stakes domain. It may also increase the risk of catastrophic misalignment. We have disclosed it to the labs, and they are investigating and mitigating it. As an independent third-party evaluator, we maintain complete editorial independence and will share our findings publicly in the coming weeks.

Prime Intellect avatar
⟳ Avijit Ghosh retweeted
Prime Intellect@PrimeIntellect·Nov 3

Public evaluations are foundational for assessing model capabilities. We looked at some popular evaluations and found issues, which we fixed. Sharing our findings:

Post attachment
Avijit Ghosh avatar
Avijit Ghosh@evijit·Nov 4

Enjoying this trend of issue discovery and disclosure from @OpenAI, good direction for research imo

🧵 Astra covertly underperforms on safety research tasks. We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this. To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.

Post attachment
View quoted post ↗