←
OSU NLP Group592 posts
OSU NLP Group avatar

OSU NLP Group

@osunlp

Natural Language Processing Group at The Ohio State University directed by @ysu_nlp @hhsun1 @shocheen

◷ Joined July 2021▧ 8 Photos and videos
Yu Su avatar
⟳ OSU NLP Group retweeted
Yu Su@ysu_nlp·1h

77% vs 22%, this is perhaps the largest gap shown between closed and open models, and it's particularly relevant because it's on representative economically valuable knowledge work. For agents to take on independent knowledge work, it's not enough to just build some static harness and call it a day. Every company is special. Every job is special. An agent must learn on the job to specialize. It needs to learn how a specific company operates: workflows, conventions, policies, software systems, exception handling, and judgment. Results on ApprenticeBench show that only the frontier models from four labs, @AnthropicAI @OpenAI @GoogleDeepMind @AIatMeta, show meaningful learning capacity. None of the open models can do it yet. And the gap is widening. There seems to be a capability wall that open models need to find a way to break out. Also impressed by Gemini 4. This benchmark is quite OOD. Gemini 4 showed strong generalization.

Major updates to the ApprenticeBench leaderboard, which tests whether an agent can learn to do an entire knowledge job end to end through continual learning. We believe in a future of specialized and decentralized intelligence, not a few closed models that everyone has to rent. However, this update shows a widening gap between closed vs. open and large vs. small models: ➤ The best closed models now outperform human testers on both score and cost, e.g., Gemini 4 Argon scores 66% at $6.03 per task, while the human tester scores 51% at $7.21 ➤ Four labs now outperform the human baseline: Anthropic, OpenAI, Google, and Meta, all closed models ➤ The best open-weight models are 55 points behind. Open-weight models seem to be stagnating while closed models continue to improve ➤ Small models are even further behind, e.g., Haiku 5.5 scores only 6% It remains a distant goal for local agents powered by open models to learn real knowledge jobs. We believe that is one of the most important gaps to fill and a fertile ground for innovation. ➤ Leaderboard: apprenticebench.com/leaderbo… ➤ Blog: neocognition.io/blog/apprent…

Post attachment
NeoCognition avatar
⟳ OSU NLP Group retweeted
NeoCognition@NeoCognition·1h

Major updates to the ApprenticeBench leaderboard, which tests whether an agent can learn to do an entire knowledge job end to end through continual learning. We believe in a future of specialized and decentralized intelligence, not a few closed models that everyone has to rent. However, this update shows a widening gap between closed vs. open and large vs. small models: ➤ The best closed models now outperform human testers on both score and cost, e.g., Gemini 4 Argon scores 66% at $6.03 per task, while the human tester scores 51% at $7.21 ➤ Four labs now outperform the human baseline: Anthropic, OpenAI, Google, and Meta, all closed models ➤ The best open-weight models are 55 points behind. Open-weight models seem to be stagnating while closed models continue to improve ➤ Small models are even further behind, e.g., Haiku 5.5 scores only 6% It remains a distant goal for local agents powered by open models to learn real knowledge jobs. We believe that is one of the most important gaps to fill and a fertile ground for innovation. ➤ Leaderboard: apprenticebench.com/leaderbo… ➤ Blog: neocognition.io/blog/apprent…

Post attachment
OSU NLP Group avatar
OSU NLP Group@osunlp·Nov 10

We’ve been cooking

Looped LLMs are heating up🔥. Open weights are becoming available, but the recipes to train these models remain hard to find. Introducing ARLoop, a family of strong, fully-open pretrained looped transformers with Attention Residuals! Our strongest model, ARLoop-1.2B, outperforms both Huginn-3.5B, a prior fully open looped LLM, and Qwen3.5-2B Base on evaluated benchmarks across commonsense reasoning, math, and coding. We release the full recipe: from data preparation and model training to intermediate checkpoints and evaluation code. ⤵️

Post attachment
Huan Sun avatar
⟳ OSU NLP Group retweeted
Huan Sun@hhsun1·Nov 10

Is it worth investing in pre-training looped transformers? Our previous work, “Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers” (how loops boost generalization in multi-hop reasoning), says YES! Today, we’re excited to share ARLoop: Pre-trained Looped Transformers with Attention Residuals, the strongest among fully-open looped LLMs. led by @yuekun_yao @osunlp ARLoop-1.2B (pretrained on 0.5T tokens only), outperforms both Huginn-3.5B, a prior fully open looped LLM, and Qwen3.5-2B Base on all eight evaluated benchmarks across commonsense reasoning, math, and coding. We release everything: model weights, training code, data processing code, and intermediate checkpoints. The key question we explore in this project: How to make looped transformers better use repeated computation? We adapt attention residuals to selectively use information across depth. It turns out looped transformers do not require access to the full residual history and sliding-window depth attention can match or outperform full-history attention at larger recurrence.

Looped LLMs are heating up🔥. Open weights are becoming available, but the recipes to train these models remain hard to find. Introducing ARLoop, a family of strong, fully-open pretrained looped transformers with Attention Residuals! Our strongest model, ARLoop-1.2B, outperforms both Huginn-3.5B, a prior fully open looped LLM, and Qwen3.5-2B Base on evaluated benchmarks across commonsense reasoning, math, and coding. We release the full recipe: from data preparation and model training to intermediate checkpoints and evaluation code. ⤵️

Post attachment
Yuting Ning avatar
⟳ OSU NLP Group retweeted
Yuting Ning@yuting_ning·Nov 2

When you ask an agent to book a restaurant, spelling out every detail is tedious. You’d rather just say “Get me a table at my usual place tomorrow.” A good personal agent should infer which restaurant from your history, and ask you for what only you know (e.g., the time and party size). It shouldn’t guess, and it shouldn’t quiz you about things it should already know. Introducing iPrefBench, my internship work at Apple 🍎: a user-centric benchmark for personalized mobile use agents, with: - 130 underspecified tasks across 26 iOS apps, grounded in a coherent persona with 3,051 past activity events - Every missing detail is either inferable from history or obtainable only by asking the user, by construction - Evaluation on both whether agents get the task right for the user and whether they only ask questions when necessary 🧵:

Post attachment
Yuting Ning avatar
⟳ OSU NLP Group retweeted
Yuting Ning@yuting_ning·Oct 27

So excited that SkillHarm received an Outstanding Paper Award at #EMNLP2026! 🏆 Huge thanks to my co-lead @Zhehao_Zhang123, our advisors @hhsun1 @ysu_nlp and all collaborators for their guidance and support!🫶 And congrats to everyone at @osunlp on all three awards!

Getting papers published probably won’t excite you anymore, but getting paper awards still do, especially THREE paper awards from #EMNLP2026 and a #NeurIPS2026 workshop on the same day @osunlp! Thrilled to share: SkillHarm, led by @yuting_ning and @Zhehao_Zhang123, gets an Outstanding Paper Award at #EMNLP2026. I particularly like that SkillHarm introduces a new attack scenario, named Self-Mutating Poisoning (SMP), where an initially benign execution silently mutates persistent skill content that defers harm until a later reuse. D3-Gym (co-led by @HananeNMoussa and @YifeiLiPKU) gets an Outstanding Paper Award at #EMNLP2026. D3-Gym has received a lot of interest from startups that aim to create scientific tasks and environments to train agents for scientific discovery. I particularly like that my students choose to stay with me rather than quitting Ph.D. to make big money! ;-) AgentCL (led by @YihengShu) was selected for a Runner-up Paper Award at the #NeurIPS2026 Workshop on Towards Test-Time Continual Learning Agents (TTCL)! Check out our attempt to rigorously evaluate continual learning in language agents. Congratulations to all co-authors and collaborators! See our original posts for details:

Sachin Kumar avatar
⟳ OSU NLP Group retweeted
Sachin Kumar@shocheen·Oct 27

Super excited to get this award!!! Congratulations to the all the coauthors of this paper, especially first authors Aruna and @orevaahia, for all their hard work.

Post attachment

We won an outstanding paper award at #EMNLP2026! 🎉🎉🎉🎉

Huan Sun avatar
⟳ OSU NLP Group retweeted
Huan Sun@hhsun1·Oct 27

Oh wow, another award by @shocheen @osunlp and collaborators! Many thanks to #EMNLP2026 committee recognition for the three paper awards!

Super excited to get this award!!! Congratulations to the all the coauthors of this paper, especially first authors Aruna and @orevaahia, for all their hard work.

Post attachment
Hanane Nour Moussa avatar
⟳ OSU NLP Group retweeted
Hanane Nour Moussa@HananeNMoussa·Oct 27

D3-Gym won an outstanding paper award 🫶🏻🥳! So happy and congrats to my co-lead @YifeiLiPKU and all my awesome collaborators!

Post attachmentPost attachment

Getting papers published probably won’t excite you anymore, but getting paper awards still do, especially THREE paper awards from #EMNLP2026 and a #NeurIPS2026 workshop on the same day @osunlp! Thrilled to share: SkillHarm, led by @yuting_ning and @Zhehao_Zhang123, gets an Outstanding Paper Award at #EMNLP2026. I particularly like that SkillHarm introduces a new attack scenario, named Self-Mutating Poisoning (SMP), where an initially benign execution silently mutates persistent skill content that defers harm until a later reuse. D3-Gym (co-led by @HananeNMoussa and @YifeiLiPKU) gets an Outstanding Paper Award at #EMNLP2026. D3-Gym has received a lot of interest from startups that aim to create scientific tasks and environments to train agents for scientific discovery. I particularly like that my students choose to stay with me rather than quitting Ph.D. to make big money! ;-) AgentCL (led by @YihengShu) was selected for a Runner-up Paper Award at the #NeurIPS2026 Workshop on Towards Test-Time Continual Learning Agents (TTCL)! Check out our attempt to rigorously evaluate continual learning in language agents. Congratulations to all co-authors and collaborators! See our original posts for details:

Zhehao Zhang avatar
⟳ OSU NLP Group retweeted
Zhehao Zhang@Zhehao_Zhang123·Oct 27

🏆 Honored to receive the Outstanding Paper Award in person at #EMNLP2026! Huge thanks to my co-lead @yuting_ning, all my wonderful collaborators, and my amazing advisors @ysu_nlp and @hhsun1 at @osunlp. This was my first-year PhD project, and I'm excited to keep building on it!

Post attachmentPost attachment

Getting papers published probably won’t excite you anymore, but getting paper awards still do, especially THREE paper awards from #EMNLP2026 and a #NeurIPS2026 workshop on the same day @osunlp! Thrilled to share: SkillHarm, led by @yuting_ning and @Zhehao_Zhang123, gets an Outstanding Paper Award at #EMNLP2026. I particularly like that SkillHarm introduces a new attack scenario, named Self-Mutating Poisoning (SMP), where an initially benign execution silently mutates persistent skill content that defers harm until a later reuse. D3-Gym (co-led by @HananeNMoussa and @YifeiLiPKU) gets an Outstanding Paper Award at #EMNLP2026. D3-Gym has received a lot of interest from startups that aim to create scientific tasks and environments to train agents for scientific discovery. I particularly like that my students choose to stay with me rather than quitting Ph.D. to make big money! ;-) AgentCL (led by @YihengShu) was selected for a Runner-up Paper Award at the #NeurIPS2026 Workshop on Towards Test-Time Continual Learning Agents (TTCL)! Check out our attempt to rigorously evaluate continual learning in language agents. Congratulations to all co-authors and collaborators! See our original posts for details:

Hanane Nour Moussa avatar
⟳ OSU NLP Group retweeted
Hanane Nour Moussa@HananeNMoussa·Oct 26

#EMNLP2026 I’ll be presenting D3-Gym at Hall H tomorrow (10/27) at 9am, come say Hi!! I’ll also be around the award ceremony later in the day ;)!

Gym environments have played a key role in advancing LMs and agents for general coding tasks. But how do we build them for scientific coding? Introducing D3-Gym, the first automatically constructed dataset of verifiable environments for data-driven scientific discovery. 🧵

Post attachment
Yu Su avatar
⟳ OSU NLP Group retweeted
Yu Su@ysu_nlp·Oct 27

My first NLP publication was at EMNLP'16. Grateful that @osunlp today received two Outstanding Paper Awards from #EMNLP2026! Congrats to all the students and collaborators!

Getting papers published probably won’t excite you anymore, but getting paper awards still do, especially THREE paper awards from #EMNLP2026 and a #NeurIPS2026 workshop on the same day @osunlp! Thrilled to share: SkillHarm, led by @yuting_ning and @Zhehao_Zhang123, gets an Outstanding Paper Award at #EMNLP2026. I particularly like that SkillHarm introduces a new attack scenario, named Self-Mutating Poisoning (SMP), where an initially benign execution silently mutates persistent skill content that defers harm until a later reuse. D3-Gym (co-led by @HananeNMoussa and @YifeiLiPKU) gets an Outstanding Paper Award at #EMNLP2026. D3-Gym has received a lot of interest from startups that aim to create scientific tasks and environments to train agents for scientific discovery. I particularly like that my students choose to stay with me rather than quitting Ph.D. to make big money! ;-) AgentCL (led by @YihengShu) was selected for a Runner-up Paper Award at the #NeurIPS2026 Workshop on Towards Test-Time Continual Learning Agents (TTCL)! Check out our attempt to rigorously evaluate continual learning in language agents. Congratulations to all co-authors and collaborators! See our original posts for details:

OSU NLP Group avatar
OSU NLP Group@osunlp·Oct 27

Great recognition of our work at #EMNLP2026!

Getting papers published probably won’t excite you anymore, but getting paper awards still do, especially THREE paper awards from #EMNLP2026 and a #NeurIPS2026 workshop on the same day @osunlp! Thrilled to share: SkillHarm, led by @yuting_ning and @Zhehao_Zhang123, gets an Outstanding Paper Award at #EMNLP2026. I particularly like that SkillHarm introduces a new attack scenario, named Self-Mutating Poisoning (SMP), where an initially benign execution silently mutates persistent skill content that defers harm until a later reuse. D3-Gym (co-led by @HananeNMoussa and @YifeiLiPKU) gets an Outstanding Paper Award at #EMNLP2026. D3-Gym has received a lot of interest from startups that aim to create scientific tasks and environments to train agents for scientific discovery. I particularly like that my students choose to stay with me rather than quitting Ph.D. to make big money! ;-) AgentCL (led by @YihengShu) was selected for a Runner-up Paper Award at the #NeurIPS2026 Workshop on Towards Test-Time Continual Learning Agents (TTCL)! Check out our attempt to rigorously evaluate continual learning in language agents. Congratulations to all co-authors and collaborators! See our original posts for details:

Huan Sun avatar
⟳ OSU NLP Group retweeted
Huan Sun@hhsun1·Oct 27

Getting papers published probably won’t excite you anymore, but getting paper awards still do, especially THREE paper awards from #EMNLP2026 and a #NeurIPS2026 workshop on the same day @osunlp! Thrilled to share: SkillHarm, led by @yuting_ning and @Zhehao_Zhang123, gets an Outstanding Paper Award at #EMNLP2026. I particularly like that SkillHarm introduces a new attack scenario, named Self-Mutating Poisoning (SMP), where an initially benign execution silently mutates persistent skill content that defers harm until a later reuse. D3-Gym (co-led by @HananeNMoussa and @YifeiLiPKU) gets an Outstanding Paper Award at #EMNLP2026. D3-Gym has received a lot of interest from startups that aim to create scientific tasks and environments to train agents for scientific discovery. I particularly like that my students choose to stay with me rather than quitting Ph.D. to make big money! ;-) AgentCL (led by @YihengShu) was selected for a Runner-up Paper Award at the #NeurIPS2026 Workshop on Towards Test-Time Continual Learning Agents (TTCL)! Check out our attempt to rigorously evaluate continual learning in language agents. Congratulations to all co-authors and collaborators! See our original posts for details:

Sachin Kumar avatar
⟳ OSU NLP Group retweeted
Sachin Kumar@shocheen·Oct 23

📢 Please check out new work from my students on instruction hierarchy alignment in vision language models with many cool results, including substantial improvements on vision centric safety benchmarks and web agents. I'm on my way to a Budapest to attend #EMNLP2026 and will present this work on Monday!

🚀 Introducing "Seeing Through Conflicts:Improving Instruction Hierarchy Alignment in Vision-Language Models" We train VLMs to reason over instruction hierarchies across text and images with RLVR. ~7K verifiable examples and no safety training data. Gains transfer to real image attacks and web agents. Joint work with Nicholas and my advisor @shocheen. Excited to discuss multimodal IH and safer VLM! 📍EMNLP 2026 Findings

Post attachment
Hanane Nour Moussa avatar
⟳ OSU NLP Group retweeted
Hanane Nour Moussa@HananeNMoussa·Oct 19

1/ Can RL improve accuracy without losing different correct ways to solve a problem? We tackle this in our new paper EliteReplay, and study diversity at the level of correct solution strategies (not just pass@k 🤓!) 📜: arxiv.org/abs/2610.22290 Thread below 🧵!

Post attachment