←
Ni Jingwei48 posts
Ni Jingwei avatar

Ni Jingwei

@NJingwei

Doctoral Researcher @ ETH Zurich, Research Interests: NLP for Social Good, Claim Detection and Verification, Causal NLP

◷ Joined December 2020▧ 8 Photos and videos
AI4Law@ICML avatar
⟳ Ni Jingwei retweeted
AI4Law@ICML@ai4law_workshop·Apr 6

🚨 Call for Papers: AI for Law Workshop @icmlconf (July 10, Seoul 🇰🇷), welcomes submissions across three themes: ⚖️ AI for Legal Reasoning 📊 AI Evaluation for Law 🌍 AI for Access to Justice ⏰ Deadline: May 22 (AoE) 📄Full paper submission via: openreview.net/group?id=ICML…

Post attachment
Ni Jingwei avatar
Ni Jingwei@NJingwei·Jan 8

🧵[7/7] Before deploying reasoning LLMs for annotation tasks: ⚠️ Test on high-disagreement cases, not just accuracy ⚠️ Consider RLHF + CoT over pure RLVR ⚠️ Keep humans in the loop where values conflict 📂 Code & data: github.com/EdisonNi-hku/Disa…

Ni Jingwei avatar
Ni Jingwei@NJingwei·Jan 8

🧵[6/7] Our key findings: ✅ RLHF + CoT prompting significantly IMPROVES disagreement modeling. But RLVR-style reasoning HARMS it. ✅ Small fine-tuned models (e.g., ModernBERT) with human labels often outperform massive reasoning models Post-training strategy matters!

Ni Jingwei avatar
Ni Jingwei@NJingwei·Jan 8

🧵[5/7] 💡 RLVR models assume objective, single-correct-answer tasks. They focus on guidelines as if all humans interpret them identically. 🌍 RLHF models? They naturally consider diverse backgrounds and perspectives—the key to capturing real human variation.

Ni Jingwei avatar
Ni Jingwei@NJingwei·Jan 8

🧵[4/7] We evaluated 60 experimental setups: 3 tasks, 5 model sizes, and Multiple prompting strategies. 📉 Shocking result: RLVR reasoning models significantly degrade disagreement modeling—especially on high-variance cases where humans legitimately disagree.

Ni Jingwei avatar
Ni Jingwei@NJingwei·Jan 8

🧵[3/7] Meanwhile, reasoning LLMs are getting more "accurate" and increasingly replacing human annotators. But here's the critical unanswered question: ❓ Does reasoning help or hurt when capturing human disagreement?

Ni Jingwei avatar
Ni Jingwei@NJingwei·Jan 8

🧵[2/7] Disagreement = signal, not noise. Recent ACL keynotes: 🎤 @barbara_plank (ACL'24): "LLMs are making NLP homogeneous. Embrace variation." 🎤 @verena_rieser (ACL'25): "Our 'gold standard' reflects diverse, contradictory values."

Ni Jingwei avatar
Ni Jingwei@NJingwei·Jan 8

🚨 Using reasoning LLMs as annotators? You might be erasing critical human disagreements. Our EACL'25 paper shows RLVR-style reasoning actually HARMS disagreement modeling—even when carefully prompted to consider it! 🙀 📄 Paper: arxiv.org/abs/2506.19467🧵👇

Post attachment
Joël Niklaus avatar
⟳ Ni Jingwei retweeted
Joël Niklaus@joelniklaus·23 Oct 2025

LEXam Update: GPT-5 Takes the Top Spot We're excited to share our latest LEXam evaluation results: - GPT-5 claims the #1 position, outperforming Gemini 2.5 Pro and setting a new state-of-the-art for legal reasoning on LEXam! - We've moved from a single GPT-4o judge to a minimum-score ensemble of GPT-4o, Qwen3-32B, and DeepSeek-V3. This ensemble was developed through iterative refinement by PhD-trained legal experts and validated via the Alt-test (Calderon et al., 2025), achieving a perfect winning rate (ω=1.00) against all three human experts. Crucially, we found that minimum-score ensembles of even open-source LLMs can match human expert judgment, making rigorous legal evaluation more accessible. - Sonnet 4.5 underperforms its predecessor Sonnet 3.7, and similarly, Qwen3-next trails behind QwQ-32B. This could suggest that the industry's current focus on coding optimization may not transfer effectively to legal reasoning tasks. - The GPT-OSS models offer exceptional accuracy-to-price ratios on LEXam. cc Yu Fan, @NJingwei, Jakob Merane, Yang Tian, @YoanHermstruwer, @YinyaHuang, @akhtarmubashara, Etienne Salimbeni, Florian Geering, Oliver Dreyer, Daniel Brunner, @LeippoldMarkus, @mrinmayasachan, @Stremitzer_Lab, Christoph Engel, @ellliottt

Post attachment
Yinya Huang avatar
⟳ Ni Jingwei retweeted
Yinya Huang@YinyaHuang·27 May 2025

🤖⚛️Can AI truly see Physics? Test your model with the newly released SeePhys Benchmark! 🚀 🖼️Covering 2,000 vision-text multimodal physics problems spanning from middle school to doctoral qualification exams, the SeePhys benchmark systematically evaluates LLMs/MLLMs on tasks integrating complex scientific diagrams with theoretical derivations. 📊Experiments reveal that even SOTA models like Gemini-2.5-Pro and o4-mini achieve accuracy rates below 55%, with over 30% error rates on simple middle-school-level problems, highlighting significant challenges in multimodal reasoning. Key Features Highlighted: 🔎Vision-Text Integration: Explicitly emphasizes multimodal reasoning failures in interpreting diagrams (e.g., circuit schematics, coordinate systems). 🔎Cross-Domain Complexity: Tests models across 7 physics domains and 8 educational tiers, exposing weaknesses in both visual grounding and logical derivation. 🔎Open-Source Design: Fully reproducible framework for diagnosing AI's "visual illiteracy" in scientific contexts. 🎖️Project led by: @kaleb962, @HengLee29423, Terry Jingchen Zhang, @YinyaHuang 💼Joint work with an exceptional team: Zirong Liu, Peixin Qu, Jixi He, Jiaqi Chen, Yu-Jie Yuan, Jianhua Han, Hang Xu, Hanhui Li, @mrinmayasachan, Xiaodan Liang 🏁The benchmark is now open for evaluation at the ICML 2025 AI for MATH Workshop. Academic and industrial teams are invited to test their models and advance multimodal physics! ⚛️Project Page: seephys.github.io 🤗Data: huggingface.co/datasets/SeeP… 📜Paper: arxiv.org/abs/2505.19099 🏆Challenge Submission: codabench.org/competitions/7… ➡️Competition Guidelines: sites.google.com/view/ai4mat…

Post attachmentPost attachmentPost attachmentPost attachment
Ni Jingwei avatar
Ni Jingwei@NJingwei·28 May 2025

🚀 Huge thanks to the community! 📈 LEXam is now the #1 trending evaluation dataset on Hugging Face! Check it out: huggingface.co/datasets?othe… 🧠 Built for deep legal reasoning across 340 law exams 👩‍⚖️ Expert LLM judge evaluations, long-form + MCQs #LegalNLP #Benchmark #LLM

🚨Time to push LLMs further! 📚LEXam: The legal reasoning benchmark you’ve been waiting for: • 340 exams, 116 courses (EN/DE) • Long-form, process-focused questions • Expert-level LLM Judges • Rich meta for targeted diagnostics • Contamination-proof, extendable MCQs [1/6]🧵

Post attachment
Ni Jingwei avatar
Ni Jingwei@NJingwei·21 May 2025

6⃣ Future-proof: Contamination & Extendability MCQs are permutable and extensible, making data contamination easy to check and new challenges easy to add. Benchmark stays tough as models evolve!🔒🔁 🔗Check out the paper! arxiv.org/abs/2505.12864 [6/6]🧵

Ni Jingwei avatar
Ni Jingwei@NJingwei·21 May 2025

4⃣ Dig Into Weaknesses with Rich Metadata Every question includes detailed metadata (course, topic, reasoning steps), so you can pinpoint where your model struggles—be it contract law, criminal law, or multi-hop reasoning. 🔎📊 [5/6]🧵

Ni Jingwei avatar
Ni Jingwei@NJingwei·21 May 2025

3⃣ LLM-as-Judge: Reliable, Scalable Evaluation Expert-level LLMs (with human spot-checks) provide consistent grading—making large-scale evaluation finally practical for legal QA. 🏛️🤖 [4/6]🧵