←
HalachaBench248 posts
HalachaBench avatar

HalachaBench

@HalachaBench

AI models on real semicha exam questions from Israel's Chief Rabbinate, in Hebrew. Provisional results; reviewers welcome: hello@halachabench.ai

◷ Joined October 2026▧ 30 Photos and videos
HalachaBench avatar
HalachaBench@HalachaBench·Nov 1

Introducing HalachaBench: we gave AI programs from the makers of ChatGPT, Claude, Gemini and others real semicha exam questions from Israel's Chief Rabbinate, in Hebrew, with no books and no internet. The best scored 80 out of 100. Many scored under 11. 🧵

Post attachment
David/Dovid Bashevkin avatar
⟳ HalachaBench retweeted
David/Dovid Bashevkin@DBashIdeas·Nov 17

We sat with the minds behind @HalachaBench to discuss the intersection of Halacha and AI.

Can AI pass a rabbinic ordination exam? And what happens if it can? This week on the 18Forty Podcast, @HalachaBench creators Ben Niewood and Sam Weiss explore what AI means for halacha, rabbinic authority, and the future of Jewish life.

1:06:35
18Forty avatar
⟳ HalachaBench retweeted
18Forty@18_forty·Nov 17

The tweet that started it all:

Introducing HalachaBench: we gave AI programs from the makers of ChatGPT, Claude, Gemini and others real semicha exam questions from Israel's Chief Rabbinate, in Hebrew, with no books and no internet. The best scored 80 out of 100. Many scored under 11. 🧵

Post attachment
18Forty avatar
⟳ HalachaBench retweeted
18Forty@18_forty·Nov 17

Can AI pass a rabbinic ordination exam? And what happens if it can? This week on the 18Forty Podcast, @HalachaBench creators Ben Niewood and Sam Weiss explore what AI means for halacha, rabbinic authority, and the future of Jewish life.

1:06:35
HalachaBench avatar
HalachaBench@HalachaBench·Nov 13

What do all those tokens mean? Answers are capped at 600 words, so nearly all of it is the model reasoning before it writes. Carbon thinks ~5x more than Argon to gain 4 points, and ~11x more than Fable 5.2 while still scoring lower. More thinking helps, but Fable gets there far more efficiently.

HalachaBench avatar
HalachaBench@HalachaBench·Nov 13

How long is "a lot"? Carbon took a median of ~13 minutes per answer, and 1 in 10 took over 20. All 180 answers took 42 hours of run time, vs ~11 for Gemini 4 Argon and under 4 for Fable 5.2. Two answers ran past our 30-minute limit and had to be rerun.

HalachaBench avatar
HalachaBench@HalachaBench·Nov 13

We have a race. 🏁 Gemini 4 Carbon storms to #2 on HalachaBench (82.3), +4 over Argon and within striking distance of champion Claude Fable 5.2 (85.8). It just thinks… a lot: ~69K tokens per answer vs Fable's ~6K. 🧠

Post attachment
HalachaBench avatar
HalachaBench@HalachaBench·Nov 12

Caveats: these use new yes / in-between / no rubrics drafted by Claude Opus 5.5 with web search and Sefaria, not yet reviewed by a rabbi. One answer per model per question, one AI judge, not pre-registered. Exploratory. Data and charts: github.com/halachabench/hash…

HalachaBench avatar
HalachaBench@HalachaBench·Nov 12

How: same 13 models, same questions, one line changed in the prompt ("the person asking is Sephardi"). Three Ashkenazi-only controls dropped (rice on Pesach, egg matzah). 168 of 169 Sephardi control answers correct. Judge: Claude Opus 5.5, told the asker is Sephardi.

HalachaBench avatar
HalachaBench@HalachaBench·Nov 12

Told the asker is Sephardi, does the model actually rule like a Sephardi posek? Claude Fable 5.1: 95% Sephardi. Grok 4.7 and Gemini 3.8 Flash: 90%. GPT-6 Luna 36%, GPT-6 Sol 38%, GPT-6.1 Terra 45%. Most of their answers fit neither tradition clearly.

Post attachment
HalachaBench avatar
HalachaBench@HalachaBench·Nov 12

Every lab shifts. GPT-5.6 Sol, both Geminis, GPT-6 Astra and Claude Haiku 5.5 shift most, 0.08 to 0.09. Claude Opus 5.5 and Sonnet 5.5 barely move. With an Ashkenazi asker Claude was the most lenient lab; with a Sephardi asker, Claude and GPT tie.

Post attachment
HalachaBench avatar
HalachaBench@HalachaBench·Nov 12

And stricter where Sephardi practice is stricter: Only glatt beef: 8% to 85% Food a non-Jewish housekeeper cooked on a flame you lit: 0% to 77% forbid it Kiddush over whiskey: 8% to 73% say no Rabbeinu Tam tefillin: 8% to 69% say put them on Chanukah candles outside: 27% to 85%

HalachaBench avatar
HalachaBench@HalachaBench·Nov 12

Biggest moves toward leniency for the Sephardi asker (share of models ruling strictly, before and after): One set of glass dishes for meat and dairy: 88% to 0% Six hours after parmesan: 100% to 15% Mustard on Pesach: 85% to 0% Wet wipes on Shabbat: 69% to 0%