

✓Building @sullyai - MTS, AI Research


We tested GPT-6.2 Astra & Sol as medical clinic receptionists. GPT-6.2 Astra made 8 unsafe calls out of 450, down from 16 for GPT-6 Astra. GPT-6.2 Sol also reduced unsafe calls from 16 to 9. But both models became more likely to reject legitimate patient requests. We built a simulated medical clinic with patient records, calendars, and AI callers. Each model acts as the receptionist, and we verify its actions against the clinic's records after every call. On identity verification tasks, GPT-6.2 Astra dropped from 39/42 successful calls to 32/42. None of its 10 failures were unsafe actions. It just refused to help authorized callers. For example, a father listed as his son's guardian called about an appointment. GPT-6 Astra helped in all 3 runs, while GPT-6.2 Astra failed all 3. The newer models are making fewer unsafe decisions, but they're also becoming less reliable at recognizing when an action is authorized. We're calling this FrontDesk-Bench. Built at @Sullyai, still a WIP. We'll open-source the benchmark and every call log once it's ready.

We've been testing decision models for a few weeks. Now they're in production in our voice agent, deciding which transcript heard the caller correctly on bilingual calls. I wrote up what we learned, including when to trust them:

I open sourced the entire workflow of how I used Claude Fable 5.2 as a film studio. muratcankoylan.github.io/fir… I’ve been seeing a bunch of posts claiming a video was made by AI, and most of them are bait. So I wanted to test it for real. Everything in this film was generated, written or found by Claude: the actors, the script, the voices, the music, the real archive footage and the edit. Here’s an experiment that cost ~$480 at API pricing plus $200 of video generation:

if you crave attention (but attention is fleeting) then you'll feel bad a few days later, but if you care about helping people or solving problems and one day people forget, it doesn't matter, because you know you did it and that the action itself is what mattered to you. so first don't find purpose in the fleeting (unless that's part of the beauty for you). next on what to do specifically, i think a lot of people try to find something that makes them happy, and obviously we should try to be happy, but in any serious pursuit there's no avoiding pain, so perhaps the real question is not what makes you happy and painless but "what will you joyfully suffer for?"


In Claude Code, I started by sharing the idea and a detailed filmmaker persona. Fable researched them, wrote its own 3,800-word persona, and made the film from it. I stayed in the loop with occasional feedback. It did the rest: - built the script and a 100-shot plan - sourced every quote and found the real archive (the 1903 newspaper, Bell’s notebook, NASA’s control rooms) - wrote the prompts, generated stills and reference images, then the shots with Seedance 2.5 on Runway: drafts first, the chosen takes finished at 1080p - cast real people’s voices from ElevenLabs's library and performed each scene as a conversation - generated the score and 95 sound effects, picked takes by measuring them, and threw out a first score that measured as 33 seconds of sub-bass - transcribed every line to check the words and set the lip sync to about a frame - wrote 7,000 lines of Python: one program cuts 98 shots on a 144 bpm grid, one draws every caption and light, one mixes the sound by measurement, because Claude can’t hear - coded every animation: 200 paper strips flown through a hall in 3D with a camera’s motion blur, a cursor made of light that sends out a ring on each click, captions that work like autocomplete, with the next words guessed in grey - coded every transition as maths on the frame, no presets: a whip tilt, a television switching off, an impact that pulls the colours apart, a glitch on the word that crashed, a baton of light that becomes the cursor on the last card - wrote its own checkers and ran them after each change: reading speed for every line of text, lip sync in frames, the music bar by bar, frozen and black frames. each fault became a code change and a new render - looked at every shot as still frames, across fifteen versions

Nobody predicts the future. Somebody starts it. A one-minute film made entirely with Claude Fable 5.2 Max in 8 hours:

There's another metaphor from Sassy the Sasquatch, in the episode where I got the image above. Being stuck in a world you take for granted doesn't mean you can't imagine a way out. That's what matters most about what's coming with AI. Not preserving the world as we know it, but creating the knowledge to navigate whatever comes next. Amazing series, btw. Worth watching. youtube.com/watch?v=vUyJq1kd…

You stand on a glacier and you'll swear it isn't moving. Every day looks exactly like the one before. Then you look up and the whole landscape is gone. That's AI rn. If you think Will is wrong, I'm sorry but you're the guy on the glacier. Progress itself is changing shape. My biggest fear used to be the government slowing all this down. Now it's basically most people's last hope, the only thing that might buy us some time to adapt before it gets ugly. Idt it will work though. You can't legislate away new knowledge, and you can't regulate discoveries nobody has made yet. We're way closer to the singularity than anyone wants to admit. and the question isn't whether humans become obsolete but more like whether we can learn fast enough to stay (while embodying it) on the field instead of watching from the sidelines, in a future we can't even picture anymore. I'm staying optimistic anyway. Pessimists don't build the thing that keeps us on the field.

Coding models like Fable 5.2 and Gemini 4 Carbon have better taste than 99.9% of humans. I’m working on a creative project with one of them, and it’s blowing my mind. From writing a script for edits to coming up with amazing prompts for the video models, and coding the transitions or animations... Or seeing it spotting a tiny lip-sync issue in its own edit and building itself a local tool to fix it. We’re in a different era man. Ignore the noise and go build with these things. People still underestimate how much your own taste changes what you get out of these models. Take your tacit knowedlge, put it in a detailed persona prompt, give the model context, and just talk with it. That’s when it starts to feel like you can end up making things you didn’t even know were possible.

When Anthropic release a new model it's always worth taking a look at their updated prompting guide - here's an entertaining tip to make Fable 5.2 less likely to be cautious in how it improves existing code: tell it "This codebase is yours" platform.claude.com/docs/en/…

We gave Jev 2,029 real phone calls. No transcripts or audio; it never heard a word. Our AI receptionist's calls were reduced to pure structure, meaning turns, tool calls, workflow stages and timing. During the calls, Jev made 38,012 turn-level forecasts at 118 ms median latency, reviewed every call with five typed questions and produced 10,145 answers in 26 seconds with 256 requests in flight. The experiment was zero-shot, with no fine-tuning or examples from our data. We compared Jev's forecasts with what actually happened in the EHR. By the halfway point, Jev could meaningfully separate calls that would book from those that wouldn't (AUC 0.78), and near the end it ranked them correctly 94% of the time. Even though Jev over-focused on visible errors our agent usually overcomes, it's still pretty incredible that it analyzed thousands of real calls in seconds for only $3.

We wrote down how we evaluate clinical AI at Sully. Two ideas you can replicate in any domain: 1. Build benchmarks from inputs that look alike but need different outputs 2. Treat every user edit as a label, weighted by what it changed We cluster transcripts and the notes generated from them separately. Two visits that sound almost the same but produce different notes are where one detail changes the right answer. Those become benchmark cases. "started metformin" -> "discussed starting metformin" A two-word edit so now it's a different patient record. That's why edit distance failed us as a quality metric. The size of an edit says nothing about its clinical importance. Our pipeline compares three things: a) the note we generated b) the note the clinician signed c) the visit transcript and context The differences are split into atomic edits, and each one is classified. Fabrication: clinician removed something the visit didn't support Omission: clinician added something the visit did support Misinterpretation: clinician changed something Style: nonclinical change Each one also gets a type (medication, plan, diagnosis) and a severity from low to critical. LLMs only run on steps where a deterministic method lost in ablations, and those steps use two judges from different model families.

It’s almost incomprehensible how wide the distance is between someone who’s working at the frontier with AI and basically everyone else right now. The crazy thing is these lines are not converging, but only growing farther apart. If the curve of AI progress had leveled off, it would certainly have given most people and companies time to catch up or implement their workflows in a standardized way. The technology would have neutralized competitive advantage because everyone would be using it the same way. But what’s happened is the opposite. The curve of AI progress is basically going vertical right now, which means if you had codified your processes 3-6 months ago you’re likely already behind again. Now imagine most companies that kicked into gear a year ago and thought that they were done. Even further behind now. I’ve never seen a technology in my life and career that improves at this rate, which consequently means you need to upgrade your expectations at a similar rate. Our brains weren’t wired for this, and our organizational change management processes certainly weren’t either. With any form of recursive self improvement, things just get wilder. It’s a time when you will need to be constantly reevaluating the things that didn’t work just a couple months ago to see if they now can be solved. AI native companies certainly have a strong advantage due to this right now, so now it comes down to can the bigger companies keep up. Wild times.
