Gemini Omni Flash is officially out! 🚀
It has been incredible working on the model pretraining, landing cool features in video editing, and pushing the 3D/camera capabilities in Omni. Seeing it live at #GoogleIO is surreal. 😍😇
Took countless days of no to little sleep. But sometimes it takes a little crazy to make something truly amazing. ✨✨✨
Go try it out! deepmind.google/models/gemin…#googleemployee@GeminiApp@GoogleDeepMind#gemini#omni
(1/10) Excited to share our new paper: "Towards Conversational Medical AI with Eyes, Ears and a Voice"
arxiv.org/abs/2605.09272
Clinical medicine depends on more than words; it relies on real-time visual cues (rashes, distress, gait), auditory signals (prosody, breathing), and guided physical exams (e.g., guiding someone through correcting their inhaler technique, or walking them through shoulder maneuvers to work up a rotator cuff injury).
But existing conversational medical AI systems are blind and deaf to this crucial information and are constrained to rigid, turn-by-turn user experience.
As an important ingredient in AI co-clinician, our new healthcare initiative at @GoogleDeepMind, we developed a first-of-its-kind real-time multimodal system that conducts natural clinical consultations over video calls while continuously watching, listening, and reasoning.
The preprint details the research progress we’ve made so far together with our collaborators at @StanfordMed and @harvardmed (@AdamRodmanMD@euanashley@DrJackOSullivan@jasongusdorf).
More details below!
Since joining @GoogleDeepMind I’ve dreamt for a decade that AI can give clinicians superpowers: amplifying reach as an always-available, trustworthy member of the care team. Delighted to share our strategic research initiative at @GoogleDeepMind towards this vision: AI co-clinician deepmind.google/blog/ai-co-c… (1/n)
9/ We are currently in a phased research approach with global collaborators ( 🇺🇸 🇦🇺 🇳🇿 🇦🇪 🇸🇬 🇮🇳) to ensure this is developed responsibly. Huge congratulations to everyone on the team and our partners for making this possible.
8/ We utilize a dual-agent architecture to decouple fluid multimodal interaction from deep clinical reasoning. A low-latency "Talker" handles empathetic dialogue, while a supervisory "Clinical Planner" monitors the encounter state.
7/ Unsurprisingly findings show that skilled physicians remain superior to AI co-clinician overall, particularly in spotting "red flags". There is still a high ceiling for further improvement, and that’s what continues to drive the team.
6/ Working with @BIDMChealth and @StanfordMed, we ran randomized simulations where the agent successfully guided "patient-actors" through physical exams, like identifying rotator cuff injuries or correcting inhaler techniques.
5/ But medicine isn't just text; it requires eyes, ears, and a voice. Building on Gemini and Project Astra, AI co-clinician can use live audio and video to talk with users and notice vital non-verbal cues, like a patient’s gait or the sound of their breathing.
4/ Precise information is vital for medication and therapy. AI co-clinician demonstrated significant progress on the RxQA benchmark and on realistic, open-ended clinical questions about medications, it outperformed currently available frontier models.
3/ For a physician, a tool is only useful if it is trustworthy. We adapted the "NOHARM" framework to test for errors. In blind, head-to-head evaluations, physicians preferred AI co-clinician’s responses to leading available tools, making zero critical errors in 97/98 queries.
2/ Healthcare systems worldwide are facing a massive challenge: a shortfall of 10 million health workers by 2030. Medicine being a team sport, we explore if AI can extend that team to offer genuine utility to both patients and physicians.
AI co-clinician is our new research initiative to help explore how multimodal agents could better support healthcare workers and patients. 🩺
Here’s a snapshot of our progress 🧵
AI co-clinician is our new research initiative to help explore how multimodal agents could better support healthcare workers and patients. 🩺
Here’s a snapshot of our progress 🧵
I am happy to introduce AI co-clinician,
@GoogleDeepMind's research initiative to explore how AI could better amplify doctor's expertise and help deliver higher quality care to patients.
We’re excited about our early results, and are taking a phased approach to our research explorations with academic and research collaborators.
Read more in our blog: deepmind.google/blog/ai-co-c…
VQA is consistently seeing tremendous progress year by year. Excited to see further progress as we now focus on specific sets of problems (reading text, consistency, external knowledge etc.).
Planning research on V+L? Do have a look at Pythia github.com/facebookresearch/…
VQA performance on a standard benchmark (VQA v2 dataset) has gone up 20% (absolute) in the last ~4 years. You can really tell the difference when interacting with these models! Check the VQA demo out here: vqa.cloudcv.org.
Checkout and discuss our @cvpr2019 poster #196 "Towards VQA Models That Can Read" at 3:20PM. Learn how current VQA models fail to answer questions about image text and our steps toward solving it. More details + TextVQA dataset + Starter Code available at textvqa.org.
Meet Shah (@Jalebi_Fafda) presenting his @facebookai AI Residency work at #CVPR2019 on a new cycle consistecy based training framework to make VQA models robust to linguistic variations in the input question.
Code + Dataset are publicly available
facebookresearch.github.io/V…