←
Meet Shah31 posts
Meet Shah avatar

Meet Shah

@meetshahdev

Research Engineer @GoogleDeepmind prev: @Waymo, @Uber_ATG, @metaai, @iitbombay

◷ Joined February 2016▧ 8 Photos and videos
Yash Bhalgat avatar
⟳ Meet Shah retweeted
Yash Bhalgat@ysbhalgat·May 19

Gemini Omni Flash is officially out! 🚀 It has been incredible working on the model pretraining, landing cool features in video editing, and pushing the 3D/camera capabilities in Omni. Seeing it live at #GoogleIO is surreal. 😍😇 Took countless days of no to little sleep. But sometimes it takes a little crazy to make something truly amazing. ✨✨✨ Go try it out! deepmind.google/models/gemin… #googleemployee @GeminiApp @GoogleDeepMind #gemini #omni

0:08
Ryu Tanno | 丹野 龍太郎 avatar
⟳ Meet Shah retweeted
Ryu Tanno | 丹野 龍太郎@RyutaroTanno·May 18

(1/10) Excited to share our new paper: "Towards Conversational Medical AI with Eyes, Ears and a Voice" arxiv.org/abs/2605.09272 Clinical medicine depends on more than words; it relies on real-time visual cues (rashes, distress, gait), auditory signals (prosody, breathing), and guided physical exams (e.g., guiding someone through correcting their inhaler technique, or walking them through shoulder maneuvers to work up a rotator cuff injury). But existing conversational medical AI systems are blind and deaf to this crucial information and are constrained to rigid, turn-by-turn user experience. As an important ingredient in AI co-clinician, our new healthcare initiative at @GoogleDeepMind, we developed a first-of-its-kind real-time multimodal system that conducts natural clinical consultations over video calls while continuously watching, listening, and reasoning. The preprint details the research progress we’ve made so far together with our collaborators at @StanfordMed and @harvardmed (@AdamRodmanMD @euanashley @DrJackOSullivan @jasongusdorf). More details below!

Post attachment

Since joining @GoogleDeepMind I’ve dreamt for a decade that AI can give clinicians superpowers: amplifying reach as an always-available, trustworthy member of the care team. Delighted to share our strategic research initiative at @GoogleDeepMind towards this vision: AI co-clinician deepmind.google/blog/ai-co-c… (1/n)

1:27
Meet Shah avatar
Meet Shah@meetshahdev·May 1

9/ We are currently in a phased research approach with global collaborators ( 🇺🇸 🇦🇺 🇳🇿 🇦🇪 🇸🇬 🇮🇳) to ensure this is developed responsibly. Huge congratulations to everyone on the team and our partners for making this possible.

Meet Shah avatar
Meet Shah@meetshahdev·May 1

8/ We utilize a dual-agent architecture to decouple fluid multimodal interaction from deep clinical reasoning. A low-latency "Talker" handles empathetic dialogue, while a supervisory "Clinical Planner" monitors the encounter state.

Post attachment
Meet Shah avatar
Meet Shah@meetshahdev·May 1

7/ Unsurprisingly findings show that skilled physicians remain superior to AI co-clinician overall, particularly in spotting "red flags". There is still a high ceiling for further improvement, and that’s what continues to drive the team.

Post attachmentPost attachment
Meet Shah avatar
Meet Shah@meetshahdev·May 1

6/ Working with @BIDMChealth and @StanfordMed, we ran randomized simulations where the agent successfully guided "patient-actors" through physical exams, like identifying rotator cuff injuries or correcting inhaler techniques.

Post attachment
Meet Shah avatar
Meet Shah@meetshahdev·May 1

5/ But medicine isn't just text; it requires eyes, ears, and a voice. Building on Gemini and Project Astra, AI co-clinician can use live audio and video to talk with users and notice vital non-verbal cues, like a patient’s gait or the sound of their breathing.

Post attachment
Meet Shah avatar
Meet Shah@meetshahdev·May 1

4/ Precise information is vital for medication and therapy. AI co-clinician demonstrated significant progress on the RxQA benchmark and on realistic, open-ended clinical questions about medications, it outperformed currently available frontier models.

Post attachment
Meet Shah avatar
Meet Shah@meetshahdev·May 1

3/ For a physician, a tool is only useful if it is trustworthy. We adapted the "NOHARM" framework to test for errors. In blind, head-to-head evaluations, physicians preferred AI co-clinician’s responses to leading available tools, making zero critical errors in 97/98 queries.

Post attachment
Meet Shah avatar
Meet Shah@meetshahdev·May 1

2/ Healthcare systems worldwide are facing a massive challenge: a shortfall of 10 million health workers by 2030. Medicine being a team sport, we explore if AI can extend that team to offer genuine utility to both patients and physicians.

Post attachment
Meet Shah avatar
Meet Shah@meetshahdev·May 1

1/ Introducing AI co-clinician: latest research milestone I’ve been a part of at @GoogleDeepmind, researching the evolution of care delivery where AI can support both patients & doctors. tech report: gstatic.com/vesper/ai_coclin… blogpost: deepmind.google/blog/ai-co-c…

AI co-clinician is our new research initiative to help explore how multimodal agents could better support healthcare workers and patients. 🩺 Here’s a snapshot of our progress 🧵

1:27
Google DeepMind avatar
⟳ Meet Shah retweeted
Google DeepMind@GoogleDeepMind·Apr 30

AI co-clinician is our new research initiative to help explore how multimodal agents could better support healthcare workers and patients. 🩺 Here’s a snapshot of our progress 🧵

1:27
Pushmeet Kohli avatar
⟳ Meet Shah retweeted
Pushmeet Kohli@pushmeet·Apr 30

I am happy to introduce AI co-clinician, @GoogleDeepMind's research initiative to explore how AI could better amplify doctor's expertise and help deliver higher quality care to patients. We’re excited about our early results, and are taking a phased approach to our research explorations with academic and research collaborators. Read more in our blog: deepmind.google/blog/ai-co-c…

Amanpreet Singh avatar
⟳ Meet Shah retweeted
Amanpreet Singh@apsdehal·21 Jun 2019

VQA is consistently seeing tremendous progress year by year. Excited to see further progress as we now focus on specific sets of problems (reading text, consistency, external knowledge etc.). Planning research on V+L? Do have a look at Pythia github.com/facebookresearch/…

VQA performance on a standard benchmark (VQA v2 dataset) has gone up 20% (absolute) in the last ~4 years. You can really tell the difference when interacting with these models! Check the VQA demo out here: vqa.cloudcv.org.

Amanpreet Singh avatar
⟳ Meet Shah retweeted
Amanpreet Singh@apsdehal·19 Jun 2019

Checkout and discuss our @cvpr2019 poster #196 "Towards VQA Models That Can Read" at 3:20PM. Learn how current VQA models fail to answer questions about image text and our steps toward solving it. More details + TextVQA dataset + Starter Code available at textvqa.org.

Post attachment