←
Philipp Schmid6,551 posts
Philipp Schmid avatar

Philipp Schmid

@_philschmid

Agents & Gemini API, MTS @GoogleDeepMind | prev: Tech Lead at @huggingface, AWS ML Hero 🤗 Sharing my own views and AI News 🧑🏻‍💻 philschmid.de

◷ Joined June 2019▧ 2,020 Photos and videos
Philipp Schmid avatar
Philipp Schmid@_philschmid·Jun 22

The Interactions API is now generally available. 🎉 The Interactions API is the simplest way to build with Gemini for humans and agents. - One API for Gemini models and agents. - Antigravity Agent with a isolated remote Linux sandbox. - Image gen with Nano Banana; music with Lyria 3, soon video with Omni. - `background=True` for async, long-running interactions. - Multimodal Tool Use & Combination. - Dedicated skills for coding agent. Building something new inside Google isn't easy, but it's possible. We spent almost a year on this because we think developers and agents deserve an API that is intuitive, familiar and easy to learn and use and evolves with improving capabilities and new behaviors. Give it a try. Feature requests, bugs, complaints all to me. 👇🏻

Post attachment
Philipp Schmid avatar
Philipp Schmid@_philschmid·2h

Agent harnesses should replace built-in multi-agent features with an MCP servers of the harness itself. Use the same loop/cli recursively to create new threads ... or search in them. Use MCP events for subscriptions. You could rebuild message board collaboration for model 1 and have orechstrator pattern for model 2 Much easier to iterate or upgrade.

Philipp Schmid avatar
Philipp Schmid@_philschmid·Nov 16

Full article:

Philipp Schmid @_philschmid Nov 16 Article Gemini 4 Carbon, Hydrogen, Decisions API, self-hosted agent sandboxes and more Last week we shipped 2 Gemini 4 models, our first Decision Model & API, self-hosted agent sandboxes in Gemini Managed Agents, a new visual knowledge benchmark, and new robotics and genomics models.

Philipp Schmid avatar
Philipp Schmid@_philschmid·Nov 16

We also shipped two domain models: 🤖 Gemini Robotics 2.5: whole-body Vision-Language-Action model across arms and humanoids hitting 96% kitting success on Mercedes-Benz lines with Apptronik Apollo 2 deepmind.google/blog/gemini-… 🧬 AlphaGenome 2: single-cell DNA model trained on 107M+ cell profiles across 68 tissues with scRNA-seq input without retraining deepmind.google/science/alph…

Philipp Schmid avatar
Philipp Schmid@_philschmid·Nov 16

Gemini Managed Agents now supports self-hosted environments. We run model inference and the agent loop on Google infrastructure while bash commands and file edits execute inside your compute over outbound HTTPS. Test it locally in TypeScript with our open-source gemini-api CLI, or deploy on Modal, E2B, Cloudflare, Daytona, Fly, Vercel, or Cloud Run: aistudio.google.com/docs/age…

Post attachment
Philipp Schmid avatar
Philipp Schmid@_philschmid·Nov 16

Our new Gemini Decisions API, powered by Gemini Decision 1, evaluates text, images, video, PDFs, or audio across up to 256K tokens against up to 1,000 isolated questions with 0 output tokens billed. Use noul for binary checks, choice across up to 255 keys, and score for ordered rubrics: aistudio.google.com/docs/dec…

Post attachment
Philipp Schmid avatar
Philipp Schmid@_philschmid·Nov 16

How to pick between Gemini 4 Hydrogen and Carbon for your workload: 🧠 Carbon: hard architecture design, debugging, scientific proofs, and security audits at $2 / $10 intro thru Nov 30 ⚡ Hydrogen: everyday coding, interactive apps, tool-using agents, and parallel subagents at $1.50 / $7.50 per 1M You can also pair Carbon and Hydrogen: use Carbon to plan and orchestrate and Hydrogen to execute: aistudio.google.com/docs/wha…

Post attachment
Philipp Schmid avatar
Philipp Schmid@_philschmid·Nov 16

Last week at @GoogleDeepMind was busy, we shipped 7 developer launches. 🧠 Gemini 4 Carbon ⚡ Gemini 4 Hydrogen ⚖️ Gemini Decisions API + Gemini Decision 1 📦 Self-hosted sandboxes for Managed Agents 📊 Multimodal Knowledge Agent benchmark 🤖 Gemini Robotics 2.5 🧬 AlphaGenome 2

Post attachment
Philipp Schmid avatar
Philipp Schmid@_philschmid·Nov 13

Great example for the Google Decision 1 audio capability.

Pipecat 1.16 adds PerceptionReader, something I'm really excited about. How someone speaks can matter just as much as what they say, and voice agents have mostly been missing that. We’ve been testing it with Google Decision 1, released yesterday. Stream it audio and you get a pretty accurate read on the prosody, tone and sarcasm of what the user said. Try talking to it at this demo here: pipecat-google-decision-demo… We've exploring many different ways of measuring expressivity... more to share on that soon! ➡️ Pipecat Docs: docs.pipecat.ai/api-referenc… ➡️ Google Decision 1: blog.google/innovation-and-a…

2:03
Philipp Schmid avatar
Philipp Schmid@_philschmid·Nov 12

Today, we're shipping Gemini 4 Hydrogen in @GoogleAIStudio, the Gemini API! It became my daily driver for day-to-day fast execution, research loops and is a great subagent, paired with Carbon. Hydrogen is great at the fast code-to-test loops. - Outperforms Argon on @ArtificialAnlys - Scores 57% on Terminal-Bench 4.0 - 4 thinking levels: low, medium, high, xhigh - $1.50/$7.50, free tier in Google AI Studio and the Gemini API Also rolling out to paid Google AI Pro and Ultra subscribers in @antigravity, @GeminiApp, @Google Search and @Google Cloud. blog.google/innovation-and-a…

Post attachment