100% open source framework for realtime voice and multimodal AI. Maintained by @trydaily engineering team with support from the Pipecat developer community.
So very, very excited. The Pipecat team is joining @DeepgramAI and, as always, staying open source and vendor neutral. Same license, same maintainers! Post from @kwindla
kwindla @kwindla Oct 29 Article Deepgram is acquiring Daily and establishing an independent Pipecat Foundation I'm excited to share that the Daily team is joining Deepgram. We will continue our work on the Pipecat open source framework and on our global network infrastructure that powers realtime audio, video,
The hard part of a multilingual voice agent isn’t translation, but instead how people switch languages mid-sentence and expect it to follow.
In this video, Merve explores how far you can get with @SarvamAI models across the whole Pipecat pipeline: hearing, thinking and speaking in Hinglish.
Sarvam models in Pipecat:
docs.pipecat.ai/api-referenc…
Demo from this video:
github.com/pipecat-ai/sarvam…
Phonon: a new open, on-device, text-to-speech model from Gradium. It's really good.
Click through, below, and you can test it in your web browser (enabled by an optimized WebGPU inference stack). Go do it, just to *feel* what it's like when time-to-first-byte is ~10ms.
There are lots of reasons to run agents/models in the cloud. But the trade-offs are cost for hosting and inference, and latency.
We have to work hard to get time-to-first-byte down below 50ms when we're running our models in the cloud. And usually for a text-to-speech model we're aiming for 100-150ms TTFB in production.
Running on-device means you can optimize to run one inference request at a time in a specific hardware and software environment. And the cost is ~$0. (Or, in very demanding applications, the power budget/battery life for the device.)
Open and on-device models make it possible for us to imagine and build new things. I expect to see people post a lot of great Phonon demos over the next few weeks!
We have some fun voice AI stuff to talk about on the @NVIDIAAI Nemotron Labs livestream tomorrow: fine-tuning audio and LLM models for specific agents and use cases. Join us live if you want to chat and ask questions. 10:00am Pacific Time on Tuesday.
Sometimes, choosing a voice for your Pipecat agent is about more than just latency. It might be about personality, accent, or how convincingly a voice fits the situation.
We explore voice realism with @usebland Speech v3, available in Pipecat as a maintained service.
📖 docs.pipecat.ai/api-referenc…
🔥 New @SarvamAI STT release — Saaras V4, with Pipecat support on launch day. 22 Indian languages + English. 10 of its Indian languages have no commercial alternative.
Watch @VinayakGavariya@SarvamForDevs demo key features
Saaras V4 is here.
Built for the messy reality of speech, accents, dialects, noise, and mixed-language conversations.
→ 5 modes: Transcribe, Translate, Transliterate, Codemix, Verbatim
→ Keyterm prompting: Guide the model toward the words that matter
→ 22 Indian languages + English
→ Real-time streaming: <150ms time-to-first-token
→ SOTA performance across all 22 Indian languages
Speech recognition, built for the real world.
See Saaras V4 in action in this demo.
Introducing PhoneLLM, an open model for voice agents.
GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost.
For voice agents, we need models that are both very low latency and very good at tool calling and instruction following.
There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem.
For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model.
But if you need your agent to respond at voice conversation speed, you can't use thinking models.
PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled.
The results are really good: accurate tool calling and concise, on-topic responses in long conversations.
And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-)
But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines.
You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today.
More details about this model, including weights on @huggingface, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...
Pipecat v1.8.0 today, with launch day support for Gemini 3.5 Transcribe, Google's new Gemini-based transcription model.
This is a very big Pipecat release. There are 233 entries in the changelog.
Some highlights:
🛡️Improvements to error handling and service failover: Processors can now report when they are no longer usable, errors are categorized, and ServiceSwitcher only fails over when a service really can't recover. STT/TTS services also stop endlessly retrying permanent failures like invalid API keys or models.
⚡ Faster pipeline startup: Pipeline setup has been reworked around the new setup() lifecycle, allowing processors and services to initialize and connect concurrently before the pipeline starts. Combined with import-time improvements, this significantly reduces startup time, especially for larger pipelines.
🗣️ More flexible turn management: Services with built-in turn detection now propose turn boundaries, while Pipecat's turn strategies make the final decision. This keeps turn management and interruption handling in one place and makes provider-native turn detection much easier to customize.
🧪 Pipecat Evals keeps getting more powerful: Run scenarios during development to iteratively measure pass rates, get machine-readable results.jsonl, evaluate turns individually, run entire directories of scenarios, and more. Great for measuring nondeterministic behaviors like interruptions and async function calls instead of testing them once and hoping for the best.
🤖 Better coding-agent experience: Pipecat Context Hub is now included with the CLI and integrated into pipecat init, including automatic setup for supported coding agents and freshness checks to help prevent agents from generating code against stale Pipecat APIs.
🔧 Function calls can now be cancelled by the LLM: Long-running async tools can opt into LLM-driven cancellation, and timed-out function calls are now automatically cancelled instead of continuing to run in the background.
🔌 New Speechify TTS, ElevenLabs Dialogue TTS, Bland TTS, Sarvam Realtime STT, Deepgram Flux on SageMaker, OpenClaw Gateway support, LiveKit runner support, MoQ client mode for connecting bots through a relay, including deployments behind NAT and more.
And, of course, there are tons of fixes and smaller improvements throughout the framework, including pipeline startup/cleanup, metrics, TTS tracking, transports, realtime services, and turn handling.
Huge thanks to everyone in the community for making everything we do possible! There are now more than 180 Pipecat integrations. ❤️
Happy hacking! 🚀
Congrats to @DeepgramAI on the GA launch of Flux TTS, now live in Pipecat 🎉
@JonPTaylor looks at how Flux TTS delivers a more consistent voice experience. Flux reads the whole conversation, not just the next line — adaptive tone, consistent pronunciation, clean interruption handling natively (no SSML markup, no style tags!) all at sub-200ms latency.
➡️ docs.pipecat.ai/api-referenc…
@aiDotEngineer Concierge is now live!
Talk to the concierge and build your custom schedule for the next three days along with a list of people ( speakers ) who you should definitely meet!
This was built on the go with @pipecat_ai and @GradiumAI
Try it out: aie-concierge-web.onrender.c…
One shotted a fun voice agent with @pipecat_ai and @GradiumAI to figure out with sessions to attend based on my profile at @aiDotEngineer .
My agent recommended @danielhanchen ( ofcourse! ) and few more.
Should I ship this so that everyone coming to conf can find their way around? Wdyt @swyx ?
World Cup Voice Commentary right in your MenuBar!
@pipecat_ai powered voice bot, not only reports on the scores but can answer any questions I have about the games!
Using only local models on my M4 Max. Under the hood:
+ Kokoro TTS for commentary
+ Gemma 4e4 (with and without thinking)
+ Nemotron 3.5 for ASR
Built with Pipecat subagents: when the menubar starts, it spins up a main loop of
[STT]
|
[LLM] <-> [subagent when a game starts]
|
[TTS]
There is one subagent per game. Schedules are pulled regularly, so I can ask stuff like “Who will the US play in the round of 32?” and have it reason over fixtures, for example.
A collector ingests live commentary (per match-id, home/away, and metadata) into a local DB and exposes it via a CLI so pipecat's subagents can query it.
Each game subagent gets an initial prompt about “its” match and tools including an /analyze-game-skill. The skill was built and tested with the early R1 games, especially the chaos of Canada-Qatar game (with injuries, goals, and two yellow cards becoming a red!)
The result: when something happens in the game, the TTS chimes in, excited or dejected, about missed chances, corners, cards, etc. Subagents maintain their own context, so with push-to-talk, which invokes the STT, I can ask about what is happening in a current, past, or future games, fixtures, etc.
The main LLM routes to the correct subagent, or can directly check via CLI, or spins up a new subagent to go hunt down answers (e.g., scores about R1 games, schedule changes, etc.)
Today’s rain delays accidentally stress-tested having multiple concurrent games in Round 3, and the subagent orchestration held up nicely.
Clips + screenshot below 👇
Halftime announcement, second goal, and the third goal.
These were voice memo recordings, aargh
My favorite annual AI event is next week in San Francisco. Come hang out with me at the @aiDotEngineer World's Fair.
Go buy a ticket. Or, if you're a student, there's a great volunteer program.
The organizing team does a great job making the World's Fair a conference with great talks, an opportunity for deep-dive learning across multiple AI engineering domains, a gathering of people building the infrastructure for AI adoption, and an opportunity to see early what everyone will be talking about next year.
I'm doing a couple of talks. One on voice interfaces, with @neilzegh. And one about experimenting with new building blocks for "AI-native" software, software patterns for applications we couldn't have built at all before now and that we'll take for granted in a few years!
I get more requests than I can keep up with, via LinkedIn and email, to meet and talk about AI engineering, voice AI, and @pipecat_ai. Most of the year, my default response is, "I would love to but I can't, if you make a PR I'll try hard to look at it." AI Engineer World's Fair is the one time of year when I can say: if you'll be at the World's Fair, come find me. We'll geek out about what you're building and all the crazy ideas we both have!
Until today, a transcription model never knew what your voice agent just asked.
The one we're launching today does.
Universal-3.5 Pro Realtime from AssemblyAI is the first realtime speech-to-text model that takes your agent's side of the conversation as context.
When your agent asks for an email and the caller rattles it off, the model already knows an email is coming, and captures it perfectly. Same for an order number read fast, an account ID, a medication, or a name spelled out over poor quality phone line.
You can turn an existing LangGraph agent into a fully functional voice agent with @pipecat_ai.
This 17-minute walkthrough shows you exactly how to do it.