←
Jon Taylor1,490 posts
Jon Taylor avatar

Jon Taylor

@JonPTaylor

🧑‍💻 Voice AI @trydaily // ᓚᘏᗢ @pipecat_ai 🎹 I have too many synths

◷ Joined July 2008▧ 246 Photos and videos
Jon Taylor avatar
Jon Taylor@JonPTaylor·12h

Spicy take: I'm not the biggest fan of node-based graph editors for building voice agents. Visually, I personally find them hard to follow, and the resulting code is often murky 🌶️. That's probably a me thing, though. But... when designed with a considered design language, there is no doubt they can be a super powerful tools - especially when paired with Pipecat Flows! We've added some primitives to Pipecat UI, as well as a quick start Graph component built on React Flow. Coding agents love em'! (Just remember to connect those wires responsibly 😉)

0:15
Jon Taylor avatar
Jon Taylor@JonPTaylor·Nov 17

We often get asked whether it's possible to build an AI experience with multiple users and one (or more) agents in the loop. This is where good diarization and a fast classifier model come in clutch. The only real prerequisites these days are good orchestration (Pipecat!), good STT (Deepgram!) and a transport and infrastructure that support multiple tracks (Daily!). Here's a one-shot Claude demo. The agent joins a meeting, and the conversation passes through Jev, which decides whether the agent is being addressed. If it isn't sure, it waits 2.5 seconds, and if no one else answers, it offers a response. All of these thresholds can be tuned. Clearly, I haven’t tuned them in this video 😅 For some use-cases, you might prefer a more natural speaking companion that is chatty and joins the conversation, for others (like this demo) you might want something that stays quiet unless spoken to. Jev adds ~100ms of latency to each turn, which is fine. You could also use a tiny classifier like Pipecat Audience (huggingface.co/pipecat-ai/au…) to make the decision on CPU in as little as 15ms.

3:39
Jon Taylor avatar
Jon Taylor@JonPTaylor·Nov 16

Getting a voice agent to read a user's emotion used to be a pipe(cat) dream. Today’s multimodal decision models, like Google Decision 1, make it easy. But they’re large general-purpose models that read tone as one task among many. A bit overkill, and it can add 100–200ms to each turn 😨 This is why we built Pipecat Voice Reader, a tiny open-weights audio model that runs on CPU and streams results in as little as 25ms. Here it is head-to-head with Decision 1. It holds its own on accuracy, which is crazy for a 28MB int-8 model (or 14mb for the tiny version.) Pair it with Pipecat Tone Reader, which reads the transcript, and you get a much cleaner signal on sarcasm vs. sincerity. Demo code is available here: github.com/pipecat-ai/pipeca…

2:07
kwindla avatar
⟳ Jon Taylor retweeted
kwindla@kwindla·Nov 13

Something like 350 million people regularly speak Hindi with lots of English words and phrases mixed in. "Hinglish" might be the most widely spoken language that, unless you live in India, you haven't heard of. But if you do live in India, and you're building voice AI agents for use cases like customer support, the need for AI models that are very good at Hinglish is going to be very much on your mind. Merve made an explainer video about the Sarvam models: LLM, TTS, STT, and decision models trained in 10 Indian languages, and uniquely good at mixed-language understanding, thinking, and speech.

The hard part of a multilingual voice agent isn’t translation, but instead how people switch languages mid-sentence and expect it to follow. In this video, Merve explores how far you can get with @SarvamAI models across the whole Pipecat pipeline: hearing, thinking and speaking in Hinglish. Sarvam models in Pipecat: docs.pipecat.ai/api-referenc… Demo from this video: github.com/pipecat-ai/sarvam…

3:58
Jon Taylor avatar
Jon Taylor@JonPTaylor·Nov 12

Pipecat 1.16 adds PerceptionReader, something I'm really excited about. How someone speaks can matter just as much as what they say, and voice agents have mostly been missing that. We’ve been testing it with Google Decision 1, released yesterday. Stream it audio and you get a pretty accurate read on the prosody, tone and sarcasm of what the user said. Try talking to it at this demo here: pipecat-google-decision-demo… We've exploring many different ways of measuring expressivity... more to share on that soon! ➡️ Pipecat Docs: docs.pipecat.ai/api-referenc… ➡️ Google Decision 1: blog.google/innovation-and-a…

2:03
Jon Taylor avatar
Jon Taylor@JonPTaylor·Nov 2

New week, new tiny model for Pipecat. Pipecat Voice Reader listens to how something is said: tone of voice, energy, laughter, speaking rate and potential sarcasm. It's a fine-tuned version of LAION’s affect-trained Whisper encoder, in two sizes (base or tiny), and runs on CPU in well under 50 ms per turn. In a Pipecat pipeline it runs beside the text tone reader, which reads what was said in the context of the previous exchange, providing broader context to the harness and metering. This is one area where a cascaded pipeline has a real edge over speech-to-speech... each turn gets explicit, calibrated readings of both the words and the delivery, with context of the single turn and the entire conversation. Available in two weights (base and tiny): huggingface.co/pipecat-ai/vo…

0:49
Jon Taylor avatar
Jon Taylor@JonPTaylor·Oct 31

Using purpose-trained classifiers that are small and fast enough to not impact the voice loop can help close the gap in naturalness between a speech-to-speech and cascade pipeline. For intent and tone though, text obviously only gets you so far. But audio models can be small too! Check out Pipecat Backchannel Detector: huggingface.co/pipecat-ai/ba… It's just 734k parameters, and does a pretty great job of figuring out when a user "mhmm"s or "oh right"s without interrupting the bot. Oh, and these backchannel events are all part of the RCS and RTVI standard we have defined in Pipecat, so supported with typed events in the client libraries too!

0:23
Jon Taylor avatar
Jon Taylor@JonPTaylor·Oct 31

We released Reader & Tagger, two tiny CPU-based classifier models that allow you to implement @pipecat_ai's Realtime Context Standard with any LLM (especially smaller ones, which are bad at following providing structured output without fine-tuning.) This demo shows parsing the tone of the user transcript, metering their mood, and steering the LLM and TTS (with style tags.) We also use it to include sounds such as chuckles or breaths, where they would fit naturally in the conversation.

3:14
Pion avatar
⟳ Jon Taylor retweeted
Pion@_pion·Oct 29

Deepgram is very lucky! The WebRTC/Voice AI ecosystem is substantially better because of the Daily team. Lots of respect for how you taught developers on how stuff really works with talks/materials.

kwindla @kwindla Oct 29 Article Deepgram is acquiring Daily and establishing an independent Pipecat Foundation I'm excited to share that the Daily team is joining Deepgram. We will continue our work on the Pipecat open source framework and on our global network infrastructure that powers realtime audio, video,

Jon Taylor avatar
Jon Taylor@JonPTaylor·Oct 23

Building on @vr000m 's amazing pipecat-local-tts-server, here is a Pipecat MacOS menubar tool for downloading and running local models for use in your agents. I'm the sort of person that runs Postgres through something like postgres.app, so I needed this. Please let me know if you'd find this useful so we can justify the time getting the app certified! (and yes, it was just an excuse to get the catmark in my toolbar)

0:18
Jon Taylor avatar
Jon Taylor@JonPTaylor·Oct 23

New Pipecat UI component: duplex timeline. I've been using this in dev for a while to visualize turns, latency, interruptions and back channeling. ui.pipecat.ai/docs/component… Our shadcn registry is "pipecat", if you're a CLI person. npx shadcn@latest registry add @/pipecat

0:11