←
Jon Taylor1,488 posts
Jon Taylor avatar✓

Jon Taylor ✓

@JonPTaylor

🧑‍💻 Voice AI @trydaily // ᓚᘏᗢ @pipecat_ai 🎹 I have too many synths

◷ Joined July 2008▧ 245 Photos and videos
Jon Taylor avatar
Jon Taylor@JonPTaylor·22h

We often get asked whether it's possible to build an AI experience with multiple users and one (or more) agents in the loop. This is where good diarization and a fast classifier model come in clutch. The only real prerequisites these days are good orchestration (Pipecat!), good STT (Deepgram!) and a transport and infrastructure that support multiple tracks (Daily!). Here's a one-shot Claude demo. The agent joins a meeting, and the conversation passes through Jev, which decides whether the agent is being addressed. If it isn't sure, it waits 2.5 seconds, and if no one else answers, it offers a response. All of these thresholds can be tuned. Clearly, I haven’t tuned them in this video 😅 For some use-cases, you might prefer a more natural speaking companion that is chatty and joins the conversation, for others (like this demo) you might want something that stays quiet unless spoken to. Jev adds ~100ms of latency to each turn, which is fine. You could also use a tiny classifier like Pipecat Audience (huggingface.co/pipecat-ai/au…) to make the decision on CPU in as little as 15ms.

3:39
Jon Taylor avatar
Jon Taylor@JonPTaylor·Nov 16

Getting a voice agent to read a user's emotion used to be a pipe(cat) dream. Today’s multimodal decision models, like Google Decision 1, make it easy. But they’re large general-purpose models that read tone as one task among many. A bit overkill, and it can add 100–200ms to each turn 😨 This is why we built Pipecat Voice Reader, a tiny open-weights audio model that runs on CPU and streams results in as little as 25ms. Here it is head-to-head with Decision 1. It holds its own on accuracy, which is crazy for a 28MB int-8 model (or 14mb for the tiny version.) Pair it with Pipecat Tone Reader, which reads the transcript, and you get a much cleaner signal on sarcasm vs. sincerity. Demo code is available here: github.com/pipecat-ai/pipeca…

2:07
kwindla avatar
⟳ Jon Taylor retweeted
kwindla@kwindla·Nov 13

Something like 350 million people regularly speak Hindi with lots of English words and phrases mixed in. "Hinglish" might be the most widely spoken language that, unless you live in India, you haven't heard of. But if you do live in India, and you're building voice AI agents for use cases like customer support, the need for AI models that are very good at Hinglish is going to be very much on your mind. Merve made an explainer video about the Sarvam models: LLM, TTS, STT, and decision models trained in 10 Indian languages, and uniquely good at mixed-language understanding, thinking, and speech.

3:58
Jon Taylor avatar
Jon Taylor@JonPTaylor·Nov 12

Pipecat 1.16 adds PerceptionReader, something I'm really excited about. How someone speaks can matter just as much as what they say, and voice agents have mostly been missing that. We’ve been testing it with Google Decision 1, released yesterday. Stream it audio and you get a pretty accurate read on the prosody, tone and sarcasm of what the user said. Try talking to it at this demo here: pipecat-google-decision-demo… We've exploring many different ways of measuring expressivity... more to share on that soon! ➡️ Pipecat Docs: docs.pipecat.ai/api-referenc… ➡️ Google Decision 1: blog.google/innovation-and-a…

2:03
Jon Taylor avatar
Jon Taylor@JonPTaylor·Nov 2

New week, new tiny model for Pipecat. Pipecat Voice Reader listens to how something is said: tone of voice, energy, laughter, speaking rate and potential sarcasm. It's a fine-tuned version of LAION’s affect-trained Whisper encoder, in two sizes (base or tiny), and runs on CPU in well under 50 ms per turn. In a Pipecat pipeline it runs beside the text tone reader, which reads what was said in the context of the previous exchange, providing broader context to the harness and metering. This is one area where a cascaded pipeline has a real edge over speech-to-speech... each turn gets explicit, calibrated readings of both the words and the delivery, with context of the single turn and the entire conversation. Available in two weights (base and tiny): huggingface.co/pipecat-ai/vo…

0:49
Jon Taylor avatar
Jon Taylor@JonPTaylor·Oct 31

Using purpose-trained classifiers that are small and fast enough to not impact the voice loop can help close the gap in naturalness between a speech-to-speech and cascade pipeline. For intent and tone though, text obviously only gets you so far. But audio models can be small too! Check out Pipecat Backchannel Detector: huggingface.co/pipecat-ai/ba… It's just 734k parameters, and does a pretty great job of figuring out when a user "mhmm"s or "oh right"s without interrupting the bot. Oh, and these backchannel events are all part of the RCS and RTVI standard we have defined in Pipecat, so supported with typed events in the client libraries too!

0:23
Jon Taylor avatar
Jon Taylor@JonPTaylor·Oct 31

We released Reader & Tagger, two tiny CPU-based classifier models that allow you to implement @pipecat_ai's Realtime Context Standard with any LLM (especially smaller ones, which are bad at following providing structured output without fine-tuning.) This demo shows parsing the tone of the user transcript, metering their mood, and steering the LLM and TTS (with style tags.) We also use it to include sounds such as chuckles or breaths, where they would fit naturally in the conversation.

3:14
Pion avatar
⟳ Jon Taylor retweeted
Pion@_pion·Oct 29

Deepgram is very lucky! The WebRTC/Voice AI ecosystem is substantially better because of the Daily team. Lots of respect for how you taught developers on how stuff really works with talks/materials.

Jon Taylor avatar
Jon Taylor@JonPTaylor·Oct 23

Building on @vr000m 's amazing pipecat-local-tts-server, here is a Pipecat MacOS menubar tool for downloading and running local models for use in your agents. I'm the sort of person that runs Postgres through something like postgres.app, so I needed this. Please let me know if you'd find this useful so we can justify the time getting the app certified! (and yes, it was just an excuse to get the catmark in my toolbar)

0:18
Jon Taylor avatar
Jon Taylor@JonPTaylor·Oct 23

New Pipecat UI component: duplex timeline. I've been using this in dev for a while to visualize turns, latency, interruptions and back channeling. ui.pipecat.ai/docs/component… Our shadcn registry is "pipecat", if you're a CLI person. npx shadcn@latest registry add @/pipecat

0:11
Jon Taylor avatar
Jon Taylor@JonPTaylor·Oct 20

Phonon from @GradiumAI is an open TTS model in 31M and 40M sizes, perfect for on-device! It’s voice promptable too, via Gradium’s Voice Designer. I love experimenting with voice and mobile. Here is Little Voices, an app where you can ask the angel and devil on your shoulder anything. It's a @pipecat_ai app in embedded Python on iPhone. • Apple SpeechAnalyzer for speech-to-text • Apple Foundation Models for the LLM • Gradium Phonon 31M for TTS. • Pipecat running in Python on the iPhone • Pipecat Audience (for who speaks next. It works out whether you’re talking to the angel or the devil.) • No network or inference costs! 📖 Repo here: github.com/pipecat-ai/ios-li…

3:16