←
kwindla6,832 posts
kwindla avatar✓

kwindla ✓

@kwindla

Infrastructure and developer tools for real-time voice, video, and AI. @trydaily // ᓚᘏᗢ // @pipecat_ai

◷ Joined September 2008▧ 1,437 Photos and videos
kwindla avatar
kwindla@kwindla·18h

Generative AI models are going to run everywhere, not just in the data center. Now we just need to feed the printouts back into the document scanner and use Gemma’s vision capabilities so the model can write notes to itself. Paper tape a-machines thought experiment in 1936 … [exactly 90 years of computing progress with vacuum tubes, relays, and silicon] … back to paper. 😁

Post attachment
kwindla avatar
kwindla@kwindla·Nov 13

Something like 350 million people regularly speak Hindi with lots of English words and phrases mixed in. "Hinglish" might be the most widely spoken language that, unless you live in India, you haven't heard of. But if you do live in India, and you're building voice AI agents for use cases like customer support, the need for AI models that are very good at Hinglish is going to be very much on your mind. Merve made an explainer video about the Sarvam models: LLM, TTS, STT, and decision models trained in 10 Indian languages, and uniquely good at mixed-language understanding, thinking, and speech.

3:58
kwindla avatar
kwindla@kwindla·Nov 13

Big models that "do everything" are truly amazing. We live in extraordinary times. But agents that wrap big models are fundamentally hard to debug, observe, maintain, extend, etc. This is currently an unsolved problem. Possible paths forward include: - Tooling that helps developers with context engineering, observability, and evals. Lots of people are doing good work in these areas. But we are very early. - New kinds of tooling that allows us to understand why deep learning models do what they do. Broadly and loosely, this is "mechanistic interpretability". Lots of good academic papers; but so far a completely unsolved problem. - Use small(-er) models that we can individually test, observe the inputs and outputs of, and fine-tune. We can do this today.

kwindla avatar
kwindla@kwindla·Nov 13

[ System One | Decision | Classifier ] models are the hot new thing. Pipecat 1.12 added a Classifiers abstraction. Since that release, we've been building a *lot* of things on top of that abstraction. Google released Decision 1 yesterday, a classifier model that can take audio input. Check out Jon's example (code and a video) below, which uses Google Decision 1. Jon's PerceptionReader class is in Pipecat 1.16 (also released yesterday). PerceptionReader makes it easy to create a voice agent that understands how a user is speaking: positive, sad, frustrated, etc.

2:03
kwindla avatar
kwindla@kwindla·Nov 12

One of the really fun things about this generative AI moment is how much innovation and interesting work is happening all over the world. Wherever I travel to, these days, there's someone on the DeepMind team to hang out with and talk about models, inference, APIs, and hardware. And when they come to SF they throw a great party!

Post attachment
kwindla avatar
kwindla@kwindla·Nov 9

Also, I highly recommend VapiCon. Last year, it seemed like the whole Voice AI world showed up in San Francisco for the conference: vapicon.ai/

kwindla avatar
kwindla@kwindla·Nov 9

I sat down with @tarush_agarwal_, CEO of @cekuraAi, to talk about his company's recent work on benchmarks. Tarush will be at VapiCon on Wednesday and Thursday this week in San Francisco. So if you're interested in benchmarks for voice AI, and will be at VapiCon, find him and say hi. Cekura has spent the last two years building simulations and evals for teams that deploy voice AI agents at scale. We talked about why Cekura is putting a lot of effort into building general benchmarks for public release, a little about their "secret sauce," what's hard about building good benchmarks, and what they're working on next. 0:00 Intro 0:29 Tarush and Cekura 0:47 Benchmarks and VapiCon 1:40 The Cekura real-world voice AI benchmark 3:51 Evals loops and how you build a multi-turn benchmark 4:51 Making simulations adaptive 6:33 Different approaches to benchmarks 7:23 Testing speech-to-speech models 8:58 How to use published benchmarks 10:58 What's next: benchmarking naturalness 12:29 Why naturalness is the next phase of voice AI

13:35
kwindla avatar
kwindla@kwindla·Nov 9

Going to the Tavus office and seeing the behind the scenes development work on the kiosk stuff is really fun. Realtime AI is a lot more than just voice and video generation (though voice and video are obviously central to the experience). Object and face tracking, directional audio signals, broader context about what's happening in the environment, richer context from integration into backend systems, the personality and tone of the agent ... all of these play a big role in making an AI agent useful and natural. Last time I visited Tavus I was there for a meeting with someone named Quinn. My name is also Kwin, same pronunciation, different spelling. I told the Tavus PAL receptionist at the office front door why I was visiting, and she said: "I'll let Quinn know you're here. Feel free to sit in the red chairs behind me. By the way, two Quinns is a lot of Quinns in one day." I couldn't disagree.

1:08
kwindla avatar
kwindla@kwindla·Nov 5

A big benchmarks release from @cekuraAi today. Good benchmarks are very difficulat develop for a bunch of reasons: - Automated testing of complex, multi-turn, conversational systems is a fundamentally hard problem. Do you use an LLM to simulate a user? That's the most common approach, these days. But how do you design things so that your benchmark isn't mostly just testing how good your sim is? (The opposite of what you actually want!) - What are you testing? Task success? On what tasks? What is success? For example, if an agent gets a buch of things wrong along the way, but an infinitely patient simulator sticks around long enough, you can probably register success on a task. On the other hand, self-correcting during a conversation is a good thing, so you want to allow some flexibility in the "path" the agent takes. Individual turn-by-turn performance? (How do you judge a good turn vs a bad turn response. There's a lot of grey area.) Specific tool calling or instruction following metrics? What about latency, interruption handling, and subjective voice quality? - There's a very large surface area for benchmark bugs. The Cekura team has been thinking hard about how to build good benchmarks.

1:23
kwindla avatar
kwindla@kwindla·Nov 3

We've gotten used to crazy progress every month in the world of frontier LLMs. And more recently, mid-sized, open weights LLMs (~20B - 200B parameters) hit that same fast-progress, upwards curve. But the new, new thing is super fast progress in small, specialized models. @JonPTaylor is training new models every week, now, for things like detecting whether speech is sarcastic. These models can take text input, audio input, or both. They can be small enough to run on CPU in <50ms. You can combine them with LLMs, decision models like Jev, other classifier models, etc. Specialized model training/fine-tuning is one of the things that coding agents make it possible to do 100x faster. So go make your consumer GPU go brrrrrrrr and train some tiny models.

0:49
kwindla avatar
kwindla@kwindla·Oct 31

specialized models 🤝 orchestration-layer standards

0:23
kwindla avatar
kwindla@kwindla·Oct 28

Come hang out on November 9th with a bunch of people building audio models and realtime AI systems!

kwindla avatar
kwindla@kwindla·Oct 26

Come hang out with Corey (and me) at Deepgram Speak on Thursday! There will be lots of good content and hallway conversations. If you try to register and there's not space, ping me here and I'll see if I can get you in.