
Career update: I’m joining @blocks for the Summer as a DL Research Intern! I’ll be working on some very cool and novel approaches to generative audio modelling, would be happy to get to know more people in this space!

Dessa pre-training @blocks. Did pure math in a past life. cashmere-y, grad student @UofT 🍁

Career update: I’m joining @blocks for the Summer as a DL Research Intern! I’ll be working on some very cool and novel approaches to generative audio modelling, would be happy to get to know more people in this space!


I'm curious how (if) u can have a good knowledge benchmark with continuation models? You judge if it creates another spkr answering the question you had or sth? is it an LLM judge? also ECAPA-tdnn is a bad model, it gets only 0.8 on real continuations, the generations in the blog are better than what it suggests

I really like their blog! Some thoughts: 9M hours at 12.5Hz taking ~180k steps => assuming bf16, they trained on approx 40-50 TB VRAM (3-4 GB200 racks). This is 13 TPP in total, or 135 TPAP. Plateau at ~3.6M hours and 31B a3B, napkin math gives ~16 active experts per layer
Introducing Interact-0, the first in a new generation of always-on models. Interact-0 is a full-duplex speech model designed around human interaction. It is the only model that knows when to speak in groups, and can be active without barging in when it shouldn’t. We share our thesis, approach, and early results below. interact.inc/blog/interact-0…

I have a feeling Muon based optimizers work much better for voice modelling, I haven’t formulated thoughts on this yet but distributing your gradient across more SV subspaces should help with vocal fidelity

I can’t imagine the beast anthropic can have in their RL environments now since they’re the only ones able to use Fable/Opus as a judge


You see, all you have to do is [REDACTED] to fix your training instabilities and you get to enjoy this:

Core Automation is literally assembling the avengers of optimization theory, John Langford is the core contributor for the Microsoft Dion repository (he also merged a PR of mine once ^-^)
John -> Core Automation coreauto.com/ After Microsoft (hunch.net/?p=13764006), I was attracted to a place which was very willing to try new things and exceptionally enabling in doing so.

introducing caramel, my attempt at training a dessa (by @jack) like conversational model purely on speech. caramel sounds very human, but can't hold a coherent conversation (yet). the model + training recipe are open source. enjoy! samples of caramel talking to itself:

You give a man a GPU, and you feed him for a day. You teach him how to kick his colleague’s jobs from the cluster and you feed him for a lifetime

They're reinventing the concept of zakah in Canada
NDP leader Avi Lewis proposes tax on wealth over $10M This money would be used to fund 1. social housing 2. public grocery stores 3. More money for Pharmacare, Dentalcare, and Medicare

As AI models deal more frequently with humans, it's increasingly important for them to have strong emotional understanding--it's a safety risk that they don't. We all imagine dystopian scenes of robots harming humans due to misunderstandings, S2S is a step towards solving that
Block's @RagavanThurair tested AI voice models by making sounds of pain. They thought he was stretching or waking up. He warns that this could become a serious safety problem as AI-powered robots gain more autonomy: "I tried to make sounds of pain. I had this experience recently where my toddler stepped on a sensitive part of my body, and I couldn't say words. All I could do is make sounds of pain. Imagine a Unitree robot steps on me the same way. I can’t say words to say, get off of me." "These models fail at that. It's like, oh, it sounds like you're stretching. Did you just wake up? It doesn't understand crying and things like that. There's a reason why we can hear emotion in voice." "I think it's actually a huge safety issue as well. These AI models are taking more and more control of things." "They're extremely smart, but if they don’t understand what we actually want or what we're trying to communicate, they can end up doing things we don't want." @blocks

Block trained its new voice AI entirely on speech, without using any text. Dessa lead @RagavanThurair says even his team didn't expect it to develop such strong language understanding: "It's trained entirely on speech, which is crazy. This company's training it just on speech, they have less data, it's a lot harder, how the hell are they gonna learn language like that? But somehow ours ended up better." "I thought we would have to have text to get to this level of conversation understanding. We didn't expect it would learn language this good." "People have done this before, like GSLM and Hertz-dev. The big gap is ours has a really strong understanding of language and you can actually have a realistic conversation with it." @blocks
Dessa is fast

I'll be on MTS live today! Tune in if you want to hear more about Dessa
MIDTERMS WRAPPED | FAL, BLOCK, CORE AUTO ON MTS x.com/i/broadcasts/2VNLMMEqe…

We're gonna spend the next twenty years laughing at how embarrassing the "tech right" was these past two years.