←
Ciru.ai - Crown 👑30,926 posts
Ciru.ai - Crown 👑 avatar

Ciru.ai - Crown 👑

@ciruai

Efficient inference, agent behavior, and rigorous evaluation AI is about the workflow, not the model. AMD Halo X Group Chat: x.ciru.ai

◷ Joined May 2009▧ 2,058 Photos and videos
Ciru.ai - Crown 👑 avatar
Ciru.ai - Crown 👑@ciruai·Nov 6

AMDots is my preview for an AMD powered local agent, still in early development. Running entirely on AMD Ryzen AI Halo 395. Before sharing the new AMDots demo, a few corrections to my October 23 video. The generated explainer had errors I should have caught before posting: The product name is AMD Ryzen AI Halo. “Gordon Halo” was incorrect. The specifications mixed two configurations. The Max+ 395 system used in that demo has 128 GB of unified memory and an NPU rated up to 50 TOPS, rather than the spoken “5 TOPS.” The newer Max+ PRO 495 Halo has 192 GB and up to 55 NPU TOPS. AMD specifications. The software name is ROCm. The narration mispronounced it and garbled several numbers, including “one hundred ninety-two gigabytes.” The output fell short of the primary-source checking I requested and missed the requested animated diagrams. The earlier voice stack used both CPU and NPU; the agent model ran on the Radeon GPU. The narration errors were problems in my software’s output and my review of it. Since then, I’ve kept building. The new demo introduces an upgraded LFM3 voice stack and CIRU’s optimized Qwen3.8-Flash-Next as the agent model: -125B parameters+51B n-gram embeddings, with 6B active per token with a 4b MTP drafter -256K native model context -CIRU’s compact weights and custom runtime optimized for AMD Strix Halo -Qwen architecture specifications · CIRU model and runtime Better voice, more capable agents, and better research, from talking through a task to following sources and producing something useful. This is what keeps me excited about the Ryzen AI Max+ 395: its unified memory and Radeon GPU give a compact desktop room for a substantial local AI workflow, with the large embedding table paged from local NVMe. I’m excited to show you how far AMDots has come. New demo below. 👇 AMD provided the Ryzen AI Halo used for this project.

12:30

We can have our own entirely local AI assistants for research and work. This is CIRU AMDots: a voice interface for a general-purpose agent team, with the AI running locally on one AMD Ryzen AI Halo. AMDots is your own little AI crew, brought to life as colorful dots. Give the main one an idea and watch its workers spring into action, exploring, researching and creating together. Keep chatting while they work, then see what they’ve made, from useful discoveries to a narrated video. The brain is MiniCPM 4.7 Ultra , CIRU HALO AGENT, my custom Q8 conversion and optimized serving stack built on OpenBMB’s vision-language MoE: 35 billion total parameters / 3 billion active per token 262,144-token model context limit Custom Ciru vLLM + AMD ROCm+DSpark On the 395: ~1,969 prompt tokens/sec and 40.5 generated tokens/sec with a roughly 64K-token prompt 122.8 aggregate generated tokens/sec across eight concurrent vision/tool requests Those are separate model-serving benchmarks; the eight-request figure is combined throughput across requests. Model, runtime and benchmark details. The voice stack uses the rest of the hardware too: GPU: conversation and worker-model inference CPU: Moonshine Small Streaming English speech recognition, 123M parameters CPU + NPU: Pocket TTS, 100M parameters, with recurrent feed-forward operations offloaded to the XDNA 2 NPU Speech synthesis stays off the GPU, leaving that compute available for the agents. Voice and workers share the language model, with live conversation taking priority. Local model inference, a visual agent workspace, and research that can become something you can watch and hear. All on a small AMD desktop. AMD provided the Ryzen AI Halo system used for this project.

11:09
Ciru.ai - Crown 👑 avatar
Ciru.ai - Crown 👑@ciruai·6h

My wife says I always get what I want somehow. Also my wife is always right. So I guess we're going to get more small Qwen4 models.

Who said we forgot about the smaller ones? 👀

Ciru.ai - Crown 👑 avatar
Ciru.ai - Crown 👑@ciruai·Nov 17

I remember the first time I tried to run a local model on Hermes/OpenClaw. It had only recently released and before then I had only used small local models for strict instruction tasks. I tried a 14b model, I was pushing it. Could barely run it. Needless to say the thing could barely do anything in the harness at all. I tried making a cron job to monitor a folder and it lied and said it did. Nothing happened. Half the time it wouldn't even respond to me because it would get lost in the harness instructions. Today I benchmarked Nanbeige4.2 3B after building an AMD tuned Q8+DSpark edition for my Ryzen AI 395. I watched it dominate Hermes. A 3B model. 85/100 on Hermes Agent 20 bench: llm.ciru.ai/research/nanbeig…

Post attachment
Ciru.ai - Crown 👑 avatar
Ciru.ai - Crown 👑@ciruai·Nov 15

This is what I call The Knowledge Gap. It's amazing to me when I watch some of my favorite tech tubers or creators who have always been good at being on the cutting edge come out with local ai advice and it blows my mind how far behind people are. Plenty on X, like @ItsmeAjayKV, and others who are always on the cutting edge. But when it comes to mainstream content like youtube there are zero content creators who seem to be on the real cutting edge of what is possible in this space. I'd say if any, @dcapitella is there. Who else are the ones releasing content that close that knowledge gap for others and why is the general, even highly technical, space still so far behind?

This is like a public announcement report. Because I still see many running Qwen3.6-35b and not upgrading to anything else cause they might be not aware. If you're still running or depending on 35b then you have to give Strata a go, run Qwen3.8-flash-next, it's better model. You're missing out on a huge upgrade !

Post attachment
Ciru.ai - Crown 👑 avatar
Ciru.ai - Crown 👑@ciruai·Nov 15

You can get two of these cheaper than a single spark. AMD is a no brainer. If your budget is 3-4k=AMD If your budget is around $8k=AMD If it's bigger= Apple M5 ultra No reason to ever buy a spark.

One thing we're learning from optimizing LLM inference on AMD Strix Halo is that generation speed really depends on what the model is doing. Our latest results with Qwen3.8-Flash-Next and DeepSeek V4.1 Flash show that pretty well. On a single @AMDRyzen Strix Halo, Qwen now reaches 122.5 tok/s in code editing. DeepSeek hits 111 tok/s in text editing across two nodes over USB4. Normal chat is slower, of course, but these workloads let us take better advantage of speculative decoding and reusing parts of the input. We're still refining both paths, and the latest improvements are waiting to be merged.

Post attachment
Ciru.ai - Crown 👑 avatar
Ciru.ai - Crown 👑@ciruai·Nov 12

I think this means my 9800x3d is dead. I will have to replace it with 9950x3d2. Won't arrive until 14th :(

Post attachment
Lotto avatar
⟳ Ciru.ai - Crown 👑 retweeted
Lotto@LottoLabs·Nov 9

Ollama should be illegal

Can I ask a dumb question? If vLLM is so much faster than Ollama, why does everyone still use Ollama?

Ciru.ai - Crown 👑 avatar
Ciru.ai - Crown 👑@ciruai·Nov 9

I'm having a horrible time with @mercury. I trusted them for my Apple M5 ultra preorders and I've been running around with support for days trying to figure out how to pay apple with my own money. Their software is ridiculous, it says I have a 20k per day limit but a 10k per month limit. So what's my real limit? I can't get an answer from them. I had a 51k limit when I ordered a couple of Ultras with it in October but apple doesn't charge until they ship, and mercury lowered my limit since then. So now what I need to know, so I can manage my payments with apple and not lose my pre-order and get sent all the way back to March is: If I convince Apple to let me charge 20k per day on my Mercury card, and pay it off every day, will it let me do this or will I be blocked by another limit? When I pay it off every day how long will it take to give me my full credit limit back to use again, because if it's not fast apple will cancel my order. If I can't get answers about how the bank works, I'll just need to move to a different bank where I don't have to worry about important payments going through.

Ciru.ai - Crown 👑 avatar
Ciru.ai - Crown 👑@ciruai·Nov 9

Interesting solution for agent native office tools. I want to see how I can use their SDK for some of my automated businesses workflows. As far as I know, there is nothing else like this. Anyone tried it out yet?

Today, we’re announcing Univer 1.4 and introducing Space. With Univer SDK, build your own spaces with file management and document, spreadsheet, and slide editing. Build an agent-native workspace. Build apps for your business. Here’s a closer look. 🧵

1:37
Ciru.ai - Crown 👑 avatar
Ciru.ai - Crown 👑@ciruai·Nov 9

This can't be true, and how is Kimi so bad on this?

Qwen-4-Flash-Early (codename hawk) added to GameFeel Bench! (and audio score was calculated wrong for qwen-4-27b, a genuine mistake that was fixed)

Post attachmentPost attachmentPost attachmentPost attachment
Ciru.ai - Crown 👑 avatar
Ciru.ai - Crown 👑@ciruai·Nov 8

Another nice padoga from my 3.8 flash next build. Never gets old.

0:32