←
Mirai Labs115 posts
Mirai Labs avatar✓

Mirai Labs ✓

@trymirai

Frontier on-device AI lab. Models, runtime & infrastructure to make on-device AI interactive, ambient & continuous.

◷ Joined January 2025▧ 41 Photos and videos
Mirai Labs avatar
Mirai Labs@trymirai·Sep 3

Today we are releasing our speculative decoding implementation in our inference engine uzu. Initially for Qwen3.6 27B, with support for Qwen3.8 27B and Muse Glimmer coming soon. On Apple M5-series chips, we outperform MTPLX (MLX + speculative decoding) by almost 2x, and llama.cpp by over 3x at comparable quantization levels, with the strongest gains achieved on mathematical reasoning and coding tasks. Run the model: Mirai-M: trymirai.com/local-models/al… Mirai-L: trymirai.com/local-models/al… Explore the benchmarks: trymirai.com/metrics Learn more about our speculative decoding implementation: trymirai.com/blog/speculativ… Our draft model, quantized checkpoint format, verification algorithm, and GPU kernels are co-designed from the ground up around the latest Apple M5 chips to take maximum advantage of GPU Neural Accelerators. Unlike popular speculative decoding architectures such as model-native MTP, which produce small draft chains of 3-4 tokens at a time, we use extremely aggressive speculative budgets of 16-32 tokens. This enables us to use Neural Accelerator-backed GEMM kernels, achieving maximum utilization of hardware arithmetic throughput.

Post attachment
Mirai Labs avatar
Mirai Labs@trymirai·Nov 5

Mirai Labs has been named to the first-ever Emergent List from @EricNewcomer and @felicis , following a nomination from @mati from ElevenLabs. We’re a frontier on-device AI lab building models, runtime, and infrastructure to make personal intelligence interactive, ambient, and continuous -with a focus on automating everyday work and life. Thank you to Mati for the nomination, to Newcomer and Felicis for the recognition, and to our team for the work that made it possible. Meet the companies on the list: newcomer.co/emergent-2026

Post attachment
:hikettei🌙 avatar
⟳ Mirai Labs retweeted
:hikettei🌙@hikettei·Nov 4

I really enjoyed making qwen3.6/3.8 go brrr with our speculative decoding research. Huge shoutout to our metal ninja for the insane work too. Can’t wait for qwen4 release😊 trymirai.com/models/alibaba-…

Post attachment
Akshay 🚀 avatar
⟳ Mirai Labs retweeted
Akshay 🚀@akshay_pachaar·Oct 30

Massive breakthrough in speculative decoding! It made a 9B LLM on my MacBook 4x faster, and 2.6x faster than regular speculative decoding. I ran Qwen3.5 9B on my M5 MacBook Pro with the same prompt, three ways: - No speculative decoding: done in 42s - Regular speculative decoding: done in 30s - DFlash + Weaver: done in 11s The video below shows all three side by side. Here's why it works: Speculative decoding lets a cheap drafter guess the next few tokens, and the main model checks them all in one pass. The fastest drafters, like DFlash, guess every position in parallel. But each guess is made on its own, so position 5 never sees what was picked at position 4. One early mismatch and everything after it gets thrown away. Weaver fixes this. It's a tiny adapter (one Transformer layer) on top of DFlash that: ↳ picks among DFlash's top candidates left to right, so each pick depends on the last ↳ grows a tree of likely continuations instead of a single chain ↳ lets the main model verify the whole tree in one pass On my Mac, every pass of the model produced 7.5 tokens instead of 1. On a B200 GPU, the paper reports 4x faster decoding across chat, math and code. @TryMirai open-sourced the whole stack in their engine, uzu, built for Apple Silicon. GitHub repo: github.com/trymirai/uzu (don't forget to star it 🌟) Trying it takes two commands: 𝗯𝗿𝗲𝘄 𝗶𝗻𝘀𝘁𝗮𝗹𝗹 𝗺𝗶𝗿𝗮𝗶 𝗺𝗶𝗿𝗮𝗶 --𝗺𝗼𝗱𝗲𝗹 𝘁𝗿𝘆𝗺𝗶𝗿𝗮𝗶/𝗤𝘄𝗲𝗻𝟯.𝟱-𝟵𝗕-𝗠 -𝗺 "𝗪𝗿𝗶𝘁𝗲 𝗮 𝗽𝘆𝘁𝗵𝗼𝗻 𝗳𝘂𝗻𝗰𝘁𝗶𝗼𝗻 𝘁𝗼 𝗺𝗲𝗿𝗴𝗲 𝘁𝘄𝗼 𝘀𝗼𝗿𝘁𝗲𝗱 𝗹𝗶𝘀𝘁𝘀" --- I wrote a full breakdown of why local inference is memory-bound and how uzu gets past it, with reproducible code. The article is quoted below. In the replies, there's an illustration of standard vs. speculative vs. uzu decoding.

0:31
Avi Chawla avatar
⟳ Mirai Labs retweeted
Avi Chawla@_avichawla·Oct 27

Big win for local LLM inference! This new inference engine runs LLMs 4x faster than llama.cpp: I ran Qwen3.5 9B on a base M5 MacBook Pro with three engines: - llama.cpp: 22.0 tokens/s - MLX: 25.1 tokens/s - uzu: 92.1 tokens/s The video below shows all three runs side by side. The first two numbers don't mean llama.cpp or MLX are badly built. But both are sitting right under a hard limit of local decoding. When an LLM generates text, every new token needs one full pass through the model, and that pass reads all of the model's weights from memory. A base M5 reads memory at about 153 GB/s. A 4-bit 9B model weighs about 5.2 GB. So at one token per pass, the ceiling is roughly 153 ÷ 5.2 ≈ 29 tokens/s, however fast the GPU computes. uzu gets past that ceiling by producing several tokens per pass instead of one: ↳ A small drafter (DFlash) proposes candidates for the next 16 positions at once. ↳ A tiny adapter (Weaver, 56.7M params) picks among them left to right, so the guesses form coherent text, and keeps a few alternative branches as a tree. ↳ The 9B model checks the whole tree in a single pass and keeps the longest path it agrees with. The main model still decides every token that ends up in the output. GitHub repo: github.com/trymirai/uzu (don't forget to star it 🌟) In my run, each pass of the 9B model produced 7.5 tokens on average. Ordinary decoding produces exactly 1. The weights and kernels are designed around this too. uzu ships its own 4-bit checkpoints, and on M5 chips it runs the verification step on the GPU's int8 matrix path. uzu is an open-source inference engine by Mirai Labs, written in Rust with Metal kernels. It has Swift, Python, TypeScript and Rust bindings, so the same engine runs inside a Mac or iOS app. I shared the GitHub repo above. If you want to dive deeper, my co-founder wrote a detailed breakdown of why local inference is memory-bound and how uzu's speculative decoding gets past it. Read it below.

0:26
Mirai Labs avatar
Mirai Labs@trymirai·Oct 22

Today we are publishing our research essay: the computer should know the way. Read it: trymirai.com/blog/the-comput… About intent, latency, and why the layer that turns what you want into actions belongs on your own Mac. We at Mirai Labs are working on Mac automation. A person expresses an intention, and the system works out how to carry it out. Many of the intentions that matter most never appear in a request log, because the designed route is too long to bother. We believe the computer should understand the intention, use all the context it has, adapt on the fly, and never interrupt the work.

Artur Chakhvadze avatar
⟳ Mirai Labs retweeted
Artur Chakhvadze@norpadon·Oct 19

We have released a new version of uzu and run the latest benchmarks on Qwen-3.8 27B we are by far the fastest at decode, and our checkpoints match Unsloth quality frontier and significantly surpass MLX ones in terms of quality github.com/trymirai/uzu

Post attachmentPost attachment
Mirai Labs avatar
Mirai Labs@trymirai·Oct 14

For context on the prefill numbers, average prompt length per benchmark: MT-Bench: 121 tokens MATH-500: 124 tokens HumanEval: 192 tokens

Mirai Labs avatar
Mirai Labs@trymirai·Oct 14

Please note that the evals used (MT-Bench, MATH-500 and HumanEval) are not very representative of the normal agentic use. We show the numbers on those specific datasets for the sake of interpretability of the results. The performance on the long context multi-turn interaction will be different.

Mirai Labs avatar
Mirai Labs@trymirai·Oct 14

882 tok/s prefill on M5 Max for Qwen3.8-27B, with uzu. Compared to Splash: prefill: 1.83x faster time to first token: 1.81x faster generation: 1.05x faster Measured on MT-Bench, MATH-500 and HumanEval.

Post attachment
Artur Chakhvadze avatar
⟳ Mirai Labs retweeted
Artur Chakhvadze@norpadon·Sep 22

We may have cooked a 2-bit version of Qwen3.8 27B which *actually works*, is not deep fried to benchmaxx, and fits in 8.5GB We have not yet finished evaluating it, and it'll take some time to properly ship it, but if someone wants to try and share feedback please dm me

Post attachmentPost attachmentPost attachment
Alexey Fateev avatar
⟳ Mirai Labs retweeted
Alexey Fateev@superalesha·Sep 26

Mirai Labs (@trymirai) compressed Qwen3.8-27B to 2.4 bits. I ported it to llama.cpp and now it runs on a 12 GB card with 128K context. With q4 KV cache the 128K window takes 11.3 GB of VRAM, with q8 KV you get 74K in 11.1 GB. Decode is 40 tok/s, still 34-35 tok/s at 62K deep, and prefill around 1000 tok/s. I measured all of it on 1 RTX 3090 at 300 W, capped to what fits in 12 GB. No requant anywhere. The GGUF carries Mirai's compressed codes bit for bit, and greedy output matches their own vLLM plugin. Everything in the video was built by this same model as an agent in pi, 11 builds from 1 prompt each, a few needed a follow up and 1 is a blooper. That run was vLLM on my 3090s. Today the same weights run in llama.cpp. Big thread, all links and recipes 👇

2:38