Qwen3.5-4B running CPU-ONLY at up to ~11+ tps on a Ryzen 5 laptop. No GPU. Running models in RAM and CPU is something the Qwen4 team is already thinking about.
💡 I built this standalone .exe because businesses are already deploying local models to save money and ensure privacy. (thumb drive friendly - just drag drop and doubleclick)
My llama.cpp recipe 👉
🧠 Qwen3.5-4B Q4_K_M GGUF
⚙️ Ryzen 5 7540U — 6C/12T
🧵 --threads 9
🧵 --threads-batch 12
⚡ --prio 2
🔄 --poll 50
📦 --batch-size 2048
📦 --ubatch-size 512
🚀 --flash-attn on
🧠 KV cache: q4_0 / q4_0
🔧 --repack
💾 --mmap
👤 --parallel 1
🚫 --device none
🚫 --gpu-layers 0
🚫 KV/op GPU offload
🚫 MTP OFF
Interesting result 👉 MTP=3 was slower (~10 tps in benchmarks).
Plain decode + 9 threads + Q4 KV hit ~11.4 tok/s.
That's about 10% faster just from tuning llama.cpp — on a basic laptop CPU.
🔢 A few days ago I asked if Tencent's new 224B Hy4-Flash could run on 1x DGX Spark. 🖥️
Well, Tencent just released the weights.
And now we have actual local performance numbers!
Hy4-Flash stats
🧠 224B total / 12B active
📚 1M maximum context
👁️ Multimodal
🔓 Open weights released TODAY
💾 GGUF quantizations already available
Look at AngelSlim's reported results at 32K context
🍎 Apple M5 Max 128GB
🍎 Q4_K_M → 44.15 tps
🍎 Q3_K → 47.37 tps
🟢 NVIDIA RTX Spark 128GB
🟢SPARK-4Bit-Fast → 33.39 tps
The Q4_K_M GGUF is approximately 168GB.
Wait. How does a 168GB model run on a 128GB machine? 🤯
ENGRAM.
The model includes approximately 63.4GB of Engram lookup tables.
Unlike ordinary transformer weights, those tables can be memory-mapped and accessed as needed rather than all residing on the GPU.
That leaves roughly 105GB of main model weights.
A 224B MoE starts looking practical on a single high-memory machine.
And the local ecosystem is already moving.
🛠️ llama.cpp support in PR #32517
💾 Five AngelSlim GGUF variants
🚀 vLLM and SGLang deployment support
🎮 Strata support reportedly coming
⚠️ The llama.cpp PR isn't merged yet, and these are the compression team's benchmarks, not independent community measurements.
Also, the published speeds are at up to 32K context, not the full 1M.
🆕New Model 👉 Earlier, I was asking if we could squeeze Tencent's 770B Hy4-Preview onto 2x DGX Sparks?
Tencent announced a 224B (Flash) model with only 12B active parameters, Hy4-Flash.
And they're planning to release the open weights. 🎉
Here are Hy4 Flash Stats ...
🧠 224B total parameters
⚡ 12B active per token
💾 Includes 64B Engram parameters
📚 1 MILLION token context
👁️ Multimodal
🚀 Over 150 tps in Tencent's optimized deployment
What's the architecture?
👉 Hybrid attention
👉 LatentMoE
👉 Engram
... and compared to Hy4 ...
💾 KV cache → 1/6 the size
...compared with Hy3
💾 KV cache → 1/24 the size
And Tencent says reinforcement learning reduced average output tokens per end-to-end task by 50%.
Sound familiar? We were just talking about similar efficiency improvements in Qwen4-Max.
API pricing per million tokens is cheap
💰 Cached input → $0.003
💰 Input → $0.070
💰 Output → $0.281
Tencent ran a human blind test with 85 WorkBuddy users across 259 everyday tasks.
Model → Satisfaction / 4 → Cost per task
🔵 DeepSeek V4.1 Flash → 3.09 → ¥1.20
🟠 Claude Haiku 5.5 High → 2.81 → ¥4.73
🔥 Hy4 Flash → 2.79 → ¥0.26
🟣 GLM-5.3 Flash → 2.53 → ¥0.81
Nearly the same satisfaction as Haiku.
🆕New Model 👉 Earlier, I was asking if we could squeeze Tencent's 770B Hy4-Preview onto 2x DGX Sparks?
Tencent just announced a 224B (Flash) model with only 12B active parameters, Hy4-Flash
And they're planning to release the open weights.
Here are Hy4 Flash Stats ...
🧠 224B total parameters
⚡ 12B active per token
💾 Includes 64B Engram parameters
📚 1 MILLION token context
👁️ Multimodal
🚀 Over 150 tps in Tencent's optimized deployment
What's the architecture?
👉 Hybrid attention
👉 LatentMoE
👉 Engram
... and compared to Hy4 ...
💾 KV cache → 1/6 the size
...compared with Hy3
💾 KV cache → 1/24 the size
And Tencent says reinforcement learning reduced average output tokens per end-to-end task by 50%.
Sound familiar? We were just talking about similar efficiency improvements in Qwen4-Max.
API pricing per million tokens is cheap
💰 Cached input → $0.003
💰 Input → $0.070
💰 Output → $0.281
Tencent ran a human blind test with 85 WorkBuddy users across 259 everyday tasks.
Model → Satisfaction / 4 → Cost per task
🔵 DeepSeek V4.1 Flash → 3.09 → ¥1.20
🟠 Claude Haiku 5.5 High → 2.81 → ¥4.73
🔥 Hy4 Flash → 2.79 → ¥0.26
🟣 GLM-5.3 Flash → 2.53 → ¥0.81
Nearly the same satisfaction as Haiku.
At ~1/18th the cost per task. 💰
Raw weight math ...
💾 8-bit → ~224GB
💾 4-bit → ~112GB
💾 3-bit → ~84GB
Hy4-Preview at 770B was ~385GB at 4-bit.
Now imagine getting Hy4 Flash onto ...
🟢 One DGX Spark 128GB
🔴 One Strix Halo 128GB
🍎 One high-memory Apple Silicon Mac
We need the actual weights, quantizations and runtime support before we know what will fit.
Tencent has officially confirmed its plans to open-source Hy4 Flash.
Hy4 Flash is available via API now.
Links in ALT
At ~1/18th the cost per task. 💰
Raw weight math ...
💾 8-bit → ~224GB
💾 4-bit → ~112GB
💾 3-bit → ~84GB
Hy4-Preview at 770B was ~385GB at 4-bit.
Now imagine getting Hy4 Flash onto ...
🟢 One DGX Spark 128GB
🔴 One Strix Halo 128GB
🍎 One high-memory Apple Silicon Mac
We need the actual weights, quantizations and runtime support before we know what will fit.
Tencent has officially confirmed its plans to open-source Hy4 Flash.
Hy4 Flash is available via API now.
Links in ALT
🏐A new model with a new approach. Another nail in the gaming studio coffin? 💀
💡 What if the next generation of AI-generated worlds isn't controlled by prompts... but by code?
@InclusionAI released Code World Model 2.0.
A very different kind of world model.
Instead of asking AI to invent both the physics and the video, CWM separates the two
🧠 CODE controls the world.
🎬 AI renders what you see. 👀
Your software defines the scene, objects, movement, camera paths and structural rules.
A diffusion model called Invoker-Render turns those instructions into photorealistic video.
Change the camera trajectory? The video follows.
Move a vehicle? The scene responds.
Change the geometry? The renderer adjusts.
And look at what the researchers demonstrated.
🎮 30 minutes of continuous generated racing video without a cut or reset.
The stats...
🧠 33.12B parameters
💾 ~132.5GB FP32 model weights
🎬 1344×768 output
⚡ 4-step causal streaming
🚀 18 FPS generation throughput on 8 GPUs
🔓 Downloadable weights + inference code
🛠️ SGLang / SGLang-Omni
The team also reports 1232 Elo on a 140-sequence video-rendering evaluation, ranking first among the tested systems.
And the inference engineering is interesting.
⚡ 9.9 → 18 FPS
The published implementation targets an eight-GPU NVIDIA setup.
And those 33B parameters are currently released in FP32.
No GGUF available or single RTX 5090 scenario yet.
The weights are public, although some components have non-commercial licensing restrictions.
🎯 Instead of game engines manually rendering every photorealistic detail, software could define the world and AI could generate the visuals.
Video Source: @TheInclusionAI
Barriers are falling. We are at the doorstep of RSI. And now the weakest link, hardware engineering, may also be falling to AI.
Case in point, AI compressed an estimated 20 weeks of semiconductor engineering into 10 days.
And it only needed 1x engineer instead of an entire front-end design team. 🧑🔬
We're watching AI move from writing software to engineering the hardware that runs it.
Check out Cognichip's Artificial Chip Intelligence (ACI).
In a semiconductor customer eval, the team started with a 55-page microarchitecture spec.
Then ACI helped automate...
🧠 Architecture exploration
🔧 RTL hardware design from scratch
🧩 Component integration
🧪 SystemVerilog testbench generation
🔍 Simulation and debugging
✅ Functional verification with >90% UVM coverage
⚡ Power, performance and area optimization
The company reports the entire front-end workflow took just 10 days with one engineer.
Norally it takes 20 weeks and a full engineering team.
Cognichip says its specialized AI models were trained on over 10 million hardware design samples.
🎯 On 1,200 open-source designs, it reports ...
📊 General-purpose AI PPA prediction error → 50–100%
📊 Cognichip specialized AI → ~10%
🤯This is astonishing.
That's pre-route prediction, not final silicon performance, but still fascinating.
Now look at the bigger picture.
🤖 Google used 93 AI agents to build a functioning experimental operating system.
⚙️ GPT-6.2 Astra just scored 76.92% on OpenAI's low-level accelerator kernel engineering benchmark.
🧠 OpenAI and Broadcom used AI during development of Jalapeño, taking an inference ASIC from initial design to tape-out in nine months.
And now specialized AI is helping automate semiconductor architecture, RTL development and verification.
We're going from AI writing applications...
👉To AI building operating systems...
👉To AI optimizing compute kernels...
👉To AI helping design the chips themselves.
Cognichip hasn't demonstrated producing a complete, manufactured chip in 10 days. Physical design, tape-out and silicon validation still come afterward.
What happens when AI dramatically reduces the cost and time needed to design custom silicon?
Source: Cognichip
🍎 New model. Apple released a 14B-parameter open-weight video generation model designed for low-latency interaction.
And the weird thing? 👉 It runs on NVIDIA CUDA, not Apple Silicon.
Here are FastFrames-L stats ...
🧠 14B parameters
🎬 832×480 video
⚡ Just 2 diffusion steps per latent frame
🔓 Downloadable weights + source code
💾 30.6GB final checkpoint
🖥️ Linux + NVIDIA CUDA 12.8
It's distilled from Wan2.1-14B.
🍎 Apple also released FastFrames-S, a smaller 1.3B model.
And look at Apple's published performance on NVIDIA B200 GPUs.
⚡ FastFrames-S (1 GPU): 36ms per latent frame
⚡ FastFrames-L (1 GPU): 158ms
⚡ FastFrames-L (8 GPUs): 45ms
Each generated latent frame produces 4 RGB video frames.
So what about running the 14B model on your RTX 5090?
The checkpoint is 30.6GB, and the 5090 has 32GB VRAM.
However, this fast video gen is implemented by loading a 11.4GB Wan text encoder onto the GPU, plus the model needs memory for activations and its KV cache.
So the stock implementation isn't a straightforward fit.
Could a Localmaxxer get it running on one RTX 5090 with text-encoder offloading or quantization?
That's the challenge!
And when does somebody port it to MLX?
Apple just released the weights and code. Let's see what the local AI community can do with them.
Video: @Apple
🆕 New Local text diffusion models from NVIDIA released! ⏩
🎉Nemotron-Labs-Sigma from NVIDIA is now available.
Wow, a totally new family of open-weight AI models that don't generate text one token at a time.
And Localmaxxers can download and run them on their own GPUs (can't wait for optimizations)
These models generate text by refining entire blocks of tokens in parallel. That's a fundamentally different approach to LLM inference.
Here are the models and stats ...
🧠 Sigma-3B → 3 billion parameters
🧠 Sigma-8B → 8 billion parameters
💬 Base + instruction-tuned versions
⚡ Few-step distilled variants
🔮 Self-speculative variants
📚 8,192-token context (8B instruct)
🔓 Open weights under NVIDIA's Nemotron license
🏠 Offline inference on your own NVIDIA GPU
Here are the performance numbers....
On a single NVIDIA H100, the 8B model achieves approximately:
⚡ 120–140 tps
Using BF16, 32 denoising steps and 64-token blocks.
The inference implementation uses PyTorch and Transformers, without requiring custom CUDA kernels.
The models can run completely offline, although NVIDIA currently requires a CUDA GPU with BF16 support. CPU inference isn't supported.
⚠️ The 8B model has only an 8K context window, and we still need independent benchmarks against comparable Qwen and Gemma models on consumer GPUs.
NVIDIA supports Ampere, Lovelace and Blackwell GPUs with BF16. That includes consumer RTX hardware, although sufficient VRAM is necessary.
NVIDIA has put 3B and 8B models in the hands of developers to experiment with locally.
Now I want to see what these can do on an RTX 3090, 4090 or 5090. 👀
Models and technical details in replies
📰 The token market is changing! 🆕 New data reveals a massive shift toward open-weight AI on Databricks-hosted models.
Open-weight AI went from 7.3% to 30.9% of enterprise AI token usage in 4 months. 📈
Also, that's a 4.2× increase.
And the data comes from more than 10,000 organizations using Databricks' AI gateway.
Databricks released its new AI Index, and the numbers tell a story about the future of open-weight AI the Local AI community already knows.
Look at the shift in token usage ...
📊 July → 7.3% open-weight
📊 November → 30.9% open-weight
🚀 Growth → 4.2×
And it's not just adoption.
Databricks also evaluated AI coding agents, where two open-weight models delivered impressive results ...
🟢 DeepSeek V4.1 → 93.1% success
🟢 GLM-5.3 → 93.1% success
Both tied for third place in its evaluation.
Why does this matter?
💰 Because economics as open weights give enterprises more choices for controlling inference costs.
🔓 Because control gives organizations choice on where and how to deploy their models.
🧠 Because capability as open-weight models are increasingly competitive in real enterprise workloads.
⚠️These numbers include Databricks hosted open-weight models.
But the bigger picture is becoming difficult to ignore.
Open-weight AI isn't just a hobbyist movement anymore.
It's becoming a serious part of enterprise AI infrastructure.
Source: Databricks AI Index