←
Anders Lie840 posts
Anders Lie avatar

Anders Lie

@anderslie

building fast local inference @usemagnitude

◷ Joined April 2014▧ 83 Photos and videos
Anders Lie avatar
Anders Lie@anderslie·Nov 15

Claude kept hitting safeguards working on inference-related work (didn't see this happen often until recently) So now I have Gemini Carbon orchestrating work and talking to Claude Desktop, and moves work to Gemini when work gets flagged.. the things ant makes us do man

not a fan of this new safeguard category

Post attachment
Anders Lie avatar
Anders Lie@anderslie·Oct 27

Got tired of worktrees, model weights, and cargo builds filling disk -> threw together a disk cleaner with actions specific to those artifacts. Personal software feels good

Post attachment
Tom Greenwald avatar
⟳ Anders Lie retweeted
Tom Greenwald@tomgreenwald·Oct 27

Want to run uncensored and abliterated models locally? Magnitude now supports that: • Paste any Hugging Face link, it suggests the best quant for your machine • Adds a speculative decoding drafter when one exists for the model • Tunes the model to your exact hardware, over 2x faster than Ollama • One click connects to your agent: Pi, OpenCode, Hermes, and more Works on Apple Silicon, NVIDIA, AMD, Intel, or just a CPU 100% open source and free to run GitHub: github.com/magnitudedev/magn… Qwen3.6 35B-A3B Uncensored finding vulnerabilities at 115 tok/s on my M4 Max ⬇️

0:51
Anders Lie avatar
Anders Lie@anderslie·Oct 15

Many long nights went into making these kernels as fast as possible across different hardware. A lot of cool techniques under the hood that make this work: - Autotuning system that fits each kernel to specific hardware -> fast on any hardware - Using hardware acceleration to the fullest extent where available (e.g. M5 tensor cores) -> faster everything - Decode kernels that unpack quantized weights just-in-time to reduce bandwidth needs -> faster decode - Quantized KV cache with fast decode kernels that compute with quantized values in-place -> less memory, faster decode - Operation packing that fits two 8-bit multiplies inside one 32-bit multiply on Macs that don't have tensor cores -> faster prefill - Filter unwinnable token candidates during unembedding by using top 4 bits as a bound -> faster decode ...and many more. This all adds up to making Magnitude the fastest engine for local AI across all models and hardware we support. And yet even more ideas for performance are in the pipeline, as well as ways to run bigger models on less hardware. Stay tuned!

We made open models run over 2x faster than Ollama, without any new hardware Introducing Magnitude, an open source inference engine that runs open models as fast as your hardware allows. It tunes its kernels on your machine to find the fastest way to run each model Works on Apple Silicon, NVIDIA, AMD, or just a CPU On average across 11 popular local models: Mac M4 Pro vs Ollama (MLX) 1.32x faster prefill, 2.19x faster decode DGX Spark vs Ollama (llama.cpp) 1.34x faster prefill, 2.17x faster decode Ollama, LM Studio, and llama.cpp ship one-size-fits-all kernels, because tuning each model for each chip by hand would take forever Magnitude tunes them automatically. When you download a model, it spends up to a minute on your actual chip trying different ways to split up the math (how many GPU threads team up, how big a chunk each one takes, how far ahead to load data) and keeps the fastest combination that doesn't change the model's output Each model also comes paired with the best speculative decoding drafter, for even more speed One click connects Magnitude to the agent you already use (Pi, OpenCode, Hermes, Codex, and more). It's also built specifically for agents: • Each agent uses about half the memory, thanks to a quantized KV cache • Memory grows and shrinks with your sessions, so your computer stays usable • Agents share prefix caches, so several can run at once without slowing down Next, we're working on getting even more out of your hardware: • Expert streaming from RAM or disk, to run models bigger than your GPU allows • Splitting one model across all your GPUs and CPU so nothing sits idle • Attention pruning that skips low-impact math, for much faster prefill Ships as a desktop app on Mac, Windows, and Linux 100% free and open source Download → magnitude.dev/download GitHub → github.com/magnitudedev/magn…

Post attachment
Anders Lie avatar
Anders Lie@anderslie·Oct 3

Cool trick for faster decode: read half the bytes during unembedding. How? Store unembedding weights in "planes" so you can read a 4-bit approximation first. Use this approximation to filter for the highest logits. Store the 5th bit in another plane, use it to eliminate more candidates. Use the last 3 bits for the final 8-bit precision logits. Even though a 4-bit representation was used to filter the logits, it's lossless because it only drops tokens that provably can't win. Fewer bytes read here generally corresponds to faster decode (tested on an M4 Pro with Qwen 3.6 35B A3B, +6.5% tok/s).

0:14
Tom Greenwald avatar
⟳ Anders Lie retweeted
Tom Greenwald@tomgreenwald·Sep 30

Magnitude has hit #1 on HN! Lots of great discussion from the community so far TLDR is that Magnitude runs open models as fast as possible on your hardware. Up to 2x faster than llama.cpp in our benchmarking It works on Mac, Linux, and Windows, on any hardware. Macbooks, NVIDIA and AMD GPUs, DGX Sparks, Strix Halos, or just a laptop CPU Our kernels are tuned on your actual device for maximum performance. And it's built for agents, where sessions are long, several run at once, and you still want to use your computer for other things Feel free to read more and join the discussion on HN :)

Post attachment
Anders Lie avatar
Anders Lie@anderslie·Sep 28

The latest generation of models is good at designing abstractions if you can point out when its needed. Previous generation needed more help designing the abstractions. Now if you can smell the bad code, you can point it out and give general guidance to redesign. Detecting these smells doesn't necessarily involve reading the code, and if the criteria for the abstraction is clear enough, it's implementation doesn't necessarily need to be read. As a result I've read much less code recently. However, no model is good at consistently identifying bad abstractions or the lack of abstractions. Maybe that's next.

Avi Chawla avatar
⟳ Anders Lie retweeted
Avi Chawla@_avichawla·Sep 23

The easiest way to find out which models you can run on your computer: Magnitude is a 100% free, open-source desktop app that profiles your machine and ranks the models across: - Speed - Accuracy - Intelligence - Memory required Then it downloads the one you pick and tunes it for your machine. Finally, connect your favorite harness (Pi, OpenCode, Claude Code, Codex, etc.) in one click. Get started here: github.com/magnitudedev/magn… (don't forget to star 🌟) My co-founder wrote a detailed article on running your favorite harnesses with local models. The article is quoted below.

0:29

Akshay 🚀 @akshay_pachaar Sep 1 Article Stop guessing which local model to run Run agent harnesses fully local, private, and offline: - No token costs - No API keys - 100% open source TL;DR This is a practitioner’s guide to running local models for agent work. Why model size,

Anders Lie avatar
Anders Lie@anderslie·Sep 21

I've been running long-horizon agents on verifiable tasks all week (kernel optimization), and found that frequent steering still multiplies their productive output. The value of work performed by agents feels logarithmic, even with a clear goal, until you check in and steer again. The agent's work pattern often "converges" after a couple hours. Usually this is not what you want. Example: it gets caught up solving small issues and recompiling, not realizing a wider architectural fix would eliminate the class of issues completely. Taking bigger leaps or identifying wider refactors often needs an explicit push. Identifying that and re-steering resets the curve. This has accidentally trained me into a polyphasic sleep schedule, so that I can re-steer long running work when it's most valuable after 2-3 hr periods.. Wonder if anyone else has had a different experience with these kinds of tasks

Post attachment
Anders Lie avatar
Anders Lie@anderslie·Sep 18

New Magnitude release! - Brand new desktop experience - Added Windows support - Easily connect to other harnesses - Track your local model usage over time Next up: Full inference engine rewrite - the fastest and most memory-efficient local models on Metal/CUDA/AMD

Which models can your machine actually run? Magnitude is a 100% free, open source desktop app that: - profiles your hardware and runs sample calculations - predicts tok/s for every model before you download - recommends the best models, from fast to smart Pick your models and it handles the rest: - downloads and tunes the models for your hardware - connects agents like pi, opencode, hermes in one click - runs models on demand as your agent works Works on whatever hardware you already own: MacBook (Intel or Apple Silicon), Mac Mini, Mac Studio, DGX Spark, Strix Halo, any NVIDIA/AMD GPU, or just a CPU Download it on macOS, Windows, or Linux: magnitude.dev Open source: github.com/magnitudedev/magn…

0:33