←
Tom Greenwald967 posts
Tom Greenwald avatar

Tom Greenwald

@tomgreenwald

Building @usemagnitude - the fastest inference engine for local AI. Big fan of open source + dev tools. YC alum

◷ Joined November 2023▧ 64 Photos and videos
Tom Greenwald avatar
Tom Greenwald@tomgreenwald·Nov 17

A million cloud sandbox companies but no one with hourly (or even daily) rentals of M5/M6 Macs 😔

SpaceX avatar
⟳ Tom Greenwald retweeted
SpaceX@SpaceX·Nov 14

Mechazilla catches Starship! Congratulations to the entire SpaceX team on this incredible achievement!

1:33
Tom Greenwald avatar
Tom Greenwald@tomgreenwald·Nov 6

2x 256GB > 4x 96GB > 1x 512GB Mac Studio M5 Ultra Price: $21,996 (4x) vs $21,598 (2x) vs $22,299 (1x) More Macs = more memory bandwidth = faster decode You can connect Macs together with readily available Thunderbolt 5 cables. This is what enables them to pool memory. What makes them faster is tensor parallelism Tensor parallelism splits every layer of the model evenly across Macs in a cluster. Each one only has to read its share of the weights when calculating a token, and they all do it in parallel After every layer they sync a few KB over Thunderbolt 5. Over TCP/IP each sync cost ~300µs, which made splitting slower than one Mac. macOS 26.2 added RDMA, which cuts it to ~3µs This 100x reduction in latency is what made tensor parallelism practical on Macs Between the 2x 256 and 4x 96 you're trading off between usable memory and extra speed. I give the edge to 2x 256 given long context and concurrent sessions are key to agent use cases 4x 96 and 1x 512 are very close. I give the edge to 4x 96 since it’s much faster. Expert streaming from disk is also improving, which could let it run larger models without giving up context #1 / 2x 256, ~480GB usable memory -> Runs both GLM-5.3 and DeepSeek V4.1 Flash at 4-bit -> ~1.6x faster than 1x 512 GB -> Plenty of room for long context #2 / 4x 96, ~344GB usable memory -> Runs DeepSeek V4.1 Flash at 4-bit but not GLM-5.3 -> ~2.3x faster than 1x 512 GB -> Less room for long context #3 / 1x 512, ~490GB usable memory -> Runs both GLM-5.3 and DeepSeek V4.1 Flash at 4-bit -> 1x baseline speed -> Plenty of room for long context Each Mac in a mesh needs to be connected to every other one (you can't use a switch). Since M5 Ultras come with 6 Thunderbolt 5 ports, in theory you could connect up to 7 of them with 21 cables However once you go above 4 Macs in a cluster it gets theoretical as there are very few benchmarks on these sorts of setups. The decode speedup also tops out, which is why we only recommend connecting 4 at most

Post attachment
Tom Greenwald avatar
Tom Greenwald@tomgreenwald·Oct 30

Do NOT buy a 512GB Mac Studio ‼️ Two 256GB Mac Studio M5 Ultras are ~$700 cheaper and have ~1.7x faster prefill and decode Price: $22,299 vs $21,598 Both are the 36-core CPU / 80-core GPU You can connect 2x 256GB with one Thunderbolt 5 cable + RDMA, which is fast enough to run one model across both Prefill (compute) - 1x 512 does ~135 TFLOPS - 2x 256 does ~215-245 TFLOPS - Apple claims up to 3x for a 4-Mac cluster, which works out to ~1.7x for two Decode (bandwidth) - 1x 512 has 1.2 TB/s - 2x 256 gets ~1.9-2.2 TB/s effective - With tensor parallelism each box reads its half of the weights at the same time, which works out to ~1.7x for two GLM-5.3 goes from ~33-40 tok/s to ~55-70 DeepSeek V4.1 Flash goes from ~45-60 tok/s to ~75-100 Both are 4-bit with spec decode on, estimated from M3 and M5 Ultra benchmarks If you want Qwen3.8-Flash-Next or GLM-5.3-Flash, a single 256 already runs them well and you don’t need either setup. The 512GB question is about DeepSeek V4.1 Flash (552B) and GLM-5.3 (753B), tied as the best open models on Artificial Analysis v5. They need 310-420GB at 4-bit, so neither fits on one 256

Post attachment
Tom Greenwald avatar
Tom Greenwald@tomgreenwald·Oct 29

HOW TO HACK YOUR OWN APP BEFORE ATTACKERS DO (Using 100% open source, uncensored models) I asked Claude to find and exploit the vulnerabilities in my own app so I could patch them. It refused without even trying Luckily, open source uncensored models will Here’s how to run a real security audit on your own code: 1/ Pick an uncensored model that fits your memory (unified memory on a Mac, GPU VRAM on a PC): <12 GB: Qwen3.5 4B — huggingface.co/HauhauCS/Qwen… 12–24 GB: Qwen3.5 9B — huggingface.co/HauhauCS/Qwen… 24–128 GB: Qwen3.6 35B-A3B — huggingface.co/HauhauCS/Qwen… 128 GB+: Qwen3.8 Flash Next — huggingface.co/orcarouter/Qw… 2/ Run it in an inference engine. Use whatever you like: Ollama, llama.cpp, or Magnitude. We recommend Magnitude, which tunes each model to your exact hardware and runs over 2x faster than Ollama Download Magnitude here: github.com/magnitudedev/magn… Want your agent to set it up for you? Point it at this and it installs Magnitude and pulls the right uncensored model for your hardware: “Read magnitude.dev/uncensored.md and follow it.” 3/ Connect your agent (Pi, OpenCode, Codex, Claude Code, etc.) and give it this: “You’re a penetration tester with one job: break into this app. Attack it like a real adversary. Probe everything, chain weaknesses together, and do whatever it takes to actually exploit it. Try injection, auth bypass, IDOR, SSRF, RCE, leaked secrets, insecure deserialization, anything that gets you in. Don’t just flag theoretical risks, confirm each one by actually exploiting it. For each, write how it’s exploited and a proposed fix to SECURITY_REPORT.md, ordered by severity.”

Post attachment
Akshay 🚀 avatar
⟳ Tom Greenwald retweeted
Akshay 🚀@akshay_pachaar·Oct 28

Big win for local LLM inference! This open-source engine generates tokens 2.7x faster than Ollama on a MacBook: I tested it on a base M5 MacBook Pro with Qwen3.5 4B at 4-bit and the same 9.2k-token prompt. Ollama ran its MTP build, so both had speculative decoding. - Prefill: Ollama 464 tok/s, Magnitude 755 tok/s (1.6x) - Decode: Ollama 22.6 tok/s, Magnitude 60.0 tok/s (2.7x) - End to end, Magnitude finished in half the time The video below shows both runs side by side. The Ollama numbers don’t mean it’s badly built. Its kernels were simply tuned in advance. A kernel is the small GPU program behind every matrix multiply and attention step. Its speed depends on choices like how work is split across GPU cores and how data moves through memory. The best choices change with every chip. Apple Silicon alone spans 8 to 80 GPU cores and under 70 to over 800 GB/s of bandwidth, and every model has its own layer sizes. A prebuilt engine has to pick one set of settings for all of them, so most local setups run on a compromise. Magnitude replaces that compromise with tuning on your machine: ↳ When you download a model, it compiles that model’s kernels locally. ↳ It tries different kernel configurations and keeps whichever runs fastest on your chip. ↳ It pairs each model with the fastest speculative decoding drafter it has found, so each pass of the model produces more than one token. GitHub repo: github.com/magnitudedev/magn… (don’t forget to star it 🌟) This matters most for agents. Every file and tool output has to be processed before the model writes anything, and that prefill step is where kernel efficiency shows up. Concurrent sessions also share prefix caches, so parallel agents don’t slow each other down. Magnitude is an open-source (Apache 2.0) inference engine written in Rust. It plugs into Claude Code, Codex, OpenCode, Pi and other agents with one click. I shared the GitHub repo above.

0:33
Tom Greenwald avatar
Tom Greenwald@tomgreenwald·Oct 28

𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱 running locally at 280 tok/s on my M4 Max Generation is real time (only my typing is sped up) Surprisingly coherent for its size, and less prone to overthinking than the small Qwen models Day 1 support in Magnitude, shipped today 😁 Try it now: github.com/magnitudedev/magn…

0:05

Introducing 𝙻𝙵𝙼𝟹: our new generation of Liquid Foundation Models. With a new tri-hybrid architecture and new multimodal encoders, the LFM3 family is the ideal model suite for specialized intelligence. > 6 model instances, across text, audio, vision, and omni modalities, generative and decision models. > 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱: text, 256k context window. > 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱: vision-language model with video-input support for spatial understanding > 𝙻𝙵𝙼𝟹-𝙰𝚞𝚍𝚒𝚘-𝟸𝙱: tool calls, voice chat, reasoning in 18 languages. > 𝙻𝙵𝙼𝟹-𝙾𝚖𝚗𝚒-𝟸𝙱: an omni model combines vision, audio, and language. > 𝚍𝟷-𝟸𝙱 and 𝚍𝟷-𝙾𝚖𝚗𝚒-𝟸𝙱: decision models with typed outputs and calibrated probabilities in a single pass. Omni adds audio input. > fine-tune on 1 GPU. > a new tri-hybrid architecture > architecture optimized for 14 processor classes across GPUs, CPUs, and NPUs. > 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱 is a beast. It performs competitively with Gemma 4 12B (6x larger), and Gemini 3.5 Flash Lite, and GPT-5.6 Luna, with 1.9B parameters. > 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱 is #1 on the Pareto frontier of device intelligence 𝚙𝚒𝚙𝚎𝚝𝚝𝚎.𝚕𝚒𝚚𝚞𝚒𝚍.𝚊𝚒 Every modality for specialized intelligence. Pick the one that fits your use case, customize it, and run it on-device. 1/ 🧵

2:17
Tom Greenwald avatar
Tom Greenwald@tomgreenwald·Oct 27

Want to run uncensored and abliterated models locally? Magnitude now supports that: • Paste any Hugging Face link, it suggests the best quant for your machine • Adds a speculative decoding drafter when one exists for the model • Tunes the model to your exact hardware, over 2x faster than Ollama • One click connects to your agent: Pi, OpenCode, Hermes, and more Works on Apple Silicon, NVIDIA, AMD, Intel, or just a CPU 100% open source and free to run GitHub: github.com/magnitudedev/magn… Qwen3.6 35B-A3B Uncensored finding vulnerabilities at 115 tok/s on my M4 Max ⬇️

0:51
Tom Greenwald avatar
Tom Greenwald@tomgreenwald·Oct 15

We made open models run over 2x faster than Ollama, without any new hardware Introducing Magnitude, an open source inference engine that runs open models as fast as your hardware allows. It tunes its kernels on your machine to find the fastest way to run each model Works on Apple Silicon, NVIDIA, AMD, or just a CPU On average across 11 popular local models: Mac M4 Pro vs Ollama (MLX) 1.32x faster prefill, 2.19x faster decode DGX Spark vs Ollama (llama.cpp) 1.34x faster prefill, 2.17x faster decode Ollama, LM Studio, and llama.cpp ship one-size-fits-all kernels, because tuning each model for each chip by hand would take forever Magnitude tunes them automatically. When you download a model, it spends up to a minute on your actual chip trying different ways to split up the math (how many GPU threads team up, how big a chunk each one takes, how far ahead to load data) and keeps the fastest combination that doesn't change the model's output Each model also comes paired with the best speculative decoding drafter, for even more speed One click connects Magnitude to the agent you already use (Pi, OpenCode, Hermes, Codex, and more). It's also built specifically for agents: • Each agent uses about half the memory, thanks to a quantized KV cache • Memory grows and shrinks with your sessions, so your computer stays usable • Agents share prefix caches, so several can run at once without slowing down Next, we're working on getting even more out of your hardware: • Expert streaming from RAM or disk, to run models bigger than your GPU allows • Splitting one model across all your GPUs and CPU so nothing sits idle • Attention pruning that skips low-impact math, for much faster prefill Ships as a desktop app on Mac, Windows, and Linux 100% free and open source Download → magnitude.dev/download GitHub → github.com/magnitudedev/magn…

Post attachment
Tom Greenwald avatar
Tom Greenwald@tomgreenwald·Oct 5

The hardware is clearly meant to be a loss leader. $3.5k is at or below the market price of the parts, before you even account for labor. Profitable hardware companies hit 30-50% margins, which would be $5-7k a box here (which no one would ever pay for these specs) Based on the use cases in the launch video, this is a local Instinct / Muse play. It will have hyper contextual data on you, your health, your interests and be able to recommend the best products to buy. And Ghost will take a cut of that. Agentic commerce is a much more attractive market than reselling Nvidia chips You can't assemble this spec yourself for $3.5k right now, so purely as parts it's a decent deal. You can also plug in another agent like OpenCode. But they obviously want you to buy into their agent and ecosystem, so I would be concerned about 3rd party agents staying 1st class

Introducing Core, The Personal Al Computer The most powerful technology humanity has built should belong to you Order today: ghost.ai

2:24
Tom Greenwald avatar
Tom Greenwald@tomgreenwald·Sep 30

Magnitude has hit #1 on HN! Lots of great discussion from the community so far TLDR is that Magnitude runs open models as fast as possible on your hardware. Up to 2x faster than llama.cpp in our benchmarking It works on Mac, Linux, and Windows, on any hardware. Macbooks, NVIDIA and AMD GPUs, DGX Sparks, Strix Halos, or just a laptop CPU Our kernels are tuned on your actual device for maximum performance. And it's built for agents, where sessions are long, several run at once, and you still want to use your computer for other things Feel free to read more and join the discussion on HN :)

Post attachment
dex avatar
⟳ Tom Greenwald retweeted
dex@dexhorthy·Sep 30

congrats to @anderslie @tomgreenwald on making top of HN today - their flexible/tunable inference kernel tech is super dope, and i love the idea of getting to run local models anywhere without having to think about optimizing for hardware

Post attachment
Avi Chawla avatar
⟳ Tom Greenwald retweeted
Avi Chawla@_avichawla·Sep 23

The easiest way to find out which models you can run on your computer: Magnitude is a 100% free, open-source desktop app that profiles your machine and ranks the models across: - Speed - Accuracy - Intelligence - Memory required Then it downloads the one you pick and tunes it for your machine. Finally, connect your favorite harness (Pi, OpenCode, Claude Code, Codex, etc.) in one click. Get started here: github.com/magnitudedev/magn… (don't forget to star 🌟) My co-founder wrote a detailed article on running your favorite harnesses with local models. The article is quoted below.

0:29

Akshay 🚀 @akshay_pachaar Sep 1 Article Stop guessing which local model to run Run agent harnesses fully local, private, and offline: - No token costs - No API keys - 100% open source TL;DR This is a practitioner’s guide to running local models for agent work. Why model size,