
we built an inference engine for deepseek v4.1-flash from scratch. the core engine is just rust + cuda. no pytorch, no sglang, no dsls, etc. ~1740 TPS decode on 4xB300, 2060TPS on 8xB300. 1M token prefill in ~12s. it's up on github

Accelerating LLMs @Basetenco - long-context and embedding inference (github.com/michaelfeil/infin…) - opinions are my own.

we built an inference engine for deepseek v4.1-flash from scratch. the core engine is just rust + cuda. no pytorch, no sglang, no dsls, etc. ~1740 TPS decode on 4xB300, 2060TPS on 8xB300. 1M token prefill in ~12s. it's up on github

Baseten runs 9B and 27B - our secrets: - we trained the models to only need a single inference pass per state, and not one prefill-step per question (e.g. perplexity flash v1.2 can be 30x more expensive) - we politely asked codex to make the inference faster, make no mistakes. Worked quite well, acoording to AA. artificialanalysis.ai/articl…

~1000 TPS on deepseek flash love us some model driven model infra
many models are now orders of magnitude better than humans at kernel work. e.g. an early checkpoint of kimi k3 handled the majority of kernel optimization for its later development. profiling, rewriting, benchmarking is well-defined and verifiable, which makes it the ideal task for an agent to hill-climb. similarly, we used our internal harness to optimize 80+ kernels and bottlenecks for deepseek v4.1 flash 1017 to create the fast endpoint. it is on artificial analysis at 991 output tokens/s and lowest time per task (pareto frontier). if you want to understand how to make flash models even faster, I explain our optimizations, covering kernel fusion, programmatic dependent launch, and host loop minimization here:

many models are now orders of magnitude better than humans at kernel work. e.g. an early checkpoint of kimi k3 handled the majority of kernel optimization for its later development. profiling, rewriting, benchmarking is well-defined and verifiable, which makes it the ideal task for an agent to hill-climb. similarly, we used our internal harness to optimize 80+ kernels and bottlenecks for deepseek v4.1 flash 1017 to create the fast endpoint. it is on artificial analysis at 991 output tokens/s and lowest time per task (pareto frontier). if you want to understand how to make flash models even faster, I explain our optimizations, covering kernel fusion, programmatic dependent launch, and host loop minimization here:
Alex Ker 🔭 @thealexker Nov 4 Article How to make flash models even faster: optimizing DeepSeek V4.1 Flash 1017 Flash models now dominate the low-cost section of the Pareto frontier. These mid-sized models deliver fast, affordable intelligence suitable for most development work. For example, DeepSeek V4.1 Flash


Baseten is offering joey, the world's fastest model: - at JEV 1.14 quality - baseten.co/blog/decisions-fo… - 0.01$ price / M

d) baselabs/elephant now running at ~50ms latency on any datacenter near you.
our decision models are: a) frontier b) free for the week on @baseten Model APIs c) completely open-source

Nemotron-3.5-Embed is now available in turbopuffer native embedding on our enterprise PDF search benchmark, Nemotron-3.5-Embed 8B improves recall@20 by ~5% over the next best model turbopuffer.com/docs/embeddi…
We’re releasing more open models to help teams across Europe build AI with their own data, on infrastructure they control. Announced at GTC Berlin, the models help developers, researchers and enterprises build specialized AI agents, train robots, design proteins and develop quantum computers. See what’s new: nvda.ws/4iwBMvF

Today we’re releasing two decision models, elephant-1 (9B) and mammoth-1 (27B), trained by the @baselabs team, available for free for one week on our Model APIs. mammoth-1 is #1 on the Jev Decision Index and elephant-1 is tied for #1 on the Artificial Analysis Decision Index. It’s a new model, but more importantly, this is an open-source model you can access, deploy, and post-train on Baseten (or wherever you do your best work). The free week ends 10/27/26 at 11:59 PM PT. Try it and let us know what you think. Try mammoth-1 here: baseten.co/library/mammoth-1… Try elephant-1 here: baseten.co/library/elephant-…

We trained the #1 decision models in the world, mammoth-1 and elephant-1! This is our first Base Labs model release and we're excited to open-source all weights and training details. We really focused on scaling high-quality data in these training runs – not only synthetically generating new decision data but stacking how we validated our data. This follows something called our 'swiss cheese' approach to looping over and improving training data. There are also a bunch more low hanging fruit when it comes to training these models, and we expect that we'll improve quickly over the coming weeks.
Today we release mammoth-1 and elephant-1! We post-trained mammoth-1, the #1 ranked best decision model in the world, and elephant-1, which is the best model under 15B (and beats Jev). We're open-sourcing all model weights and provide a technical report detailing all our training and data methods.



While working on GLM5.2 chat-templating for Basetenkenizer, found a small bug, thats now fixed with latest minijinja 2.22 (x.com/feilsystem/status/2081…) and will render the chat-template correctly. Will be included in future releases. github.com/mitsuhiko/minijin…
We shipped the worlds fastest tokenizer for KimiK3. Basetenkenizer now available on day 0 on Baseten and on pypi.


This is now supported in the dynamo-tokenizers crate. Great work from Michael and the rest of the B10 team!
We shipped the worlds fastest tokenizer for KimiK3. Basetenkenizer now available on day 0 on Baseten and on pypi.



We shipped the worlds fastest tokenizer for KimiK3. Basetenkenizer now available on day 0 on Baseten and on pypi.

