←
Baseten3,187 posts
Baseten avatar

Baseten

@baseten

Inference is everything.

◷ Joined March 2021▧ 766 Photos and videos
Baseten avatar
Baseten@baseten·Nov 3

Inference, training, routing, governance, and sandboxes are the building blocks of every AI product today. We’ve learned a lot from our customers about what it takes to build that system well: @OpenEvidence trains models that outperform the closed frontier, then serves them on Dedicated Inference. @datadoghq post-trains on Loops. @Opendoor routes every AI request across the company through Grid. This week we have three launches expanding our inference platform: - Grid: gateway, intelligent router, and guardrails for every model your company uses (launching tomorrow) - Sandboxes: stateful, VM-isolated environments for agents that scale to thousands of instances in seconds (launching Thursday) - Loops: serverless SFT and RL on open-weight models, deployable straight to Dedicated Inference (GA Friday) @amiruci, @saimaddali, @paul_s3i, and @oneill_c break down what we're launching. You might notice the visuals have a new look as well.

3:08
Baseten avatar
Baseten@baseten·12h

Less than 24 hours until All-Gather. Researchers from OpenAI, Microsoft AI, Thinking Machines Lab, Goodfire, NVIDIA, Cognition, Stanford, and more are all coming together to push the frontier forward. A huge thank you to everyone who applied. For those not attending: lots of content to come.

0:15
Baseten avatar
Baseten@baseten·14h

Baseten is now available on Microsoft Foundry! Teams building on Azure can now run frontier open models powered by Baseten inside Foundry, with the billing, security, and observability they already use. Existing Azure commitments can be directly applied to open models on Baseten. Read more in our announcement: baseten.co/blog/baseten-is-n…

Post attachment
Dannie Herzberg avatar
⟳ Baseten retweeted
Dannie Herzberg@DannieHerz·15h

Today we’re bringing Baseten inference to Microsoft Foundry. With Baseten on Foundry, enterprises get our inference performance alongside the security, governance, and billing they already rely on in Azure. Thank you to the Microsoft Foundry team for building this with us. baseten.co/blog/baseten-is-n…

Baseten avatar
Baseten@baseten·17h

.@get_unblocked is building the context layer for agentic software development. To scale AI code reviews grounded in each team’s code, conversations, and past decisions, it runs open models on Baseten. We help the Unblocked team achieve: - 3.1x lower cost per review - 20x lower price per token - 4x lower stall rate vs. other providers More details here: baseten.co/resources/custome…

Post attachment
Baseten avatar
Baseten@baseten·19h

Is there a best harness for each popular open model? We partnered with @runta to build FrontierHarness Eval v1.1, built by running 83B tokens across 22 harnesses and 5 open models. We found that cost per pass can vary by up to 4.4x across harnesses. If you’re running a coding agent with open models, maintain your own evals and re-test your harness on new model releases.

FrontierHarness v1.1 results: when you switch models, consider switching harnesses too, at least if you care about your token bill. The full run: 83B tokens across 27,280 agent runs, each in its own @runta runtime, inference by @baseten. Codex CLI ranks 1st on cost with GLM-5.3, and 10th of 22 with Kimi-K3. On any one model, cost per pass varies by up to 4.4x across harnesses. Only Pi never left the top 6 on cost. 🧵

Post attachment
Baseten avatar
Baseten@baseten·Nov 17

A few days ago, we announced ~1,740 TPS decode for DeepSeek V4.1 Flash on 4xB300. On Vera Rubin, it runs up to 1.7x faster. We're working on open-sourcing it right now.

Today we’re sharing our latest results in pushing low-latency performance for Deepseek v4.1 Flash. It’s running at ~2,500 tok/s on Vera Rubin, batch size 1, TP4, 1.9ms per step It’s up to 1.7x faster than our GB300 numbers, and we’re working with @sgl_project and @radixark to get it all into OSS. Here’s what @TriDao2 and @stel4th shared at the SGLang summit last week:

0:16
Phil Howes avatar
⟳ Baseten retweeted
Phil Howes@saltyph·Nov 16

Today we’re sharing our latest results in pushing low-latency performance for Deepseek v4.1 Flash. It’s running at ~2,500 tok/s on Vera Rubin, batch size 1, TP4, 1.9ms per step It’s up to 1.7x faster than our GB300 numbers, and we’re working with @sgl_project and @radixark to get it all into OSS. Here’s what @TriDao2 and @stel4th shared at the SGLang summit last week:

0:16
Ian Carrasco avatar
⟳ Baseten retweeted
Ian Carrasco@ia_n_ai·Nov 16

We've been taking our STT and TTS work at Baseten a step further and building it into a full voice agent stack. Our goal is to provide a very snappy, low latency experience and offer full model flexibility without sacrificing conversational and reasoning capabilities. Here's a quick demo of an agent I built on our voice agent stack, pairing Qwen3-ASR for speech recognition, Gemma 4 31B for reasoning, and Qwen3-TTS for speech generation (demo below). Read more about how it works here: baseten.co/blog/introducing-…

1:39
halex avatar
⟳ Baseten retweeted
halex@Halex623·Nov 16

I repeated this on Vera Rubin, to see both the limits of next-gen hardware and how well an agent would generalize to an architecture that has limited public information. Not only did it surpass our initial performance predictions, but it even found optimizations that generalized to Blackwell which made our B300 step time 10% faster! Since Opus 5.5 came out we hit an inflection point for performance engineering; with solid verification agents can hillclimb to infinity. The human input mainly came in the form of encouragement and gentle steering towards more ambitious optimizations.

We built an inference engine for DeepSeek V4.1-Flash from scratch in Rust and CUDA. 4xB300: ~1,740 TPS decode 8xB300: ~2,060 TPS decode 1M-token prefill in ~12s It's open-source.

Baseten avatar
Baseten@baseten·Nov 16

Introducing the Baseten Voice Agent Stack: optimized infrastructure to run and customize voice agents. Every model and the orchestrator are hosted for low end-to-end latency, each component scales on its own, and you can swap or post-train any model. We built Baseten Realtime Voice on it using open-weight models: 73.5% on τ²-Bench Voice, more than double the cascaded baseline, with sub-second turns at ~$0.01/min. Both are in private preview now: baseten.co/blog/introducing-…

Post attachment
Baseten avatar
Baseten@baseten·Nov 16

Not every coding task needs a frontier model. Open models meet the quality bar for much of that work at a fraction of the cost. We put together a guide on how to evaluate open models for coding agents, covering: - Choosing a pilot team and workload - Measuring quality on your own merged PRs - Tracking cache-adjusted cost per successful task - Maximizing ease of use: keep your harness + get a map from current models to open alternatives Read it here: baseten.co/resources/guide/e…

Post attachment
Baseten avatar
Baseten@baseten·Nov 13

We built an inference engine for DeepSeek V4.1-Flash from scratch in Rust and CUDA. 4xB300: ~1,740 TPS decode 8xB300: ~2,060 TPS decode 1M-token prefill in ~12s It's open-source.

we built an inference engine for deepseek v4.1-flash from scratch. the core engine is just rust + cuda. no pytorch, no sglang, no dsls, etc. ~1740 TPS decode on 4xB300, 2060TPS on 8xB300. 1M token prefill in ~12s. it's up on github

Baseten avatar
Baseten@baseten·Nov 13

Honored to be #11 on the AI 100 list alongside so many of our friends and partners. Thank you, @BessemerVP!

Post attachment
Phil Howes avatar
⟳ Baseten retweeted
Phil Howes@saltyph·Nov 13

we built an inference engine for deepseek v4.1-flash from scratch. the core engine is just rust + cuda. no pytorch, no sglang, no dsls, etc. ~1740 TPS decode on 4xB300, 2060TPS on 8xB300. 1M token prefill in ~12s. it's up on github