LLM inference is too slow, too expensive, and too hard to scale.
🚨 Introducing llm-d, a Kubernetes-native distributed inference framework, to change that—using vLLM (@vllm_project), smart scheduling, and disaggregated compute.
Here’s how it works—and how you can use it today:
Muse-Glimmer 30B, now in FP8-block. Our quantized checkpoint of the multimodal model, at roughly half the memory and disk.
Block-wise FP8 on the linear layers, vision tower kept in full precision. Quantized with LLM Compressor, serve on @vllm_project.
Evals coming shortly.
huggingface.co/RedHatAI/Muse…
Today we're also opening the weights for Muse Glimmer, a great 30B parameter dense model that can run locally. Soon we'll also release the weights for Muse Spark 1.2, our latest foundation model. Meta is a strong supporter of open source and I'm proud of these releases. Congrats to @alexandr_wang and the MSL team for all your great work on these models.
Kimi K3 open weights landed yesterday. By that afternoon, Red Hat AI Inference was serving it on one 8x B300 node. 2.8T params.
Day-0 preview images exist so you can experiment the moment weights drop. 👇
🇬🇧 London, June 10.
@vllm_project & @_llm_d_ Inference Meetup, hosted by Red Hat AI, @nvidia, and @SteliaAI.
Talks on vLLM updates, speculative decoding, llm-d in production, AI safety, and more.
Plus food, drinks, and the people building this stuff. luma.com/iuecyow4
Llama 70B as a cloud endpoint costs exponentially more than Llama 8B.
For teams where a smaller model meets the quality bar, that gap is hard to ignore. And with INT4 quantization: 4x smaller, 2x faster, less than 1% accuracy loss.
The right model isn't always the biggest one.
redhat.com/en/blog/when-less…
Calling Boston area startups building with AI. 🤙
We're kicking off 2026 with the first event in a new monthly, in person hackathon series hosted by @RedHat and @IBM in Boston’s Seaport District.
This one day hackathon is designed specifically for local startups that want to move faster from idea to working prototype.
Instead of a fixed theme, you bring a real AI problem your team is actively facing. We help you build a proof of concept using open source, enterprise ready templates from aitemplates.io, including MCP Server, AI Agent, and UI templates.
What you will get:
⚡ Rapid prototyping without boilerplate
🧠 Hands on guidance from Red Hat AI architects
🤝 Connections with other Boston based AI startups and ecosystem partners
If you are a Boston startup looking to turn an AI challenge into something real, this is for you.
Event details are shared after registration.
Register now: luma.com/i3q8df0x
The @RedHat_AI team contributes a lot to vLLM and does amazing work for the open-source community. Great to see vLLM performing so well compared to TRT-LLM on H200! vLLM comes pretty close to B200, with the @NVIDIAAI team working on closing the gap for GPTOSS within the next couple of updates.
InferenceMAX, vLLM TPU, compressed-tensors, MoE support via transformers, DeepSeek-OCR, and more.
Here’s what’s new in the @vllm_project community over the past two weeks:
InferenceMAX, vLLM TPU, compressed-tensors, MoE support via transformers, DeepSeek-OCR, and more.
Here’s what’s new in the @vllm_project community over the past two weeks:
4 tracks. 12 sessions. 1 day of learning.
Join us on Oct. 16 for Red Hat AI Day of Learning, a free virtual event for developers, engineers & practitioners.
Tracks:
⚡ Fast & efficient inference
🎯 Model customization
🤖 Agentic AI
🌐 Scaling AI over hybrid cloud
Sessions include:
· Intro to vLLM and how to get started
· Model optimization with LLM Compressor
· Lossless LLM inference acceleration w/ Speculators
· End-to-end model customization
· Synthetic data generation and data processing
· Continual learning of LLMs with Training Hub
· Build open source agentic AI solutions
· Intro to Model Context Protocol (MCP)
· Intro to Llama Stack
· Intro to distributed inference
· Distributed inference with llm-d
· Scaling AI Infrastructure
👉 Register free: redhat.com/en/events/webinar…
Qwen3-Next dropped yesterday and you can run it with Red Hat AI today.
✅ Day-zero support in vLLM
✅ Day-one deployment with Red Hat AI
Step-by-step guide: developers.redhat.com/articl…
The future of AI is open.
Thanks to the @lmcache team for joining forces with Red Hat on llm-d!
llm-d is a new open source project for scalable, efficient distributed LLM inference with @vllm_project.
Learn more about our collaboration here: blog.lmcache.ai/2025-05-22-r…
Really excited to see the emergence of llm-d @addvin ! Inference is the biggest workload in human history and the open source tools need to keep evolving to serve it
The llm-d project is a major step forward for the #opensource AI ecosystem, and we are proud to be one of the founding contributors, reflecting our commitment to collaboration as a catalyst for innovation in generative AI.
As generative and agentic AI continue to evolve, scalable, high-performance inference will be critical to unlocking their full potential.
That’s why we’re partnering with @RedHat and other contributors to grow the llm-d community and accelerate its capabilities—powered by our contributions, including innovations from NVIDIA Dynamo such as NIXL.
🔗 Explore and contribute on GitHub: nvda.ws/3FlttSL
📰 Read the launch blog: nvda.ws/3FgF0Tn
🎙️ Hear from NVIDIA’s VP of Engineering & AI Frameworks, Ujval Kapasi → nvda.ws/45pyVhU
The llm-d project is a major step forward for the #opensource AI ecosystem, and we are proud to be one of the founding contributors, reflecting our commitment to collaboration as a catalyst for innovation in generative AI.
As generative and agentic AI continue to evolve, scalable, high-performance inference will be critical to unlocking their full potential.
That’s why we’re partnering with @RedHat and other contributors to grow the llm-d community and accelerate its capabilities—powered by our contributions, including innovations from NVIDIA Dynamo such as NIXL.
🔗 Explore and contribute on GitHub: nvda.ws/3FlttSL
📰 Read the launch blog: nvda.ws/3FgF0Tn
🎙️ Hear from NVIDIA’s VP of Engineering & AI Frameworks, Ujval Kapasi → nvda.ws/45pyVhU
DeepSeek’s Open Source Week drops A LOT of exciting goodies! We’re hosting vLLM Office Hours tomorrow—learn what they are, how they integrate with vLLM, & ask questions!
Date: Thursday, Thu, Feb 27
Time: 2PM ET / 11AM PT
Register: neuralmagic.com/community-of…#deepseek#ai
At @RedHat, we believe the future of AI is open. That's why I'm incredibly excited about our acquisition of @NeuralMagic. Together, we're furthering our commitment to our customers and the open source community to deliver on the future of AI—and that starts today.
Today, Red Hat completed the acquisition of @NeuralMagic, a pioneer in software and algorithms that accelerate #genai inference workloads. Read how we are accelerating our vision for #ai’s future: red.ht/408kJ8K.