←
Red Hat AI2,711 posts
Red Hat AI avatar✓

Red Hat AI ✓

@RedHat_AI

Accelerating AI innovation with open platforms and community. The future of AI is open.

◷ Joined May 2018▧ 617 Photos and videos
Red Hat AI avatar
Red Hat AI@RedHat_AI·Sep 16

Everything you need to start self-hosting an open LLM. Run it on your own hardware. No API keys. No per-token bill. Nothing leaves your machine. The full path with @vllm_project: batch inference in Python, an OpenAI-compatible API server in one command, and quantized models that cut an 8B from ~16GB of weights to a quarter of that while keeping 98-100% accuracy. Walkthrough by @cedricclyburn.

10:56
Eldar Kurtić avatar
⟳ Red Hat AI retweeted
Eldar Kurtić@_EldarKurtic·19h

We teamed up with @inferact to bring you an even faster speculator for Kimi K3.1 Pro. vLLM serve and enjoy! And as usual, the model card has evaluation benchmarks as far as the eye can see.

Post attachmentPost attachment
vLLM avatar
⟳ Red Hat AI retweeted
vLLM@vllm_project·20h

More accepted tokens. Bounded draft KV memory. 🔥 The new DSpark for @Kimi_Moonshot’s Kimi K3.1 Pro, jointly developed by @Inferact and the @RedHat_AI Speculators team with vLLM + Speculators, brings: • Up to 30% higher acceptance length vs. the preview (HumanEval). • A 2K sliding window in every draft layer, keeping draft KV memory bounded even at 1M-token context. Efficient draft architectures need an inference stack that can serve them. vLLM’s redesigned KV management supports P/D disaggregation and different parallelism configurations, bringing sliding-window draft models into production. Comparison: DSpark k=7, temperature 1.0, top_p 0.95. Acceptance length includes the bonus token. Weights + vLLM config below. 👇

Post attachment
Red Hat AI avatar
Red Hat AI@RedHat_AI·20h

Red Hat ran its own engineering org on open weights as customer zero. What the EnMaaS pilot (Praxis gateway + llm-d + vLLM) showed: -> Scale: more than 1 trillion tokens processed in a single 30-day window. -> Adoption: give developers frictionless access and they take it. Usage started 70/30 in favor of commercial frontier APIs and flipped to 30/70 toward internally hosted open weight models, with open weights gaining share every day. -> Workload: Praxis intelligently routed the majority of routine coding and evaluation traffic away from external APIs to hosted open weight models. -> Cost: shifting that traffic from frontier to self-hosted models on llm-d cut effective token spend 40 to 70% vs prior frontier pricing, depending on prompt caching and routing. Here's how we did it: redhat.com/en/blog/EnMaaS-Re…

Post attachment
Red Hat AI avatar
Red Hat AI@RedHat_AI·23h

Inference performance isn't one knob. It's parallelism, scheduling, KV cache offloading, and prefill/decode disaggregation, and when each actually pays off. Tyler Michael Smith (vLLM core maintainer) and Yuchen Fama break it down live Nov 18, 2pm ET: redhat.com/en/events/webinar…

Red Hat avatar
⟳ Red Hat AI retweeted
Red Hat@RedHat·Nov 11

How is @0LatHQ powering time-sensitive AI applications? By using Red Hat AI Factory with NVIDIA as the foundation for Zerogrid, its distributed inference network. Using Red Hat Advanced Cluster Management, they manage edge nodes powered by NVIDIA Blackwell GPUs across decentralized datacenters from a single control plane. Learn more: red.ht/4sYlPCf #KubeCon+#CloudNativeCon

Post attachment
Red Hat avatar
⟳ Red Hat AI retweeted
Red Hat@RedHat·Nov 10

Red Hat Enterprise Linux 10.3 and 9.9 are here! Strengthen your hybrid cloud foundation against AI-driven threats with sealed images, native post-quantum security, and self-managed AI troubleshooting. Explore the new releases: red.ht/4z4M9ft #KubeCon+#CloudNativeCon

Post attachment
vLLM avatar
⟳ Red Hat AI retweeted
vLLM@vllm_project·Nov 11

vLLM EPD, now upgraded 🚀 DeepInfer demonstrated the value of EPD disaggregation. vLLM’s latest EPD improvements deliver lower TTFT and higher throughput per GPU, counting all encoder GPUs, integrated with vLLM Router, SMG, and llm-d! Read more here!👇 vllm.ai/blog/2026-11-11-enco…

Red Hat AI avatar
Red Hat AI@RedHat_AI·Nov 11

Red Hat AI Elevate, a virtual tech showcase on Dec 9. A look at Red Hat AI 3.6, the Red Hat AI Factory with @nvidia, and the built-in guardrails for running agents in production without blowing up governance or cost. Save your seat: redhat.com/en/events/webinar…

Red Hat AI avatar
Red Hat AI@RedHat_AI·Nov 10

In @NVIDIAAIDev's OpenShell, your agent never holds the real secret. It only sees a placeholder, and the real key gets swapped in at request time, bound to the one endpoint it's allowed to hit. All of the agent's outbound traffic goes through a default-deny policy proxy, so even if it gets prompt-injected and tries to curl your creds off somewhere, it can't reach anything you didn't allow. You can even pick which programs get to make the call: GitHub CLI yes, bare curl no. It runs on OpenShift too, though that path is still experimental. @LegareKerrison's blog below has loads of great getting started resources - check it out!

Misha Goin avatar
⟳ Red Hat AI retweeted
Misha Goin@mgoin_·Nov 10

come here me blab about microscaling formats in vllm at the end 🤠

Red Hat AI avatar
Red Hat AI@RedHat_AI·Nov 10

New vLLM Office Hours on advanced model compression, with Kyle Sayers and @_soyr_. They cover: - LLM Compressor's sequential pipeline and data parallel schemes - Settings most users never change - What observers do and how they affect quantization error - Expert pruning with REAP and MoNE youtube.com/watch?v=2KZQX_X5…