←
Mark Kurtz183 posts
Mark Kurtz avatar

Mark Kurtz

@markurtz_

AI @ @RedHat, former CTO @ @NeuralMagic (acquired), active ML researcher and contributor

◷ Joined April 2017▧ 21 Photos and videos
Red Hat AI avatar
⟳ Mark Kurtz retweeted
Red Hat AI@RedHat_AI·16 Sep 2025

🚀 Thrilled to announce GuideLLM v0.3.0! This release is highlighted by a brand new Web UI, containerized benchmarking, and powerful dataset preprocessing. GuideLLM GitHub: github.com/vllm-project/guid… (Thread 👇)

Post attachment
Eldar Kurtić avatar
⟳ Mark Kurtz retweeted
Eldar Kurtić@_EldarKurtic·25 Jun 2025

Our flagship paper on how far careful quantization can really go in practice got accepted as an oral at ACL 2025 (top 8%)! 🥳 Turns out, old-school methods like GPTQ, SmoothQuant, and RTN are quite good when tuned properly. All of the tricks are already in LLM-Compressor!

Post attachment
Red Hat AI avatar
⟳ Mark Kurtz retweeted
Red Hat AI@RedHat_AI·18 Jun 2025

🚨 Introducing the Axolotl-LLM Compressor integration, designed to make fine-tuning sparse models easier and more efficient than ever! Now you can fine-tune sparse models for specific data while preserving their sparse structure and recovering any accuracy lost during pruning. cc. @axolotl_ai What you can achieve with this integration: - 99% accuracy recovery or better - 5X smaller models - 3X faster inference than the baseline. How it works: 1️⃣ Start by sparsifying a model using LLM Compressor (github.com/vllm-project/llm-…) or choose a sparse foundation model from RedHat AI on Hugging Face, such as Sparse-Llama-3.1-8B-2of4 (huggingface.co/RedHatAI/Spar…). You can also bring your own sparse model, as long as it’s compatible with @huggingface transformers. 2️⃣ Next, fine-tune the sparse model using the new Axolotl-LLM Compressor integration (docs.axolotl.ai/docs/custom_…). 3️⃣ Finally, apply post-training quantization (github.com/vllm-project/llm-…) using LLM Compressor to compress the model further. Once you’ve fine-tuned and quantized your model, deploy it effortlessly with @vllm_project for fast, scalable inference. 🚀 Explore the full steps here: developers.redhat.com/articl…

Post attachmentPost attachment
Red Hat AI avatar
⟳ Mark Kurtz retweeted
Red Hat AI@RedHat_AI·18 Apr 2025

Missed the session? No worries! Watch the recording on YouTube: youtube.com/live/T8XDkZuv7O4 View the slides: docs.google.com/presentation… Questions? Drop them in comments and @markurtz_ will get back to you.

Red Hat AI@RedHat_AI17 Apr 2025

Join us this Friday for Random Samples, a weekly AI talk series from @RedHat_AI Innovation Team. Topic: The State of LLM Compression — From Research to Production We’ll explore quantization, sparsity, academic vs. real-world benchmarks, and more. Join details in comments 👇

Post attachment
Red Hat AI avatar
⟳ Mark Kurtz retweeted
Red Hat AI@RedHat_AI·17 Apr 2025

Join us this Friday for Random Samples, a weekly AI talk series from @RedHat_AI Innovation Team. Topic: The State of LLM Compression — From Research to Production We’ll explore quantization, sparsity, academic vs. real-world benchmarks, and more. Join details in comments 👇

Post attachment
Red Hat AI avatar
⟳ Mark Kurtz retweeted
Red Hat AI@RedHat_AI·5 Apr 2025

Llama 4 Herd is here! It brings a lot of goodies, like MoE architecture and native multimodality, enabling developers to build personalized multimodal experiences. With Day 0 support in vLLM, you can deploy Llama 4 with @vllm_project now! Let's dig into it. (a thread)

Post attachment
Mark Kurtz avatar
Mark Kurtz@markurtz_·1 Apr 2025

To quote a recent tweet: vLLM is the de facto standard open-source LLM inference engine for large-scale, enterprise-grade production deployments.

Post attachment

vLLM is the open source inference engine behind many of the fastest and most efficient LLM deployments today. If you’re building GenAI applications, vLLM is probably already in your stack — or should be. Here's why 🧵

Mark Kurtz avatar
Mark Kurtz@markurtz_·12 Feb 2025

I’m a bit late in sharing this, but at the start of the year, I embarked on an exciting new journey—joining @RedHat following its acquisition of @neuralmagic, where I had the privilege of serving as CTO. While waiting for the deal to finalize during a holiday trip, I saw this fedora-shaped iceberg (it looked a bit more like it from the front!) floating by and ran outside to admire it. It was a perfect reflection point—not just on the incredible work we accomplished at Neural Magic, but on how much more lies beneath the surface, waiting to unfold. Fittingly, the Fedora is an iconic symbol for Red Hat, representing the power of open-source communities, collaboration, and innovation. The AI space follows the same principles. The industry often spotlights curated, surface-level demos, but the real breakthroughs happen below in product enablement, fine-tuning, optimization, and scaling, where innovation transforms AI from research into real-world impact across industries and communities. What you see is just the tip; what truly matters is the foundation beneath. At Red Hat, I'm thrilled to scale our commitment to making AI open, efficient, and accessible, all within the incredible open-source communities that power Red Hat and the future of technology. Exciting things ahead! Read more about the acquisition and Red Hat's AI vision: redhat.com/en/about/press-re… #opensourcedevelopment #ai #machinelearning #mlops #community #startups

Post attachment
jamiegoldstein avatar
⟳ Mark Kurtz retweeted
jamiegoldstein@jamieagoldstein·8 Jan 2025

In 2018, we raised a glass to toast the start of our partnership with @neuralmagic. Nearly seven years later, we raised another to mark the successful sale of the company to Red Hat - a well-deserved milestone for an incredible team and groundbreaking technology.

Mark Kurtz avatar
Mark Kurtz@markurtz_·25 Nov 2024

Incredibly excited to continue our mission of open, efficient AI!

Red Hat AI@RedHat_AI25 Nov 2024

Introducing 2:4 Sparse Llama: The first sparse foundation model built on Llama 3.1 8B. With 98% recovery on Open LLM Leaderboard v1 and full recovery on fine-tuning tasks (math, coding, chat), it’s more efficient and open source! Get from Hugging Face: huggingface.co/neuralmagic/S…

Post attachment
brian stevens avatar
⟳ Mark Kurtz retweeted
brian stevens@addvin·12 Nov 2024

I’m thrilled to announce that Neural Magic has signed a definitive agreement to join forces with Red Hat, Inc. At Neural Magic our vision is that the future of AI is open, and we have been on a mission to enable enterprises to capture the powerful innovation from AI, while at the same time being free of friction and restriction. Open-source models, optimization algorithms, and inferencing systems are the heart of this - enabling enterprises to own their own AI models, privately customize them to their datasets, and deploy them on their private multi-vendor infrastructure - whether cloud, datacenter, or edge. For the last 5 years, we have been contributing open research, open-source model quantization and sparsification tools, and open-source pre-optimized models to the community. As part of our increasing shift to support GPUs, we have been recognized as the top commercial contributor to the amazing vLLM project. There is a saying in the startup world that an acquisition is an exit. This is not. It is an entrance. A beginning to create an open source world for AI, where an open ecosystem flourishes, and customers and the community are the benefactors. Red Hat shares this vision, and together we will advance the world of truly “open” AI. It’s a great day for open source AI. Read the Press Release: redhat.com/en/about/press-re…

Post attachment
Mark Kurtz avatar
Mark Kurtz@markurtz_·5 Nov 2024

The results are in: Trade-offs between accuracy & performance in LLM quantization After hundreds of thousands of evals and benchmarks from our research team at @neuralmagic, I'm excited to share our findings on LLM quantization—now available as a paper on arXiv: arxiv.org/abs/2411.02355 Some key insights: 1/ w8a8 quantization delivers up to 8x speedup on high-performance GPUs, making it ideal for larger models or server deployments. 2/ For smaller models, w4a16 quantization can reduce cost per request by up to 7x, especially for synchronous use cases. 3/ FP8 (w8a8) is essentially lossless in our benchmarks. INT8 (w8a8) & INT4 (w4a16) also retain very high fidelity. 4/ AWQ and GPTQ are close for academic benchmarks, but AWQ struggles with real-world performance. 📄 Full paper: arxiv.org/abs/2411.02355 Interested in reducing your model costs, boosting performance, or scaling efficiently? DM me—let’s chat! ✉️ And stay tuned—🚀 - New compressed model launch soon - Multi-modal model results incoming - More benchmarks, algorithms, and evaluations on the way #llms #quantization #optimization #genai #ai

Post attachmentPost attachmentPost attachmentPost attachment
Mark Kurtz avatar
Mark Kurtz@markurtz_·17 Oct 2024

To paraphrase a popular post from @zjasper on this platform: give me in4 or give me death! In all seriousness, there is no reason why people should waste power, energy, and performance deploying 16-bit LLMs. These models are enormous, and with that, there's a ton of redundancy inside that can easily be removed through model compression with negligible effects on accuracy. To prove this, our research team at @neuralmagic ran over 500,000 evals. The result is that quantized models function nearly exactly the same as their bf16 counterparts. We compiled our results into a blog post alongside our open-sourced models and evals. Take a look through for the full details: neuralmagic.com/blog/we-ran-… #llms #llama #genai #optimization #mlops #research #opensource

Post attachmentPost attachmentPost attachment
Mark Kurtz avatar
Mark Kurtz@markurtz_·14 Oct 2024

It was a privilege to sit down with Chris Brandt from FUTR.tv and explore the state of AI today. We covered everything from the promise of smaller, specialized models to the real risks enterprises face when adopting AI. Key points in the podcast: 1️⃣  Why is AI replacing creative fields and not mundane tasks 2️⃣ The pitfalls of larger, general models and the importance of scaling smaller models 3️⃣ The power of open-source AI 4️⃣ Smarter algorithms for the future Watch the full talk on YouTube: youtube.com/watch?v=4ogqU_jf… Or listen to the podcast: futr.tv/thefeed/2024/10/14/a… What’s your take on these trends? Are enterprises ready for the shift to AI? How will creativity and automation coexist?

Santiago avatar
⟳ Mark Kurtz retweeted
Santiago@svpino·25 Sep 2024

You can now optimize and make any open-source LLM faster: 1. pip install llmcompressor 2. apply quantization with 1 line of code Two benefits: 1. Your LLM will run faster during inference time. 2. You will save a ton of money on hardware Here are a couple of examples: • Llama 3.1 405b requires 2 8x80GB nodes. You can optimize it using LLM Compressor to run in a single 4x80GB node with 99.9% recovery. That's 400% savings! • Llama 3.1 70b requires 2 x 80GB GPUs. After you optimize it, it can run on a single 80GB GPU. LLM Compressor is open-source, integrates with HugginFace model repositories, and is compatible with the most popular open-source inferencing systems, such as @vllm_project and @huggingface. Here is the repository: github.com/vllm-project/llm-…

Post attachment
Red Hat AI avatar
⟳ Mark Kurtz retweeted
Red Hat AI@RedHat_AI·16 Sep 2024

.@markurtz_, CTO of Neural Magic, shares insights on building a career in AI, from hands-on expertise to understanding AI’s limitations. He emphasizes mentorship and open-source projects in advancing education. Read his tips for impactful AI solutions!👇 medium.com/authority-magazin…