←
Hyperstition39 posts
Hyperstition avatar

Hyperstition

@hyperstition_cc

For too long has coordination been the Achilles’ heel to human progress. DMs open for collabs.

◷ Joined September 2026▧ 17 Photos and videos
Hyperstition avatar
Hyperstition@hyperstition_cc·Nov 5

We invented a recursive megakernel generator called Hyperkernel, the most complete solution to megakernels and GPU efficiency. Hyperkernel-generated megakernels achieve state of the art performance on 11 different domains: inference (DeepSeek, Kimi and GLM), pretraining (Qwen), hardware fault recovery, robotics (π₀-FAST, ACT) diffusion and real time world models (Matrix-Game), full self driving (OpenPilot), voice models (Moshi), genomics (Evo 2) and protein folding (ESMFold). All of this in a single kernel launch. For context, Hyperkernel generated fully optimized megakernels for robotics, genomics and voice in under 50 minutes, and doubled the efficiency for >5 models generally using only about 60-100 lines of code per megakernel. This speaks to the unparalleled standard for code quality in the automated generations. Our first release today open sources the megakernels themselves. Over the next couple weeks, we will release the entire Hyperkernel stack, along with the complete serving stack for Kimi and GLM (20-30% inference efficiency gains across all concurrencies). Out of the box GPU machinery performs terribly for unorthodox research. We built Hyperkernel only 20 days ago for internal use; but given its multi-modality, we have decided to open source it. Creative research should not be bottlenecked by the lack of dedicated infra teams. Hyperkernels offer a one stop shop infra stack for all research. We would love for the open source community to add to it :) GitHub repo: github.com/Hyperstitioncc/Hy… NOTE: We primarily use NVIDIA B300s, but our kernel generation extends to other hardware (B200s, AMD, TPUs). 1/9 We absolutely crush NVIDIA and achieve the best low latency inference for DeepSeek v4.1 Flash. Compared with NVIDIA’s published vLLM results, Hyperkernel delivers: • 36.7% higher per-user generation speed at concurrency 1. • 28.2% higher per-user speed while serving two users, versus NVIDIA serving one. • 15% higher per user speed while serving four users, versus NVIDIA serving one. • 90.6% higher per-user generation speed at concurrency 4, compared with the highest published C4 AgentX interactivity for TP4 B300s. • 28.5% higher per-user generation speed at concurrency 8, compared with NVIDIA's published result. • 36% faster median per-request decode speed.

Post attachment
Hyperstition avatar
Hyperstition@hyperstition_cc·19h

We benchmarked 512 sequences × 256 decode tokens on one NVIDIA B200 (180 GB) per engine using vLLM 0.33, excluding prefill. LoRA used rank-1 adapters on every linear layer; “Gather” uses our reference kernel. As adapter diversity grows, default and tuned vLLM LoRA slow down, while Hypertune stays flat.

Post attachment
Hyperstition avatar
Hyperstition@hyperstition_cc·19h

Eval trajectories for co-resident adapters in 13 different domains. LoRA cannot handle co-residence even if it undergoes training to accommodate it.

Post attachment
Hyperstition avatar
Hyperstition@hyperstition_cc·19h

Part 1: We now enter the era of hyper-composability. For the first time, it is economically feasible to serve individually fine tuned models at scale. And today, we open source it all. We can now serve heterogeneous batches of custom fine-tunes, without incurring LoRA’s per matrix bandwidth overhead. This allows us to economically serve this in large batches without incurring penalties for co-residing with default model users in the same batch. With Hypertune, we solve co-residence and are able to train co-resident adapters. Co-residence is the ability to compose independently trained fine-tunes into one model, without capability regression from adapter interference. Hypertune further matches LoRA in RL performance across 13 domains with 100x-500x fewer parameters. Serving heterogenous LoRA fine-tunes in one batch is prohibitively expensive. Each fine-tune variant needs its own low-rank adapter at every matrix and is only used for a few tokens, leading to many narrow matmuls and poor utilization. This is exacerbated for MoEs, where an adapter only engages when the token simultaneously selects a certain fine-tune and expert. This combinatorial blowup kills any remaining prospects of fast inference for fine-tuned / custom models. Hypertune introduces soft prefill, a trained KV adapter. Soft prefill can be trained using GRPO, SFT or any other first order method. As a consequence, Hypertune circumvents the LoRA overhead because soft prefills live entirely in the KV cache. This allows us to easily integrate it into existing serving stacks by prepending the soft prefills into paged KV blocks. Note: Context compression using cartridges requires copious amounts of synthetically generated self-study data for context distillation. Hypertune, instead, accommodates a panoply of gradient based methods for a wider range of objectives. Caveat: Comparisons with full-parameter GRPO are not required, because under RL, LoRA already matches it. LoRA Without Regret by @thinkymachines finds that LoRA performs equivalently to full fine-tuning for RL even at very small ranks, down to rank 1. The takeaway: Our gains indicate that substantial amounts of in-context learning capacity is left on the table by discrete prompts. This could be either because soft prompts reach conditioning states that no token sequence can express, or because their continuous parameterization makes good conditioning states far easier to find via gradient descent (even if a discrete equivalent exists). Companies and individuals can finally fine tune models on their private data, making models hyper-optimized to their needs, all without dramatically overpaying (75% cheaper than LoRA for inference). Here's the open sourced repo: github.com/Hyperstitioncc/Hy… Part 2 coming out later this week.

0:11
Deven Pietrzak avatar
⟳ Hyperstition retweeted
Deven Pietrzak@DevenPzak·Nov 12

Pretraining might not be teaching attention where to look. It may just be refining a decision made by luck in the first 10% of the run. Top: one model’s attention at step 1k and step 10k, barely changed. Bottom: two models, same training recipe, data, and data ordering, different initialization seed. By step 10k, they’re looking at different things. 🧵(1/4)

Post attachment

Today we report a startling pretraining finding: Attention circuits STOP meaningfully evolving much sooner than we expect. In other words, the model has decided where to look very early on. It is known that different seeds create starkly different results; here we isolate a specific mechanism that causes it. The setup: We perform two identical training runs with the same data order and different initializations and find that by the first thousand steps each run’s attention resembles its own step-10k pattern MORE than the individual finished runs resemble each other. The initialization plays a greater role in the attention patterns than training dynamics, which implies that the final pattern is randomly chosen. NOTE: Randomly chosen, not from the space of all the possible attention patterns that it could be, but from the different attention pattern basins that it could converge to—which one it ends up in appears to be init dependent. Some differences in the loss are attributable to these different attention patterns. This means the final performance is somewhat dependent on a random choice. Depending on how lucky with pattern formation one gets in the first thousand steps, we could end up much better or worse in different contexts. This further implies training could simply find the optimal patterns, or at least better ones, if the attention could evolve instead of being stuck. A story for why attention evolves less than we’d hope falls out of the softmax derivative. We can observe a positive feedback loop we call the credit ratcheting. Let oᵢ be the head output and δᵢ = ∂L/∂oᵢ the loss gradient at that output. Define key k’s advantage for query i as Aᵢₖ = −(vₖ − oᵢ)ᵀδᵢ i.e. how much the loss drops if oᵢ moves toward vₖ, relative to the current attention-weighted mix. Thus, the gradient with respect to sᵢₖ = kₖᵀqᵢ / √d is ∂L/∂sᵢₖ = −aᵢₖAᵢₖ A key can have a huge advantage and have its score still barely move, because the advantage is multiplied by the attention it already gets. It’s the same form as a softmax policy gradient, π(a)·A(a): attention is the policy, and it mostly learns about the actions it already takes. In aggregate, the query gradient is gᵢ = Jᵢᵀδᵢ where Jᵢ = (1/√d) ∑ₖ aᵢₖ(vₖ − oᵢ)kₖᵀ. Across 166,656 Jacobians in Qwen2.5-1.5B, ≥99% of tokens in each of the 3 heads we probed had squared-spectrum effective rank <10, in a 128-dim head. The update to the query is being filtered through an exceedingly narrow channel. The values have the same issue. Key k’s share of the value gradient is Gᵢₖ = aᵢₖδᵢxₖᵀ, so at similar δᵢxₖᵀ, 10× the attention ≈ 10× the credit. This means the values mostly adapt to positions already looking at them. This compounds! Attention allocates credit, credit updates values, and values can pull attention further toward the winners. The most natural approach to fix this is to precondition the query gradient with the Fisher of the attention distribution (NATQ): Fᵢ = Covₐᵢ(k) / d uᵢ = (Fᵢ + λᵢI)⁻¹gᵢ g̃ᵢ = uᵢ · (‖gᵢ‖₂ / ‖uᵢ‖₂) Even this blunt version does a lot. At 124M, entropy rose from 3.75 to 4.69 nats, top-key attention fell from 0.238 to 0.149, and in layers 4–7, keys the baseline gave under 0.1% attention got 67% of the gain. Validation loss rose by 0.082 nats. A more targeted intervention need not make this concession. Here’s the complete blog: hyperstition.cc/stuck-in-a-l… We frame this as an open problem that offers a stark contrast in the way we perceive pretraining. If we can find a way to allow for attention to evolve throughout training, opposed to just the very start, we may find inordinate gains on model capability. Internally we half joke that the alternative read is that attention might simply not be real past the first few steps, and it’s all the MLPs. If training is able to work as well as it does without really changing the attention patterns much, maybe the MLP is the real workhorse that’s driving the learning.

0:18
Hyperstition avatar
Hyperstition@hyperstition_cc·Nov 11

Today we report a startling pretraining finding: Attention circuits STOP meaningfully evolving much sooner than we expect. In other words, the model has decided where to look very early on. It is known that different seeds create starkly different results; here we isolate a specific mechanism that causes it. The setup: We perform two identical training runs with the same data order and different initializations and find that by the first thousand steps each run’s attention resembles its own step-10k pattern MORE than the individual finished runs resemble each other. The initialization plays a greater role in the attention patterns than training dynamics, which implies that the final pattern is randomly chosen. NOTE: Randomly chosen, not from the space of all the possible attention patterns that it could be, but from the different attention pattern basins that it could converge to—which one it ends up in appears to be init dependent. Some differences in the loss are attributable to these different attention patterns. This means the final performance is somewhat dependent on a random choice. Depending on how lucky with pattern formation one gets in the first thousand steps, we could end up much better or worse in different contexts. This further implies training could simply find the optimal patterns, or at least better ones, if the attention could evolve instead of being stuck. A story for why attention evolves less than we’d hope falls out of the softmax derivative. We can observe a positive feedback loop we call the credit ratcheting. Let oᵢ be the head output and δᵢ = ∂L/∂oᵢ the loss gradient at that output. Define key k’s advantage for query i as Aᵢₖ = −(vₖ − oᵢ)ᵀδᵢ i.e. how much the loss drops if oᵢ moves toward vₖ, relative to the current attention-weighted mix. Thus, the gradient with respect to sᵢₖ = kₖᵀqᵢ / √d is ∂L/∂sᵢₖ = −aᵢₖAᵢₖ A key can have a huge advantage and have its score still barely move, because the advantage is multiplied by the attention it already gets. It’s the same form as a softmax policy gradient, π(a)·A(a): attention is the policy, and it mostly learns about the actions it already takes. In aggregate, the query gradient is gᵢ = Jᵢᵀδᵢ where Jᵢ = (1/√d) ∑ₖ aᵢₖ(vₖ − oᵢ)kₖᵀ. Across 166,656 Jacobians in Qwen2.5-1.5B, ≥99% of tokens in each of the 3 heads we probed had squared-spectrum effective rank <10, in a 128-dim head. The update to the query is being filtered through an exceedingly narrow channel. The values have the same issue. Key k’s share of the value gradient is Gᵢₖ = aᵢₖδᵢxₖᵀ, so at similar δᵢxₖᵀ, 10× the attention ≈ 10× the credit. This means the values mostly adapt to positions already looking at them. This compounds! Attention allocates credit, credit updates values, and values can pull attention further toward the winners. The most natural approach to fix this is to precondition the query gradient with the Fisher of the attention distribution (NATQ): Fᵢ = Covₐᵢ(k) / d uᵢ = (Fᵢ + λᵢI)⁻¹gᵢ g̃ᵢ = uᵢ · (‖gᵢ‖₂ / ‖uᵢ‖₂) Even this blunt version does a lot. At 124M, entropy rose from 3.75 to 4.69 nats, top-key attention fell from 0.238 to 0.149, and in layers 4–7, keys the baseline gave under 0.1% attention got 67% of the gain. Validation loss rose by 0.082 nats. A more targeted intervention need not make this concession. Here’s the complete blog: hyperstition.cc/stuck-in-a-l… We frame this as an open problem that offers a stark contrast in the way we perceive pretraining. If we can find a way to allow for attention to evolve throughout training, opposed to just the very start, we may find inordinate gains on model capability. Internally we half joke that the alternative read is that attention might simply not be real past the first few steps, and it’s all the MLPs. If training is able to work as well as it does without really changing the attention patterns much, maybe the MLP is the real workhorse that’s driving the learning.

0:18
Hyperstition avatar
Hyperstition@hyperstition_cc·Nov 5

Why solving hang detection matters for pretraining efficiency: NCCL takes minutes to detect hardware faults, and can require manual intervention in the case of a silent hang. Large scale pretraining runs have faults every few hours, and teams lose 10-15% on wall time to fault overhead. Our heartbeat protocols detect NVLink, InfiniBand and GPU faults in seconds, and reload the minimal affected model state from healthy peers. Then, the megakernel is regenerated to preserve MFU in the degraded cluster. This requires recalculating optimal sharding of experts and tensors, as well as assigning less work to degraded DP groups.

3/n We have the most complete pretraining megakernel with hardware fault detection and replanning. 1. We achieve an out of the box 31% MFU on standard mixed precision (BF16 and FP32) pretraining of a 30B MoE Qwen architecture on one node and 30% MFU on data parallel pretraining over multiple nodes. 2. We have tensor parallelism, expert parallelism and data parallelism. 3. Hyperkernel automatically generates efficient megakernels that fuse low intensity processes and schedule the tiles to load balance over applicable streaming multiprocessors. This generalizes across all cluster topologies. 4. Hyperkernel automatically detects individual GPU, NVLink, and InfiniBand failures within seconds. Performs live triage mid-step so as to only recompute tensors which were lost rather than naively reverting to a previous checkpoint. Autonomously rewrites the megakernel to preserve MFU on the now partially degraded cluster.

0:24
Hyperstition avatar
Hyperstition@hyperstition_cc·Nov 5

Robotics is more latency bound than we think. A big reason π0 style VLAs carries a separate denoising head is that token by token decoding has been too slow to serve. pi.website/research/fast Autoregressive decoding at batch 1 is latency-bound. On a typical stack, every token has a python step overhead, a stream of kernel launches and a host sync. Our π0-FAST megakernel runs the whole chunk in one launch: vision prefill, then the full decode loop on the GPU, with KV cache resident, sampling on device. We also have ops overlap across layers with no kernel boundaries. 94 ms, vs 243 ms for PyTorch + CUDA graphs. Credit: @Nahomes_15

9/n Our megakernel on robotics: π₀-FAST, from Physical Intelligence @physical_int, is a vision-language-action model for dexterous robot control. Instead of generating actions with a flow-matching action expert, as π₀ does, it compresses them into discrete tokens and decodes them autoregressively. Our megakernel ran the vision encoding prefill and generation of 60 tokens in 94 ms, while a custom reference PyTorch + CUDA graph decoding ran it in 243ms, with megakernels reducing latency by 61% at batch one in BF16.

0:15
Jerry Tworek avatar
⟳ Hyperstition retweeted
Jerry Tworek@MillionInt·Nov 5

Megakernels are the future and incredibly cracked teams seem to be pushing the boundaries of possibility each day getting further and further. Looking forward to seeing more from this team 😀

We invented a recursive megakernel generator called Hyperkernel, the most complete solution to megakernels and GPU efficiency. Hyperkernel-generated megakernels achieve state of the art performance on 11 different domains: inference (DeepSeek, Kimi and GLM), pretraining (Qwen), hardware fault recovery, robotics (π₀-FAST, ACT) diffusion and real time world models (Matrix-Game), full self driving (OpenPilot), voice models (Moshi), genomics (Evo 2) and protein folding (ESMFold). All of this in a single kernel launch. For context, Hyperkernel generated fully optimized megakernels for robotics, genomics and voice in under 50 minutes, and doubled the efficiency for >5 models generally using only about 60-100 lines of code per megakernel. This speaks to the unparalleled standard for code quality in the automated generations. Our first release today open sources the megakernels themselves. Over the next couple weeks, we will release the entire Hyperkernel stack, along with the complete serving stack for Kimi and GLM (20-30% inference efficiency gains across all concurrencies). Out of the box GPU machinery performs terribly for unorthodox research. We built Hyperkernel only 20 days ago for internal use; but given its multi-modality, we have decided to open source it. Creative research should not be bottlenecked by the lack of dedicated infra teams. Hyperkernels offer a one stop shop infra stack for all research. We would love for the open source community to add to it :) GitHub repo: github.com/Hyperstitioncc/Hy… NOTE: We primarily use NVIDIA B300s, but our kernel generation extends to other hardware (B200s, AMD, TPUs). 1/9 We absolutely crush NVIDIA and achieve the best low latency inference for DeepSeek v4.1 Flash. Compared with NVIDIA’s published vLLM results, Hyperkernel delivers: • 36.7% higher per-user generation speed at concurrency 1. • 28.2% higher per-user speed while serving two users, versus NVIDIA serving one. • 15% higher per user speed while serving four users, versus NVIDIA serving one. • 90.6% higher per-user generation speed at concurrency 4, compared with the highest published C4 AgentX interactivity for TP4 B300s. • 28.5% higher per-user generation speed at concurrency 8, compared with NVIDIA's published result. • 36% faster median per-request decode speed.

Post attachment
Hyperstition avatar
Hyperstition@hyperstition_cc·Nov 5

11/11 Our megakernel on a two handed manipulation robot policy: ACT, from Stanford's ALOHA project, is a robot policy that does two-handed manipulation from a few dozen human demonstrations. Since it runs every camera frame through the control loop, inference time is the primary determinant for how fast the robot can react. Our megakernel ran it in 0.81 ms per frame versus 2.21 ms for Hugging Face LeRobot in BF16 with torch.compile, 2.7× faster, with unchanged task success.

Post attachment
Hyperstition avatar
Hyperstition@hyperstition_cc·Nov 5

10/n Our megakernel on voice models: Moshi is a full-duplex speech-to-speech model behind real-time voice agents. Our megakernel ran the Moshi language model and the deformer in 6.8 ms per frame versus 9.45 ms for the PyTorch reference with CUDA graphs. This is a 28% lower measured latency at batch one.

Post attachment
Hyperstition avatar
Hyperstition@hyperstition_cc·Nov 5

9/n Our megakernel on robotics: π₀-FAST, from Physical Intelligence @physical_int, is a vision-language-action model for dexterous robot control. Instead of generating actions with a flow-matching action expert, as π₀ does, it compresses them into discrete tokens and decodes them autoregressively. Our megakernel ran the vision encoding prefill and generation of 60 tokens in 94 ms, while a custom reference PyTorch + CUDA graph decoding ran it in 243ms, with megakernels reducing latency by 61% at batch one in BF16.

0:15
Hyperstition avatar
Hyperstition@hyperstition_cc·Nov 5

8/n Our megakernel on protein folding: ESMFold predicts a protein's 3D structure directly from its amino-acid sequence. Our megakernel folded a 164-residue protein end to end in 195 ms, against 677 ms for the PyTorch + CUDA graph reference: a 71% lower measured latency at batch one, with structures matching the reference within 1 Å.

0:50
Hyperstition avatar
Hyperstition@hyperstition_cc·Nov 5

7/n Our megakernel on real time world models: Matrix-Game 3.0 is an interactive world generator model. Our megakernel runs this generation at 37.4 FPS while a PyTorch + CUDA graph reference runs it at 11.6 FPS, giving us 3.2x higher throughput on matched starting image prompts and controls applied.

0:30