←
Hao AI Lab768 posts
Hao AI Lab avatar

Hao AI Lab

@haoailab

Hao AI Lab at UCSD. Our mission is to democratize large machine learning models, algorithms, and their underlying systems.

◷ Joined March 2024▧ 282 Photos and videos
Hao AI Lab avatar
Hao AI Lab@haoailab·Nov 11

Towards Verifiable LLM Infra 🚀 Suppose coding agents can build a fast inference engine from scratch, and one engineer orchestrates 10,000 agents in parallel: how do we know the engine they wrote is actually correct? Having agents review other agents mostly wastes tokens compared to empirical and formal verification. We introduce vstack, a 5-stage formal verification workflow for coding agents, and veriserve, a verifiable PD-disaggregated serving engine: uncovering 3 classes of real RDMA and cache races in production serving engines, and backing veriserve's KV-page lifecycle design model with a 226-module, 516,973-obligation TLAPS safety proof at +0.18% mean p50 TPOT overhead on GB200. Blog: haoailab.com/blogs/verifiabl… veriserve: github.com/hao-ai-lab/verise… vstack: github.com/hao-ai-lab/vstack

Hao AI Lab avatar
Hao AI Lab@haoailab·19h

(6/8) Lost cancellation: under DP attention, a rapid cancel can overtake its own generation request and be dropped, leaving the request running to max_new_tokens. This breaks cancellation finality: once an abort request succeeds, the cancelled request must not run to completion. With the default DP control routing, a generation request goes directly to its owning rank while an abort goes to rank 0 and is broadcast across ranks, with no ordering guarantee between the two paths. If the abort arrives first, the scheduler has not seen the request yet, drops the abort, and runs the request to its token limit on the next poll. On GB200, 33 of 204 aborts sent within 20 ms were lost and ran to max_new_tokens, dropping to 0 of 204 once aborts travel in order on each rank's work socket. PR: github.com/sgl-project/sglan…

Hao AI Lab avatar
Hao AI Lab@haoailab·19h

(5/8) Multi-GPU rank mismatch: under DP attention with the prefill delayer and interval min batch size, two DP ranks enter different collectives and exit. This breaks multi-GPU lockstep: every rank sharing a communication group must run the exact same collectives in the exact same order each step. Each rank checks its own running batch size when deciding whether to defer prefill. When one rank meets the threshold and another sits below it, the busy rank skips prefill and goes straight to the per-round sync while the idle rank enters the prefill-delayer negotiation on the same CPU group, causing the communication library to abort. Having a deferring rank still join the delayer negotiation with zero prefill keeps every collective aligned, completing 30 of 30 cross-threshold runs on GB200 with no abort. PR: github.com/sgl-project/sglan…

Hao AI Lab avatar
Hao AI Lab@haoailab·19h

(4/8) Streaming stop-string trimming issue: in streaming mode, SGLang can send the client the start of a stop string when a later character is split across tokens. This breaks stop-string trimming: streaming and non-streaming generation must stop before the exact same stop string. Byte-level tokenizers often split emoji and CJK characters across multiple tokens. SGLang holds back streaming text while the decoded tail matches a stop-string prefix, but a token carrying only the first bytes of a multi-byte character decodes to U+FFFD, breaking that prefix check and flushing the printable head of the stop string early. On GB200 with Qwen3-4B under default flags, the stop-string prefix was emitted untrimmed in 8 of 8 streaming comparisons across chat and completions endpoints, and 0 of 8 after also checking the tail stripped of trailing U+FFFD. PR: github.com/sgl-project/sglan…

Hao AI Lab avatar
Hao AI Lab@haoailab·19h

(3/8) Decode server stall: on a PD-disaggregated decode server, a single retracted request freezes the transfer queue while health checks still pass. This breaks liveness: requests whose KV cache has already arrived must be able to run and release their reserved memory so retracted work can resume. Under KV pressure, the decode scheduler retracts a request and returns early every step to hold off new preallocations. That early return also skips polling the transfer queue, so arrived transfers keep holding their pages and decode reserves while the retracted request never fits, even as /health still returns 200. On GB200, the decode server stalled for 364.69 s while /health answered 200 in 363 of 363 probes. Gating only new preallocations lets arrived transfers finish and clears the stall. PR: github.com/sgl-project/sglan…

Hao AI Lab avatar
Hao AI Lab@haoailab·19h

(2/8) Weight-update race condition: after an in-place pause in an RL loop, a weight update can overwrite weights while an in-flight step is still reading them. This breaks rollout policy isolation: a forward step launched before a pause must finish reading the old weights before a weight update changes them. With the default overlap scheduler, an in-place pause returns right away while the already-launched step keeps running on the GPU's forward stream. Parameter writes run on the scheduler stream, whose barrier only waits for input metadata reads before the model layers, so new weights can land mid-step with no error. On GB200, the in-flight step computed its token, logprob, and KV from the updated weights in 8 of 8 trials before the fix, and 0 of 8 after. Adding a GPU stream wait before parameter writes ensures the forward step finishes first. PR: github.com/sgl-project/sglan…

Hao AI Lab avatar
Hao AI Lab@haoailab·19h

(1/8) Agentic Formal Verification on SGLang 🚀 Last week, we introduced vstack for agentic formal verification in AI infra; now we're working with the @sgl_project community on its mature, robust serving stack. vstack complements regular unit and e2e CI by comprehensively exploring async interleavings and state transitions, catching concurrency and multi-step bugs before they hit production. Below we share 5 of the 20+ issues and fix PRs we proposed with vstack, from weight-update race conditions to decode server stalls🧵 🔌 Try vstack here: github.com/hao-ai-lab/vstack

Hao AI Lab avatar
Hao AI Lab@haoailab·Nov 13

(7/7) Our fixes are going upstream now, and we're continuing to work with the vLLM team and the open-source community to bring more verifiable AI infra. Huge thanks to the @vllm_project team for building such a robust engine and for their valuable feedback! vLLM verification work by @AlexZhang152005 and @Junda_Su08 , with special thanks to @Yuxuan_Zhang13 , @yisun0618 , @Junda_Chen_ and @FuYichao123 for building this together at @haoailab !

Hao AI Lab avatar
Hao AI Lab@haoailab·Nov 13

(6/7) Cross-instance mix-up: a question about one image is answered from another image's cached data. This breaks content-based caching: two different images must never share a cache key. Normally, vLLM identifies each image by a content hash. But when both the prefix cache and image processing cache are off, it uses a counter starting at 1 on every server. Since that label still enters an external cache store key, two vLLM servers sharing the store give different images the same key. So when the second server is asked about its own image, it can find the first server's cached data under that key and answer from that other image, with a normal cache hit and no error. Across two servers sharing an external KV cache store, the second answered from another image's cached data. Changing image labeling fixes this: each server adds its own random part, so labels across servers no longer match. PR: github.com/vllm-project/vllm…

Hao AI Lab avatar
Hao AI Lab@haoailab·Nov 13

(5/7) Stale adapter: after a LoRA adapter is reloaded in place, new requests can still run on the old weights. This breaks adapter updates: once a reload reports success, new requests for that adapter must use the new weights. An in-place reload keeps the adapter's name and ID but points them to new files. A request admitted before the reload still points to the old files. So when vLLM serves more adapters than it keeps in memory and must load the adapter again, it can load it from that old request, so new requests run on the old weights while vLLM reports the new version. An in-place LoRA reload while serving left new requests on old weights. Tracking each adapter's newest successful reload ensures vLLM loads from it, so old requests no longer bring back old weights. PR: github.com/vllm-project/vllm…

Hao AI Lab avatar
Hao AI Lab@haoailab·Nov 13

(4/7) Silent out-of-bounds write: a long streaming session overwrites a shared cache block. This breaks memory isolation: a request must never write outside its own memory. With streaming input, vLLM checks each new chunk's length but not the whole session's, so a session can exceed the model's maximum length. vLLM reserves no memory for those extra tokens, so their data spills outside the session's memory and can land in a shared cache block. A later request reusing that block can get different output, with no error. A long streaming session past the model's length limit overwrote a shared prefix cache block. Bounding the worker write prevents that, dropping token data past the limit. PR: github.com/vllm-project/vllm…

Hao AI Lab avatar
Hao AI Lab@haoailab·Nov 13

(3/7) Server lockout: with a queue limit set, a few failed requests make vLLM answer every new request with "busy" until a restart. This breaks a basic availability promise: a request that fails must give back its place in the queue and in the load balancer. When a queue limit is set, vLLM counts each request against it before handing it to the engine. If that hand-off fails, nothing returns the slot, so the client gets an error while its slot stays taken. Thus a few failed requests make the server refuse all work, though its health check still passes. (Without a limit, the leaked requests just keep using memory.) With data parallelism, the same failure also leaves a routing entry, so the load balancer keeps sending new traffic to one engine. With a queue limit set, 2 failed requests made vLLM refuse the next 2 normal requests 2 of 2 times, while under data parallelism 8 of 8 went to one engine. Now a failed request returns its slot and routing entry, so both normal requests succeed and traffic splits 4 and 4 again. Our PR fixes both leaks. PR: github.com/vllm-project/vllm…

Hao AI Lab avatar
Hao AI Lab@haoailab·Nov 13

(2/7) Silent wrong output: hybrid models can continue from an all-zero cache block, and then cache it. This breaks prefix caching: a cache hit must give a request the same result that it would have computed itself. On hybrid models with fine-grained prefix matching turned on, vLLM marks a cache block as reusable as soon as it plans the block, before the block is actually written. So when a second request in the same step reuses that block, it copies it too early and gets all zeros. The second request then continues from those zeros. Since the bad block stays in the cache, later requests can pick it up as well, while every response still looks normal. On GB200, the copy source was all zeros in 32 of 32 trials before the fix, and 0 after. A small fix defers a request with an unwritten copy source to the next scheduler step, after the source is populated. PR: github.com/vllm-project/vllm…

Hao AI Lab avatar
Hao AI Lab@haoailab·Nov 13

(1/7) Agentic Formal Verification on vLLM 🚀 This week, we introduced vstack for agentic formal verification in AI infra; today, we're working with @vllm_project , putting vstack to test: although vLLM is already very mature, some bugs only surface in rare combinations that are hard for human reviewers to catch. With vstack's agentic formal verification, we identified 17 (and growing!) bugs, from silent wrong output to server lockout. Below we share 5 of them, each with a fix PR ⬇️ 🔌 vstack plugin: github.com/hao-ai-lab/vstack 📋 Bugs Tracker: docs.google.com/spreadsheets…