Install
Easy, fast, and cheap LLM serving for everyone vLLM is a fast and easy-to-use library for LLM inference and serving. Originally developed in the Sky Computing Lab at UC Berkeley, vLLM has evolved into a community-driven project with contributions from both academia and industry. vLLM is fast with: Efficient management
- 37articles · 30d
- 2+ day agolatest article
- Aug 15, 2026earliest in window
- 0%with images
- 177avg words
- Science & Technology 37
- Software Dev. 37
- Computers & Electronics 36
- Jobs & Education 1
Please confirm you are human
This browser or connection looks automated. Press and continuously hold the control for 3 seconds to enable Google-hosted web results and, when separately allowed, AI-assisted answers.
A successful check enables 100 search requests. Interactive access does not authorize scraping, systematic collection, or reuse of search output.
News
dspark
2+ day, 21+ hour ago (47+ words) vLLM CPU DeepSeek-V4 DSpark (speculative decoding) — deferred, not implemented. DSparkDeepseekV4ForCausalLM is only instantiated when the speculative- decoding registry resolves it by name for a DSpark speculative config; a plain (non-spec) DeepSeek-V4 CPU model never constructs this class. This stub exists solely…...
ple_layer
1+ week, 5+ day ago (122+ words) Return whether a 64-bit integer is prime. Build deterministic hash multipliers for one PLE layer. Build per-head vocabulary sizes, offsets, and total row count. Return the count-th prime strictly greater than start. Mix an integer into a deterministic unsigned 64-bit…...
low_latency_gemm
1+ week, 5+ day ago (20+ words) vLLM Qwen4Exp low-latency GEMM hook for AMD ROCm. Keep the standard vLLM linear methods on AMD ROCm....
indexer_qsa
1+ week, 5+ day ago (49+ words) Replicated Q/K projection plus paged, weight-free QSA selection. Normalize pooled K and apply the first token's exact group position. Project replicated Q/K, normalize+rotate Q, and preserve raw K. Use vLLM's portable RMSNorm implementation on ROCm. Apply the main…...
hy_v4_reasoning_parser
1+ week, 6+ day ago (286+ words) Reasoning extraction for HYV4, on plain data. Count reasoning tokens (0 in no_think). The start token is injected at the END of the prompt, so it is normally absent from token_ids. Only skip a leading start token when it actually appears first (legacy / no…...
flashmla_sparse
1+ week, 6+ day ago (254+ words) Sink-capable FlashMLA sparse backend for HY V4 (NVIDIA). HY V4 adds a per-head learnable attention sink on top of sparse MLA. The vendored FlashMLA kernels already accept an attn_sink argument, but vLLM's shared FLASHMLA_SPARSE backend neither advertises sink support nor forwards the tensor, so…...
hy_v4_tool_parser
1+ week, 6+ day ago (620+ words) Pure tool-call parsing logic for HYV4. Returns plain dicts; no vLLM types. Holds only parser state: the structural token strings/ids, the compiled regexes, and the streaming incremental state. Stateless argument parsing is delegated to the module-level utilities above. Streaming has…...
sse_keep_alive
2+ week, 6+ day ago (48+ words) vLLM SSE keep-alive comments for idle streaming responses. Emit an SSE keep-alive comment when generator is idle. A non-positive or non-finite interval returns generator unchanged, so the default path has no overhead. Otherwise a keep-alive comment is yielded whenever no…...
sharded_rdt_fake
2+ week, 6+ day ago (525+ words) Op-chain recording for the sharded-RDT backend. The consumer asks the trainer for the exact slice a worker consumes, described as an op chain replayed on the trainer's live tensor. FakeRDTTensor builds the chain by intercepting the model's own weight loaders;…...
sharded_rdt_engine
2+ week, 6+ day ago (1746+ words) Sharded Ray Direct Transport (RDT) weight transfer engine (consumer side). Pulls only the slice each vLLM worker consumes under tensor/expert parallelism, not the full HF-format tensor. One thing at four resolutions, over three lifetimes. FetchKey -- (name, op_chain), "which slice of…...