Sparse from the cache up.
An inference engine that makes sparse KV methods part of the runtime architecture, not an afterthought in attention code.
- Type
- Open-source inference engine
- Focus
- Long-context sparse inference
- Runtime
- Python, CUDA, Triton
- License
- Apache 2.0
The bottleneck is structural.
Sparse methods change how tokens are retained, selected, read, and scheduled. Treating them as a thin attention patch leaves the cache and controller out of sync.
Cache-manager first
Method-specific runtime state lives beside cache allocation and view construction, while attention remains generic.
Policy-owned scheduling
Each sparse method declares the prefill behavior it actually needs instead of relying on benchmark-side exceptions.
Comparable evidence
Benchmark records include method, prefill policy, chunk size, prompt length, batch size, and DeltaKV checkpoints.
Multiple sparse families
Physical eviction, logical masking, query-aware selection, and hybrid compression share one explicit runtime model.
One engine, several sparse semantics.
The method registry is the contract between cache behavior, prefill policy, and the attention read view.
Physical eviction
StreamingLLM, SnapKV, PyramidKV
Retain a selected subset of tokens and physically compact the cache.
Logical masking
OmniKV
Keep storage intact while narrowing the attention read view.
Query-aware
Quest
Select cache pages at decode time using the current query.
Hybrid compression
DeltaKV
Keep a small full-precision pool and compress older context, with training, inference, offload, and benchmark tooling in the same repository.
6
Sparse method families
Dense baseline excluded. The public README documents streaming, selection, masking, and compression paths.
2
Prefill policies
Chunked prefill plus a method-gated full long-prefill path for transformations that require it.
1
Source of truth
The method registry owns runtime policy so benchmark scripts do not silently change execution semantics.