KVCacheStore is a high-performance Alibaba Cloud storage service designed for large language model (LLM) inference. It provides KV access interfaces and supports hundreds of PiB of storage, hundreds of billions of KV pairs, tens of TB/s of throughput, hundreds of millions of IOPS, and sub-millisecond latency.
Overview
KVCacheStore provides a high-performance, multi-tier KVCache storage service designed for AI inference scenarios. It reduces TTFT, improves TPS and cache hit ratio, and effectively lowers inference cost.
Supports access through the user-space KV interface, enabling zero-copy data transfer and zero kernel-mode switching between the inference engine and storage, reducing metadata interaction overhead.
Supports affinity deployment with compute resources. Architecturally, KVCacheStore is positioned at the G3.5 layer, delivering performance superior to remote shared storage. A single compute node can provide 40 GB/s throughput and achieve millions of QPS through the Batch interface.
KVCacheStore is deployed independently of compute resources and supports elastic scaling. A single KVCacheStore can accommodate hundreds of billions of KV entries, meeting the KV Cache capacity requirements of mainstream customers in the industry.
Scenarios
KVCache caching for large model inference.
Access interface
Supports the user-space KV interface, including Put/BatchPut, Get/BatchGet, Exist/BatchExist, and Delete/BatchDelete. Enables zero-copy data transfer and zero kernel-mode switching between the inference engine and storage, reducing metadata interaction overhead.
Mount method
Supports mounting on Intelligent Computing Lingjun bare metal nodes.
Specifications
Category | Metric | Specification |
Compute-side access performance | Throughput per client | 40 GB/s. |
IOPS per client | Millions of IOPS for 4 KB random reads. | |
Scale | Maximum capacity per KVCacheStore | Up to 100 PiB. |
Minimum capacity/capacity increment | The minimum capacity is 300 TiB. Capacity can be increased in increments of 300 TiB. | |
Number of KV pairs | 500 billion KV entries. | |
KV performance | KV QPS for inference engines | The BatchPut and BatchGet interfaces can achieve millions of QPS. The BatchExist and BatchDelete interfaces can achieve millions of QPS. |
Storage performance per KVCacheStore | Maximum throughput | Tens of TB/s. |
Maximum IOPS | Hundreds of millions. |