KVCacheStore:Best practices for integrating the KVCacheStore SDK

Use case 1: Prefix caching for inference KVCache

In large model inference, identical or similar prefixes are repeatedly computed, such as shared system prompts, few-shot examples, and multi-turn conversation histories. This increases GPU computation, Time to First Token (TTFT), and inference costs.

After integrating with KVCacheStore, the inference engine can:

1. Use ll_batch_query to query whether a prefix is a cache hit.

2. On a cache hit, use ll_batch_get_items to read the KVCache and skip the corresponding computation.

3. On a cache miss, compute normally and use ll_batch_put to write back the results.

Deployment approach:

• When the KVCache volume is large with a clear hot/cold distinction, offload historical prefix caches from GPU memory (G1/G2 tier) to KVCacheStore (G3.5 tier), freeing GPU memory for current requests.

• The hit rate directly determines the benefit. We recommend including the model name, version, and prefix content hash in the key, such as model-a/v1/prefix/<hash>. Different models should use separate prefixes to avoid data conflicts.

• The per-token KVCache size can be estimated using the following formula:

  2 × Number of layers × Number of KV heads × Head dimension × Number of bytes per element

  For example, for Qwen2.5-72B at bf16 precision, the KVCache size per token is approximately:

  2 × 80 × 8 × 128 × 2 ≈ 320 KiB/Token

  The total capacity is approximately:

  KVCache size per token × Average cached sequence length × Number of prefixes to cache

• After capacity is full, the least recently used cache entries may be subject to LRU eviction. Therefore, always maintain the complete flow of "query for cache hit → read; cache miss → recompute and write back". For implementation details, see Practice 1: Integrate prefix caching into an inference engine.

Use case 2: Cross-machine KVCache transfer and elastic scaling

During inference cluster scale-out/scale-in, failover, or Prefill/Decode disaggregated deployment, recomputing existing KVCache on new nodes increases latency in the early scale-out phase.

KVCacheStore is deployed independently of compute nodes. Nodes that mount the same KVCacheStore share a single storage pool. KV entries written by one node can be read by other nodes.

Deployment requirements:

• Mount KVCacheStore on all VSCs that may carry inference workloads. Write and read operations are not constrained to specific node locations.

• During scale-out/scale-in, you only need to deploy kvcs-efc on the compute node and install the SDK. No data migration or warm-up tasks are required.

• After task migration, the existing KVCache remains in the storage pool (subject to default LRU eviction). New nodes can read directly by key for cache hits. Missed prefixes are automatically recomputed and written back without manual intervention.

Use case 3: Share a KVCache storage pool among multiple models and applications

Multiple models or applications can share the same KVCacheStore and isolate data through key prefixes. Each application maintains an independent read/write lifecycle without interference. For example:

• model-a/v1/...

• model-b/v2/...

• application-c/...

Deployment approach:

• Isolation is achieved on the application side through key prefixes (such as model-a/... and model-b/...). No additional configuration is required on the KVCacheStore side.

• LowLevel mode enables LRU eviction (GC) by default, and it cannot be disabled: when capacity is full, data is automatically reclaimed in least-recently-used order. Applications do not need to build their own cache management logic. However, avoid mixing workloads with significantly different eviction cycles in a single KVCacheStore, because high-frequency write workloads accelerate the eviction of caches belonging to low-frequency workloads.

• For workloads that require strict isolation, you can create multiple KVCacheStore instances for physical isolation through management API (instances do not share capacity or performance).

Practice 1: Integrate prefix caching into an inference engine

This practice covers the complete closed loop of "cache hit check → retrieve cache → miss-and-recompute with write-back", which is the core pattern for inference engine integration. The example assumes that the application already generates stable keys using the model name, model version, and prefix hash.

from kvcs import KvcsLowLevelClient, KvcsError, LLGetItemOpt, LLPutItem

client = KvcsLowLevelClient(efc_socket="/var/run/kvcs/efc-grpc.sock")

def get_or_compute_prefix_cache(prefix_key: str, compute_fn):
    """Return the cached KVCache on a hit, or call compute_fn to recompute and write it back on a miss."""
    # 1. Check whether the prefix is cached.
    try:
        qr = client.ll_batch_query([prefix_key])
        if qr[0].status == "ok":
            # 2. Cache hit: Retrieve the KVCache and skip the corresponding computation.
            got = client.ll_batch_get_items(
                [LLGetItemOpt(key=prefix_key)], deadline_s=10.0)
            if got[0].status > 0:  # status is the number of bytes returned on a cache hit.
                return got[0].data
    except KvcsError:
        pass  # Treat a batch query/read failure, such as a timeout, as a miss and recompute.

    # 3. Cache miss: Compute as usual and write the result back for subsequent requests.
    kv_cache = compute_fn()
    put_results = client.ll_batch_put(
        [LLPutItem(key=prefix_key, value=kv_cache)], deadline_s=10.0)
    if put_results[0].status != 0:
        # A write failure does not affect this inference because KVCache can be recomputed. Log the failure.
        print("prefix cache write failed:", put_results[0].status)
    return kv_cache

func (p *PrefixCache) GetOrCompute(ctx context.Context, prefixKey string,

    compute func() []byte) ([]byte, error) {


    // 1. Check whether the prefix is a cache hit

    qr, _ := p.client.LLBatchQuery(ctx, []string{prefixKey})

    if len(qr) > 0 && qr[0].Status == "ok" {
        // 2. Cache hit: retrieve the KVCache and skip the corresponding computation

        got, _ := p.client.LLBatchGetItems(ctx, []kvcs.LLGetItemOpt{

            {Key: prefixKey},
        })
            return got[0].Data, nil
            return got[0].Data, nil
        }
    }

    kvCache := compute()
    kvCache := compute()

    putResults, _ := p.client.LLBatchPut(ctx, []kvcs.LLPutItem{


        {Key: prefixKey, Value: [][]byte{kvCache}},

    })
    if len(putResults) > 0 && putResults[0].Status != 0 {
    }
        log.Printf("prefix cache write failed: %d", putResults[0].Status)
    }
    return kvCache, nil
}

use std::time::Duration;

fn get_or_compute_prefix_cache(
    client: &KvcsLowLevelClient,
    prefix_key: &str,
    compute: impl FnOnce() -> Vec<u8>,
) -> Vec<u8> {
    let dl = Some(Duration::from_secs(10));
    let opts = LLBatchOptions::default();

    // 1. Check whether the prefix is cached.
    let qr = match client.ll_batch_query(&[prefix_key], opts) {
        Ok(qr) => qr,
        Err(_) => Vec::new(), // Treat a query failure as a miss and recompute.
    };
    if !qr.is_empty() && qr[0].status == "ok" {
        // 2. Cache hit: Retrieve the KVCache and skip the corresponding computation.
        if let Ok(got) = client.ll_batch_get_items(
            &[LLGetItemOpt { key: prefix_key }], dl, opts) {
            if !got.is_empty() && got[0].status > 0 {
                return got[0].data.clone().unwrap_or_default();
            }
        } // Treat a read failure as a miss and recompute.
    }

    // 3. Cache miss: Compute as usual and write the result back for subsequent requests.
    let kv_cache = compute();
    if let Ok(put_results) = client.ll_batch_put(
        &[LLPutItem { key: prefix_key, value: &[kv_cache.as_slice()] }],
        dl, opts) {
        if !put_results.is_empty() && put_results[0].status != 0 {
            // A write failure does not affect this inference because KVCache can be recomputed. Log the failure.
            eprintln!("prefix cache write failed: {}", put_results[0].status);
        }
    } // A batch write failure also does not affect this inference.
    kv_cache
}

Practice 2: High-throughput reads and buffer reuse

Read-path throughput is a key metric for inference integration. This practice demonstrates the Copy-into + buffer pool read pattern: preallocate buffers once and reuse them across reads to avoid allocating new memory for each read. Expand buffers on demand when insufficient.

from kvcs import KvcsLowLevelClient, KvcsError, LLGetItemOpt

client = KvcsLowLevelClient(
    efc_socket="/var/run/kvcs/efc-grpc.sock",
    max_value_size=16 * 1024 * 1024,   # Configure this based on the maximum value size of the workload.
)

# Buffer pool: Preallocate based on max_value_size and reuse across requests.
buf_pool = [bytearray(16 * 1024 * 1024) for _ in range(256)]

def read_kv_cache(keys):
    bufs = buf_pool[:len(keys)]
    try:
        got = client.ll_batch_get_into_items(
            [LLGetItemOpt(key=k) for k in keys],
            bufs, deadline_s=10.0)
    except (KvcsError, ValueError):
    except (KvcsError, ValueError):
        return [None] * len(keys)

    result = [ ]

    for i, r in enumerate(got):
        if r.status > 0:
        if r.status > 0:
        else:
        else:
    return result

// Buffer pool: Preallocate based on MaxValueSize and reuse across requests.

func (r *KVReader) ReadKVCache(ctx context.Context, keys []string) [][]byte {


    items := make([]kvcs.LLGetItemOpt, len(keys))


    bufs := make([][]byte, len(keys))

    for i, k := range keys {
        items[i] = kvcs.LLGetItemOpt{Key: k}
    }
    }
    into, err := r.client.LLBatchGetIntoItems(ctx, items, bufs)
    if err != nil {
    }
    }

    out := make([][]byte, len(keys))

    for i, it := range into {
        if it.Status == kvcs.OK {
            // the buffer is reused and overwritten by the next call.

            out[i] = append([]byte(nil), bufs[i][:it.Size]...)

        } // Treat a non-OK result as a miss and let the caller fall back.
    }
    return out
}

fn read_kv_cache(client: &KvcsLowLevelClient, keys: &[&str],
                 buffers: &mut [Vec<u8>]) -> Vec<Option<Vec<u8>>> {
    let dl = Some(Duration::from_secs(10));
    let opts = LLBatchOptions::default();

    let items: Vec<LLGetItemOpt> =
        keys.iter().map(|k| LLGetItemOpt { key: k }).collect();

    // Buffer pool: Preallocate based on max_value_size and reuse across requests.
    let mut bufs: Vec<&mut [u8]> =
        buffers.iter_mut().map(|b| b.as_mut_slice()).collect();

    match client.ll_batch_get_into_items(&items, &mut bufs, dl, opts) {
        Ok(results) => results.iter().enumerate().map(|(i, r)| {
            if r.status == Status::Ok {
                Some(buffers[i][..r.size].to_vec())
            } else {
            } else {
            }
        }).collect(),
        }).collect(),
    }
}

Practice 3: Multi-file-system isolation

When the KVCacheStore server uses a pooled-disk (CPFS) backend with multiple file systems mounted, you can use the per-batch option mountpoint_index to specify the target file system for the current batch of operations (0 is the default file system), achieving physical isolation between workloads. This capability takes effect only on pooled-disk backends. For direct-disk backends, this parameter is ignored, and writes go to the default disk without errors.

from kvcs import LLBatchOptions, LLPutItem

# Write to the file system dedicated to model-b (mountpoint_index=1).
results = client.ll_batch_put(
    [LLPutItem(key="model-b/layer-0/prefix-x", value=b"...")],
    deadline_s=10.0,
    opts=LLBatchOptions(mountpoint_index=1),
)

On the Go side, pass this through kvcs.WithMountpointIndex(1). On the Rust side, construct LLBatchOptions { mountpoint_index: 1 }. Only on pooled-disk backends, when the specified index is not configured in the server-side mount points, the batch returns ENODEV.

1. Concurrency

get_workers, set_workers, and simple_workers control the number of workers for read, write, and simple operations (Query/Delete) respectively, with default values of 4, 4, and 2.

Start with the default values and adjust incrementally based on your workload:

• Estimate the number of workers as "peak QPS × average request latency (in seconds)".

• For read-intensive workloads, increase get_workers first.

• For write-intensive workloads, increase set_workers first.

• After increasing the number of workers, reassess shared memory and thread scheduling overhead.

• Adjust only one category of parameters at a time, and record the changes in throughput, latency, and error rate.

2. Batch size and memory configuration

max_keys_per_batch is the capacity parameter for the SDK internal ring and shared memory. When a single submission exceeds this value, the SDK automatically splits it.

max_value_size defaults to 4 MiB, with a maximum of 4 GiB. When a value exceeds the limit, the write result returns E2BIG per key.

Tuning recommendations:

• When values are generally smaller than 1 MiB, keep the default of 4 MiB.

• When values approach or exceed 4 MiB, increase max_value_size and reduce max_keys_per_batch or the number of workers accordingly to avoid exponential growth of shared memory.

• Prepare buffers based on the actual value size distribution. Avoid using the maximum specification for all requests.

3. Performance testing

Use a benchmark tool to verify throughput and latency at your production scale. The kvcs-bench provided in the repository supports multi-node benchmark orchestration. You can submit various worker/batch-size parameter matrices in a single run and aggregate steady-state metrics:

cd kvcs-test/benchmark
./kvcs-bench \
  --environment /path/to/environment.json \
  batch --ops get --workers 2,4,8,16 --batch-sizes 128

The benchmark should cover: typical value sizes, read/write ratios, peak concurrency, and 1.5× peak concurrency. Clean up benchmark data after testing to avoid occupying production capacity.

4. Asynchronous write-back

Practice 1 uses synchronous write-back, which is simple and reliable. In production environments, the write-back data volume after prefill can be large (a single long-sequence request can reach MB to GB level), and waiting synchronously adds storage latency directly to inference latency. Since KVCache is recomputable, a write-back failure does not affect the current inference — it only affects future hit rate. Therefore, asynchronous write-back is recommended for production environments to move write-back off the inference critical path. A correct implementation must satisfy three key requirements:

1. buffer ownership until write-back completes: The SDK holds a zero-copy reference to the value, and the buffer must not be reused before ll_batch_put returns. The inference engine's KV block pool recycles buffers. During write-back, the buffer must remain exclusively owned. The safest approach is to transfer buffer ownership to the write-back thread, which returns the buffer after put completes.

2. bounded queue + drop policy: The write-back queue must have a capacity limit. When backlogged, drop new writes directly — this only reduces the hit rate without affecting correctness. An unbounded queue will exhaust memory during storage jitter.

3. per-key status tracked in monitoring: After switching to asynchronous mode, write results have no synchronous consumer. Failures accumulate silently. You must feed write failure rates and drop rates into monitoring (see runtime metrics). Otherwise, a decline in hit rate will be impossible to diagnose.

Python example (covering all four aspects: buffer handoff, bounded queue, drop policy, and counting):

import queue
import threading

write_queue = queue.Queue(maxsize=64)  # Bounded queue: maximum backpressure.
stats = {"ok": 0, "fail": 0}

def writer_main():
    """Use a dedicated write-back thread so that put does not occupy an inference thread."""
    while True:
        item = write_queue.get()
        if item is None:  # Exit signal.
            return
        key, kv_cache = item
        results = client.ll_batch_put(
            [LLPutItem(key=key, value=kv_cache)], deadline_s=1.0)
        if results[0].status == 0:
            stats["ok"] += 1
        else:
            stats["fail"] += 1  # Monitor failures because asynchronous writes have no synchronous consumer.
        # After put returns, the kv_cache reference can be safely released.

threading.Thread(target=writer_main, daemon=True).start()

def put_async(key: str, kv_cache: bytes):
def put_async(key: str, kv_cache: bytes):
    try:
        write_queue.put_nowait((key, kv_cache))
    except queue.Full:
        pass

For small KV entries, prototyping phases, or when there is a natural scheduling gap between prefill and decode, keep the synchronous write-back from Practice 1. There is no need to force asynchronous mode. For high-concurrency, long-sequence production scenarios, asynchronous write-back is recommended.

5. Runtime metrics

Enable enable_metrics during initialization and use get_metrics() / get_metrics_text() (or GetMetrics() / GetMetricsText() in Go) to obtain SDK runtime metrics for validating tuning results and production monitoring. Metrics include: success/cache miss/error counts by operation type (put/get/delete/exists), cumulative key counts and byte counts, batch latency distribution, in-flight batch count and in-flight key count, and client-side latency breakdown by phase.