autumn-rs L2 adapter#
An LMCache L2 adapter backed by an autumn-rs cluster: a distributed storage engine whose replicated, ordered KV layer holds the chunks. The cluster is shared, durable and large, so it serves as a warm tier that survives restarts and is shared across LMCache instances. The client reads and writes straight into the L1 buffers with the GIL released; there is no C++ extension to build.
Maintainer |
dongmao zhang (@thesues) |
License |
Apache-2.0 |
LMCache |
>= 0.5.7 (tested with 0.5.7.post1) |
autumn-rs client |
>= 0.1.2 |
Python |
>= 3.11, Linux x86_64 / aarch64 |
Support |
Install#
pip install "lmcache-l2-plugins[autumn] @ git+https://github.com/LMCache/lmcache_l2_plugins"
The autumn extra pulls in LMCache >= 0.5.7 and the autumn-rs client
(autumn-rs on PyPI). The client speaks TCP everywhere. With
"transport": "ucx" it moves chunks over RDMA through the host’s UCX
(>= 1.17, libucp.so.0 on the loader path), which it loads only then.
Over UCX the adapter registers its L1 arena when it is created, so chunks in the arena move straight between L1 and the cluster without a copy. This pins the whole arena up front, so the memlock limit must cover it.
The transport is process-wide: every autumn adapter in one LMCache server process must use the same transport.
Prepare the cluster#
Keys are stored under a namespace, kvc by default, which must exist before
the adapter starts. Create it cut into 16 partitions so chunks spread over all
of them from the start:
autumn-op namespace-create --name kvc --parts 16
Keys are {namespace}/{h}/lmcache/{storage key}, where h is the chunk
hash’s first hex digit, the digit the namespace is cut on when it is created
with --parts 16.
Start the LMCache server#
lmcache server --l1-size-gb 40 --eviction-policy LRU \
--l2-adapter '{
"type": "plugin",
"module_path": "lmcache_l2_plugins.autumn",
"class_name": "AutumnL2Adapter",
"adapter_params": {"manager": "10.0.0.1:9001", "transport": "ucx"}
}'
LMCache finds the config class AutumnL2AdapterConfig by its name; no
config_class_name is needed. LMCache’s own L2 keys (shared, eviction,
serde, persist_enabled) go at the top level of the JSON, beside type;
inside adapter_params they are ignored. LMCache’s logs and metrics label the
adapter plugin; its status reports "type": "autumn".
adapter_params fields:
Field |
Default |
Meaning |
|---|---|---|
|
required |
Comma-separated |
|
|
autumn namespace for the keys. |
|
|
An autumn principal credential file, as |
|
|
Batches queued or in flight (> 0). The client runs gets and puts one batch at a time. |
|
|
|
|
|
Read large values straight from the storage nodes instead of through the partition servers. Needs the client to reach the storage nodes. |
|
|
Expiry of each stored chunk in seconds; |
|
|
Max L2 capacity in GB for usage tracking / aggregate eviction. |
Authorization on, chunks expire after a day, a capacity cap:
--l2-adapter '{"type": "plugin", "module_path": "lmcache_l2_plugins.autumn", "class_name": "AutumnL2Adapter", "adapter_params": {"manager": "10.0.0.1:9001,10.0.0.2:9001", "credential_file": "/etc/autumn/kvc.cred", "ttl_secs": 86400, "max_capacity_gb": 500}}'
Partition-server memory tier#
Partition servers started with --mem-tier-bytes <budget> --mem-tier-prefix kvc/
acknowledge a chunk from DRAM and append it to the replicated log right after,
and serve reads from memory. A partition-server crash loses only chunks whose
append had not landed yet, which a cache can afford.
Tests#
For the real-server CPU e2e against LMCache dev, including a verified L1
clear and L2 readback, see the e2e guide.
From the repository root:
pip install -e ".[autumn,test]"
pytest tests/autumn
The unit tests replace the autumn client with an in-memory fake and need no
cluster. Every adapter in them is built from an --l2-adapter JSON of type
plugin, the way lmcache server builds it. One test runs against a live
cluster when AUTUMN_MANAGER (ip:port) is set; add
AUTUMN_TRANSPORT=ucx to run it over UCX through a registered arena, and
AUTUMN_CREDENTIAL_FILE for a protected namespace.
Validation status, with LMCache 0.5.7.post1 and autumn-rs 0.1.2 on one host:
Check |
Result |
|---|---|
Unit tests |
29 passed, 1 skipped (the live test) |
Live test against a 3-node cluster, TCP |
passed |
Live test against a 3-node cluster, UCX over RoCE |
passed |
|
adapter created, L1 arena registered, clean shutdown |
The same adapter code, built in-tree, was run end to end with vLLM 0.28 and LMCache’s MP connector: Qwen3-VL-32B, 6 documents of ~17K tokens, written by one instance and loaded by a fresh lmcache server and vLLM. All answers were correct, 99.1% of prefix tokens came from L2, and TTFT was 0.30 s over UCX, 1.34 s over TCP, against 2.78 s recomputing. Cross-host runs are not measured.