GPU server

Run the server on Linux with one NVIDIA card; the default base is Gemma 4 31B, which needs a 96 GB card.

One NVIDIA card, Linux and four commands start the measured system. The first start downloads the checkpoint, so it takes minutes, not seconds.

Figure · The first start

Install, download, warm up: the first start is minutes, and the weights are most of it.

The 31B's FP8 repository is about 31 GB to download (62 GB for Google's bf16 weights) and the Qwen base about 36 GB. These are one session's times, with the network that session had.

Source: the README's "On a GPU, with pip" and docs/running.md at the pinned release (Lab 2, 2026-10-09; AMD EPYC 9534).

TL;DR

git clone https://github.com/aminry/decisio
cd decisio
uv sync --extra serve --frozen
uv run python -m decisio.serve.vllm_engine

Needs Linux with one NVIDIA card: taken verbatim from the repository's tested docs.

The server listens on http://127.0.0.1:8000; GET /health answers once it is ready, and Get started sends a first question to it.

Word	Meaning
base	the model the server serves; with no flag, Gemma 4 31B
checkpoint	the model’s weights, downloaded into the Hugging Face cache on the first start
FP8	the 8-bit floating-point format the 31B’s weights are stored in, 30.6 GiB in memory
uv	the Python package manager the commands use

The requirements and the flags

Requirements: Linux, one NVIDIA card (measured on an RTX PRO 6000 Blackwell with 96 GB), a driver that supports CUDA 13.0 (the runtime uv.lock pins), Python 3.12 and uv.

git clone https://github.com/aminry/decisio
cd decisio
uv sync --extra serve --frozen
uv run python -m decisio.serve.vllm_engine

Needs Linux with one NVIDIA card: taken verbatim from the repository's tested docs.

With no flag the server serves Gemma 4 31B (--base gemma-4-31b), the default base since 0.10.0, from its FP8 repository tachara-ai/decisio-gemma-4-31b; it needs the 96 GB card. --model google/gemma-4-31B-it serves Google's weights at the pinned revision instead, quantized to FP8 each time they load (--revision does the same, naming a revision of Google's checkpoint).
A third party served the 31B on a smaller card than ours: JevBench's maintainer ran decisio 0.8.0 with --base gemma-4-31b (Google's weights, quantized to FP8 on load) on one H100 80 GB (Lium) and "no flag of theirs was changed to make it fit". That is the board's own account of its run of 2026-10-06 on the v1.6.0/v1.6.1 pool, in the row decisio-gemma-4-31b-v080 (JevBench v1.6.1, https://benchmarkheaven.com/api/jevbench/v1.6.1); we did not run it, and nothing here changes what we state: the 31B is measured on, and needs, a 96 GB card. The row also notes that 23 of the pool's items, of about 80,000 tokens, are over the 32,768-token context.
--base qwen3.6-35b-a3b serves Qwen/Qwen3.6-35B-A3B-FP8 and --base gemma-4-12b serves google/gemma-4-12B-it, each at its pinned revision and with its own settings (the README's "Choosing a base"; docs/cli.md lists every setting per base).
With a --model and no --base, the base is detected from the checkpoint's config.json, so a local copy of any base's checkpoint brings its own settings; a base's pinned revision applies whenever --model names the base's own Hugging Face repository and --revision is not given.
The first start downloads the checkpoint into the Hugging Face cache and warms the engine: about 31 GB for the 31B's FP8 repository (62 GB for Google's bf16 weights) and about 36 GB for the Qwen base (runs/2026-10-01_docker-first-gpu-start/); /health answers once it is ready.
Upgrading from a release before 0.10.0, where the container served the Qwen base: docs/upgrading.md.
The decisio plugin registers its model classes with vLLM through an entry point, so vLLM 0.30.0 loads them without any patch; the package must be installed (as uv sync does) for vLLM's engine processes to find it (docs/design/vllm-plugin.md).
patches/ holds an optional latency patch series, off by default and not needed for correct answers (patches/README.md).
The server refuses to start when VLLM_USE_DEEP_GEMM is set to anything other than 0, unless --allow-deep-gemm is given.
It listens on 127.0.0.1:8000 (--host, --port); put a reverse proxy in front of it to expose it.
--image-model adds a second engine on the same card for requests that carry images; EVAL_CARD.md section 1 has the memory shares it was measured with.
vLLM's engine runs in the server's process by default; with a second engine (--image-model, --head-engine, --one-engine) it runs in a process of its own (--engine-process, docs/cli.md).
A later, different question about a state read before, sent in its own request, reads the state from the prefix cache; on the Gemma bases that needs the state's boundary registered, and --register-boundary sets when (below).
vLLM's prefix_cache_retention_interval would also keep the boundary, but it keeps every sliding-window block of each state, and with the cache filled to 1.3 times its pool the earliest states lost even their repeated question, where the default (0) kept them at twice the pool; it stays at the default. The Qwen base needs neither: its state is padded to end on its 1,056-token block, the latest checkpoint of a request with one question. /health reports cache_hit_unit, hash_unit, registers_state_boundary and register_boundary.

Per-request latency depends on the host's CPU and on the card's power limit, as well as the card.

The CPU: with the engine in the server's process, a Qwen question on a cached state took 39.4 ms of server time on an AMD EPYC 7452 (Zen 2) host, 18.5 ms on an AMD Ryzen 9 9950X (Zen 5) and 20.2 ms on an AMD Ryzen Threadripper 9960X, with the same card model and vLLM (runs/2026-10-05_engine-death-gates/, runs/2026-10-06_latency-0.8.1/, runs/2026-10-06_latency-585w/). The Gemma bases differed far less between hosts (Gemma 4 12B: 28.0, 23.4 and 24.8 ms).
The power limit: first reads on the Gemma bases are bound by the card. With the card's enforced limit at 585 W (default 600 W), the software power cap held the clock back on 72 to 92% of busy samples on the Gemma bases, single requests included, and on 0 to 2% on the Qwen base (runs/2026-10-06_latency-585w/); at 400 W their first reads were slower still (runs/2026-10-06_latency-0.8.1/). The latency figures in the README and EVAL_CARD.md are from the 585 W session.

Quote latency with the host's CPU model and the card's power limit beside it (nvidia-smi -q -d POWER).

A later question about the same state (`--register-boundary`)

On the Gemma bases vLLM keeps only a finished request's latest sliding-window checkpoint, which lies inside its question. So the server registers each state's boundary with the warm-up that multi-question requests already send: the state and one token. Without it, a later, different question about the state read the whole state again, which costs about a first read of it: on the 31B at 3,000 tokens, 439.0 ms for the first read against 43.3 ms for a different question read from the cache (one card at 585 W, an AMD Ryzen Threadripper 9960X; runs/2026-10-06_latency-585w/). The one measurement of the old behaviour at 31,000 tokens is from before 0.8.1, on another host: 9.2 s instead of 0.17 (531 ms instead of 47 at 3,000 tokens there). The server remembers the last 4,096 states it registered and sends the warm-up again only when a request finds the boundary gone, so a repeated question costs no extra engine call. Several questions in one request are not affected, and the Qwen base needs no registration (its padded state already ends on a single question's latest checkpoint).

--register-boundary sets when a single question on a new state registers it:

after: the question is answered first, reading the state fresh, and the warm-up is sent once the response has been handed to the server (in library use, once answer() returns). The caller does not wait for it.
before (the default of both Gemma bases since 0.11.1; the 12B's was after from 0.9.0 to 0.11.0): the warm-up goes ahead of the question, in the caller's request, as in 0.8.1. On one card at 585 W with an AMD Ryzen Threadripper 9960X that added +16, +25 and +28 ms to a first read at 300, 1,000 and 3,000 tokens on Gemma 4 12B, and +21, +34 and +34 ms on Gemma 4 31B (runs/2026-10-06_latency-585w/, and register_boundary.md there for every figure below).
off: never; a later, different question about a state reads it again, as before 0.8.1. For traffic that asks one question per state, where a registration would only cost throughput.

Why the 12B's default moved from after to before in 0.11.1: Lab 2's grid of 2026-10-09 (12B, after, AMD EPYC 9534, RTX PRO 6000 at 600 W, decisio 0.10.0; RLCD experiments/2026-10-08_lab2_latency_grid/results/analysis_12b_one_question_after.md) found that one question on each new state, sent back to back, took 774 ms at 3,000 tokens instead of about 390 and 1.6 s instead of 0.8 at 6,000, because each request waited behind the previous state's registration (385 and 796 ms outside the engine's stages, one read of the state); batch mode, three-question requests and the 31B's before showed no gap. before reads the state once plus one token inside the request, so nothing is pending behind a response. Lab 2 then confirmed it directly (2026-10-09, one server on decisio 0.10.0, RLCD experiments/2026-10-08_lab2_latency_grid/results/box/confirm_12b/): one question per new state, ten states, server time of the first request and the median of the nine later ones. Back to back, after took 378 and 771 ms at 3,000 tokens and 812 and 1,613 ms at 6,000; before 413 and 417 ms and 834 and 834 ms; off 383 and 395 ms and 816 and 818 ms. With the client waiting for the registrar between states, after took 396 and 396 ms and 818 and 827 ms. So the doubling is only under after and only back to back; before costs 22 ms (3,000 tokens) and 17 ms (6,000) over off. off shows the cost of no registration and nothing else: it is the baseline the other two are read against, and it does not show how a later, different question fares, since without a registration that question reads the state again. The response's timing.state_boundary.ran_before was 0 on every request of every order: it counts only a registration the request sends itself, and no field records the wait for the registrar's background warm-up.

Why after is still there: on the same card, with the queue idle, after took the registration off the caller's first read on both bases (12B 33.3, 90.3 and 237.1 ms against 56.7, 107.7 and 261.9 at 300, 1,000 and 3,000 tokens; 31B 52.0, 149.2 and 415.5 against 78.6, 157.2 and 435.4). Under load, with 32 and 64 clients sending first and follow-up questions about new 1,000-token states, the 12B kept 0.99 and 0.98 of the throughput it had with before, but the 31B kept only 0.61 and 0.74, with p95 1.69 and 1.62 times; so the 31B registers before the question.

What after guarantees, and its limits:

A follow-up question sent after the first answer reads the state from the cache. A registration that is due goes before every later request's questions: a background thread sends it as soon as the engine is free, and a request that takes the engine first sends it ahead of its own questions.
The bound on that follow-up: it may wait, at most for the engine call already in progress and then the warm-up, which reads the whole state again, since vLLM kept no checkpoint at the boundary. The warm-up costs about one first read of the state: 34, 88 and 238 ms at 300, 1,000 and 3,000 tokens on Gemma 4 12B, and 52, 146 and 407 ms on Gemma 4 31B (the reads without registration). Measured on 3,000-token states, a follow-up read the state from the cache on 20 of 20 at 0, 50, 200 and 1,000 ms after the answer, and took 267, 222, 70 and 50 ms on the 12B (441, 392, 244 and 46 ms on the 31B), against 34 to 46 ms with before.
A second question sent before the first answer is out sends the warm-up itself, ahead of its question, as before does.
A follow-up reads the state again only when its registration was dropped (more than 256 pending, the oldest dropped first), when its warm-up failed, or when the state was evicted from the cache since. /health reports state_boundary: the registrations pending, and how many were deferred, registered, dropped and failed.
The cost moves from latency to throughput: a new state is read twice on the card, once for the answer and once for the warm-up, where before read it once plus one token. A request about another state that arrives during a warm-up waits for it, so one client sending new states back to back pays the previous state's warm-up: on the 12B it saved 4.9 and 22.1 ms at 300 and 1,000 tokens and lost 36.5 ms at 3,000.
The cache can hold fewer states under after: on the 31B, with states filling 1.3 times its reported pool, the 8 earliest were evicted under after and kept under before, and at 32 clients 82 of 96 follow-ups found their state against 96 of 96; the 12B kept the 8 earliest under both orders.
When the process ends (a library caller returning, or the server stopping), the warm-up in progress is waited for, at most 60 s, and the pending ones are dropped: an interpreter that shut down under a warm-up aborted the process on a card.
The first answer about a new state is a fresh read again. On Gemma 4 12B a fresh read and a cached read of the same question can differ, by up to 0.113, and one near-tied choice of 60 new states changed (EVAL_CARD.md 6.5), so with after a question asked again can differ from its first answer by that much; with before the two are equal. On Gemma 4 31B the two reads were identical.
Each request's timing.state_boundary shows what it did: registered (warm-ups sent ahead of its question), found (states already registered), deferred (warm-ups sent after its response), and ran_before and ran_before_ms (other requests' due warm-ups it sent first, and their time). ran_before counts only a registration that request sends itself. The time a request waited for the engine, behind another request or behind a background registration, is timing.waited_ms (wait= in the x-decisio-stages header), and timing.waited_behind_registration (the header x-decisio-wait-behind: registration) says when a registration was running as it arrived.

What next

Choose your machine: the other four paths.
Flags and options: every flag and its default per base.
Health and monitoring: what /health says once it is up.