# GPU server

> Run the server on Linux with one NVIDIA card; the default base is Gemma 4 31B, which needs a 96 GB card.

Source: https://opendecisio.com/docs/quickstart/gpu-server/

One NVIDIA card, Linux and four commands start the measured system. The first start downloads the checkpoint, so it takes minutes, not seconds.

 

[Figure: Install, download, warm up: the first start is minutes, and the weights are most of it. Measured on one RTX PRO 6000 at 600 W with decisio 0.11.0 from PyPI: pip install "decisio[serve]" 2 min 4 s; 31B, first start (with download) 7 min 54 s; 31B, weights on disk 4 min 3 s; 12B, first start (with download) 4 min 12 s.]

 

## TL;DR

 

```
git clone https://github.com/aminry/decisio
cd decisio
uv sync --extra serve --frozen
uv run python -m decisio.serve.vllm_engine
```

 

Needs Linux with one NVIDIA card: taken verbatim from the repository's tested docs.

 

The server listens on `http://127.0.0.1:8000`; `GET /health` answers once it is ready, and [Get started](https://opendecisio.com/docs/) sends a first question to it.

  

| Word | Meaning |
| --- | --- |
| base | the model the server serves; with no flag, Gemma 4 31B |
| checkpoint | the model’s weights, downloaded into the Hugging Face cache on the first start |
| FP8 | the 8-bit floating-point format the 31B’s weights are stored in, 30.6 GiB in memory |
| uv | the Python package manager the commands use |

 

Common mistake

The 31B is measured on, and needs, a 96 GB card. On a smaller card pick another base with `--base`: [Choosing a base](https://opendecisio.com/docs/bases/) says which fits what.

 

## The requirements and the flags

 

Requirements: Linux, one NVIDIA card (measured on an RTX PRO 6000 Blackwell with 96 GB), a driver that supports CUDA 13.0 (the runtime `uv.lock` pins), Python 3.12 and [uv](https://docs.astral.sh/uv/).

 

```
git clone https://github.com/aminry/decisio
cd decisio
uv sync --extra serve --frozen
uv run python -m decisio.serve.vllm_engine
```

 

Needs Linux with one NVIDIA card: taken verbatim from the repository's tested docs.

 

- With no flag the server serves Gemma 4 31B (`--base gemma-4-31b`), the default base since 0.10.0, from its FP8 repository `tachara-ai/decisio-gemma-4-31b`; it needs the 96 GB card. `--model google/gemma-4-31B-it` serves Google's weights at the pinned revision instead, quantized to FP8 each time they load (`--revision` does the same, naming a revision of Google's checkpoint).
- A third party served the 31B on a smaller card than ours: JevBench's maintainer ran decisio 0.8.0 with `--base gemma-4-31b` (Google's weights, quantized to FP8 on load) on one H100 80 GB (Lium) and "no flag of theirs was changed to make it fit". That is the board's own account of its run of 2026-10-06 on the v1.6.0/v1.6.1 pool, in the row `decisio-gemma-4-31b-v080` (JevBench v1.6.1, `https://benchmarkheaven.com/api/jevbench/v1.6.1`); we did not run it, and nothing here changes what we state: the 31B is measured on, and needs, a 96 GB card. The row also notes that 23 of the pool's items, of about 80,000 tokens, are over the 32,768-token context.
- `--base qwen3.6-35b-a3b` serves [`Qwen/Qwen3.6-35B-A3B-FP8`](https://huggingface.co/Qwen/Qwen3.6-35B-A3B-FP8) and `--base gemma-4-12b` serves [`google/gemma-4-12B-it`](https://huggingface.co/google/gemma-4-12B-it), each at its pinned revision and with its own settings (the README's "Choosing a base"; [`docs/cli.md`](https://opendecisio.com/docs/flags/) lists every setting per base).
- With a `--model` and no `--base`, the base is detected from the checkpoint's `config.json`, so a local copy of any base's checkpoint brings its own settings; a base's pinned revision applies whenever `--model` names the base's own Hugging Face repository and `--revision` is not given.
- The first start downloads the checkpoint into the Hugging Face cache and warms the engine: about 31 GB for the 31B's FP8 repository (62 GB for Google's bf16 weights) and about 36 GB for the Qwen base ([`runs/2026-10-01_docker-first-gpu-start/`](https://github.com/aminry/decisio/tree/v0.11.1/runs/2026-10-01_docker-first-gpu-start)); `/health` answers once it is ready.
- Upgrading from a release before 0.10.0, where the container served the Qwen base: [`docs/upgrading.md`](https://opendecisio.com/docs/upgrading/).
- The decisio plugin registers its model classes with vLLM through an entry point, so vLLM 0.30.0 loads them without any patch; the package must be installed (as `uv sync` does) for vLLM's engine processes to find it ([`docs/design/vllm-plugin.md`](https://github.com/aminry/decisio/blob/v0.11.1/docs/design/vllm-plugin.md)).
- [`patches/`](https://github.com/aminry/decisio/tree/v0.11.1/patches) holds an optional latency patch series, off by default and not needed for correct answers ([`patches/README.md`](https://github.com/aminry/decisio/blob/v0.11.1/patches/README.md)).
- The server refuses to start when `VLLM_USE_DEEP_GEMM` is set to anything other than `0`, unless `--allow-deep-gemm` is given.
- It listens on `127.0.0.1:8000` (`--host`, `--port`); put a reverse proxy in front of it to expose it.
- `--image-model` adds a second engine on the same card for requests that carry images; [`EVAL_CARD.md`](https://opendecisio.com/docs/evaluation-card/) section 1 has the memory shares it was measured with.
- vLLM's engine runs in the server's process by default; with a second engine (`--image-model`, `--head-engine`, `--one-engine`) it runs in a process of its own (`--engine-process`, [`docs/cli.md`](https://opendecisio.com/docs/flags/)).
- A later, different question about a state read before, sent in its own request, reads the state from the prefix cache; on the Gemma bases that needs the state's boundary registered, and `--register-boundary` sets when (below).
- vLLM's `prefix_cache_retention_interval` would also keep the boundary, but it keeps every sliding-window block of each state, and with the cache filled to 1.3 times its pool the earliest states lost even their repeated question, where the default (0) kept them at twice the pool; it stays at the default. The Qwen base needs neither: its state is padded to end on its 1,056-token block, the latest checkpoint of a request with one question. `/health` reports `cache_hit_unit`, `hash_unit`, `registers_state_boundary` and `register_boundary`.

 

Per-request latency depends on the host's CPU and on the card's power limit, as well as the card.

 

- The CPU: with the engine in the server's process, a Qwen question on a cached state took 39.4 ms of server time on an AMD EPYC 7452 (Zen 2) host, 18.5 ms on an AMD Ryzen 9 9950X (Zen 5) and 20.2 ms on an AMD Ryzen Threadripper 9960X, with the same card model and vLLM ([`runs/2026-10-05_engine-death-gates/`](https://github.com/aminry/decisio/tree/v0.11.1/runs/2026-10-05_engine-death-gates), [`runs/2026-10-06_latency-0.8.1/`](https://github.com/aminry/decisio/tree/v0.11.1/runs/2026-10-06_latency-0.8.1), [`runs/2026-10-06_latency-585w/`](https://github.com/aminry/decisio/tree/v0.11.1/runs/2026-10-06_latency-585w)). The Gemma bases differed far less between hosts (Gemma 4 12B: 28.0, 23.4 and 24.8 ms).
- The power limit: first reads on the Gemma bases are bound by the card. With the card's enforced limit at 585 W (default 600 W), the software power cap held the clock back on 72 to 92% of busy samples on the Gemma bases, single requests included, and on 0 to 2% on the Qwen base ([`runs/2026-10-06_latency-585w/`](https://github.com/aminry/decisio/tree/v0.11.1/runs/2026-10-06_latency-585w)); at 400 W their first reads were slower still ([`runs/2026-10-06_latency-0.8.1/`](https://github.com/aminry/decisio/tree/v0.11.1/runs/2026-10-06_latency-0.8.1)). The latency figures in the README and [`EVAL_CARD.md`](https://opendecisio.com/docs/evaluation-card/) are from the 585 W session.

 

Quote latency with the host's CPU model and the card's power limit beside it (`nvidia-smi -q -d POWER`).

 

## A later question about the same state (`--register-boundary`)

 

On the Gemma bases vLLM keeps only a finished request's latest sliding-window checkpoint, which lies inside its question. So the server registers each state's boundary with the warm-up that multi-question requests already send: the state and one token. Without it, a later, different question about the state read the whole state again, which costs about a first read of it: on the 31B at 3,000 tokens, 439.0 ms for the first read against 43.3 ms for a different question read from the cache (one card at 585 W, an AMD Ryzen Threadripper 9960X; [`runs/2026-10-06_latency-585w/`](https://github.com/aminry/decisio/tree/v0.11.1/runs/2026-10-06_latency-585w)). The one measurement of the old behaviour at 31,000 tokens is from before 0.8.1, on another host: 9.2 s instead of 0.17 (531 ms instead of 47 at 3,000 tokens there). The server remembers the last 4,096 states it registered and sends the warm-up again only when a request finds the boundary gone, so a repeated question costs no extra engine call. Several questions in one request are not affected, and the Qwen base needs no registration (its padded state already ends on a single question's latest checkpoint).

 

`--register-boundary` sets when a single question on a new state registers it:

 

- `after`: the question is answered first, reading the state fresh, and the warm-up is sent once the response has been handed to the server (in library use, once `answer()` returns). The caller does not wait for it.
- `before` (the default of both Gemma bases since 0.11.1; the 12B's was `after` from 0.9.0 to 0.11.0): the warm-up goes ahead of the question, in the caller's request, as in 0.8.1. On one card at 585 W with an AMD Ryzen Threadripper 9960X that added +16, +25 and +28 ms to a first read at 300, 1,000 and 3,000 tokens on Gemma 4 12B, and +21, +34 and +34 ms on Gemma 4 31B ([`runs/2026-10-06_latency-585w/`](https://github.com/aminry/decisio/tree/v0.11.1/runs/2026-10-06_latency-585w), and `register_boundary.md` there for every figure below).
- `off`: never; a later, different question about a state reads it again, as before 0.8.1. For traffic that asks one question per state, where a registration would only cost throughput.

 

Why the 12B's default moved from `after` to `before` in 0.11.1: Lab 2's grid of 2026-10-09 (12B, `after`, AMD EPYC 9534, RTX PRO 6000 at 600 W, decisio 0.10.0; RLCD `experiments/2026-10-08_lab2_latency_grid/results/analysis_12b_one_question_after.md`) found that one question on each new state, sent back to back, took 774 ms at 3,000 tokens instead of about 390 and 1.6 s instead of 0.8 at 6,000, because each request waited behind the previous state's registration (385 and 796 ms outside the engine's stages, one read of the state); `batch` mode, three-question requests and the 31B's `before` showed no gap. `before` reads the state once plus one token inside the request, so nothing is pending behind a response. Lab 2 then confirmed it directly (2026-10-09, one server on decisio 0.10.0, RLCD `experiments/2026-10-08_lab2_latency_grid/results/box/confirm_12b/`): one question per new state, ten states, server time of the first request and the median of the nine later ones. Back to back, `after` took 378 and 771 ms at 3,000 tokens and 812 and 1,613 ms at 6,000; `before` 413 and 417 ms and 834 and 834 ms; `off` 383 and 395 ms and 816 and 818 ms. With the client waiting for the registrar between states, `after` took 396 and 396 ms and 818 and 827 ms. So the doubling is only under `after` and only back to back; `before` costs 22 ms (3,000 tokens) and 17 ms (6,000) over `off`. `off` shows the cost of no registration and nothing else: it is the baseline the other two are read against, and it does not show how a later, different question fares, since without a registration that question reads the state again. The response's `timing.state_boundary.ran_before` was 0 on every request of every order: it counts only a registration the request sends itself, and no field records the wait for the registrar's background warm-up.

 

Why `after` is still there: on the same card, with the queue idle, `after` took the registration off the caller's first read on both bases (12B 33.3, 90.3 and 237.1 ms against 56.7, 107.7 and 261.9 at 300, 1,000 and 3,000 tokens; 31B 52.0, 149.2 and 415.5 against 78.6, 157.2 and 435.4). Under load, with 32 and 64 clients sending first and follow-up questions about new 1,000-token states, the 12B kept 0.99 and 0.98 of the throughput it had with `before`, but the 31B kept only 0.61 and 0.74, with p95 1.69 and 1.62 times; so the 31B registers before the question.

 

What `after` guarantees, and its limits:

 

- A follow-up question sent after the first answer reads the state from the cache. A registration that is due goes before every later request's questions: a background thread sends it as soon as the engine is free, and a request that takes the engine first sends it ahead of its own questions.
- The bound on that follow-up: it may wait, at most for the engine call already in progress and then the warm-up, which reads the whole state again, since vLLM kept no checkpoint at the boundary. The warm-up costs about one first read of the state: 34, 88 and 238 ms at 300, 1,000 and 3,000 tokens on Gemma 4 12B, and 52, 146 and 407 ms on Gemma 4 31B (the reads without registration). Measured on 3,000-token states, a follow-up read the state from the cache on 20 of 20 at 0, 50, 200 and 1,000 ms after the answer, and took 267, 222, 70 and 50 ms on the 12B (441, 392, 244 and 46 ms on the 31B), against 34 to 46 ms with `before`.
- A second question sent before the first answer is out sends the warm-up itself, ahead of its question, as `before` does.
- A follow-up reads the state again only when its registration was dropped (more than 256 pending, the oldest dropped first), when its warm-up failed, or when the state was evicted from the cache since. `/health` reports `state_boundary`: the registrations pending, and how many were deferred, registered, dropped and failed.
- The cost moves from latency to throughput: a new state is read twice on the card, once for the answer and once for the warm-up, where `before` read it once plus one token. A request about another state that arrives during a warm-up waits for it, so one client sending new states back to back pays the previous state's warm-up: on the 12B it saved 4.9 and 22.1 ms at 300 and 1,000 tokens and lost 36.5 ms at 3,000.
- The cache can hold fewer states under `after`: on the 31B, with states filling 1.3 times its reported pool, the 8 earliest were evicted under `after` and kept under `before`, and at 32 clients 82 of 96 follow-ups found their state against 96 of 96; the 12B kept the 8 earliest under both orders.
- When the process ends (a library caller returning, or the server stopping), the warm-up in progress is waited for, at most 60 s, and the pending ones are dropped: an interpreter that shut down under a warm-up aborted the process on a card.
- The first answer about a new state is a fresh read again. On Gemma 4 12B a fresh read and a cached read of the same question can differ, by up to 0.113, and one near-tied choice of 60 new states changed ([`EVAL_CARD.md`](https://opendecisio.com/docs/evaluation-card/) 6.5), so with `after` a question asked again can differ from its first answer by that much; with `before` the two are equal. On Gemma 4 31B the two reads were identical.
- Each request's `timing.state_boundary` shows what it did: `registered` (warm-ups sent ahead of its question), `found` (states already registered), `deferred` (warm-ups sent after its response), and `ran_before` and `ran_before_ms` (other requests' due warm-ups it sent first, and their time). `ran_before` counts only a registration that request sends itself. The time a request waited for the engine, behind another request or behind a background registration, is `timing.waited_ms` (`wait=` in the `x-decisio-stages` header), and `timing.waited_behind_registration` (the header `x-decisio-wait-behind: registration`) says when a registration was running as it arrived.

 

## What next

 

- [Choose your machine](https://opendecisio.com/docs/quickstart/): the other four paths.
- [Flags and options](https://opendecisio.com/docs/flags/): every flag and its default per base.
- [Health and monitoring](https://opendecisio.com/docs/health/): what `/health` says once it is up.