Skip to content
Decisio

DocsRun it

7 min read · updated

Health and monitoring

What /health reports, what each response carries, and what a dead engine looks like.

A server tells you what it is serving and what each request cost; nothing needs a separate agent. Two things are worth watching: /health, and the headers of the responses.

Figure · What to watch

/health says what is being served; the headers say what each request cost.

Once the engine has died, /health answers 503 with the reason.

Source: docs/api.md "GET /health" and "Response headers" at the pinned release.

TL;DR

Section titled “TL;DR”

With a server running (Get started starts one), check what it serves, then ask one question and print the headers that say what it cost:

curl -s http://127.0.0.1:8000/health; echo
curl -s -D - -o /dev/null http://127.0.0.1:8000/v1/systemone \
  -H 'Content-Type: application/json' \
  -d '{"state": "Both charges are 49.00 EUR, same day. Please refund one.", "questions": {"urgent": {"type": "noul"}}}' \
  | grep -i '^x-decisio'
# x-decisio-server-ms: the server's time for the request
# x-decisio-route: the route that served it (text or image)
# x-decisio-stages: where the engine's time went

Run against the CPU stand-in at v0.11.1 on 2026-10-14.

WordMeaning
/healthone object a run record can quote: the engine, the base, its checkpoint and revision, the temperatures and the prompt
server timex-decisio-server-ms, without the network
stagesx-decisio-stages: prepare, warm, questions, readout, engine and wait; engine + wait is the server time within a few ms
exit code 70what a server exits with after its engine has died

1. What /health says

Section titled “1. What /health says”

What the server is serving, in one object a run record can quote:

  • ok, and the engine's facts: the vLLM version, the GPU, the mode, the padding (pad_unit, pad_where), the quantization and the cache data types.
  • block_size and match_unit, the KV cache's block and the unit a prefix match is counted in; cache_hit_unit, the step prefix-cache hits come in, and hash_unit, the step prefixes are hashed at (on the Gemma 4 12B base, whose cache keeps groups of 16- and 64-token blocks, hits come in 64-token steps; 32 on the 31B; 1,056 on the Qwen base); registers_state_boundary, whether a single question registers its state's boundary in the cache for the next question, and register_boundary, when (after the response or before the question; docs/running.md); with after, state_boundary counts the registrations pending, deferred, registered, dropped and failed.
  • base and profile: the base, the checkpoint and its revision, where vLLM's engine runs (engine_process), the temperature for each question type, and the prompt (layout, answer position, label forms, system turn, yes/no and option rendering, multi-question scoring, padding).
  • prompt_format: the prompt's layout settings.
  • systemone: the System One route's settings (rendering rules, noul_commit, abstention and its tasks, orders, tasks and which are applied, the temperatures).
  • head_engine, the intent head's hidden-state reader (the serving engine by default, a second engine with --head-engine), and image_engine when one is running.

Once the engine has died, /health answers 503 with {"ok": false, "engine": "dead", "reason": "...", "exit_code": 70}, every other route answers 503 with the reason, and the server exits with code 70 (docs/running.md, "When the engine dies").

2. What each response says

Section titled “2. What each response says”
{"model": "decisio-gemma-4-31b-it-letters",
 "answers": {
   "urgent":   {"type": "noul", "noul": ...},
   "category": {"type": "choice", "choice": "access", "confidence": ..., "probabilities": {"billing": ..., "access": ..., ...}},
   "impact":   {"type": "score", "score": ..., "confidence": ..., "legend": {"0": "...", ...}, "probabilities": {"0": ..., ...}}},
 "usage": {"input_tokens": ..., "output_tokens": ...}}

A shape to fill in, not a command to run.

Answer fieldMeaning
noulthe probability of yes
choicethe most probable option's key; among exactly tied options, the key that sorts first in Unicode code-point order, so the answer never depends on the order the keys arrive in
scorethe probability-weighted mean level
probabilitiesthe distribution over the options (choice) or the levels by index (score)
confidenceSystem One's confidence: for a choice, the top probability rescaled from uniform; for a score, the concentration at the mode
legendeach level's index and description
unknown_probability, abstainedwith imajev's abstention extension (--abstain-option): the probability of the offered "can't tell" option and whether the server abstained

usage.input_tokens counts the state and each question's own text; usage.output_tokens is one per question.

Response headers:

  • x-decisio-server-ms: the server's time for the request.
  • x-decisio-route: the route that served it (text or image).
  • x-decisio-tasks: the registered tasks applied, when any were.
  • x-decisio-stages: where the engine's time went, for example prepare=...;warm=...;questions=...;readout=...;engine=...;wait=.... engine is the engine call itself; wait is the time before it, from the request reaching the engine until it took the engine's lock, so engine + wait is the server time within the request's own overhead (a few ms). The wait is 0 on an idle server.
  • x-decisio-wait-behind: registration when a boundary registration (--register-boundary after) was running when the request arrived, so the wait was behind it; absent otherwise. On POST /v1/answer the same facts are timing.waited_ms and timing.waited_behind_registration.

3. When the engine dies

Section titled “3. When the engine dies”

An engine whose forward pass fails (a CUDA error, an out-of-memory error) or whose engine-core process exits does not serve again, so the server ends itself and leaves the restart to whatever runs it.

A death is confirmed before it is declared, so that a request which trips a bug cannot take a healthy server down:

  • vLLM's own word is enough: its EngineDeadError, or its flag for an engine-core process that is gone.
  • Any other unexpected error fails that request with 500, as it always has, and the server then sends the engine one probe: a forward pass over a one-token prompt, given 5 seconds, or three times the engine's slowest recent call when that is longer, so that a slow engine is not taken for a dead one (the server times one probe per engine at start-up, so its first request already allows for a slow engine).
  • If the probe answers, the engine is alive and the server stays up; the server log says so (the engine answered a probe in ... ms).
  • If the probe fails or does not answer in that time, the engine is dead. An in-process engine that has failed hangs rather than answering, so this is how its death shows.
  • A probe still waiting when the engine is declared dead another way (vLLM's flag, another request) stops at once, and its request gets its 503.
  • An error caused by the request (a malformed question, a state longer than the context: 422, docs/api.md) is answered with a 4xx and the engine is not probed.

From the moment the engine is dead:

  • every request, including those already waiting for the engine, is answered at once with 503 and the reason, and the engine is not called again;
  • /health answers 503 with {"ok": false, "engine": "dead", "reason": "...", "exit_code": 70};
  • after 2 seconds, once every request in flight has its answer (waiting at most 8 seconds more), the server stops its engine processes and exits with code 70; a request the dead engine never returns has its connection closed by the exit.

The in-process arrangement therefore exits about 7 seconds after the failing request (the probe's 5 seconds, then the grace period's 2). In the separate arrangement (--engine-process separate, the default with a second engine) vLLM reports the death itself, so the server exits about 2 seconds after it. The server also watches vLLM's flag between requests, so a death between requests turns /health to 503 within a second, before any request arrives.

Run the server under a restart policy, so that the exit brings up a new one:

  • compose: restart: unless-stopped, as compose.yaml has it;
  • systemd: Restart=on-failure;
  • Kubernetes: the default restartPolicy: Always, with a liveness probe on /health.

Docker restarts a container when its process exits, not when its HEALTHCHECK reports it unhealthy, so it is the exit that brings the server back; the image's check reports the dying container unhealthy in the meantime. The new server loads the engine again (the start times above), and requests sent until it is healthy are refused.

In decisio 0.1.0 to 0.7.1 none of this happens: after the engine dies /health keeps answering 200 and the process keeps running, so no restart policy acts. In the separate arrangement every request then fails with 500; with --engine-process in (0.7.0 and 0.7.1) every request after the failing one hangs.

What next

Section titled “What next”