Batch concurrent requests

Batched mode is opt-in: it scores every request waiting for the engine together, in one call, so concurrent requests share the model’s forward passes. A request that arrives at an idle server is still scored at once, alone, and gets the serial answer bit for bit, so a single client sees no change.

Figure · Request scheduling

Serial handles requests one at a time; batched can combine eligible waiting requests.

Serial

1. Call for AOne request uses the engine.
2. Call for BThe next request follows.
3. Call for CThe next request follows.

Batched, when eligible requests are waiting

1. Requests A, B, CSeveral clients have requests waiting.
2. One shared callEligible requests share the engine call.
3. Separate answersEach client receives its own response.

An idle server still answers a lone request at once. Certified-task reads, thinking and image requests take their documented separate paths.

Source: concurrency in the pinned running guide. Box widths do not represent latency or a speed ratio.

Under concurrent load it answers more requests a second: the Gemma 4 12B in FP8 answered 8.9 and 9.2 requests a second at 8 and 16 clients, against 7.2 one at a time, on requests about new texts of about 1,000 tokens. The gain depends on the card, the model and how often texts repeat, so measure it on your own traffic.

Measured with decisio 0.24.0’s tokenizer fix before its release, the Gemma 4 12B in FP8 on one RTX 5090 at 0.95 of the card, vLLM 0.31.0, 2026-10-21.

This reads the mode your server runs; without a flag it is serial:

curl -s http://127.0.0.1:8000/health \
  | python3 -c 'import json, sys; print(json.dumps(json.load(sys.stdin)["concurrency"], indent=1))'

Run against the CPU stand-in at v0.46.0 on 2026-11-13. It printed:

{
 "mode": "serial"
}

1. Turn it on

Start the server with --concurrency batched:

pip install "decisio[serve]==0.46.0"
python -m decisio.serve.vllm_engine --concurrency batched

Shown, not run: it needs Linux with one NVIDIA card, which this check does not have.

• Without the flag the server stays serial: one request at a time through the engine, and the same request gives the same answer whatever else arrives.
• --batch-max-requests caps one call at 16 requests by default, counted in requests, not questions. The oldest waiting request is always taken.
• One call holds the requests of one adapter, or of the base alone.

2. Know what stays one at a time

• A certified task’s questions, and every task registration and certification read, take the serial path on a batched server too.
• A thinking request, an image request and a request a task’s intent head answers take the engine alone.
• So certified answers are made and served on the reads a serial server would make.

3. Know what can move

• Answers on the 12B can be made not to move. With --batch-invariant on, off by default, a request gives the same bits in any batch of up to 16 reads, in bf16 and FP8, at a cost that grows with the input.
• Answers on the 31B, and on the 12B by default, can. An answer no certified task covers can differ slightly when it is scored beside other requests, because the engine’s arithmetic follows the shape of the batch. Keep serial where your application requires repeatable request-by-request results.
• What was measured. Only the 12B in FP8 has a published batched throughput figure; none is published for the 31B or the 12B in bf16.

4. Read what it did

• x-decisio-batch says, when a request shared a call, with how many requests and questions: requests=N; rows=M. It is absent when the request was scored alone.
• wait= in x-decisio-stages is the time the request waited for its call.
• /health reports the mode under concurrency, and in batched mode the calls made and the requests each call held.
• A request that fails on its own, such as a text over the length, fails alone; the call’s other requests are scored again without it.

Where it runs and what it does to certificates are in Requirements and limits.

What next

• Scale: many questions about one text, and the first read.
• Requirements and limits: batching in production.
• Server flags: --concurrency and --batch-max-requests in full.