# Limits

> What the numbers do not say: the conditions, the hardware and the measurements that belong beside every figure.

Source: https://opendecisio.com/docs/limits/

A number without its conditions misleads. These are the conditions that belong beside every figure on this site, and the limits of the system itself.

 

[Figure: Eight conditions that belong beside every figure. Limits: One card, one session: Every figure is from one card and one run per configuration. Hardware: The 31B is measured on, and needs, a 96 GB card (or a 64 GB Mac); the 12B runs on a 32 GB Mac or a 16 GB laptop. The host and the power limit: Latency depends on the CPU and on the card's power limit: a card at 600 W may read new states faster than the 585 W measured. Repeatability: Answers repeat within a session on one machine; across sessions near-ties can move. Images: The image route was measured with the Qwen base only. What was fitted on what: The temperatures were fitted on a private suite, so its ECE figures are in-sample. Size: A prompt holds up to 32,768 tokens and a question up to 255 options. Self-run, not board scores: The Decision Index values are our own run, pending the maintainers' validation.]

 

## TL;DR

 

Two limits you can check yourself, with a server running ([Get started](https://opendecisio.com/docs/) starts one): a request with no questions, and a choice with 256 options, which is one more than allowed:

 

```
import json
import urllib.error
import urllib.request

def status(questions):
    request = urllib.request.Request(
        "http://127.0.0.1:8000/v1/systemone",
        data=json.dumps({"state": "A short text.", "questions": questions}).encode(),
        headers={"Content-Type": "application/json"},
    )
    try:
        return urllib.request.urlopen(request).status
    except urllib.error.HTTPError as error:
        return error.code

print("no questions:", status({}))
print("256 options:", status({"q": {"type": "choice", "instructions": "Which?", "criteria": {f"option_{i:03d}": None for i in range(256)}}}))
# no questions: 422
# 256 options: 400
```

 

Run against the CPU stand-in at v0.11.1 on 2026-10-14.

  

| Word | Meaning |
| --- | --- |
| context | what a prompt can hold: the state with one question, up to 32,768 tokens |
| session | one run on one machine; every figure is from one card and one run per configuration |
| power limit | the card’s enforced limit in watts; the latency figures are from 585 W, default 600 W |

 

## 1. The limits of the measurements

 

- One card per run and one run per configuration.
- JevBench's v1.5 reading is taken on JevBench's 231 published questions, of the board's 904 open items, with no judge tier and no sealed half; its published items have been public since v1.2 and may be in any model's pretraining data.
- The Decision Index rows of sections 3, 6 and 7 were rebuilt with the kit for four benchmarks; their counts and the rows' sha256 match (`decisio.bench.di_rows`), but a partial rebuild cannot be checked against the whole suite's hash. Section 8's runs used the whole suite, rebuilt with the kit and matched against the lab's hashes.
- The ImajevBench record was made without the rendering rules and the key-order tie-break; 2 of its 254 items ended in exact ties that the harness rejected and are counted wrong.
- The second-engine head mode with the registered text-only class is unmeasured for latency.
- Latency is server-side on the card's localhost; cost is the card-hour price divided by measured throughput, with nothing else counted.
- Latency depends on the host and the card's power limit. The latency rows marked 585 W come from one RTX PRO 6000 Blackwell Workstation Edition whose power limit was 585 W (default 600 W), on an AMD Ryzen Threadripper 9960X host ([`runs/2026-10-06_latency-585w/`](https://github.com/aminry/decisio/tree/v0.11.1/runs/2026-10-06_latency-585w)). On the Gemma bases the limit held the clock back on 72 to 92% of the busy samples, one request at a time included, so a card at 600 W may read new states faster; on the Qwen base it did on 0 to 2%. A cached Qwen question depends on the CPU instead: 39.4 ms in-process on an AMD EPYC 7452 host, 18.5 ms on an AMD Ryzen 9 9950X and 20.2 ms on the Threadripper, the same card model and vLLM ([`runs/2026-10-05_engine-death-gates/`](https://github.com/aminry/decisio/tree/v0.11.1/runs/2026-10-05_engine-death-gates), [`runs/2026-10-06_latency-0.8.1/`](https://github.com/aminry/decisio/tree/v0.11.1/runs/2026-10-06_latency-0.8.1)). Latency rows not marked come from hosts whose CPU and power limit were not recorded.
- The per-item Decision Index records keep each request's id, the payload's sha256 and our response, not the item text (GPQA's authors ask that its items not be published in plain text).

 

## 2. The Gemma 4 12B base

 

- One card and one session; the Qwen base's numbers beside it are from the same session.
- The Gemma base does not fit a 32 GB card on vLLM's defaults. Simulated on the 96 GB card with the engine's share cut to 28.8 GB (vLLM's 0.90 of 32 GB), it did not start at 32,768 or at 16,384 tokens of context: its 22.8 GiB of weights left no room for the KV cache. Its path to 32 GB machines is an MLX build, in preparation.
- The image route and `--pad-policy row` were not measured on this base; `--multi-question warm` only on the 272 travel requests of section 6.5.
- An option-count temperature (one that depends on a question's number of options, fitted on the suite only) was tested for this base on 2026-10-07 against a rule written before the run, and is not adopted: the out-of-fold log-loss gain was not wholly below zero (its upper end +0.0005); the temperatures of 6.4 stay as served (the private RLCD repository, `experiments/2026-10-07_lab1_option_count_temperature/RESULTS.md`).
- The limits of section 5 on JevBench, the Decision Index rows and the board apply as written; nothing was submitted to JevBench's board, and the Decision Index's whole suite was self-run and submitted (section 8).

 

## 3. The Gemma 4 31B base

 

- At bf16 the checkpoint does not serve a 32,768-token context on one 96 GB card: its weights (57.91 GiB) leave 20.24 GiB for the KV cache at vLLM's 0.90 share, while one 32,768-token request needs 27.51 GiB; vLLM estimates 24,096 tokens as the most that fits (`bf16_start.txt`), hence FP8 on load.
- Several questions in one request, scored one engine call each as on the other bases, equalled each question sent alone on all 272 travel requests; batched (`--multi-question warm`, engine in the server's process) they repeated exactly and matched each question alone except 6 of 1,088 answers, by at most 0.005, with no choice changed ([`runs/2026-10-05_engine-process/`](https://github.com/aminry/decisio/tree/v0.11.1/runs/2026-10-05_engine-process)). Against each question in its own engine call, batched answers changed no choice on the suite's 1,400 items, JevBench's 231 published questions or the travel requests, by at most 0.0073 ([`runs/2026-10-05_engine-death-gates/`](https://github.com/aminry/decisio/tree/v0.11.1/runs/2026-10-05_engine-death-gates)); the suite and JevBench were not measured against each question sent alone.
- Slower than both served bases on states it has not seen before, the more so the longer the state (one question on a new 3,000-token state: 438 ms, against 85 ms on the Qwen base; 585 W).
- One card and one session; the intent heads over three draws, the other bases' over six.
- **Repeatability is within one session on one machine.** On 2026-10-06 the served FP8 profile at v0.8.0, with identical prompts and profile, did not reproduce this section's record: over 1,631 suite and JevBench items every readout differed (median |dp| per task 1e-9 to 3e-6), and near-tied items moved by up to 0.98, changing 28 top answers (1.7%). The engine arrangement (in the server's process, 0.8.0's default) and the driver (590.48.01, against the record's 610.43.02) differed; section 8's run of 2026-10-05, at v0.6.0 with the engine in its own process and the record's driver, had reproduced its four Decision Index values exactly. A comparison of this base with an earlier record must therefore be paired in the same session (Track 7's stage C, recorded in the private RLCD repository at `experiments/2026-10-05_t7_nvfp4_stage_c/RESULTS.md`).
- The figures in this section are the vLLM FP8 path's; the MLX path is another engine, gated against them ([`runs/2026-10-07_mlx-gemma-31b/`](https://github.com/aminry/decisio/tree/v0.11.1/runs/2026-10-07_mlx-gemma-31b)) and documented for states up to 16,383 tokens; the image route was not measured and is not served on a Mac.
- An option-count temperature (one that depends on a question's number of options, fitted on the suite only) was tested for this base on 2026-10-07 against a rule written before the run, and is not adopted: the board-style calibration estimate fell 27.0%, where the rule required a third; the temperatures of 7.4 stay as served (the private RLCD repository, `experiments/2026-10-07_lab1_option_count_temperature/RESULTS.md`).
- The limits of section 5 on JevBench, the Decision Index rows and the board apply as written; nothing was submitted to JevBench's board, and the Decision Index's whole suite was self-run and submitted (section 8).

 

## What next

 

- [Repeatability](https://opendecisio.com/docs/repeatability/): what repeats and what can move.
- [The evaluation card](https://opendecisio.com/docs/evaluation-card/): the sections these come from, with every measurement.
- [Choose your machine](https://opendecisio.com/docs/quickstart/): the hardware each path needs.