Skip to content
Decisio

DocsUnderstand

6 min read · updated

Limits

What the numbers do not say: the conditions, the hardware and the measurements that belong beside every figure.

A number without its conditions misleads. These are the conditions that belong beside every figure on this site, and the limits of the system itself.

Figure · What the numbers do not say

Eight conditions that belong beside every figure.

Each card links to the page that says more.

Source: EVAL_CARD.md sections 4, 5, 6.5, 6.6 and 7.5, the README and docs/running.md at the pinned release.

TL;DR

Section titled “TL;DR”

Two limits you can check yourself, with a server running (Get started starts one): a request with no questions, and a choice with 256 options, which is one more than allowed:

import json
import urllib.error
import urllib.request




def status(questions):
    request = urllib.request.Request(
        "http://127.0.0.1:8000/v1/systemone",
        data=json.dumps({"state": "A short text.", "questions": questions}).encode(),
        headers={"Content-Type": "application/json"},
    )
    try:
        return urllib.request.urlopen(request).status
    except urllib.error.HTTPError as error:
        return error.code




print("no questions:", status({}))
print("256 options:", status({"q": {"type": "choice", "instructions": "Which?", "criteria": {f"option_{i:03d}": None for i in range(256)}}}))
# no questions: 422
# 256 options: 400

Run against the CPU stand-in at v0.11.1 on 2026-10-14.

WordMeaning
contextwhat a prompt can hold: the state with one question, up to 32,768 tokens
sessionone run on one machine; every figure is from one card and one run per configuration
power limitthe card’s enforced limit in watts; the latency figures are from 585 W, default 600 W

1. The limits of the measurements

Section titled “1. The limits of the measurements”
  • One card per run and one run per configuration.
  • JevBench's v1.5 reading is taken on JevBench's 231 published questions, of the board's 904 open items, with no judge tier and no sealed half; its published items have been public since v1.2 and may be in any model's pretraining data.
  • The Decision Index rows of sections 3, 6 and 7 were rebuilt with the kit for four benchmarks; their counts and the rows' sha256 match (decisio.bench.di_rows), but a partial rebuild cannot be checked against the whole suite's hash. Section 8's runs used the whole suite, rebuilt with the kit and matched against the lab's hashes.
  • The ImajevBench record was made without the rendering rules and the key-order tie-break; 2 of its 254 items ended in exact ties that the harness rejected and are counted wrong.
  • The second-engine head mode with the registered text-only class is unmeasured for latency.
  • Latency is server-side on the card's localhost; cost is the card-hour price divided by measured throughput, with nothing else counted.
  • Latency depends on the host and the card's power limit. The latency rows marked 585 W come from one RTX PRO 6000 Blackwell Workstation Edition whose power limit was 585 W (default 600 W), on an AMD Ryzen Threadripper 9960X host (runs/2026-10-06_latency-585w/). On the Gemma bases the limit held the clock back on 72 to 92% of the busy samples, one request at a time included, so a card at 600 W may read new states faster; on the Qwen base it did on 0 to 2%. A cached Qwen question depends on the CPU instead: 39.4 ms in-process on an AMD EPYC 7452 host, 18.5 ms on an AMD Ryzen 9 9950X and 20.2 ms on the Threadripper, the same card model and vLLM (runs/2026-10-05_engine-death-gates/, runs/2026-10-06_latency-0.8.1/). Latency rows not marked come from hosts whose CPU and power limit were not recorded.
  • The per-item Decision Index records keep each request's id, the payload's sha256 and our response, not the item text (GPQA's authors ask that its items not be published in plain text).

2. The Gemma 4 12B base

Section titled “2. The Gemma 4 12B base”
  • One card and one session; the Qwen base's numbers beside it are from the same session.
  • The Gemma base does not fit a 32 GB card on vLLM's defaults. Simulated on the 96 GB card with the engine's share cut to 28.8 GB (vLLM's 0.90 of 32 GB), it did not start at 32,768 or at 16,384 tokens of context: its 22.8 GiB of weights left no room for the KV cache. Its path to 32 GB machines is an MLX build, in preparation.
  • The image route and --pad-policy row were not measured on this base; --multi-question warm only on the 272 travel requests of section 6.5.
  • An option-count temperature (one that depends on a question's number of options, fitted on the suite only) was tested for this base on 2026-10-07 against a rule written before the run, and is not adopted: the out-of-fold log-loss gain was not wholly below zero (its upper end +0.0005); the temperatures of 6.4 stay as served (the private RLCD repository, experiments/2026-10-07_lab1_option_count_temperature/RESULTS.md).
  • The limits of section 5 on JevBench, the Decision Index rows and the board apply as written; nothing was submitted to JevBench's board, and the Decision Index's whole suite was self-run and submitted (section 8).

3. The Gemma 4 31B base

Section titled “3. The Gemma 4 31B base”
  • At bf16 the checkpoint does not serve a 32,768-token context on one 96 GB card: its weights (57.91 GiB) leave 20.24 GiB for the KV cache at vLLM's 0.90 share, while one 32,768-token request needs 27.51 GiB; vLLM estimates 24,096 tokens as the most that fits (bf16_start.txt), hence FP8 on load.
  • Several questions in one request, scored one engine call each as on the other bases, equalled each question sent alone on all 272 travel requests; batched (--multi-question warm, engine in the server's process) they repeated exactly and matched each question alone except 6 of 1,088 answers, by at most 0.005, with no choice changed (runs/2026-10-05_engine-process/). Against each question in its own engine call, batched answers changed no choice on the suite's 1,400 items, JevBench's 231 published questions or the travel requests, by at most 0.0073 (runs/2026-10-05_engine-death-gates/); the suite and JevBench were not measured against each question sent alone.
  • Slower than both served bases on states it has not seen before, the more so the longer the state (one question on a new 3,000-token state: 438 ms, against 85 ms on the Qwen base; 585 W).
  • One card and one session; the intent heads over three draws, the other bases' over six.
  • Repeatability is within one session on one machine. On 2026-10-06 the served FP8 profile at v0.8.0, with identical prompts and profile, did not reproduce this section's record: over 1,631 suite and JevBench items every readout differed (median |dp| per task 1e-9 to 3e-6), and near-tied items moved by up to 0.98, changing 28 top answers (1.7%). The engine arrangement (in the server's process, 0.8.0's default) and the driver (590.48.01, against the record's 610.43.02) differed; section 8's run of 2026-10-05, at v0.6.0 with the engine in its own process and the record's driver, had reproduced its four Decision Index values exactly. A comparison of this base with an earlier record must therefore be paired in the same session (Track 7's stage C, recorded in the private RLCD repository at experiments/2026-10-05_t7_nvfp4_stage_c/RESULTS.md).
  • The figures in this section are the vLLM FP8 path's; the MLX path is another engine, gated against them (runs/2026-10-07_mlx-gemma-31b/) and documented for states up to 16,383 tokens; the image route was not measured and is not served on a Mac.
  • An option-count temperature (one that depends on a question's number of options, fitted on the suite only) was tested for this base on 2026-10-07 against a rule written before the run, and is not adopted: the board-style calibration estimate fell 27.0%, where the rule required a third; the temperatures of 7.4 stay as served (the private RLCD repository, experiments/2026-10-07_lab1_option_count_temperature/RESULTS.md).
  • The limits of section 5 on JevBench, the Decision Index rows and the board apply as written; nothing was submitted to JevBench's board, and the Decision Index's whole suite was self-run and submitted (section 8).

What next

Section titled “What next”