Repeatability

A model’s answers move at the scale of any change of rounding: the batch a question rides in, where a long row is split, the card, the driver. So what repeats is stated for a configuration and a request form, base by base.

Figure · What repeats

Within a configuration, answers repeat; across sessions and rounding, near-ties can move.

BaseThe same request againA first read against a cached readQuestions batched against aloneAnother session

Qwen3.6-35B-A3Bthe same probabilities every timeidenticalthe same probabilities, bit for bitmatched an earlier record on all 1,400 suite items in 4 of 5 restarts; on another card, to 1.1e-16

Gemma 4 12Bnot bit for bitup to 0.113 on 60 new states; one choice changedthe same choice, probabilities within 0.035up to 0.128 on 212 of 231 items; two near-tied choices changed

Gemma 4 31Bexact within one session on one machineidenticalthe same choice, probabilities within 0.0073near-tied answers moved; 28 top answers changed (1.7%) over 1,631 items

Every bit-identity claim holds for a fixed configuration and request form: any change of rounding moves this model's answers.

Source: EVAL_CARD.md sections 4, 6.5 and 7.5, docs/cli.md "Determinism" and the README's "Choosing a base" at the pinned release.

TL;DR

With a server running (Get started starts one), this script sends the same request five times and counts the distinct answers:

import json
import urllib.request

body = json.dumps({
    "state": "I was charged twice for October. Both charges are 49.00 EUR, same day. Please refund one.",
    "questions": {"urgent": {"type": "noul", "instructions": "Does this ticket need a response within the hour?"}},
}).encode()


def ask():
    request = urllib.request.Request("http://127.0.0.1:8000/v1/systemone", data=body, headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(request) as response:
        return json.dumps(json.load(response)["answers"], sort_keys=True)


print("distinct answers in 5 sends:", len({ask() for _ in range(5)}))
# distinct answers in 5 sends: 1

Run against the CPU stand-in at v0.11.1 on 2026-10-14.

The README’s three-question example, on the 31B under its served defaults, gave the same response in 30 of 30 repeats on one server.

Word	Meaning
bit for bit	the same number to the last digit
first read	the server reading a state it has not seen; a later read comes from the prefix cache
batched	several questions scored in one engine step instead of one engine call each
near-tied	two options whose probabilities are close, so a small move can change the choice

Common mistake

Comparing an answer with an earlier record from another session as if they were the same experiment. On one day the served 31B profile, with identical prompts and profile, did not reproduce an earlier record: every readout differed, and near-tied items moved by up to 0.98, changing 28 top answers (1.7%); the engine arrangement and the driver had changed. Pair a comparison within one session.

1. Why answers move

The Gemma 4 12B base’s final logits are bf16 after a soft cap, so near the top they lie on a grid of 0.0625 to 0.125, and a state read for the first time and the same state read from the prefix cache can land a step apart. Questions scored in one batch are not the same forward pass as each scored alone: on the Qwen base the dependence was traced to vLLM 0.30.0’s forward, whenever more than one sequence shares an engine step. Every change of rounding moves the answers, which is why each claim holds for a fixed configuration and request form.

2. What the server does about it

On a running server with the Qwen base, a question returns the same probabilities every time, alone or among other questions: each question is scored in its own engine call. Across server restarts, single-question answers have matched an earlier record on the whole suite in most starts measured, and moved slightly without changing a choice in the others. The Gemma base does not repeat bit for bit: its logits come out in bf16 after its soft cap, and a state read for the first time and read again from the prefix cache can land a step apart; several questions in one request stay within the bound in the README's settings table. Any change of rounding moves this model's answers, the batch, the padding or where a long row is split, so every bit-identity claim holds for a fixed configuration and request form. Measurements: EVAL_CARD.md sections 4 and 6.5.

What next

• Flags and options: --multi-question, --register-boundary and --engine-process, which decide what repeats.
• What the probability means: temperatures and calibration.
• The evaluation card: sections 6.5 and 7.5 have every measurement.