Benchmarks
Each base with its own defaults on one RTX PRO 6000 Blackwell: the Qwen and Gemma 4 12B bases in one session (runs/2026-10-04_gemma-base/), Gemma 4 31B in its own (runs/2026-10-04_gemma-4-31b/), image input in another (runs/2026-09-27_image-input/). Measured by us with the public harnesses (the Decision Index kit 0.2.1) and recorded in runs/; none is a board score.
How Decisio compares
decisio's three bases set beside TypeSafe's Jev 1.13 and the three open-weights entries with the highest Decision Index on its public board, as of 2026-10-05.
Our values are self-run: the Decision Index runs are submitted in apolinario/decision-index#62 and pending the maintainers' validation, and the JevBench counts are this repository's records, not board entries.
Jev's and the other entries' values come only from the public boards, with each board's revision, the fields read and the date read in runs/2026-10-05_comparison/.
No figure here comes from any call of ours to TypeSafe's API, and nothing here is a rank.
How a cell is marked. Each of our cells is ahead of Jev, level or behind (bold on this site when level or ahead), by one rule for every cell, written and committed before anything was computed (RULE.md).
A difference counts only when it lies outside a 95% interval for the benchmark's size, h = 1.96 √(2m(1 − m)/n), with m the two values' mean and n the benchmark's items; otherwise the cell is level.
For the index and the areas, the same per-benchmark variance is carried through the board's formula.
The one lower-is-better metric, ForecastBench's Brier loss, is compared in its own direction.
Only our cells are marked; the other entries' columns are shown as the board has them.
The headline
Bold: level with or ahead of Jev 1.13, by the comparison's rule; plain: behind.
| Measure | DecisioQwen3.6-35B-A3B | DecisioGemma 4 12B | DecisioGemma 4 31B | Jev 1.13 | Surogate Rune26B-A4B v3 |
|---|---|---|---|---|---|
| Decision Index 0.2.1 (0 to 100) | 48.06: behind Jev | 49.43: behind Jev | 57.58: level with Jev | 57.91 | 57.44 |
| Knowledge & Reasoning | 32.45: behind Jev | 31.37: behind Jev | 45.57: behind Jev | 51.40 | 43.36 |
| Language Understanding | 51.39: behind Jev | 51.66: behind Jev | 61.02: level with Jev | 62.02 | 63.06 |
| Retrieval & Classification | 53.56: level with Jev | 58.00: ahead of Jev | 62.75: ahead of Jev | 55.42 | 63.52 |
| Tools & Automation | 68.43: behind Jev | 71.62: behind Jev | 75.04: level with Jev | 75.09 | 71.22 |
| Arts & Human Taste | 31.57: behind Jev | 32.59: behind Jev | 37.43: level with Jev | 37.66 | 41.90 |
| On JevBench's 231 published questions, correct | 200: level with Jev | 200: level with Jev | 213: ahead of Jev | 200 | not compared (h) |
| JevBench hard tier, correct | 82 of 111 | 82 of 111 | 93 of 111 | not compared (a) | not compared (h) |
| System | Latency, median per request | Price or self-hosting cost, one pass over the 38 index benchmarks | Context length | What it is |
|---|---|---|---|---|
| DecisioQwen3.6-35B-A3B | 70.5 ms (b) | $6.98 (e) | 32,768 tokens per prompt, the state plus one question | official Qwen weights, frozen, letters readout |
| DecisioGemma 4 12B | 37.8 ms (b) | $6.92 (e) | 32,768 tokens per prompt | official Gemma weights, frozen, letters readout |
| DecisioGemma 4 31B | 56.0 ms (b) | $10.28 (e) | 32,768 tokens per prompt | official Gemma weights, frozen, FP8 on load, letters readout |
| Jev 1.13 | 524.1 ms (c) | $6.05 (f) | 64k tokens per request; 32k tokens for the state plus the longest question (g) | hosted model, TypeSafe's API |
| Surogate Rune26B-A4B v3 | 120.5 ms (d) | $16.53 (e) | not on the board | full fine-tune of google/gemma-4-26B-A4B-it |
| Decider chatGemma-4-31B | 108.5 ms (d) | $22.18 (e) | not on the board | an inference technique on google/gemma-4-31B-it |
| pplx-decider-v1-27b | 101.4 ms (d) | $24.94 (e) | not on the board | full fine-tune of Qwen/Qwen3.8-27B |
(a) JevBench's board reports Jev's hard tier over 220 questions, 109 of them held out and never published, so it is not the same item set as the 111 published ones we can run.
(b) Our server time on one RTX PRO 6000 Blackwell: the Decision Index kit's HTTP wall time to a server on the same machine, one request at a time, over all 150,759 requests.
(c) The board's own measurement of TypeSafe's hosted API: an HTTPS round trip over the internet from its lab, one request at a time, on its latency sample. It includes the network, so it is not the same measurement as (b) and (d), and it carries no mark.
(d) The board's on-card median on one RTX PRO 6000, one request at a time, on its latency sample.
(e) Self-hosted, estimated as mean request latency × 150,759 requests at $1.48 per card-hour, what the card for our runs cost on vast.ai; for the other entries, with the board's mean latency. One request at a time throughout; a server answering requests concurrently would cost less per pass.
(f) The board's recorded API cost of Jev's run over the same benchmarks, at TypeSafe's tariff of $0.042 per million input tokens, output tokens free (jev.results[].usd).
(g) From TypeSafe's documentation (section 3).
(h) Not on JevBench's board.
Capabilities
What a buyer asks, answered for decisio from its documentation and for Jev from TypeSafe's public documentation.
| Question | decisio | Jev 1.13 | Sources |
|---|---|---|---|
| Where does my data go? | Stays on the machine that runs it | TypeSafe's hosted API; not used for training | Sources |
| What does it cost? | No per-call price; you pay for the card | $42 per billion input tokens; output free | Sources |
| Does it run on my own GPU, Mac or laptop? | Yes: GPU, Mac with MLX, or a laptop with Ollama | No: Hosted API only; its documentation describes no self-hosted option | Sources |
| Longest document? | 32,768 tokens per prompt | 32k tokens for the state plus the longest question (64k per request) | Sources |
| Options per question? | Up to 255 | Up to 255 on a choice | Sources |
| Many questions in one read? | Yes: The state is read once and shared by every question | Yes: Every question in a request is evaluated in parallel | Sources |
| Images? | Yes: Photos in the state; measured on the Qwen base only | No: Text only | Sources |
| Probabilities I can trust? | Yes: Fitted per base; the 31B measures 0.035 and 0.091 on the standard and hard tiers | Yes: Trained for calibrated decisions, in its documentation | Sources |
| Can I teach it with my own examples? | Yes: POST /v1/tasks fits calibration and an intent head to your labelled examples; the model stays frozen | No: Not fine-tuned on customer data; shaped through the request | Sources |
| Same answer every time? | Qwen: bit for bit. The 31B: the same answers within a session on one machine | Not stated in its documentation; an alias moves when a release ships | Sources |
| Licence? | Apache-2.0 | A hosted service under TypeSafe's customer agreements | Sources |
Sources, row by row
Each statement about Jev is what TypeSafe's public documentation states, quoted in snapshots/typesafe_docs.json with each page's hash, read 2026-10-05 at 21:44 UTC.
Where a row says the documentation does not state something, that is about the pages read, not a claim about the service.
- Where does my data go?
- What does it cost?
- decisio (Price)
- no per-call price; the card's cost (section 1, e)
- Jev 1.13 (Price)
- "$42 / $0.042" per billion / million tokens, "Charged per input token. Output tokens are free." (models)
- Does it run on my own GPU, Mac or laptop?
- decisio (Self-hosting)
- yes: one 96 GB card on vLLM, the Docker image, or a Mac (below)
- Jev 1.13 (Self-hosting)
- served by "the same endpoint,
POST /v1/systemone"; the pages read describe the hosted API and no self-hosted option (models)
- decisio (Mac and Ollama)
- Qwen, Gemma 4 12B and Gemma 4 31B (up to 16,383 tokens of state) on Apple silicon with MLX; the Qwen base on Ollama (
aminroudaki/decisio), with Ollama's own prompt and no calibration - Jev 1.13 (Mac and Ollama)
- the pages read describe the hosted API only
- Longest document?
- decisio (Context)
- 32,768 tokens per prompt: the state plus one question
- Jev 1.13 (Context)
- "64k tokens per request; 32k tokens for
stateplus the longest question" (models)
- Options per question?
- decisio (Options per question)
- up to 255 named options per choice
- Jev 1.13 (Options per question)
- "a maximum of 255 options per Choice"; a Score "accepts up to 10" levels (API)
- Many questions in one read?
- decisio (Several questions per request)
- the state is read once and shared by every question in the request
- Jev 1.13 (Several questions per request)
- "System One models evaluate every question in a request in parallel." (primitives)
- Images?
- decisio (Image input)
- photos in the state, through a second engine on the same card (
--image-model); measured on the Qwen base only - Jev 1.13 (Image input)
- "Text only. String, JSON object, or array of text values. No image, audio, or video input." (models)
- Probabilities I can trust?
- decisio (Calibration)
- fitted temperatures per base; JevBench ECE on the standard and hard tiers: Qwen 0.121 and 0.043, Gemma 4 12B 0.033 and 0.085, Gemma 4 31B 0.035 and 0.091
- Jev 1.13 (Calibration)
- "System One models are trained for calibrated decisions: their probabilities are optimized against outcomes to reflect uncertainty." (System One)
- Can I teach it with my own examples?
- decisio (Teaching from labelled examples)
POST /v1/tasksfits a per-task calibration and, for long option lists, an intent head on your labelled examples; the model stays frozen- Jev 1.13 (Teaching from labelled examples)
- "Jev is not fine-tuned or LoRA-adapted with customer data", and "You shape its answers to your domain through the request rather than through per-account weights" (models)
- Same answer every time?
- decisio (Repeatability)
- checkpoints at pinned revisions; Qwen: a repeated request returns the same probabilities bit for bit; Gemma 4 12B: the same choice, probabilities within 0.035; Gemma 4 31B: a repeated request returns the same answers with the engine in the server's process, within one session on one machine; across sessions near-tied answers can move (
EVAL_CARD.mdsections 5 to 7) - Jev 1.13 (Repeatability)
- "An alias moves when a new release ships, so the answers behind it can change without a change on your side." The pages read do not state whether a repeated request returns the same probabilities (models)
- Licence?
- decisio (Licence)
- Apache-2.0; the weights are Qwen's under Apache-2.0 and Google's Gemma 4 under Apache-2.0
- Jev 1.13 (Licence)
- a hosted service under TypeSafe's customer agreements, which "govern your use of TypeSafe" (legal)
Every index benchmark
One row per benchmark, grouped by area, with the coverage-adjusted value that feeds the index (raw on the board); ForecastBench shows its Brier loss, where lower is better.
n is the number of items behind the interval: the smallest of the benchmark's cases, our scored requests and Jev's cases.
Each interval is in runs/2026-10-05_comparison/results.json, and tables.md there is these tables as the script writes them.
Bold: level with or ahead of Jev 1.13, by the comparison's rule; plain: behind.
Knowledge & Reasoning
| Benchmark | n | Qwen3.6-35B-A3B | Gemma 4 12B | Gemma 4 31B | Jev 1.13 | Rune26B-A4B v3 | Decider chatGemma-4-31B | pplx-decider-v1-27b |
|---|---|---|---|---|---|---|---|---|
| GPQA Diamond, accuracy | 196 | 0.510: behind Jev | 0.388: behind Jev | 0.520: behind Jev | 0.786 | 0.469 | 0.490 | 0.495 |
| GSM8K, accuracy | 1,319 | 0.544: behind Jev | 0.599: behind Jev | 0.805: level with Jev | 0.799 | 0.797 | 0.826 | 0.674 |
| ChessBench, accuracy | 5,000 | 0.162: level with Jev | 0.165: level with Jev | 0.220: ahead of Jev | 0.172 | 0.187 | 0.223 | 0.172 |
| MuSR, accuracy | 752 | 0.581: behind Jev | 0.581: behind Jev | 0.640: level with Jev | 0.661 | 0.649 | 0.642 | 0.618 |
| SATA-Bench, case exact accuracy | 1,650 | 0.232: behind Jev | 0.264: level with Jev | 0.323: ahead of Jev | 0.264 | 0.348 | 0.293 | 0.299 |
| CRUXEval, accuracy | 570 | 0.625: behind Jev | 0.654: behind Jev | 0.832: ahead of Jev | 0.730 | 0.744 | 0.793 | 0.749 |
| CLadder, accuracy | 5,000 | 0.622: behind Jev | 0.634: behind Jev | 0.764: ahead of Jev | 0.726 | 0.727 | 0.746 | 0.745 |
| HLE, accuracy | 501 | 0.114: behind Jev | 0.106: behind Jev | 0.150: behind Jev | 0.204 | 0.106 | 0.148 | 0.120 |
| MMLU-Pro, accuracy | 12,032 | 0.613: behind Jev | 0.550: behind Jev | 0.694: behind Jev | 0.827 | 0.672 | 0.696 | 0.645 |
| BBH, accuracy | 5,507 | 0.694: behind Jev | 0.691: behind Jev | 0.762: behind Jev | 0.929 | 0.811 | 0.762 | 0.781 |
Language Understanding
| Benchmark | n | Qwen3.6-35B-A3B | Gemma 4 12B | Gemma 4 31B | Jev 1.13 | Rune26B-A4B v3 | Decider chatGemma-4-31B | pplx-decider-v1-27b |
|---|---|---|---|---|---|---|---|---|
| ContractNLI, macro-F1 | 123 | 0.733: level with Jev | 0.743: level with Jev | 0.726: level with Jev | 0.717 | 0.770 | 0.736 | 0.781 |
| ANLI, macro-F1 | 3,200 | 0.686: behind Jev | 0.702: behind Jev | 0.737: level with Jev | 0.748 | 0.735 | 0.732 | 0.707 |
| WinoGrande, accuracy | 1,267 | 0.808: behind Jev | 0.730: behind Jev | 0.852: behind Jev | 0.919 | 0.860 | 0.837 | 0.852 |
| HellaSwag, accuracy | 10,042 | 0.940: level with Jev | 0.867: behind Jev | 0.923: behind Jev | 0.945 | 0.924 | 0.920 | 0.940 |
| ACOS, per-review F1 | 400 | 0.174: behind Jev | 0.155: behind Jev | 0.215: behind Jev | 0.295 | 0.268 | 0.188 | 0.204 |
| FinEntity, macro-F1 | 979 | 0.803: behind Jev | 0.905: ahead of Jev | 0.931: ahead of Jev | 0.870 | 0.884 | 0.909 | 0.929 |
| iSarcasmEval, Sarcasm F1 · track A, English | 1,400 | 0.474: level with Jev | 0.427: behind Jev | 0.506: level with Jev | 0.505 | 0.604 | 0.511 | 0.603 |
| VAST, macro-F1 | 3,006 | 0.490: behind Jev | 0.591: behind Jev | 0.687: ahead of Jev | 0.646 | 0.766 | 0.732 | 0.708 |
| NLI4CT, macro-F1 | 5,500 | 0.772: behind Jev | 0.813: behind Jev | 0.835: level with Jev | 0.841 | 0.804 | 0.829 | 0.848 |
| RAGTruth, F1 on hallucinated class | 2,700 | 0.702: behind Jev | 0.696: behind Jev | 0.783: level with Jev | 0.765 | 0.768 | 0.769 | 0.803 |
Retrieval & Classification
| Benchmark | n | Qwen3.6-35B-A3B | Gemma 4 12B | Gemma 4 31B | Jev 1.13 | Rune26B-A4B v3 | Decider chatGemma-4-31B | pplx-decider-v1-27b |
|---|---|---|---|---|---|---|---|---|
| BANKING77, macro-F1 | 3,080 | 0.746: behind Jev | 0.729: behind Jev | 0.784: level with Jev | 0.797 | 0.838 | 0.791 | 0.790 |
| CLINC150+OOS, macro-F1 | 5,500 | 0.822: behind Jev | 0.872: behind Jev | 0.901: level with Jev | 0.893 | 0.874 | 0.912 | 0.878 |
| BRIGHT, nDCG@10 | 220 | 0.444: level with Jev | 0.471: level with Jev | 0.439: level with Jev | 0.475 | 0.463 | 0.436 | 0.487 |
| Amazon ESCI, macro-F1 | 5,000 | 0.450: behind Jev | 0.535: level with Jev | 0.540: level with Jev | 0.552 | 0.553 | 0.531 | 0.551 |
| PhishNChips phishing decisions, accuracy | 2,000 | 0.752: ahead of Jev | 0.813: ahead of Jev | 0.875: ahead of Jev | 0.625 | 0.809 | 0.875 | 0.600 |
| HoVer, accuracy | 4,000 | 0.700: behind Jev | 0.693: behind Jev | 0.756: ahead of Jev | 0.729 | 0.807 | 0.764 | 0.742 |
Tools & Automation
| Benchmark | n | Qwen3.6-35B-A3B | Gemma 4 12B | Gemma 4 31B | Jev 1.13 | Rune26B-A4B v3 | Decider chatGemma-4-31B | pplx-decider-v1-27b |
|---|---|---|---|---|---|---|---|---|
| BFCL, case exact accuracy | 1,694 | 0.960: level with Jev | 0.962: level with Jev | 0.979: ahead of Jev | 0.958 | 0.948 | 0.980 | 0.976 |
| ToolRet, nDCG@10 | 685 | 0.625: level with Jev | 0.622: level with Jev | 0.613: level with Jev | 0.653 | 0.644 | 0.636 | 0.674 |
| API-Bank, accuracy | 508 | 0.817: behind Jev | 0.870: level with Jev | 0.854: level with Jev | 0.882 | 0.833 | 0.850 | 0.841 |
| Home appliance simulator, case exact accuracy | 88 | 0.398: level with Jev | 0.443: level with Jev | 0.614: level with Jev | 0.523 | 0.466 | 0.625 | 0.739 |
| When2Call, accuracy | 3,652 | 0.714: behind Jev | 0.761: behind Jev | 0.772: behind Jev | 0.810 | 0.760 | 0.769 | 0.817 |
Arts & Human Taste
| Benchmark | n | Qwen3.6-35B-A3B | Gemma 4 12B | Gemma 4 31B | Jev 1.13 | Rune26B-A4B v3 | Decider chatGemma-4-31B | pplx-decider-v1-27b |
|---|---|---|---|---|---|---|---|---|
| BPoMP, accuracy | 811 | 0.904: level with Jev | 0.910: level with Jev | 0.894: level with Jev | 0.909 | 0.900 | 0.906 | 0.939 |
| Humicroedit, accuracy | 2,628 | 0.608: level with Jev | 0.617: level with Jev | 0.625: level with Jev | 0.619 | 0.622 | 0.638 | 0.624 |
| POP909-CL, accuracy | 2,000 | 0.142: behind Jev | 0.060: behind Jev | 0.250: ahead of Jev | 0.166 | 0.661 | 0.279 | 0.378 |
| cfcolor, accuracy | 5,000 | 0.600: behind Jev | 0.624: behind Jev | 0.639: level with Jev | 0.644 | 0.626 | 0.626 | 0.644 |
| ForecastBench, Brier loss (lower is better) | 10,139 | 0.228: behind Jev | 0.202: behind Jev | 0.207: behind Jev | 0.174 | 0.203 | 0.202 | 0.195 |
| Habermas Machine, accuracy | 1,676 | 0.447: level with Jev | 0.465: level with Jev | 0.484: level with Jev | 0.459 | 0.422 | 0.479 | 0.418 |
| New Yorker, accuracy | 528 | 0.688: level with Jev | 0.633: behind Jev | 0.739: level with Jev | 0.701 | 0.741 | 0.739 | 0.703 |
Hosted chat models on JevBench's 231 published questions
The JevBench operator's published counts for the general chat models it measured through their providers' APIs, by the board's model IDs (revision v1.4.2.2, public_accuracy × 231); they are not decision models, have no Decision Index value, and carry no marks.
| Model, as the board names it | Board ID | Correct of 231 |
|---|---|---|
| GPT-6 Luna, low reasoning effort | gpt-6-luna-low | 229 |
| GPT-6 Luna, default medium reasoning effort | gpt-6-luna | 230 |
| GPT-5.6 Luna, low reasoning effort | gpt-5.6-luna | 225 |
| Gemini 3.1 Flash-Lite | gemini-3.1-flash-lite | 201 |
| DeepSeek V4.1 Flash, thinking by default | deepseek-flash | 226 |
| Qwen3.8 27B, on Chutes | qwen3.8-27b | 166 |
Source: docs/comparison.md Jev's GPQA Diamond is the board's jev.benchmarks['25'].raw (0.7857, 154 of 196 scored); the board file also holds jev.results['25'].score = 0.7828, which its GPQA Diamond column and benchmark page display as 78.3. Field: EVAL_CARD.md
Each base on the public harnesses
Accuracy and macro-F1, higher is better.
| Measure | Qwen3.6-35B-A3BRecord: runs/2026-10-04_gemma-base/ | Gemma 4 12BRecord: runs/2026-10-04_gemma-base/ | Gemma 4 31B (default)Record: runs/2026-10-04_gemma-4-31b/ |
|---|---|---|---|
| JevBench published items, easy / standard / hard | 1.000 / 0.972 / 0.739 | 1.000 / 0.972 / 0.739 | 1.000 / 1.000 / 0.838 |
| BANKING77, macro-F1 | 0.746 | 0.729 | 0.785 |
| CLINC150+OOS, macro-F1 | 0.822 | 0.871 | 0.902 |
| GPQA Diamond, accuracy | 0.510 | 0.378 | 0.520 |
| MMLU-Pro, accuracy | 0.613 | 0.549 | 0.694 |
| Intent heads, BANKING77 / CLINC150 | 0.840 / 0.912 | 0.832 / 0.908 | 0.844 / 0.970 |
Image input, ImajevBenchRecord: runs/2026-09-27_image-input/ | 0.791 | n/m | n/m |
| New 300-token state | 49.9 ms | 53.6 ms | 80.7 ms |
| 1,000-token state, cached | 20.2 ms | 24.8 ms | 31.7 ms |
It went in under the maintainer's decision, past a pre-registered rule it missed by 0.2 to 1.1 points on three of four benchmarks; EVAL_CARD.md section 7 states the rule, the numbers and the reason.
Server time
The latency rows were measured in one session on decisio 0.8.1's served defaults: the engine in the server's process, and on the Gemma bases a single question on a new state registering its boundary first (runs/2026-10-06_latency-585w/).
The card was one RTX PRO 6000 Blackwell Workstation Edition with its power limit at 585 W (default 600 W), on an AMD Ryzen Threadripper 9960X host.
On the Gemma bases that limit held the clock back during most first reads, so a card at 600 W may read new states faster; on the Qwen base it did not.
In the same session, the boundary registration added +16 to +28 ms (12B) and +21 to +34 ms (31B) to the first read of a new state of 300 to 3,000 tokens, and a second, different question then read the state from the cache on 20 of 20 states per base.
Changes paired within one session:
Gemma 4 12B against Qwen3.6-35B-A3B, paired
What drives the choice, measured on one RTX PRO 6000 Blackwell in one session, each base with its own defaults, paired over the same items (95% bootstrap intervals; runs/2026-10-04_gemma-base/):
| Measure | Gemma 4 12B | Qwen3.6-35B-A3B | Gemma minus Qwen |
|---|---|---|---|
| JevBench v1.5 open-set reading, I_open (equal types) | 64.4 | 49.4 | +15.0 [+7.0, +23.6] |
| JevBench v1.5, yes/no / score | 49.0 / 63.7 | 13.7 / 57.2 | |
| Yes/no answers between 0.20 and 0.80 (74 items) | 15% | 39% | |
| JevBench, published items, correct | 0.0 points [-4.3, +4.3] | ||
| Decision Index accuracy, CLINC150+OOS | +4.5 [+3.6, +5.5] | ||
| Decision Index accuracy, BANKING77 | -1.4 [-2.7, -0.1] | ||
| Decision Index accuracy, GPQA Diamond | -13.6 [-21.7, -5.6] | ||
| Decision Index accuracy, MMLU-Pro | -6.4 [-7.2, -5.5] | ||
| 1,400-item suite, accuracy | 0.735 | 0.770 | -3.5 [-5.5, -1.6] |
One question on a new 3,000-token state, server time (0.8.1's served defaults, the engine in the server's process; one RTX PRO 6000 at 585 W, AMD Ryzen Threadripper 9960X; one session, runs/2026-10-06_latency-585w/) | 258.6 ms | 85.2 ms |
Record: runs/2026-10-04_gemma-base/
Calibration
The calibration of each base is in the capabilities table above, under "Probabilities I can trust?", with its sources.
EVAL_CARD.md has the full tables, the calibration figures and the disclosures of what was fitted on what (sections 4, 6.4 and 7.4).
The Decision Index, whole suite: self-runs
Self-run, submitted to the board and pending the maintainers' validation; no rank is claimed.
| Base | Tag | Decision Index 0.2.1 | Raw index | Breadth | Request latency, median / p95 |
|---|---|---|---|---|---|
| Qwen3.6-35B-A3B (FP8) | v0.4.0 | 48.06 | 60.45 | 46.55 | 70.5 / 261.2 ms |
| Gemma 4 12B (bf16) | v0.4.0 | 49.43 | 61.47 | 47.57 | 37.8 / 275.9 ms |
| Gemma 4 31B (FP8 on load) | v0.6.0 | 57.58 | 67.33 | 56.50 | 56.0 / 439.3 ms |
Source: EVAL_CARD.md
JevBench, the public board
The board's reading of decisio, revision v1.6.1, in view A: its headline composite, higher is better. The board's 31B row is decisio v0.8.0 on Gemma 4 31B; it has no row for later releases of that base.
- Second among open-weights systems on JevBench, level with Jev, and the highest-ranked with frozen weights.
- It rests on ranks and on overlap, not on separation: decisio's 95% interval overlaps those of H2O-Lightning-4B, Jev 1.13.0, decider-12b v1 and Cygnet.
- On JevBench v1.6.1, fourth of 135, in view A, the board's headline composite; views B and C rank it third and fourth.
- Level with Jev: the two 95% intervals overlap.
- Second on the sealed set, by the sealed half of the intelligence axis.
| Rank of 135 | System | Score | 95% interval | Open-board rank of 128 | Items answered |
|---|---|---|---|---|---|
| 2 | H2O-Lightning-4B | 72.52 | 69.79 to 73.79 | 1 | 1,500 of 1,500 |
| 4 | decisio, Gemma 4 31B, v0.8.0 | 71.69 | 69.86 to 72.81 | 2 | 1,477 of 1,500 |
| 5 | Jev 1.13.0 | 71.49 | 69.14 to 72.40 | not on it | 1,477 of 1,500 |
| 6 | Quyet-1.0-Large | 71.39 | 69.84 to 72.21 | 3 | 1,500 of 1,500 |
| 12 | deck-31B | 68.68 | 67.07 to 69.74 | 8 | 1,477 of 1,500 |
| 14 | decisio, Gemma 4 12B, v0.8.0 | 68.17 | 57.37 to 69.65 | 10 | 1,477 of 1,500 |
| 15 | decisio, Gemma 4 12B, v0.9.0 | 67.76 | 54.38 to 69.56 | 11 | 1,477 of 1,500 |
- H2O-Lightning-4B59.0261.05 open
- decisio, Gemma 4 31B, v0.8.073.2667.59 open
- Jev 1.13.061.6465.60 open
- Quyet-1.0-Large71.5675.24 open
- deck-31B75.8270.23 open
- decisio, Gemma 4 12B, v0.8.051.5753.08 open
- decisio, Gemma 4 12B, v0.9.051.5353.07 open
JevBench, axis by axis
The ten highest open-weights systems and Jev on the public board, each cell shaded by its rank in the column; higher is better on every axis.
- On JevBench v1.6.1, among the ten highest open-weights systems and Jev: 2nd overall, 2nd on sealed questions (73.3, never published) and 3rd on intelligence (70.4 against Jev's 63.6).
- On JevBench v1.6.1, within 2 points of the highest on calibration (88.7; Jev 90.6) and on speed (91.2; H2O-Lightning-4B 92.6).
- On JevBench v1.6.1, 9th on the board's cost axis (51.7; H2O-Lightning-4B 60.3 is highest). On your own hardware you pay for the card, not per token.
On the composite, decisio's 95% interval overlaps each of the other three compared on this site, so none of them leads. The four axes have no published intervals; the composite has its 95% interval, shown in small text. The board's rows for decisio are v0.8.0.
Every number
| System | Overall | Intelligence | Sealed questions | Calibration | Speed | Cost |
|---|---|---|---|---|---|---|
| H2O-Lightning-4B | 72.52 | 60.0 | 59.0 | 90.0 | 92.6 | 60.3 |
| Decisio 31B v0.8.0 (ours) | 71.69 | 70.4 | 73.3 | 88.7 | 91.2 | 51.7 |
| Jev 1.13.0 (reference) | 71.49 | 63.6 | 61.6 | 90.6 | 91.5 | 54.7 |
| Quyet-1.0-Large | 71.39 | 73.4 | 71.6 | 90.0 | 86.9 | 50.5 |
| decider-12b v2 | 70.88 | 63.0 | 63.6 | 82.0 | 90.5 | 57.8 |
| decider-12b v1 | 70.42 | 60.6 | 59.5 | 83.8 | 90.5 | 57.8 |
| torchcast-decision-12b | 69.91 | 60.5 | 61.1 | 82.9 | 91.7 | 56.4 |
| Winnow-12B Q8 | 68.89 | 59.5 | 59.2 | 83.0 | 86.7 | 56.6 |
| deck-31B | 68.68 | 73.0 | 75.8 | 82.2 | 86.7 | 49.7 |
| Cygnet | 68.55 | 54.8 | 51.3 | 87.0 | 91.8 | 56.4 |
| Decisio 12B v0.8.0 (ours) | 68.17 | 52.3 | 51.6 | 89.2 | 87.0 | 59.3 |
The highest value in each column is in bold. Source: the board's file for revision v1.6.1.
The rows read as frozen weights
| Rank of 135 | System, as the board describes it | Score | 95% interval |
|---|---|---|---|
| 4 | decisio v0.8.0 on gemma-4-31B-it (frozen, FP8 on load, one prefill per state, self-hosted) | 71.69 | 69.86 to 72.81 |
| 9 | decider-12b v1, stock Gemma-4-12B-it (Mapika) | 70.42 | 68.09 to 71.63 |
| 12 | deck-31B (krishna765, frozen Gemma-4-31B-it, TorchAO FP8 dynamic) | 68.68 | 67.07 to 69.74 |
| 13 | Cygnet (blockbrain, frozen Gemma-4-12B-it) | 68.55 | 64.42 to 69.94 |
| 14 | decisio v0.8.0 on gemma-4-12B-it (frozen, one forward pass, self-hosted) | 68.17 | 57.37 to 69.65 |
| 15 | decisio v0.9.0 on gemma-4-12B-it (frozen, one prefill per question, self-hosted) | 67.76 | 54.38 to 69.56 |
| 20 | decider chat on Gemma-4-31B-it (Mapika, frozen base, inference technique) | 66.08 | 63.83 to 67.01 |
Scores this close cannot be separated: decisio's interval overlaps H2O-Lightning-4B, Jev 1.13.0 and Quyet-1.0-Large. The 12B rows' intervals are wide on the low side.
"Frozen weights" is our reading of each row's display text on the board, which has no such class; the table above shows the text.
Our temperatures were not fitted on JevBench, but we have read its published items (the evaluation card, section 7.4).
Source: the JevBench board, its file for revision v1.6.1 (benchmarkheaven.com/api/jevbench/v1.6.1, sha256 5cd8c1332226), read 2026-10-09.
Many questions about one document
On one box, Decisio 31B answers 50 questions about one 3,000-token document in 1.08 s and about one 6,000-token document in 1.73 s; Decisio 12B takes 1.57 s and 1.97 s, and H2O-Lightning-4B 2.05 s and 2.82 s.
On one box, Jeff-1.0-Large and Quyet-1.0-Large, the two systems ranked above Decisio on JevBench's capability score, take about 44 times (3,000 tokens) and 66 times (6,000 tokens) as long as Decisio 31B to answer 50 questions about one document, and neither accepts a 32,000-token document.
At 32,000 tokens, Decisio 31B answers 50 questions about one document in 10.1 s, Decisio 12B in 6.76 s and H2O-Lightning-4B in 9.05 s; Jeff-1.0-Large accepts documents of up to 8,192 tokens and Quyet-1.0-Large up to 6,000, so neither accepts a 32,000-token document.
On one box, the first question about a new 3,000-token document takes a median of 112 ms on H2O-Lightning-4B, 295 ms on Decisio 12B, 490 ms on Decisio 31B, and 895 ms on Jeff-1.0-Large and 895 ms on Quyet-1.0-Large.
Measured by Tachara AI Lab. One box, one session: an RTX PRO 6000 at 600 W (enforced), each system on its own published server, SM clock p50 2,220 to 2,610 MHz (the box row); median of 5 or 10 documents (the count is in every cell); synthetic support tickets by the Gemma 4 tokenizer; this compares time, not accuracy. This chart: five systems were measured in one session on one box; the other five on the earlier box, drawn as hollow rings.
What these numbers do not show
Source: EVAL_CARD.md
- One card per run and one run per configuration.
- JevBench's v1.5 reading is taken on JevBench's 231 published questions, of the board's 904 open items, with no judge tier and no sealed half; its published items have been public since v1.2 and may be in any model's pretraining data.
- The Decision Index rows of sections 3, 6 and 7 were rebuilt with the kit for four benchmarks; their counts and the rows' sha256 match (
decisio.bench.di_rows), but a partial rebuild cannot be checked against the whole suite's hash. Section 8's runs used the whole suite, rebuilt with the kit and matched against the lab's hashes. - The ImajevBench record was made without the rendering rules and the key-order tie-break; 2 of its 254 items ended in exact ties that the harness rejected and are counted wrong.
- The second-engine head mode with the registered text-only class is unmeasured for latency.
- Latency is server-side on the card's localhost; cost is the card-hour price divided by measured throughput, with nothing else counted.
- Latency depends on the host and the card's power limit.
The latency rows marked 585 W come from one RTX PRO 6000 Blackwell Workstation Edition whose power limit was 585 W (default 600 W), on an AMD Ryzen Threadripper 9960X host (
runs/2026-10-06_latency-585w/). On the Gemma bases the limit held the clock back on 72 to 92% of the busy samples, one request at a time included, so a card at 600 W may read new states faster; on the Qwen base it did on 0 to 2%. A cached Qwen question depends on the CPU instead: 39.4 ms in-process on an AMD EPYC 7452 host, 18.5 ms on an AMD Ryzen 9 9950X and 20.2 ms on the Threadripper, the same card model and vLLM (runs/2026-10-05_engine-death-gates/,runs/2026-10-06_latency-0.8.1/). Latency rows not marked come from hosts whose CPU and power limit were not recorded. - The per-item Decision Index records keep each request's id, the payload's sha256 and our response, not the item text (GPQA's authors ask that its items not be published in plain text).
Every table
The evaluation card has the full tables for each base, the calibration figures and what was fitted on what: the evaluation card.