Decisio

Demos

Pick an error rate, see what it can automate

We taught Decisio two public sets of customer intents and a coding agent's command rules from a few labelled examples each, then asked it to certify which answers can go out with no one checking them. Every message, answer and certificate on this page comes from those runs.

Teach your first taskWhat certified meansHow it was run

Inbox autopilot

A support inbox, split by the certificate

These are real messages from BANKING77's test split, answered by Gemma 4 31B after 10 labelled examples per intent. Pick an error target: the answers the certificate covers go out on their own, and the rest go to a person.

A stricter target sends more of the inbox to a person. That is what the promise costs.

Error target

Sent automatically

certified: the answer goes out as it is

  1. When should I receive my virtual card?

    getting_virtual_card0.96right

  2. What is the limit to number of transactions I can do with a disposable card?

    disposable_card_limits0.97right

  3. What do I need to do to verify the source of my funds?

    verify_source_of_funds0.98right

  4. How do I retrieve a trapped card from a ATM?

    card_swallowed0.96right

  5. I want to change my personal details.

    edit_personal_details0.98right

Sent to a person

not certified: a person checks the answer first

  1. I have a 1 euro fee on my statement.

    extra_charge_on_statement0.83right

  2. Where can I find the card PIN?

    get_physical_card0.31right

  3. Can I use American Express to add money into my account?

    supported_cards_and_currencies0.62right

Sent automatically, of all 3,080 test messages
1,335 43.3% of the inbox
Wrong among those
43 3.2% of them
The certificate's threshold
0.9413 certified at or above it

BANKING77's test split, credited under How it was run. Gemma 4 31B from its FP8 checkpoint on one RTX PRO 6000 Blackwell, decisio 0.14.0: taught with 770 examples, certified on 1,535 held-out ones. The 48 messages are a seeded random sample of the test split, not picked by hand.

The share automated

Stricter targets automate less

Each bar is the share of a test split the certificate covered, and how often those answers were wrong. Two intent sets, two bases, 10 labelled examples per intent.

  • 77.6% to 84.2%answered right, untaught and taught: BANKING77 on Gemma 4 31B
  • 73.1% to 82.3%answered right, untaught and taught: BANKING77 on Gemma 4 12B
  • 92.0% to 95.2%answered right, untaught and taught: CLINC150 on Gemma 4 31B
  • 87.3% to 93.1%answered right, untaught and taught: CLINC150 on Gemma 4 12B
The share of each test split certified, by error targetBANKING77, Gemma 4 31B: at 10%, 78.8% certified, 8.1% of them wrong; at 5%, 43.3% certified, 3.2% of them wrong; at 2%, nothing certified. BANKING77, Gemma 4 12B: at 10%, 73.1% certified, 8.5% of them wrong; at 5%, nothing certified; at 2%, nothing certified. CLINC150, Gemma 4 31B: at 10%, 100.0% certified, 4.8% of them wrong; at 5%, 100.0% certified, 4.8% of them wrong; at 2%, 90.9% certified, 2.2% of them wrong. CLINC150, Gemma 4 12B: at 10%, 100.0% certified, 6.8% of them wrong; at 5%, 95.2% certified, 4.5% of them wrong; at 2%, 82.6% certified, 2.4% of them wrong.BANKING77 Gemma 4 31B 10% 78.8%, 8.1% wrong 5% 43.3%, 3.2% wrong 2% nothing certified BANKING77 Gemma 4 12B 10% 73.1%, 8.5% wrong 5% nothing certified 2% nothing certified CLINC150 Gemma 4 31B 10% 100.0%, 4.8% wrong 5% 100.0%, 4.8% wrong 2% 90.9%, 2.2% wrong CLINC150 Gemma 4 12B 10% 100.0%, 6.8% wrong 5% 95.2%, 4.5% wrong 2% 82.6%, 2.4% wrong 0% 50% 100% The share of each test split certified, by error targetBANKING77, Gemma 4 31B: at 10%, 78.8% certified, 8.1% of them wrong; at 5%, 43.3% certified, 3.2% of them wrong; at 2%, nothing certified. BANKING77, Gemma 4 12B: at 10%, 73.1% certified, 8.5% of them wrong; at 5%, nothing certified; at 2%, nothing certified. CLINC150, Gemma 4 31B: at 10%, 100.0% certified, 4.8% of them wrong; at 5%, 100.0% certified, 4.8% of them wrong; at 2%, 90.9% certified, 2.2% of them wrong. CLINC150, Gemma 4 12B: at 10%, 100.0% certified, 6.8% of them wrong; at 5%, 95.2% certified, 4.5% of them wrong; at 2%, 82.6% certified, 2.4% of them wrong.BANKING77Gemma 4 31B 10% 78.8%, 8.1% wrong 5% 43.3%, 3.2% wrong 2% nothing certified BANKING77Gemma 4 12B 10% 73.1%, 8.5% wrong 5% nothing certified 2% nothing certified CLINC150Gemma 4 31B 10% 100.0%, 4.8% wrong 5% 100.0%, 4.8% wrong 2% 90.9%, 2.2% wrong CLINC150Gemma 4 12B 10% 100.0%, 6.8% wrong 5% 95.2%, 4.5% wrong 2% 82.6%, 2.4% wrong

At 10% the certificate covers most of each split. At 2% BANKING77 certifies nothing on either base: when the taught model is most sure, it is still wrong a little more than 2% of the time, and no number of held-out examples changes that. CLINC150's intents barely overlap, so even a strict target covers most of it.

Which examples you teach with matters at 5%

Three random draws of 10 examples per intent on BANKING77, each certified the same way. At 10% the draws agree; at 5% a handful of confident mistakes among the top answers decides how much is certified.

Three teaching draws, by error targetGemma 4 31B at 5%: 43.3%, 10.8%, 54.5%; Gemma 4 31B at 10%: 78.8%, 79.4%, 79.0%; Gemma 4 12B at 5%: 0.0%, 16.8%, 30.7%; Gemma 4 12B at 10%: 73.1%, 71.2%, 70.8%.Gemma 4 31B 10.8% to 54.5% 5% Gemma 4 31B 78.8% to 79.4% 10% Gemma 4 12B 0.0% to 30.7% 5% Gemma 4 12B 70.8% to 73.1% 10% 0% 50% 100% Three teaching draws, by error targetGemma 4 31B at 5%: 43.3%, 10.8%, 54.5%; Gemma 4 31B at 10%: 78.8%, 79.4%, 79.0%; Gemma 4 12B at 5%: 0.0%, 16.8%, 30.7%; Gemma 4 12B at 10%: 73.1%, 71.2%, 70.8%.Gemma 4 31B10.8% to 54.5% 5% Gemma 4 31B78.8% to 79.4% 10% Gemma 4 12B0.0% to 30.7% 5% Gemma 4 12B70.8% to 73.1% 10% 0% 50% 100%

The promise

Two hundred re-splits, one line to stay under

A certificate promises that, nine times in ten, at most the target share of its certified answers is wrong. To watch that hold, each task's held-out and test answers were pooled and split again 200 times: certify on one part, count the mistakes on the other.

Each dot is one split. A dot above the line is a split where the promise missed, and nine in ten is the bar.

The 200 re-splitsBANKING77, Gemma 4 31B: 200 re-splits at a 5% target. The promise held on 191 of them; 9 came in over the target; 16 certified nothing.0% 2% 4% 6% 8% 10% 0% 25% 50% 75% 100% target 5% share of the split certified wrong among the certified The 200 re-splitsBANKING77, Gemma 4 31B: 200 re-splits at a 5% target. The promise held on 191 of them; 9 came in over the target; 16 certified nothing.0% 2% 4% 6% 8% 10% 0% 50% 100% target 5% share of the split certified wrong among the certified
The promise held
191 of 200 splits
Over the target
9 splits
Certified nothing
16 splits, counted as held

On the official test split

Both 2% certificates on CLINC150 came in a little over the target on its official test split. That split is harder than the validation split the certificates were made on: there Gemma 4 12B is right 94.3% of the time and Gemma 4 31B 96.7%, against the test-split accuracy in the tiles above.

A certificate promises its rate for traffic like its examples, and that test split is not quite like them. Pick CLINC150 and 2% above to see the same answers pooled and re-split, which removes the difference.

Drift alarm

Traffic it was never taught, and a monitor that noticed late

The CLINC150 task was taught its 150 intents and no way to say a message fits none of them. A replay of 5,363 messages then mixed in off-topic ones, from none at message 1,000 to 30 in every 100 by message 4,000, and each one certified was a wrong answer sent out.

off-topic messages per 100 wrong answers certified per 100

The drift replayCLINC150, Gemma 4 31B: off-topic messages rise from none at message 1,000 to 30 in every 100 at message 4,000. The monitor's first alert came after message 5,200.0 10 20 30 40 50 0 1,000 4,000 5,363 the monitor at each reading: ok or alert first alert, after message 5,200 The drift replayCLINC150, Gemma 4 31B: off-topic messages rise from none at message 1,000 to 30 in every 100 at message 4,000. The monitor's first alert came after message 5,200.0 20 40 0 1,000 4,000 5,363 the monitor at each reading: ok or alert first alert after 5,200
The certificate's threshold
0.06 100.0% of the stream certified
Wrong answers certified before the alert
1,027
And after it, to message 5,300
32

The monitor has no labels: it watches the mix of answers and says when the mix moves. In decisio 0.14.0, where these replays ran on 2026-10-16, 2026-10-17 and 2026-10-18, an alert changed no answer and withdrew no certificate.

The fix: teach it to say none of these

Give the task a 151st option, out of scope, taught from 10 off-topic examples, and certify it on held-out examples that include 100 off-topic ones. An off-topic message then has a right answer, and the certificate counts it.

The same drift stream, off-topic rising from 0 to 30%: 150 options, then the out-of-scope option taught. decisio-gemma-4-31b FP8, decisio 0.14.0, 2026-10-18.
No out-of-scope optionOut of scope taught
Wrong among the certified20.1%8.4%
The alarm, first after message5,2002,600
Wrong answers certified before it1,027167
  • At a 10% target every message was automated and 8.4% were wrong: the promise held.
  • At 5% the same 8.4% missed the promise, because the certificate's examples were 3.2% off-topic, against the stream's 30%.

The promise covers traffic like the examples; the alarm flags when traffic shifts away from them.

From decisio 0.16.0: the alarm suspends certification

From 0.16.0 the confidence alarm suspends certification by default for a task certified on 0.16.0: answers keep coming, but none is marked certified until the traffic looks like the examples again. A task certified on an earlier version has no such alarm until it is certified again. Measured on the 0.16.0 release candidate, decisio-gemma-4-31b FP8.

  • On a stream whose traffic jumps at message 1,500, the first suspended answer came 74 messages after the jump, with 26 wrong answers certified before it. Certification resumed 316 messages after the traffic returned to normal.
  • On the CLINC150 stream above, the first suspension came at message 1,570, with 40 wrong answers certified before it, against 1,027 under 0.14.0.
  • On normal traffic it suspended nothing.

Gatekeeper

Allow, ask or block, for a coding agent's commands

A coding agent proposes a shell command, and a team has 16 written rules, each saying allow, ask or block. Gemma 4 12B was taught the rules from 20 labelled commands per rule and asked which rule covers each command; the action follows from the rule.

  • 34% to 40%of the commands decided on its own at a 5% error target, with the wrong action in 1.6% to 2.3% of those
  • 70% to 75%decided on its own at a 10% target
  • 16 of 1,799certified commands let through when the policy says ask or block, across the four taught versions of the task

That is about 1 in 100. Four were real model mistakes, such as gh stack merge 7; twelve were strict items in the policy's own lists, commands the rule text supports, such as test runs through python -m. All 16 count as errors, because the policy is the ground truth.

When a new kind of command arrives

Database and migration commands, a kind no rule names, rose to 30% of the traffic. None of the 184 was certified, and 160 went to "ask" through the policy's catch-all rule, "any other program".

The lesson beside CLINC150's: a catch-all option is what keeps certification safe when traffic shifts. CLINC150's task had none, so its off-topic messages went out certified as some other intent.

The 16 rules and their actions
  • inspect allow
  • git_read allow
  • git_local_write allow
  • git_discard ask
  • git_push ask
  • git_push_force_or_protected block
  • test_build_lint allow
  • run_scripts ask
  • install_packages ask
  • edit_workspace allow
  • delete_in_workspace ask
  • outside_workspace block
  • secrets block
  • network_and_cloud ask
  • remote_code_or_privilege block
  • other ask

Gemma 4 12B, the 6-bit MLX conversion mlx-community/gemma-4-12B-it-6bit, decisio 0.14.0, 2026-10-17 and 2026-10-18. Three draws of 20 commands per rule, each read on the same 1,481 test commands; one more version, taught with 10 per rule, was read across two server starts. The sources are under How it was run.

Method

How it was run

Decisio 0.14.0, every serving setting at its default, on 2026-10-16, 2026-10-17 and 2026-10-18. The model's weights never changed: teaching fits a small head on its hidden states.

The bases and the machines
  • Gemma 4 31B, its FP8 checkpoint, on one RTX PRO 6000 Blackwell (96 GB), reading 16 messages at a time.
  • Gemma 4 12B, the 6-bit MLX conversion, on one 64 GB Apple silicon Mac, one message at a time, under a memory cap.

Nothing here is a speed figure.

The intent sets and the commands
  • BANKING77: 77 banking intents. Casanueva, Temcinas, Gerz, Henderson and Vulic, Efficient Intent Detection with Dual Sentence Encoders (2020), PolyAI. CC BY 4.0.
  • CLINC150: 150 intents across 10 domains, with off-topic queries. Larson et al., An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction (EMNLP 2019), Clinc. CC BY 3.0.

The messages on this page are from their test splits, unchanged.

The gatekeeper's commands:

  • SWE-Hero OpenHands trajectories: NVIDIA, nvidia/SWE-Hero-openhands-trajectories (shell commands, shard 0). CC BY 4.0.
  • NL2SH-ALFA: westenfelder/NL2SH-ALFA (from nl2bash, tldr-pages, NL2CMD, LinuxCommands and InterCode). MIT (its parts keep their own licences; tldr-pages CC BY 4.0).
  • documentation shell lines: github/docs, kubernetes/website and MicrosoftDocs/azure-devops-docs (shell lines of the documentation). CC BY 4.0.
  • Templated commands and the database commands of the drift replay, generated for this demo by seeded scripts. CC0.
Teaching and the certificate

Each message is one question over every intent, "Which of these labels fits the text?", and the confidence is the answer's top probability. A task was registered with its teaching examples, and the server fitted an intent head on them.

The certificate was made on a held-out split that shares no example with the teaching set: drawn from BANKING77's training split, and all of CLINC150's validation split. It is Learn then Test with Clopper-Pearson bounds.

What it promises: with probability at least 90% over the draw of held-out examples, at most the target share of the certified answers is wrong, for traffic like those examples, on the serving stack it was made on.

The re-splits and the drift replay

For each task, the held-out and test answers were pooled and split again 200 times, by intent, into parts of the original sizes. A split that certified nothing counts as held, and the counts show how often that happened.

The drift stream holds every in-scope CLINC150 test message once, plus off-topic messages from its out-of-scope list, read one at a time through the task with its 5% certificate. The monitor's state was read every 100 messages.

Found and fixed in decisio 0.16.0

In 0.14.0, registration and serving scored a certificate's examples by different paths, so on the card a few confidences moved between the two. 0.16.0 certifies on the served path; a certificate made on an earlier version keeps the gap until it is made again.

Re-read through the served path at registration, 200 of 200 held-out examples gave the same confidence: the largest shift was 0.0.

What to keep in mind
  • A certificate holds on the serving stack it was made on: one made on the Mac does not carry to the card, or the reverse.
  • It holds for messages like its held-out examples, and it cannot cover an answer the task cannot give, such as out of scope.
  • Some mistakes are labels that overlap: "my wallet was stolen with my cards in it" is labelled compromised_card and was answered lost_or_stolen_card. They count as mistakes here, as they would in a real inbox.
  • One machine per base, one draw at 5 and 20 examples per intent, and no legal text, long documents, other languages or adversarial messages.