Watch for drift

A certified task watches its own traffic with three checks that need no labels. Certification is suspended while the confidence alarm is on, so a shift does not quietly fill your queue with wrong certified answers:

Figure · Watching for drift

The alarm suspends certification soon after the traffic shifts.

The shaded area is the share of messages the task has no option for, from 0% to its peak; each of them is a wrong answer. While certification is suspended, answers keep coming, none marked certified.

Source: the teach-and-certify drift stream, decisio-gemma-4-31b FP8 on one RTX PRO 6000 Blackwell, measured on the decisio 0.16.0 release candidate.

1. Know what it watches

Check	What it sees	What it does
The serving setup	any change to the setup the certificate was made on	suspends certification at once: serving_stack_changed
The confidences	how often an answer’s top probability falls below the certification examples’ 5th percentile	suspends certification while it alerts: traffic_changed, with confidence among its signals
The answer mix	how often each option was chosen in the last 500 answers, checked every 100	reports traffic_changed with answer_mix; changes nothing

• The confidence alert clears once the traffic is back to normal for about 300 answers.
• While certification is suspended, the answers themselves do not change: they come back marked certified: false, with the reason.
• The answer mix is weak on a task with many options, where a shift spread over them barely moves the mix. That is why only the confidence check suspends.

2. Read the status

GET /v1/tasks/{id} gives a task’s status: ok, suspended with what changed, or traffic_changed with the signals that alerted. For the task Certify answers registers:

curl -s http://127.0.0.1:8000/v1/tasks/ticket-routing \
  | uv run python -c 'import json, sys; s = json.load(sys.stdin)["status"]; print(json.dumps({k: s.get(k) for k in ("state", "text")}, indent=1))'

Run against the CPU stand-in at v0.18.1 on 2026-10-19. It printed:

{
 "state": "ok",
 "text": "ok"
}

GET /v1/tasks lists every task’s status.

3. Alert on it

GET /metrics serves Prometheus text for certified tasks:

curl -s http://127.0.0.1:8000/metrics | grep '^decisio_task_'

Run against the CPU stand-in at v0.18.1 on 2026-10-19. It printed, trimmed to the fields this page uses:

decisio_task_state{task="ticket-routing",state="ok"} 1
decisio_task_state{task="ticket-routing",state="traffic_changed"} 0
decisio_task_state{task="ticket-routing",state="suspended"} 0
decisio_task_answers_total{task="ticket-routing"} 0
decisio_task_certified_answers_total{task="ticket-routing"} 0
decisio_task_mix_alerts_total{task="ticket-routing"} 0
decisio_task_window_answers{task="ticket-routing"} 0
decisio_task_mix_line{task="ticket-routing"} 0.23266866666666663
decisio_task_confidence_statistic{task="ticket-routing"} 0.0
decisio_task_confidence_limit{task="ticket-routing"} 8.617628257164982
decisio_task_confidence_alerts_total{task="ticket-routing"} 0

Metric	What it counts
decisio_task_state{task,state}	each task’s state
decisio_task_confidence_statistic, _limit, _alerts_total	the confidence check: where it stands, where it alerts, and how often it has
decisio_task_mix_distance, decisio_task_mix_line	the answer mix, and the line it alerts at

It also counts each task’s answers and certified answers. The counters live in memory and start again when the server does.

4. Check it with labels

The checks cannot see a wrong answer, because they have no labels. The only direct check that the promise holds on live traffic is to label a sample of it and certify again.

• Give the task an “other” option. A message that fits none of the options gets a wrong answer, and the checks only notice once enough of them arrive.
• To report without suspending, certify with "suspend_on_alert": false in the certify block.

What next

• Certify answers: the promise these checks protect, and how to certify again.
• Operate: health, the exit code and what else to watch in production.
• Certified answers on real tasks: a drift alarm on real traffic, run on the release it was measured on.