Skip to content
Decisio

DocsGuides

2 min read · updated

Long documents and many options

Prompts up to 32,768 tokens and questions with up to 255 options; what each costs, and where a registered head helps.

Two limits matter when a question grows: how long the text is, and how many options the answer has. Both are generous, and both have a cost worth knowing before you plan around them.

Figure · Long documents, many options

Up to 32,768 tokens and 255 options; a head helps most when the list is long.

A prompt, the state with one question, can hold up to 32,768 tokens, and a question can have up to 255 options.

Source: figures.same_box_speed of launch_facts.json (one RTX PRO 6000 at 600 W) and the tasks table of docs/tasks.md at the pinned release (Qwen3.6-35B-A3B, 10 labelled examples per intent, mean of six draws).

TL;DR

Section titled “TL;DR”

With a server running (Get started starts one), this script asks one choice question with 120 options and prints how many options came back:

import json
import urllib.request


options = {f"option_{i:03d}": f"The ticket belongs to category {i}" for i in range(120)}
body = {
    "state": "I was charged twice for October. Both charges are 49.00 EUR, same day. Please refund one.",
    "questions": {"category": {"type": "choice", "instructions": "Which category does this ticket belong to?", "criteria": options}},
}
request = urllib.request.Request(
    "http://127.0.0.1:8000/v1/systemone",
    data=json.dumps(body).encode(),
    headers={"Content-Type": "application/json"},
)
with urllib.request.urlopen(request) as response:
    answer = json.load(response)["answers"]["category"]
print("options answered:", len(answer["probabilities"]))
# options answered: 120

Run against the CPU stand-in at v0.11.1 on 2026-10-14.

WordMeaning
contextwhat a prompt can hold: the state with one question, up to 32,768 tokens
optionone answer a choice question can give; up to 255 in a question
heada registered correction on the model’s hidden state, kept only for questions with 10 or more options

1. Long documents

Section titled “1. Long documents”

A prompt, the state with one question, can hold up to 32,768 tokens; one longer is answered with 422 and a detail that says the rendered prompt is longer than the context, and the server stays up. The first question about a document pays for reading it, and that grows with the length: on the 31B, 175 ms at 1,000 tokens, 490 at 3,000, 1,018 at 6,000 and 8,655 at 32,000. Every later question about it reads from the cache.

2. Many options

Section titled “2. Many options”

A choice can have up to 255 options; a request with more gets 400. A long list of fine-grained labels is where a generalist’s reading of the label names is weakest, and where a registered head helps most: on the two intent sets of 77 and 150 options, heads from 10 examples per intent added 7 to 11 points of accuracy. A head is only fitted for lists of 10 or more options, and the list must be fixed: options that change per input cannot be registered.

What next

Section titled “What next”