# Long documents and many options

> Prompts up to 32,768 tokens and questions with up to 255 options; what each costs, and where a registered head helps.

Source: https://opendecisio.com/docs/guides/long-and-wide/

Two limits matter when a question grows: how long the text is, and how many options the answer has. Both are generous, and both have a cost worth knowing before you plan around them.

 

[Figure: Up to 32,768 tokens and 255 options; a head helps most when the list is long. On the 31B the first question about a document takes 175 ms at 1,000 tokens, 490 ms at 3,000 tokens, 1018 ms at 6,000 tokens, 8655 ms at 32,000 tokens. With a registered intent head, accuracy on BANKING77 (77 options) goes from 0.747 to 0.840 and CLINC150 (150 options) goes from 0.820 to 0.912.]

 

## TL;DR

 

With a server running ([Get started](https://opendecisio.com/docs/) starts one), this script asks one choice question with 120 options and prints how many options came back:

 

```
import json
import urllib.request

options = {f"option_{i:03d}": f"The ticket belongs to category {i}" for i in range(120)}
body = {
    "state": "I was charged twice for October. Both charges are 49.00 EUR, same day. Please refund one.",
    "questions": {"category": {"type": "choice", "instructions": "Which category does this ticket belong to?", "criteria": options}},
}
request = urllib.request.Request(
    "http://127.0.0.1:8000/v1/systemone",
    data=json.dumps(body).encode(),
    headers={"Content-Type": "application/json"},
)
with urllib.request.urlopen(request) as response:
    answer = json.load(response)["answers"]["category"]
print("options answered:", len(answer["probabilities"]))
# options answered: 120
```

 

Run against the CPU stand-in at v0.11.1 on 2026-10-14.

  

| Word | Meaning |
| --- | --- |
| context | what a prompt can hold: the state with one question, up to 32,768 tokens |
| option | one answer a choice question can give; up to 255 in a question |
| head | a registered correction on the model’s hidden state, kept only for questions with 10 or more options |

 

## 1 Long documents

 

A prompt, the state with one question, can hold up to 32,768 tokens; one longer is answered with 422 and a `detail` that says the rendered prompt is longer than the context, and the server stays up. The first question about a document pays for reading it, and that grows with the length: on the 31B, 175 ms at 1,000 tokens, 490 at 3,000, 1,018 at 6,000 and 8,655 at 32,000. Every later question about it reads from the cache.

 

## 2 Many options

 

A choice can have up to 255 options; a request with more gets 400. A long list of fine-grained labels is where a generalist’s reading of the label names is weakest, and where a registered head helps most: on the two intent sets of 77 and 150 options, heads from 10 examples per intent added 7 to 11 points of accuracy. A head is only fitted for lists of 10 or more options, and the list must be fixed: options that change per input cannot be registered.

 

Common mistake

Reordering, adding or renaming an option makes a different task, which needs its own registration. A head is also bound to the option order it was fitted on.

 

## What next

 

- [Teach it your question](https://opendecisio.com/docs/tasks/): register the question and its options.
- [Make it fast](https://opendecisio.com/docs/guides/latency/): the first read of a long state, and what changes it.
- [Many questions about one text](https://opendecisio.com/docs/guides/many-questions/): ask more of the text you have read.