Request lifecycle
Everything that happens between your POST and the last token of the response.
One POST /v1/chat/completions passes through the stages below. Most of them are
invisible when everything works; each one is a place a request can end early, and
knowing which stage produced an error is usually the whole of debugging it.
The stages
Read the body
The body is read with a hard cap of 8 MiB — over it, 413.
Check idempotency
Only for a non-streamed request carrying an Idempotency-Key. The key is either free — the request
takes the lock and proceeds — or it has been seen, in which case you get the
stored response back without a second call to any model, or a 409 if the
original is still in flight.
Work out what the request needs
The raw body is scanned for capabilities, because they decide which backends may serve it:
| In your request | Capability required |
|---|---|
an image_url content part | image input |
a file or input_file content part | PDF input |
reasoning or reasoning_effort | reasoning |
tools (non-web) or a non-empty functions | tools |
web_search_options, plugins: [{ id: "web" }], a web-search tool | tools |
A plain chat request requires nothing and can go anywhere the model is served.
Build the candidate list
Routing resolves your model into an ordered list of targets — each one a
provider plus that provider's own name for the model. The order is configured,
not scored per request.
Try candidates, in order
For each candidate that is still usable:
- Estimate the prompt tokens, and take the output ceiling from your
max_tokensor a default. - Place a hold on your balance for that estimate. This happens per attempt — a request that tries two backends places and releases two holds.
- Send the body to that backend.
- On success, go to the next stage. On failure, void the hold immediately, classify the error, and decide whether to try the next candidate.
Return, settle, record
The response is written to you. Settlement is then enqueued as a background job: the hold is released, the real cost is charged, and the remainder returns to your balance — which is why a cost can be marked provisional for a short while. See Billing.
Last, one durable record of the request is written with its disposition, every attempt made, token counts, cost and latency. That record is what the Usage page shows you.
Your disconnect does not abort the call
Once a backend call has started it runs on a detached context: hanging up does not cancel it. The result is still needed to settle the hold correctly and to write an honest record.
Where errors come from
Every stage above can end the request, and the Errors page lists what each ending looks like on the wire. Two are worth knowing here:
- A failure before any bytes were sent becomes an HTTP status.
- A failure after streaming has started cannot — the status line has already gone out. The stream carries an error event and then ends. See Streaming.