Streaming

Set stream: true and the response arrives as server-sent events, in the OpenAI format: one data: line per chunk, each carrying choices[].delta, terminated by data: [DONE].

Any OpenAI-compatible client reads this without changes.

Usage arrives at the end

The final chunks carry a usage object with the token counts for the whole response. That is the same block a non-streamed response returns, and the same numbers you are billed on.

Failures after the first chunk

This is the one place where streaming behaves differently from what you might expect, and it is worth handling explicitly.

Once the first chunk has been sent, the HTTP status is already 200 and cannot be taken back. A failure after that point therefore cannot be an HTTP error. What you get instead is an error event in the stream, in the OpenAI error shape, followed by [DONE]:

data: {"error":{"message":"...","type":"upstream_error"}}

data: [DONE]

Note also that a failure this late cannot be retried on another backend — see Failover & retries. Failover happens while the request is being set up; after that the choice is committed.

What a broken stream costs

Tokens that were generated and delivered are billed. The request appears in the usage log with the partial disposition. See Request dispositions.

Disconnecting

If you hang up mid-stream, generation is not cancelled — the request runs to completion so it can be billed and recorded accurately, and you are charged for what it produced. If you want to stop paying for an answer you no longer need, cap it with max_tokens rather than dropping the connection.

Idempotency does not apply

Idempotency-Key is ignored on streamed requests. A stream cannot be replayed from a cache, so retry logic on streamed calls has to be yours.