Streaming
A streaming request is answered 200 the moment its order is in the book, so the stream carries the wait as well as the answer. This page covers chat and completions; Responses and Messages use the event sequences in their API references.
The frames
Server-sent events over the open connection, with the headers below. Lines that start with a colon are comments: the OpenAI SDKs skip them, and a hand-written parser should too, before calling JSON.parse on the rest.
| Frame | What it means |
|---|---|
| : queued bid=0.05 | Sent as soon as the order is in the book, with the price it rests at. |
| : queued level=0.0143 position=12 | Where the order sits and what the level is, sent whenever the stream has been quiet for a moment. A comment rather than an event, so SDKs ignore it. |
| : queued | Both of the comments above, on a fixed-price model: it has no bid or level to report, and requests there take turns across accounts, so a place in its book is no rank. |
| : waiting | The same keep-alive once the order has left the book: admitted and waiting for the first token, or mid-generation with no token for a moment. |
| : requeued (transient backend failure), waiting | The backend failed before any output and the order went back into the book. Nothing was charged and nothing is lost. A failure after any output is a 502 in band instead. |
| data: {...} | An ordinary OpenAI chunk, forwarded as the model sends it. |
| data: {"usage": {...}, "choices": []} | The receipt, sent after billing has settled and only when the request asked for it with stream_options.include_usage. |
| data: {"error": {...}} | A failure after the 200, in the same envelope as an HTTP error, carrying the same message, type and code. |
| data: [DONE] | The last frame either way, after a completed response or after an error. |
The queue comments are the normal case rather than an edge. A request whose price limit is under the level can sit in the book for the whole of its timeout before the first token, and the comments are what keep the connection alive through it. Position comments name the order's place in the book so a client can show progress. A client that times out on silence should time out on frames of any kind. Only position comments carry a queue rank.
| Header | Value |
|---|---|
| content-type | text/event-stream; charset=utf-8 |
| cache-control | no-cache, no-transform |
| x-accel-buffering | no |
The chunks
Chat chunks are the model's own chat.completion.chunk objects. Three fields in a delta arrive as fragments and need assembling on the client.
| Field | How it arrives |
|---|---|
| delta.content | Text fragments, in order. |
| delta.reasoning_content | A reasoning model's chain of thought, as string fragments, in its own field beside the content. |
| delta.tool_calls[].function.arguments | Split across chunks. Concatenate by the call's index; the id, type and name come on whichever fragment carries them, normally the first. |
On /v1/completions each chunk is a text_completion with choices[0].text. The model's own usage chunk is consumed by the gateway and never forwarded; the receipt below is the one that reaches you.
The receipt in a stream
Send stream_options: {"include_usage": true} and one more chunk arrives after the last token: an empty choices list and a usage block carrying x_market. On a live chat or completions stream, usage is sent only when requested. An idempotent streaming replay carries the whole result and usage in one chunk, followed by [DONE].
import osfrom openai import OpenAIclient = OpenAI(base_url="https://api.openfill.ai/v1", api_key=os.environ["OPENFILL_API_KEY"]) stream = client.chat.completions.create( model="market-preview", messages=[{"role": "user", "content": "hello"}], stream=True, stream_options={"include_usage": True}, extra_body={"price_limit": 0.10},)for chunk in stream: if chunk.choices: print(chunk.choices[0].delta.content or "", end="") if chunk.usage: print("\n", chunk.usage) # carries x_marketimport OpenAI from "openai";const client = new OpenAI({ baseURL: "https://api.openfill.ai/v1", apiKey: process.env.OPENFILL_API_KEY }); const stream = await client.chat.completions.create( { model: "market-preview", messages: [{ role: "user", content: "hello" }], stream: true, stream_options: { include_usage: true }, }, { headers: { "X-Price-Limit": "0.10" } },);for await (const chunk of stream) { process.stdout.write(chunk.choices[0]?.delta?.content ?? ""); if (chunk.usage) console.log("\n", chunk.usage); // carries x_market}Errors on a streaming request
A streaming request commits its status the moment the order is queued, so the response is 200 before the order has cleared. A failure before that point is an ordinary HTTP error from the status table. Every failure after it arrives inside the stream instead, as an OpenAI error envelope carrying the same message, type and code, followed by data: [DONE]. That covers 400, 408, 409, 429, 500, 502 and 503.
400 is on that list because the model itself can refuse a request, for example a prompt longer than the context window. A non-streaming request gets that back as a real 400; the same request with stream: true is already committed to 200 by the time the model sees it, so the refusal arrives in band like any other. Whatever was delivered before an error stays delivered, and a stream that fails before its answer is complete is never charged. The one exception is a 500 after the last token, when the answer was produced but billing could not be confirmed: its message distinguishes a recorded charge from settlement that could not be confirmed. Check the order and usage before starting the work again.
Caution
An in-band error has no status code and no Retry-After header, and SDK retry logic keys off the status, so it does not engage. A streaming client branches on the error event and applies the backoff itself.
Closing the connection
Close a streaming connection while the order is queued and the order is cancelled, with no charge. Close it while the order is executing and the generation runs to completion and is charged; the tokens go to a socket nobody is reading. A gateway restart is the exception: a request still waiting in the book when a deploy lands is answered a restart error inside the stream and cancelled with no charge, so the retry queues on the new gateway and runs once. A detached or batch order holds no connection and is picked up where it stood. To hold a place in the book without holding a socket, send the request detached instead.