Streaming
A streaming request is answered 200 the moment its order is in the book, so the stream carries the wait as well as the answer. This page is every frame it can carry, in order.
The frames
Server-sent events over the open connection, with the headers below. Lines that start with a colon are comments: the OpenAI SDKs skip them, and a hand-written parser should too, before calling JSON.parse on the rest.
| Frame | What it means |
|---|---|
| : queued bid=0.05 | Sent as soon as the order is in the book, with the price it rests at. |
| : queued level=0.0143 position=12 | Where the order sits and what the level is, sent whenever the stream has been quiet for a moment. A comment rather than an event, so SDKs ignore it. |
| : waiting | The same keep-alive once the order has left the book: admitted, and waiting for the first token. |
| data: {...} | An ordinary OpenAI chunk, forwarded as the model sends it. |
| : requeued (transient backend failure), waiting | The backend failed before any output and the order went back into the book. Nothing was charged and nothing is lost. |
| data: {"usage": {...}, "choices": []} | The receipt, sent after billing has settled and only when the request asked for it with stream_options.include_usage. |
| data: {"error": {...}} | A failure after the 200, in the same envelope as an HTTP error, carrying the same message, type and code. |
| data: [DONE] | The last frame either way, after a completed response or after an error. |
The queue comments are the normal case rather than an edge. A request whose max price is under the level can sit in the book for the whole of its timeout before the first token, and the comments are what keep the connection alive through it. Each names the order's place in the book, so a client can show progress, and a client that times out on silence should time out on frames of any kind.
| Header | Value |
|---|---|
| content-type | text/event-stream; charset=utf-8 |
| cache-control | no-cache, no-transform |
| x-accel-buffering | no |
The chunks
Chat chunks are the model's own chat.completion.chunk objects. Three fields in a delta need assembling on the client, and every OpenAI SDK does it.
| Field | How it arrives |
|---|---|
| delta.content | Text fragments, in order. |
| delta.reasoning_content | A reasoning model's chain of thought, as string fragments, ahead of the content. |
| delta.tool_calls[].function.arguments | Split across chunks. Concatenate by the call's index; the id, type and name arrive on the first fragment. |
On /v1/completions each chunk is a text_completion with choices[0].text. The model's own usage chunk is consumed by the gateway and never forwarded; the receipt below is the one that reaches you.
The receipt in a stream
Send stream_options: {"include_usage": true} and one more chunk arrives after the last token: an empty choices list and a usage block carrying x_market. Without it no usage is sent on a stream.
import osfrom openai import OpenAIclient = OpenAI(base_url="https://api.openfill.ai/v1", api_key=os.environ["OPENFILL_API_KEY"]) stream = client.chat.completions.create( model="placeholder-large", messages=[{"role": "user", "content": "hello"}], stream=True, stream_options={"include_usage": True}, extra_body={"max_price": 0.10},)for chunk in stream: if chunk.choices: print(chunk.choices[0].delta.content or "", end="") if chunk.usage: print("\n", chunk.usage) # carries x_marketimport OpenAI from "openai";const client = new OpenAI({ baseURL: "https://api.openfill.ai/v1", apiKey: process.env.OPENFILL_API_KEY }); const stream = await client.chat.completions.create( { model: "placeholder-large", messages: [{ role: "user", content: "hello" }], stream: true, stream_options: { include_usage: true }, }, { headers: { "X-Max-Price": "0.10" } },);for await (const chunk of stream) { process.stdout.write(chunk.choices[0]?.delta?.content ?? ""); if (chunk.usage) console.log("\n", chunk.usage); // carries x_market}Errors on a streaming request
A streaming request commits its status the moment the order is queued, so the response is 200 before the order has cleared. A failure before that point is an ordinary HTTP error from the status table. Every failure after it arrives inside the stream instead, as an OpenAI error envelope carrying the same message, type and code, followed by data: [DONE]. That covers 400, 408, 409, 429, 500, 502 and 503.
400 is on that list because the model itself can refuse a request, for example a prompt longer than the context window. A non-streaming request gets that back as a real 400; the same request with stream: true is already committed to 200 by the time the model sees it, so the refusal arrives in band like any other. Whatever was delivered before an error stays delivered, and a request that fails after streaming began is never charged.
Caution
An in-band error has no status code and no Retry-After header, and SDK retry logic keys off the status, so it does not engage. A streaming client branches on the error event and applies the backoff itself.
Closing the connection
Close a streaming connection while the order is queued and the order is cancelled, with no charge. Close it while the order is executing and the generation runs to completion and is charged; the tokens go to a socket nobody is reading. To hold a place in the book without holding a socket, send the request detached instead.