Batch API
The same shape as OpenAI's Batch API, so existing tooling works with a base_url change. The one difference is the price: instead of a fixed discount, the batch names a price limit and every line on a market model clears at the level of its own moment.
Submitting a batch
import jsonimport osimport timefrom openai import OpenAIclient = OpenAI(base_url="https://api.openfill.ai/v1", api_key=os.environ["OPENFILL_API_KEY"]) # 1. write and upload one request per linerequest = { "custom_id": "job-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "market-preview", "messages": [{"role": "user", "content": "hi"}]},}with open("requests.jsonl", "w") as source: source.write(json.dumps(request) + "\n")with open("requests.jsonl", "rb") as source: f = client.files.create(file=source, purpose="batch") # 2. create the batch (extra_body sets the price for every line)batch = client.batches.create( input_file_id=f.id, endpoint="/v1/chat/completions", completion_window="24h", extra_body={"price_limit": 0.02}, # a low price waits for cheap troughs) # 3. wait for a terminal state, then collect both files when presentwhile batch.status in {"validating", "in_progress", "cancelling"}: time.sleep(5) batch = client.batches.retrieve(batch.id)for file_id, path in [(batch.output_file_id, "output.jsonl"), (batch.error_file_id, "errors.jsonl")]: if file_id: client.files.content(file_id).write_to_file(path)if batch.errors: print(batch.errors)The batch price applies to every line on a market model, and each line is charged the level at its own execution, up to that price. Lines in one batch can clear at different levels, and a line that never executes is never charged. A line can override the price with body.price_limit.
Every line on a market model needs a price: its own body.price_limit, the batch price, or a default for its model on the key or account. A batch can be created without a default; a line with no resolved price lands in the error file as invalid_request. Batch lines always run with price locking off.
A line on a fixed-price model needs no price. Any price_limit on the line or the batch is ignored for it, and it is charged that model's prices as they stood when the batch was created. A line whose model's prices changed after that is not run or charged, and lands in the error file under prices_changed with a message saying so. One batch can mix the two kinds line by line.
A line on a preview of the market is not run or charged, and lands in the error file with the message a request to it answers with 400 model_preview.
The deadline
completion_window accepts "24h" only, matching OpenAI, and the batch object echoes it. The deadline itself is max_wait on the batch, in seconds, and defaults to a day; expires_at on the batch is the instant. A positive integer up to 2592000 is accepted and then clamped to the configured order timeout range on GET /v1/rate-limits. Lines still queued when it passes are marked expired, are never charged, and are reported in error_file_id, with completed lines that have retrievable results in output_file_id. The batch itself is then marked expired.
Cancel with POST /v1/batches/{id}/cancel. Queued lines are cancelled and never charged; a line already executing finishes and is charged, and lands in the output file. Batch orders do not count against your interactive queue cap; they have a cap of their own, in the size limits table on this page.
The input file
One JSON object per line, in the OpenAI shape. Blank lines are skipped, and the whole file is validated when the batch starts rather than at upload.
| Field | Rule |
|---|---|
| custom_id | A string of 1 to 256 characters, unique in the file. A duplicate fails the whole batch. |
| method | POST. |
| url | The batch's endpoint: /v1/chat/completions, /v1/completions, /v1/responses or /v1/messages. A line's body is in that endpoint's shape, and so is its output line; a line the endpoint would refuse lands in the error file with the same sentence. |
| body | A request in the endpoint's shape, validated and translated as on that endpoint. Its storage fields follow that wire's rules. |
What a line cannot do
| Field | Outcome |
|---|---|
| stream: true | The line is rejected. |
| price_lock, max_wait, expires_after | Validated when the endpoint declares them, then ignored. The batch deadline applies, and lines never lock. |
| detached | Ignored on chat and completions. detached: true is refused on Responses and Messages by their translators. |
| n or best_of above 1 | The line is rejected, as an interactive request would be. |
| No price_limit anywhere | On a market model, the line is rejected with a message naming its model, under invalid_request in the error file. Precedence is the line, then the batch, then the key, then the account. A line on a fixed-price model needs none. |
A rejected line is reported in the error file with invalid_request and a message naming the field, and the rest of the batch runs. Chat, completions and Responses use the model's full output length when no cap is supplied. Messages still requires max_tokens on each line.
A line's image URLs are fetched when the line becomes an order, as the batch works through the file, and a batch at your cap on queued batch orders waits in validating until lines finish, so a line late in a large file can be fetched as late as the batch's deadline (its max_wait). A URL that expires, such as a presigned one, has to last until then. A line whose URL cannot be fetched lands in the error file under invalid_request with the reason. While URLs cannot be fetched on our side, the line waits and is tried again until the deadline, and a line still waiting then lands under image_fetch_unavailable without having run or been charged.
Following a batch
| Status | Meaning |
|---|---|
| validating | The input file is being read and each line turned into an order. A batch waits here while your queued batch orders are at their cap. |
| in_progress | Every line is an order, and a line on a market model clears at its own level. Counts move as lines finish. |
| cancelling | You cancelled it. Queued lines are being cancelled; executing lines finish and are charged. |
| completed | Every line ended. The output file is written when at least one line has a result, and the error file when at least one does not; a file that would be empty is not created and its id stays null. |
| expired | The deadline passed with lines still queued. Those lines are in the error file; the rest are in the output file. |
| cancelled | Finished after a cancel, with output and error files for what ran and what did not. |
| failed | Validation failed before any line ran. errors on the batch says why. |
request_counts is the OpenAI trio: total, completed and failed, where failed folds in expired and cancelled lines. x_market on the batch takes the sum apart.
| Field | Meaning |
|---|---|
| x_market.active | Lines queued or executing right now. |
| x_market.pending_result | Lines that finished but whose body is still being written. |
| x_market.expired | Lines still queued when the deadline passed. Never charged. |
| x_market.canceled | Lines cancelled while queued. Never charged. |
| x_market.bid | The batch's price_limit, or null when the batch was created without one and each line is priced by its own price_limit or by a key or account default. |
Poll GET /v1/batches/{id} at a gentle interval; it shares the per-account order budget on rate limits. For one line's level and charge, list its orders: GET /v1/orders?batch={id}&include=usage,pricing.
The output and error files
Both are JSONL, keyed by custom_id. Order is not guaranteed, so match on it. A line that was not valid JSON has no custom_id of its own and is reported under a synthetic one beginning __invalid_. Each output line carries the completion under response.body, with usage.x_market inside it, and response.request_id is the order id.
{"id": "batch_req_...", "custom_id": "job-1", "response": {"status_code": 200, "request_id": "<order_id>", "body": { ...the completion... }}, "error": null}{"id": "batch_req_...", "custom_id": "job-2", "response": null, "error": {"code": "expired", "message": "..."}}| Code | Meaning |
|---|---|
| invalid_request | The line failed validation. The message names the field. |
| expired | The line was still queued when the deadline passed. On a fixed-price model the message says it did not start before the batch's deadline: resubmit the line, create the batch with a longer max_wait, or add credits if your balance ran out. |
| prices_changed | The model's prices changed after the batch was created, so the line was not run or charged. Resubmitted as it is, it runs at the new prices. |
| cancelled | The line was queued when you cancelled the batch. |
| backend_error | The line ran and the model failed it. The message is the order's own error. |
| image_fetch_unavailable | The line's image URLs could not be fetched on our side before the batch's deadline. The line was not run or charged; resubmitted, it runs. |
| result_unavailable | The line ran and its result was lost before the file was assembled. It was refunded. |
| output_file_too_large | The line ran but the output file reached its size ceiling first. It was refunded; split the batch. This line keeps a response object with the order id and a null body beside its error, so branch on error. |
Download either with GET /v1/files/{id}/content. A refunded line is never both charged and reported failed.
Size limits
| Size limit | Value | When you reach it |
|---|---|---|
| Requests per input file | 10,000 | The batch fails validation. |
| Input file size | 50 MB | 413 on upload. |
| Batches running at once, per account | 5 | 429 on the next create, with Retry-After. |
| Queued batch orders, per account | 5,000 | The batch stays in validating and fans out the rest as lines finish. |
| Batch submissions per minute, per account | 1,200 | 429 with Retry-After. |
Storage and collection
Caution
Batch is the one place storage cannot be off. An output file is kept for at least 1 day after the batch completes, even with storage off. A storage period longer than that keeps it for longer. After that it is deleted permanently and cannot be recovered.
Every batch object carries x_retention.output_expires_at, the instant the output file is deleted. It is null until the batch completes, and it accounts for a storage period lowered after the file was written.
When your setting leaves the output on that minimum and the batch produced an output file, OpenFill emails the deletion time and a download command as soon as the batch finishes.
batch = client.batches.retrieve(batch.id)batch.x_retention out = client.files.content(batch.output_file_id){"output_days": 1, "output_expires_at": 1785657600, "output_deleted": false}What is kept, and for how long
An input file expires that long after you upload it, and an output file that long after the batch completes.
Raising your storage period applies to batches that finish after the change. An expiry is set when a file is written, so a file that already exists keeps the one it was given.
Once the batch has assembled its output, the per-order results inside it are evicted on your normal storage schedule. The output file is the durable copy.