Streaming

Set "stream": true to receive the response incrementally as server-sent events.

stream = client.chat.completions.create(
    model="qwen3.8-27b",
    messages=[{"role": "user", "content": "Summarise this report."}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Keepalive comments

While the model is processing a long prompt, it may produce no output for a significant period. During those gaps the API sends SSE comment lines to keep the connection healthy:

: ping

data: {"id":"chatcmpl-...","choices":[{"delta":{"content":"Hi"}}]}

data: [DONE]

Per the SSE specification, any line beginning with : is a comment and must be ignored. Every conformant SSE client already does this, including the official OpenAI SDKs — you do not need to write any special handling.

This only matters if you have written your own SSE parser. If you have, make sure it skips comment lines rather than treating them as malformed input or as the end of the stream.

Comment lines are only ever sent on text/event-stream responses. A non-streaming response is always plain JSON with nothing injected into it.

Why long prompts should stream

A non-streaming request must complete within a server-side deadline. If the model has not finished by then you receive a 504 UPSTREAM_TIMEOUT — a clean JSON error, but still a failure.

A streaming request has no such bound. Bytes begin flowing immediately and the connection stays healthy for as long as generation takes.

If you are sending a large context or expecting a long answer, stream it.

Handling disconnects

If your client disconnects mid-stream, the request is cancelled upstream and generation stops. You are not billed for output you never received, and no work is left running.

To cancel deliberately, close the stream:

stream = client.chat.completions.create(..., stream=True)
for chunk in stream:
    if some_condition:
        stream.close()   # cancels the request upstream
        break