Handle LLM streaming responses

Check support before enabling a stream

Streaming lets a client consume partial output. It does not guarantee lower total latency, lower cost or a complete answer. Check the selected route’s declared capabilities in /v1/models and its model guide. If streaming is not declared, start with a non-streaming request and confirm support before enabling it.

For an OpenAI Chat Completions route with verified streaming support, set MODEL_ID to its exact public ID. This command requires jq and makes one billable generation request; it intentionally has no automatic retry.

jq -n --arg model "$MODEL_ID" \
  '{model: $model, messages: [{role: "user", content: "Explain HTTP in three steps."}], stream: true}' \
  > request.json
curl --no-buffer --fail-with-body --max-time 120 \
  "https://newrouters.com/v1/chat/completions" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @request.json

--no-buffer changes curl output buffering; it does not configure the model. The command is for inspecting the event stream, not a production parser. Keep request.json out of logs if you replace the demonstration prompt with sensitive content.

Parse events, not network chunks

A network chunk may contain half an event or several events. Use an SSE parser or the selected protocol’s SDK with correct endpoint configuration. Preserve incomplete data between reads, decode UTF-8 incrementally, and dispatch only complete events. Comments and empty deltas must not become user-visible answer text.

Protocol	Streaming entry point	What to parse
Chat Completions	/v1/chat/completions, stream: true	SSE data, choice deltas and terminal [DONE]
Responses	/v1/responses, stream: true	Named lifecycle and output events
Claude Messages	/v1/messages, stream: true	Message/content-block events and stop state
Gemini	/v1beta/models/{model}:streamGenerateContent?alt=sse	Candidate parts and finish information

Only use an entry point supported by the selected route. Do not apply the Chat Completions [DONE] parser to every protocol. Usage may arrive separately from text; keep it separate and treat absent usage as unknown, not zero.

Make partial output explicit

Track connecting, receiving, complete and interrupted states. Mark an answer complete only after the expected protocol completion signal. A closed socket or HTTP 200 alone is insufficient. Surface truncation or refusal distinctly from a successful complete answer.

When a client disconnects, stop rendering and release client resources. Cancellation does not prove that upstream generation stopped or that no usage was incurred. Do not restart the entire request automatically after partial text: this can duplicate output and spend. Follow errors and retries and measure time to first text separately from total time in your evaluation.