Streaming responses
How Janus relays Server-Sent Events and still counts tokens.
Transparent relay
Set "stream": true exactly as you would against the provider. Janus relays each event as it arrives, without buffering, so time-to-first-token is unchanged apart from the gateway’s own overhead. When a security policy checks responses, a small trailing window of text is held back until it has been scanned, and for child accounts Child Safety releases replies a paragraph at a time.
Token counts come from the provider’s usage frames. For OpenAI-compatible, Ollama and Vertex upstreams Janus adds stream_options.include_usage to streamed chat and completions requests that do not set stream_options, so the counts are present. If a stream ends without them, Janus estimates output from the text it relayed.
Other body changes — a renamed model’s native name, security redactions, routing or Child Safety instructions — are described on their own pages.
Quotas during a stream
By default a stream that crosses a limit mid-flight is allowed to finish and the next request is refused. If an administrator set the quota to hard-kill, the stream is cut as soon as the limit is crossed.
Because the HTTP status is already 200, a stream that is cut ends with a final data: {"error": …} frame carrying policy.quota_exceeded, policy.security_blocked, policy.response_too_large or upstream.stream_interrupted. Check the last frame for an error before treating the answer as complete.