Configuring upstreams

Before you start

You need the provider base URL and a credential. Credentials are encrypted with AES-256-GCM using JANUS_ENCRYPTION_KEY before they touch the database, and only the first and last four characters are ever displayed again.

AWS Bedrock and Google Vertex are paid cloud services: you supply your own account. Bedrock credentials are stored as ACCESS_KEY_ID:SECRET_ACCESS_KEY; Vertex takes the service-account JSON key.

Choosing a provider fills in its documented base URL and a suggested name. Both stay editable: change the URL for a proxy or a regional endpoint, and name the upstream whatever you like. Self-hosted engines have no fixed address, so their field shows the engine’s default port as an example.

A base URL that ends in an API version (…/v1, Google’s …/v1beta/openai) is used as the API root, the way the provider documents it for OpenAI clients; Janus appends /chat/completions or /models without adding another /v1. A base without one, such as https://openrouter.ai/api, gets /v1 added.

Procedure

1. Administration → Gateway → Upstreams → Add upstream. Choose the provider type: it is immutable afterwards, because it determines how every stored model is addressed.

2. Choose Create and discover models. Janus starts discovery right away and records what the provider reports. For a self-hosted engine (vLLM, llama.cpp, Ollama, TEI), check the server flags in “Self-hosted engines” first: they decide whether speed, cache savings and the context window are exact.

3. Administration → Gateway → Models. New models arrive as “Pending approval” and are not served until you enable them. A model the upstream stops listing is marked stale. When a stale model is listed again it returns to the state it had before — enabled, disabled or awaiting approval — so a short gap in the listing never switches a model off for good. If an upstream that has enabled models answers with an empty list, Janus treats the check as failed and changes nothing; three failed checks in a row raise an “unreachable” alert. When enabled models stop being listed, administrators get a notification naming them.

4. Open Edit to review automatic metadata and pricing warnings, set any overrides, then enable. A model cannot be enabled until it has a rate card ($0 is allowed for self-hosted models); unsupported tiers and per-image rates are never flattened into token prices.

5. Administration → Gateway → Grants. Grant the model to a person, a group, a team, a service token, or everyone.

Pricing from the bundled reference seed

Rate cards are always per million tokens; see “Pricing by modality” for how speech, transcription, image and realtime models — which providers quote per minute, per character or per tier — translate into that unit. Janus ships a reference rate-card seed for common OpenAI, Anthropic, Mistral and Google Gemini models so a fresh install does not need every price typed by hand. Discovery prices a model from this seed automatically when the upstream is the provider’s official endpoint (api.openai.com, api.anthropic.com or generativelanguage.googleapis.com); xAI and OpenRouter publish their own prices, which discovery reads too. GET /api/v1/admin/ratecards/reference lists the seed; POST /api/v1/admin/ratecards/apply prices every discovered model that matches the seed by name and has no rate card yet — use it for models reached some other way, such as Mistral or a proxy. It is available through the API only. Models you already priced are never touched, and re-applying is a no-op.

The seed reflects public list prices at the time it was authored — it is a starting point, not a live price feed. Verify against the provider’s current pricing before relying on cost attribution, and adjust any figure per model afterwards.

The seed also carries each model’s context window (context_window_tokens). Applying it fills in the context window for every matching model that does not have one yet — including models you already priced by hand — but never overwrites a value that is already set.

curl -X POST ${JANUS_PUBLIC_URL}/api/v1/admin/ratecards/apply \
  -H "Cookie: janus_session=…" -H "X-Janus-CSRF: …"

Google Gemini

Choose Google Gemini (AI Studio key) and paste an API key from Google AI Studio. Janus talks to Google’s OpenAI-compatible endpoint, https://generativelanguage.googleapis.com/v1beta/openai, so chat, streaming, function calling, image input and embeddings use the OpenAI request format unchanged.

Discovery lists the models the key can call, with their context windows. Live-API (realtime voice) models are left out because the OpenAI-compatible endpoint cannot call them.

Gemini reports thinking tokens only in its total, and Google bills them at the output rate, so Janus meters output as total minus prompt: the request log and quotas match the bill. Implicit prompt-cache hits are recorded as cached input.

Prices for Gemini 2.5 Flash and Flash-Lite, 3 Flash Preview, 3.1 Flash-Lite, 3.5 Flash and Flash-Lite and Embedding 2 come from the bundled reference. Models priced by prompt size (2.5 Pro, 3.1 Pro) or with a scheduled price change (3.6 to 3.8 Flash, which double on 1 January 2027) arrive unpriced: enter the price yourself.

For a Google Cloud project with a service account instead of an AI Studio key, use Google Vertex AI.

Thinking settings and provider limits

Chat clients such as Open WebUI send the same settings to every model: a reasoning effort (none, minimal, low, medium, high, xhigh), a temperature from a preset, and tools. Providers each refuse a different part of that. GPT-6 Sol and Astra take low to xhigh only; Gemini takes none to high but not minimal or xhigh; Grok refuses none; Claude 5 refuses any temperature or top_p; GPT-5 and GPT-6 take only the default temperature and refuse function tools with a reasoning effort on chat completions.

Janus fits the request instead of returning the provider’s error. When a provider refuses a setting and says what it refused, Janus changes it and sends the request again before anything reaches the caller (a refused request is not billed): a reasoning level becomes the nearest level the model takes, a refused sampling setting is left out, and a GPT-5 or GPT-6 request with tools goes through OpenAI’s Responses API and comes back as a normal chat completion. What it learns is kept per model for a day on each gateway replica, so later requests go out fitted without a refusal.

Every change is shown on the response in the X-Janus-Reasoning-Adjusted header and on the request in the request log, for example “minimal->low temperature->omitted tools->responses”. An error Janus cannot fix is passed through unchanged.

Anthropic: reasoning_effort is translated into the thinking form each Claude model takes. Claude 4.6 and later (all of Claude 5) get adaptive thinking with that effort and a summary of the thinking, which Janus returns as reasoning_content (none and minimal ask for the lowest effort, low; adaptive models cannot switch thinking off), and the output cap is raised to 32,000 tokens when the caller set none. Earlier Claude models get a thinking budget (1,024 to 63,999 tokens by level, kept within the model’s output limit) and none leaves thinking off. Thinking needs the default temperature, so a temperature is left out when thinking is on. A request that sends Anthropic’s own thinking object is passed through as it is.

Gemini: with thinking on, Gemini signs each tool call and refuses the next turn when the signature is missing, which most chat clients drop. Janus remembers each signature as it relays the tool call and puts it back when the call is replayed; a call it did not see (another replica, an older conversation) gets the placeholder Google documents for that case.

The same model on more than one upstream

Two upstreams can serve a model under the same name — the same vLLM model on two nodes, or an old and a new endpoint while a pod moves. Each copy is its own catalog entry with its own rate card and grants, and the Models page marks it with a “+N upstream(s)” badge. A request for the name goes to the most recently discovered copy the caller is granted, so granting the new copy is enough to move traffic; the old copy’s grants never apply to the new one, and an ungranted copy is never used as a fallback.

If only the address changed, edit the existing upstream’s base URL instead of adding a new one: the upstream keeps its models, rate cards and grants, and traffic follows the new address immediately.

For one stable name users never have to change, create a managed model on the Managed models page, grant it, and repoint it whenever the model moves. Grants on a managed model are independent of the model it points at, so repointing moves every caller without touching a single grant.

To use several copies at once, make the managed model a load-balanced pool (below).

Load-balanced pools (Business)

A managed model can list several models — typically the same model served by several vLLM or llama.cpp servers — and Janus spreads its traffic across them. Open Managed models → Edit, and under Load balancing use Add a model. Callers keep using the one alias name; grants, quotas and the fallback stay on the alias, and the request log records which server actually answered.

Balancing: Failover sends traffic to the servers with the lowest priority number and keeps higher numbers as standbys; servers that share a priority also share conversations. A server added under Failover gets the next number, so it starts as a standby; under the other policies it joins the serving tier. Round robin takes turns by weight. Least loaded picks the server with the shortest queue (waiting requests count double). Context aware compares how much of each server’s context capacity is taken by the conversations it is running, so one 100,000-token conversation counts for more than several short chats — set a server’s Context capacity to override what it reports. Whatever the policy, each server’s Priority (0–100) groups servers into tiers: lower numbers serve first and higher numbers only take traffic when every lower-priority server is out.

Keep conversations together: every turn of a conversation goes to the server that already holds its prompt cache, which makes follow-up turns much faster. Janus recognizes a conversation from an X-Janus-Session, X-Session-Id or X-Conversation-Id header, the OpenAI prompt_cache_key field, or — when a client sends none — from its tools, system prompt and first message, which stay the same on every turn. Bounded (the default) moves a conversation to its second-choice server only while its own server is busier than the pool average by the chosen percentage and by a meaningful absolute amount; Strict moves it only when its server is out of service; Off balances every request separately.

Every gateway replica makes the same choice for the same conversation without sharing any state on the request path, so pools work the same with one replica or many. Live load is read every 2 seconds from each server’s /metrics endpoint (vLLM: requests running and waiting, KV-cache use and size; llama.cpp: run llama-server with --metrics, and --slots for context figures). Servers without metrics (cloud APIs, Ollama) are balanced on Janus’s own in-flight counts and marked “No live metrics”. The full flag list per engine is in “Self-hosted engines”.

Failures: a server that refuses a request before sending anything (connection error, 502, 503, 504, 429) is skipped and the request goes to the next server in the same call, so the caller does not see the error; nothing is retried once a response has started. Three consecutive failures (connection errors, 502, 503, 504 — not 429, which means the server is busy) take a server out for 5 seconds, doubling up to 2 minutes, and every replica takes it out together. When every server is out, the alias’s fallback model serves (response header X-Janus-Fallback-Reason: pool_exhausted); without a fallback the request fails with a message naming the pool.

When a member fails before answering, Janus retries on the next member only if that member gets the same Security Gateway checks, isn’t blocked by Child Safety for the caller, and the caller’s quota on it isn’t used up; a member that fails any of these is skipped. Bind a stricter policy to the pool’s name rather than to one member if every member should be checked the same way. Security Gateway classifiers are never pool members.

When a server is taken out of rotation, administrators get one “Pool member taken out of rotation” alert per ejection.

Pooled responses carry X-Janus-Pool-Member (model@upstream) and X-Janus-Pool-Reason (policy, affinity, spill, moved, retry or only). The editor’s Servers right now panel shows each server’s state, queue, context use and prompt-cache hit rate, refreshed every few seconds.

Licensing: a pool of more than one model, or any policy other than Failover, needs a Business license with load balancing. When the license lapses, existing pools keep balancing and can still be shrunk, re-weighted or have servers switched off; adding or re-enabling servers and changing the policy are blocked.

Verification

Administration → Settings → System status shows each upstream’s last check, latency, and last error. A user should now see the model in GET /v1/models within seconds.

Rollback

Disabling an upstream stops all traffic through it and disables its models, while retaining every historical usage event that references it.

Deleting an upstream first shows what depends on it: the models it hosts (all disabled by the delete), the managed models whose target lives there (they keep resolving by name but can no longer be served unless they have a fallback), and the direct grants on those models. While an enabled managed model still targets the upstream the delete is blocked — repoint or disable the alias on the Managed models page, or tick the explicit “delete anyway” acknowledgement (DELETE …?force=true) and repoint afterwards. An opt-in also removes the direct grants on the upstream’s models (?purge_grants=true) rather than leaving them attached to models nobody can call.