Self-hosted engines

Why engine settings matter

Janus works with any of these engines out of the box, but several figures are only as good as what the engine reports. Exact token counts and cost, the speed users see on the Models page, the context window on each model, prompt-cache savings and live load for pools all come from the engine. Without them Janus falls back to its own estimates, which are marked as such but are less accurate.

Each engine section below lists what to turn on, what Janus reads, and what you lose without it. After changing flags, restart the server and run discovery again from Administration → Gateway → Upstreams so the context window is refreshed.

vLLM

Provider type: vLLM. Base URL: the server root with /v1, for example http://vllm.example:8000/v1.

--enable-per-request-metrics — returns each request’s timing (prefill time and decode time) with its usage. Janus records decode speed as reported by the provider; without it, speed is measured on the gateway’s clock and includes prompt processing, so long prompts make the model look slow. Requires engine statistics: do not also pass --disable-log-stats.

--enable-force-include-usage — puts a usage block on every stream, even when a client does not ask for one. Janus already asks on the requests it forwards, so this is a safety net for exact metering.

--enable-prompt-tokens-details — reports cached prompt tokens, so prompt-cache savings are billed at the cached rate and shown on the request log. Pair it with --enable-prefix-caching.

--max-model-len — the context window Janus reads from /v1/models (max_model_len). Set it to what the server can actually hold; discovery copies it onto the model.

Live load for pools: /metrics is on by default. Janus reads requests running and waiting, KV-cache use and prefix-cache hit rate from it every 2 seconds.

--served-model-name — the model name users call. Keep it identical on every server of a pool so they appear as the same model.

vllm serve /models/qwen3-32b \
  --served-model-name qwen3-32b --max-model-len 32768 \
  --enable-prefix-caching --enable-prompt-tokens-details \
  --enable-force-include-usage --enable-per-request-metrics

llama.cpp (llama-server)

Provider type: llama.cpp server. Base URL: the server root with /v1, for example http://llama.example:8080/v1.

Speed and token counts: llama-server returns its own timings (prompt and generation tokens per second) with every response. Janus records them as reported by the provider with no extra flag.

--metrics — enables /metrics, which pools use for live queue length and KV-cache use. Without it the server is balanced on Janus’s own in-flight count, and the Servers right now panel marks its load “estimated”.

--slots — enables /slots, which gives pools each server’s real context capacity and current use for context-aware balancing.

--ctx-size and --parallel — the served context is ctx-size, which /v1/models reports as meta.n_ctx and Janus copies onto the model. With --parallel N and a non-unified KV cache (--no-kv-unified), each request gets ctx-size ÷ N, so set ctx-size to N × the context you want per request.

--alias — the model name users call; without it the name is the GGUF file path. Keep it identical across a pool.

Prompt caching is on by default (cache_prompt), and cached tokens are reported as prompt_tokens_details.cached_tokens.

llama-server -m /models/Qwen3-32B-Q4_K_M.gguf --alias qwen3-32b \
  --ctx-size 65536 --parallel 2 --no-kv-unified \
  --jinja --metrics --slots

Ollama

Provider type: Ollama. Base URL: the server root without /v1, for example http://ollama.example:11434. Discovery reads /api/tags and /api/show; traffic goes through Ollama’s OpenAI-compatible endpoint.

Speed: the OpenAI-compatible endpoint returns no timings, so speed is measured on the gateway’s clock (marked Calculated by gateway in the request log). Treat it as a lower bound.

Context window: discovery reads the model’s trained context length from /api/show, but Ollama serves only num_ctx (by default a few thousand tokens) unless told otherwise. Set OLLAMA_CONTEXT_LENGTH on the server, or a PARAMETER num_ctx in the Modelfile, and override the model’s context window in Administration → Gateway → Models to match. Otherwise long prompts are silently truncated by Ollama.

Pools: Ollama has no /metrics endpoint, so pooled Ollama servers are balanced on Janus’s own in-flight counts. Set OLLAMA_NUM_PARALLEL to how many requests each server should run at once.

OLLAMA_KEEP_ALIVE — keep models loaded (for example 24h). A model that has to load first makes the first request after idle very slow, which shows up in the speed and latency figures.

Hugging Face TEI

Provider type: Hugging Face TEI. Base URL: the server root, for example http://tei.example:8080. TEI serves one embedding, reranker or classifier model per process; discovery reads /info and fills in the model name, its input cap (max_input_length) and, for a guard classifier, its security role.

--served-model-name — the model name users call; without it Janus uses the last part of the model id.

TEI has no chat endpoint: a TEI upstream only proxies /v1/embeddings. Use it for embeddings and for the Prompt Guard classifier (see Security gateway).

Other OpenAI-compatible servers

SGLang, LM Studio, LocalAI, TGI’s OpenAI endpoint and similar servers work with the Other OpenAI-compatible endpoint provider type. Check three things on the first requests in the request log: the counting method should say Reported by upstream (not Estimated from byte count), streamed requests should carry token counts (the server must honor stream_options.include_usage), and the model’s context window should be filled in after discovery. Set any missing context window by hand in Administration → Gateway → Models.

Checking an engine is set up well

Open the request log, filter by the model, and open a recent request. Counting method should be Reported by upstream. Throughput should say Measured by upstream for vLLM (with per-request metrics) and llama.cpp; Calculated by gateway means Janus timed it. Cached input tokens should appear on repeat prompts if prefix caching is on.

For pools, Managed models → Edit → Servers right now should mark each vLLM or llama.cpp server’s load “live” and show its queue and context use; “estimated” means its metrics are not being read.