Route evaluation with a judge
Check whether a route is as good as the default: a judge model compares sampled answers blind, twice with the order swapped.
What it is
For a sample of a route's requests, after the caller has their answer, Janus asks the router's default model the same question out of band. A judge model compares the two answers without knowing which is which, twice with the order swapped, and a disagreement counts as a tie.
Why you want it
Sending work to a cheaper or self-hosted model is only worth it if the answers hold up. Evaluation measures that on real traffic instead of a benchmark that may not match how people use AI.
Each route judged against the default on real traffic: blind verdicts with the order swapped, a confidence range and the cost difference.
Actual Janus interface · Demo identities and synthetic usageHow it works
- Default sample 5% of a route's requests, ending after 14 days unless changed
- Per route: better, tie and worse counts, 'at least as good' with a 95% range, judge consistency and cost against the default
- Daily count and daily budget reserved before anything is spent; pauses while the gateway is busy
- Copies sent to the default and the judge pass the caller's security policies; a sample a policy would block or redact is skipped
- Only verdicts, scores, a one-line reason, cost and timing are stored, never prompts or answers
- Evaluation calls never count against the caller's quota
In the product
Screenshot above. Actual Janus interface with demo identities and synthetic usage.
See it on your own network.
The Community edition is free for up to 25 people. The 30-day Business trial unlocks every Business feature.