Preference-aware LLM routing evaluation

RouteJudge

An Open Platform for Reproducible and Preference-Aware LLM Routing

RouteJudge evaluates LLM routers through anonymous pairwise user preferences, budget-aware model selection, and router-level attribution.

Try Platform View ORBIT Read Paper

Router-Level Evaluation Anonymous Pairwise Preference Budget-Aware Routing ORBIT-Compatible

Abstract

Existing LLM routing evaluation relies heavily on static benchmarks, golden answers, or automated scores. RouteJudge instead evaluates whether router decisions lead to responses preferred by real users under realistic query distributions, heterogeneous preferences, and deployment constraints. For each query, multiple routing strategies recommend models under the same model pool and budget constraints. The selected model outputs are shown through anonymous pairwise comparison, and user preferences are attributed back to the routers behind the compared responses.

Teaser: RouteJudge Workflow

RouteJudge converts online preference feedback into router-level analytics while preserving a shared model pool, budget constraint, and evaluation record.

Workflow figure from the repository. If the browser cannot render embedded PDFs, open the workflow PDF.

1 Constrain

User Query + Budget → Budget-Feasible Model Space.

2 Route

Router Committee → Duel Pair Selection.

3 Judge

Anonymous Preference Arena collects A/B/Tie/Both Bad labels.

4 Analyze

Router-Level Attribution → Evaluation Analytics.

Anonymous Preference Arena

The user sees only anonymous responses during judgment. Model identities, router votes, costs, and latencies are hidden during user judgment.

Illustrative interface mockup Preference judgment

Response A

A concise answer that states assumptions, compares the alternatives, and provides a direct recommendation for the user query.

Response B

A broader answer that gives additional caveats, examples, and more detailed reasoning about cost and latency trade-offs.

Model identities, router votes, costs, and latencies are hidden during user judgment.

Illustrative interface mockup Router-level attribution

Router	Selected Model	Participated	Outcome
EmbedLLM	m_A	Yes	Win
EquiRouter	m_B	Yes	Loss
Avengers-Pro	m_K	No	NonPart

Example rows are generic and do not represent measured leaderboard results.

Key Features

RouteJudge is designed for preference-aware, cost-aware, and task-conditioned analysis of LLM routers.

A

Anonymous pairwise preference arena

Collects user preferences without exposing model or router identity.

R

Router-level preference attribution

Attributes user judgments to routers behind compared responses.

B

Budget-feasible model filtering

Constrains all routers to the same feasible model pool.

C

Cost and latency tracking

Records deployment-relevant signals alongside preference labels.

E

Preference score / win rate / Elo rating

Provides complementary views of preference-based router quality.

P

Cost-quality Pareto analysis

Studies the trade-off between preference outcomes and inference cost.

T

Task-conditioned diagnostics

Breaks down router behavior by task type and metadata.

N

Participation-rate analysis

Interprets preference scores together with duel exposure.

O

ORBIT-based router submission and integration

Connects standardized router development to online RouteJudge evaluation.

RouteJudge × ORBIT

ORBIT is the standardized development and integration layer for RouteJudge. Researchers implement routers in ORBIT, evaluate them offline under unified protocols, and submit compatible routers for RouteJudge historical replay and online preference-based evaluation.

01Implement Router in ORBIT

Follow the unified router interface.

02Offline Evaluation

Run under shared protocols before submission.

03PR Submission

Package configs, dependencies, seeds, and entry points.

04Historical Replay

Evaluate decisions on recorded RouteJudge data.

05Online RouteJudge Evaluation

Collect preference feedback from live comparisons.

For Researchers

A concise integration path for new routing methods.

Implement a router following the ORBIT router interface.
Expose a unified prediction function that takes query representation and the feasible model set as input.
Add configs, dependencies, seeds, checkpoints, and preprocessing instructions.
Run offline evaluation under ORBIT.
Submit a pull request.
After integration, the router can be evaluated through RouteJudge.

Evaluation Metrics

RouteJudge reports router behavior without fabricating fixed universal rankings.

Preference Score

Aggregates preference labels attributed to each router.

Win Rate

Measures how often a router-selected response wins pairwise judgments.

Elo Rating

Maintains relative ratings from pairwise outcomes.

Cost-Quality Pareto

Compares preference quality against inference cost.

Task-Conditioned Performance

Analyzes behavior across task categories.

Pairwise Router Comparison

Directly compares routers when their selected models meet.

Participation Rate

Tracks how often each router enters evaluated duels.

Routing Behavior Diagnostics

Inspects model selection, budget use, latency, cost, and metadata.

Authors

Guannan Lai^1,2,3, Haoran Hu^1,2, Han-Jia Ye^1,2

¹ School of Artificial Intelligence, Nanjing University

² National Key Laboratory for Novel Software Technology, Nanjing University

³ SinapisAI

Citation

@misc{lai2026routejudge_arxiv,
  title         = {RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing},
  author        = {Lai, Guannan and Hu, Haoran and Ye, Han-Jia},
  year          = {2026},
  eprint        = {2606.18774},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2606.18774v1}
}

@inproceedings{lai2026routejudge,
  title     = {RouteJudge: Preference-Based Evaluation of {LLM} Routers under Pluralistic User Preferences},
  author    = {Guannan Lai and Haoran Hu and Han-Jia Ye},
  booktitle = {Pluralistic Alignment Workshop at ICML 2026},
  year      = {2026}
}