Smart Cost Routing

Smart Cost Routing analyzes prompt complexity and intent, then selects an eligible model balancing quality, availability and cost. It combines curated benchmark data, real-time feedback from your users, and multi-armed bandit exploration to continuously improve routing decisions.

Smart Cost seeks cheaper eligible models while respecting quality and exploration limits. Complex requests bypass cost optimization and continue through the governing routing strategy.

Every request passes through six stages before a provider is called:

Request → Complexity & Intent Classification → Candidate Filtering → Scoring → Selection → Provider Call → Feedback Loop

1. Complexity Classification scores the prompt on a 0-to-1 scale using weighted heuristics.
2. Intent Detection identifies the task type (code, math, reasoning, general).
3. Candidate Filtering removes models that are too expensive or too low quality.
4. Scoring ranks remaining candidates using benchmark data, feedback history, and cost savings.
5. Selection picks a model via exploit/explore strategy.
6. Feedback Loop records outcome data that improves future decisions.

Every prompt receives a complexity score between 0 and 1 based on a weighted heuristic analysis. Each signal is normalized to a 0-1 range, then multiplied by its weight.

The weighted sum maps to one of three tiers:

Complex prompts bypass Smart Cost and continue with Feedback-driven or the declared rule strategy.

In parallel with complexity scoring, the classifier detects the task type by scanning message content for keywords and structural signals. The detected intent determines which benchmarks matter most when scoring candidate models.

Priority determines which intent wins when multiple are detected. A prompt containing both code blocks and the word analyze is classified as Code, not Reasoning.

This is the core of Smart Cost Routing. The system maintains a curated database of model capabilities, scored across standardized benchmarks. This data drives the quality estimation that makes routing decisions possible.

Benchmark scores are aggregated from multiple authoritative sources and cross-referenced for accuracy:

• Provider documentation (official model cards from OpenAI, Anthropic, Google, etc.)
• HuggingFace Open LLM Leaderboard
• Epoch AI Benchmarks
• Artificial Analysis
• LLM Stats / Klu AI / Onyx Leaderboards

When sources disagree, we use the provider’s official numbers as the primary reference and flag discrepancies for manual review.

Each benchmark measures a specific capability. The routing system uses different benchmarks depending on the detected intent.

Benchmark coverage depends on the deployed catalog. Models without benchmark data receive a default score of 0.5 (neutral, neither boosted nor penalized).

The Midas aggregation system fetches pricing and benchmark data for all supported models. Each model entry contains:

• Pricing: input, output, and cached token costs
• Benchmarks: normalized to a 0-1 scale for cross-model comparison
• Capabilities: context window, multimodal support, function calling
• Strengths and weaknesses: qualitative notes per model
• Recommendations: suggested use cases

Data is refreshed whenever providers publish updates. The full dataset is embedded in the gateway binary at build time, enabling zero-latency lookups with no external API calls during request routing.

The quality score for a given model and intent is computed as a weighted average of available benchmark scores:

benchmark_score = Σ(vᵢ × wᵢ) / Σ(wᵢ)

Where vᵢ is the model’s score on benchmark i and wᵢ is the weight for that benchmark given the detected intent. The denominator sums only the weights of benchmarks that are actually present for the model, so missing benchmarks do not penalize the score.

Code Intent:

Math Intent:

Reasoning Intent:

General Intent:

GPT-4o with Code intent:

Suppose GPT-4o has these benchmark scores: HumanEval = 0.902, MMLU = 0.887. SWE-bench and LiveCodeBench are not available for this model.

Sum of contributions: 0.3157 + 0.0887 = 0.4044 Sum of available weights: 0.35 + 0.10 = 0.45

Quality score = 0.4044 / 0.45 = 0.898

Only enabled provider/model pairs in the effective rule’s Smart Cost configuration are candidates. An empty list means no optimization; other catalog models are not discovered automatically.

When Feedback-driven is active, it controls quality, approved targets and the single exploration budget. Smart Cost can break an exact quality tie among approved targets, or contribute external models exclusively in exploration bands. An external model with good feedback remains excluded from exploit until explicitly added to rule targets. See the full decision flow.

• Configured tier must match the request’s tier.
• Provider needs a key and catalog support for requested tools/structured output.
• Input/output prices for both candidate and baseline must be known; unknown does not mean free.
• Candidate midpoint token cost must be strictly lower than baseline cost.
• Constraints, confidence and required shadow validation apply, including to untested candidates.

Without active Feedback-driven, the baseline is the declared strategy’s first choice. Tested candidates need at least 10 requests in the 7-day window and quality ≥ min_quality. Quality uses average manual feedback, auto feedback when manual is absent, or benchmarks when feedback is absent; dynamic source weights belong to Feedback-driven.

Exploit ranks candidates with:

score = 0.4 × success_rate + 0.4 × quality + 0.2 × savings_ratio

Simple and Moderate use this score in separate configured tier pools. Complex bypasses to the rule strategy; in joint mode, Feedback-driven continues among approved targets.

One draw defines explore_untested in [0, r × p), explore in [r × p, r), and exploit in [r, 1). r is total exploration rate and p its reserved untested share. Explore uses tested candidates meeting quality; ExploreUntested uses candidates with fewer than 10 requests and skips only the statistical quality floor. Both use Thompson sampling with HTTP successes and failures.

Zero rate never explores, including cold start. Zero untested share never tests new models. Empty pools fall back to exploit or bypass without reallocating the budget. History failure bypasses optimization; a valid empty history permits trials only in the reserved band. Missing feedback does not enable Smart Cost automatically.

1. Go to Routing in the dashboard.
2. Create or edit a routing rule.
3. Enable Smart Cost Routing.
4. Configure the following parameters:
   • Exploration Rate: Percentage of requests used for testing less-explored models (default: 10%).
   • Minimum Quality: The minimum acceptable quality score for a candidate model (default: 70%).
   • Providers: Restrict which providers and models can be used as routing targets.

Every response routed through Smart Cost Routing includes headers that tell you what happened:

Use these headers in your application to log routing decisions, build dashboards, or trigger alerts when routing behavior changes unexpectedly.

Smart Cost Routing is available on the Pro plan.