Floopy collects quality feedback on LLM responses and uses it as the primary signal for routing optimization. Every feedback submission feeds directly into Smart Cost Routing and Smart Selector, creating a self-improving loop: the more feedback you submit, the better your routing decisions become.

Without feedback, routing relies on public benchmark scores. With feedback, routing learns what actually works for your prompts, your users, and your quality bar.

graph LR
    S1[Session feedback] --> CH[(ClickHouse)]
    S2[Auto feedback] --> CH
    S3[Manual feedback] --> CH
    CH -->|Bounded history query| W[Feedback-driven weights]
    S4[Benchmark prior] --> W
    W --> D[Routing decision]

Feedback-driven reads request and feedback aggregates from ClickHouse during planning, with bounded query timeouts. Auto, manual and session signals remain separate. Successful empty history uses benchmark priors; a query error preserves the approved baseline. Smart Selector is a separate feature with its own cached variant scores.

Submit session-level feedback via the feedback endpoint. Use the same API key you use for chat requests. The session_id is the value you send in the floopy-session-id header during chat requests.

Endpoint: POST /v1/feedback

await fetch("https://api.floopy.ai/v1/feedback", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.FLOOPY_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: '{"session_id":"sess_abc123","score":8,"useful":true}',
});

import requests
import os

requests.post(
    "https://api.floopy.ai/v1/feedback",
    headers={"Authorization": f"Bearer {os.environ['FLOOPY_API_KEY']}"},
    json={"session_id": "sess_abc123", "score": 8, "useful": True},
)

curl https://api.floopy.ai/v1/feedback \
  -H "Authorization: Bearer $FLOOPY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "sess_abc123",
    "score": 8,
    "useful": true
  }'

Session feedback is stored in the ClickHouse session_feedback table, linked to all requests in that session via session_id. There is no need to update individual requests — the feedback applies to the entire session automatically.

This enables aggregation across multiple axes:

• Per model: Average NPS of gpt-4o vs claude-sonnet-4-6 for your workload
• Per provider: Aggregate quality across all models from a single provider
• Per prompt pattern: Quality trends for specific types of requests
• Per time window: Detect quality degradation over hours, days, or weeks

Routing reads ClickHouse aggregates with a timeout. Requests are deduplicated before feedback joins, so multiple feedbacks do not multiply sample counts. Success means HTTP 2xx. Auto, manual and session are separate signals: auto/composite is scaled from 0–100, session from 0–10, and manual thumbs to 0 or 1. Session feedback joins requests within the same organization and session.

When active, Feedback-driven controls quality ranking, approved targets and exploration. Missing feedback does not disable its switch or enable Smart Cost. In joint mode, Smart Cost introduces external candidates only in Explore or ExploreUntested; better quality does not lose to savings, and external models never automatically join exploit. See the decision flow.

Each weight multiplies its own source’s average. Missing sources are omitted and remaining weights renormalized; one average is not reused as auto, manual and session. Phases use organization counts, while scores and samples belong to the provider/model pair.

Without Feedback-driven, Smart Cost retains fixed weights:

performance_score = 0.4 × success_rate + 0.4 × quality + 0.2 × savings_ratio

Quality uses manual average, then auto, then benchmarks. Selection savings compare midpoint token cost with baseline; logged savings use actual tokens and known prices. These weights do not replace Feedback-driven quality in joint mode.

The equal weighting between success rate and quality reflects a core design principle: a model that never fails but produces mediocre output is no better than a model that produces excellent output but fails frequently. Both matter equally. Cost savings gets 20% because it is the reason Smart Cost Routing exists — but it should never override reliability or quality.

Standalone Smart Cost requires at least 10 requests before exploitation. In joint mode, Feedback-driven may exploit approved targets using priors even before that threshold; external candidates with fewer than 10 requests remain confined to ExploreUntested. No model is guaranteed traffic outside the configured exploration budget.

Smart Selector is a separate feature. Its presets and algorithm options below do not govern the joint Feedback-driven/Smart Cost policy.

Smart Selector uses feedback at a more granular level. Instead of averaging all four dimensions into a single quality score, it applies configurable weights to each dimension independently:

composite = relevance × w₁ + coherence × w₂ + helpfulness × w₃ + safety × w₄ + cost_efficiency × w₅

Cost efficiency is calculated as (min_cost / variant_cost) × 100, where min_cost is the cheapest model in the variant set. This rewards cheaper models proportionally rather than using a binary threshold.

Choose a preset that matches your use case. A customer support chatbot benefits from Safety-Critical. An internal code generation tool benefits from Quality-First. A high-volume classification pipeline benefits from Cost-Optimized.

Smart Selector uses multi-armed bandit algorithms to balance exploration (testing models) and exploitation (using the best model). Three strategies are available:

• Epsilon-Greedy: Exploits the best model (1 - ε) of the time, explores a random model ε of the time. Simple and predictable. Good default.
• UCB1 (Upper Confidence Bound): Selects the model that maximizes score + √(2 × ln(total) / n), where total is total requests across all models and n is requests for this model. The square root term is a confidence bonus that shrinks as a model gets more data. Models with fewer observations get a larger exploration bonus, ensuring under-tested models are tried.
• Thompson Sampling: Maintains a Beta distribution for each model’s expected quality, parameterized by successes and failures. Samples from each distribution and picks the highest sample. Naturally balances exploration and exploitation — uncertain models (wide distributions) occasionally sample high, getting tested. Converges faster than UCB1 in practice.

History spans 7 days. Tested means at least 10 requests, not 10 feedbacks. One draw splits the budget into ExploreUntested (fewer than 10 requests), Explore (tested with sufficient quality) and Exploit (approved targets). Exploration uses Thompson sampling with HTTP successes and failures, without round-robin or guaranteed immediate coverage.

ExploreUntested skips only the statistical quality floor. Constraints, prices, support and required shadow approval still apply. An external candidate reaching 10 requests can move into Explore, but enters Exploit only after explicit inclusion in approved targets.

Zero rate, empty pools or low confidence can prevent trials. History failure keeps the configured baseline and disables exploration; successful empty history permits exploration within budget. Tested models below the floor do not receive Explore until their windowed quality recovers; automatic recovery through traffic outside those limits is not promised.

Feedback data surfaces in three places in the dashboard:

• Requests page: Each request shows its feedback scores inline, so you can correlate quality with model, latency, and cost.
• Smart Selector detail: The experiment view shows per-variant feedback distributions, making it clear which model your users prefer.
• Smart Cost Savings widget: Compares feedback scores between cheap-routed requests and default-model requests, proving that cost savings are not coming at the expense of quality.

If your product doesn’t collect NPS or user feedback today, Floopy still improves routing from day one. Auto feedback (LLM-as-judge on every response) combined with public benchmark scores starts the learning loop immediately. Session NPS is the strongest signal, but the system works without it — you get the optimization benefit even before building a feedback collection pipeline. The weight transition table above shows how signals ramp in as data accumulates.

• Collect session feedback from end users. Even a simple NPS prompt (“How would you rate this conversation? 0-10”) after a session gives the routing engine high-confidence signal. Aim for feedback on at least 10% of sessions.
• Always include both score and useful. Both fields are required and give the routing engine the full picture — NPS measures satisfaction while useful captures practical value.
• Automate feedback collection. Use LLM-as-judge (a cheap model evaluating the output of an expensive model) or heuristic scoring (response length, format compliance, user engagement signals) to generate auto feedback at scale without manual review. This serves as the automated baseline while session feedback builds up.
• Monitor the exploration rate. If exploration is consistently routing to a model that scores poorly, consider removing that model from the candidate set.
• Start with the Balanced preset, then adjust. After 1,000+ requests with feedback, review the quality signal breakdown to see which weights match your actual quality priorities.

If you’re migrating from a per-request feedback system (Helicone, LangSmith, PromptLayer), the translation is straightforward: instead of rating each individual response, rate the end-to-end conversation. One NPS score per session covers every routing decision in that session — multi-turn, tool calls, chained reasoning. You can continue capturing per-request data in parallel for debugging, but the routing engine consumes session-level signal for optimization. Existing per-request ratings can be migrated by aggregating them under each session_id; the first one you POST under a session_id wins.

Feedback collection is available on all plans. Feedback-driven routing (Smart Cost Routing and Smart Selector integration) is available on the Pro plan.

Even when end-user feedback is thin, the routing engine keeps its per-(subject, complexity) Model Ranking fresh via a nightly continuous-evaluation pass. A Coolify cron job (continuous-eval, daily at 04:00 UTC) samples up to 50 recent (request, response) pairs from each of the 18 buckets (6 subjects × 3 complexities) and asks an LLM-as-judge to grade each response on a 0–100 scale. Scores are written to the continuous_eval_scores table and rolled into continuous_eval_latest for keyed lookups.

The dashboard ranking query blends the continuous-eval score with user feedback at read time:

final_score = (1 − CONTINUOUS_EVAL_WEIGHT) × user_feedback_score
            +  CONTINUOUS_EVAL_WEIGHT      × continuous_eval_score

The default weight is 0.3 (30% continuous-eval, 70% user feedback). When only one side is present the final score falls back to whichever exists; when neither exists the ranker falls back to the benchmark prior.

Only requests from organisations that have shares_to_shared_pool = true are sampled, matching the privacy boundary used by the Shared Intelligence Pool. The judge model is configurable via CONTINUOUS_EVAL_JUDGE_MODEL (default meta-llama/Llama-3.3-70B-Instruct-Turbo via Together API).

The admin dashboard (/admin/continuous-eval) showing run history and per-bucket score trends is planned for v3.5 — this release ships the scoring pipeline only.