Self-Hosted vs Managed LLMOps: When to Choose Each

The LLMOps category — the layer that observes, evaluates, and routes traffic between your application and one or more LLM providers — has split along an architectural axis that doesn’t get named often enough. On one side: self-hosted, source-available platforms you run inside your own infrastructure. On the other: managed SaaS that runs as a hosted service. Both shapes solve similar problems. They don’t make the same trade-offs, and the decision between them deserves more than a feature checklist.

This post is the framework we wish we’d seen when we started — neutral, axis-first, with no claim that one side is inherently better. If you’re evaluating LLMOps platforms in 2026, the managed-vs-self-hosted choice is upstream of every feature comparison. Get it wrong and you’re rebuilding a year in.

The axis, named clearly

A self-hosted LLMOps platform ships as code (often open-source, often Rust, Go, or Python) that you deploy into your own VPC, cluster, or bare-metal environment. You run the binaries. You operate the database. You handle upgrades, secrets, scaling, on-call. In return, every byte of request and response data stays inside your perimeter, and you control the upgrade cadence completely.

A managed LLMOps platform runs the same general capabilities — gateway, observability, evaluation, routing — as a hosted service. You point your SDK at an endpoint and pay per request, per session, or per seat. The vendor handles uptime, upgrades, scale, and security posture. In return, you accept that some metadata (request shape, latency, evaluation scores, sometimes redacted content) crosses an organizational boundary.

Most platforms pick a side and stay there. A few claim both, but the operational shape of “self-hosted with a managed control plane” is genuinely different from either pure choice and worth its own discussion later.

When self-hosted is the right call

Self-hosting LLMOps makes sense, often emphatically, when:

• Your data cannot leave the perimeter. Healthcare, defense, finance, and a growing list of EU-regulated workloads have policies that prevent any third party — even with a BAA, even with a SOC 2 — from seeing prompts or responses. A self-hosted platform is the only architecturally clean answer.
• You already have the DevOps capacity. Teams that run their own Postgres, Kafka, and Kubernetes are not adding meaningful overhead by running one more stateful service. The marginal cost of self-hosting LLMOps for these teams is small; the gains in control are real.
• You need full upgrade control. Some organizations have compliance reasons to pin every dependency. A self-hosted platform lets you ship version N+1 on your timeline, after your security review, into your release window.
• Your routing logic must encode proprietary business rules. A self-hosted platform you can fork is the cleanest way to express routing logic that you’d never want exposed in a vendor’s product roadmap.
• You want to learn from every request, in isolation. If your traffic is unique enough that cross-tenant signal would dilute rather than improve your routing, self-hosting and tuning purely on your own data is the right design.

TensorZero is the strongest example of this approach in the open-source LLMOps space — Rust, source-available, designed for teams that want infrastructure ownership and developer-defined feedback loops. It’s the right pick for teams where the bullets above describe their reality.

When managed is the right call

Managed LLMOps makes sense, often emphatically, when:

• DevOps is a constraint, not a strength. Most product teams under 50 engineers do not have spare capacity to run another stateful service well. The hours spent operating a self-hosted LLMOps stack are hours not spent on the product.
• You want cross-tenant intelligence. This is the most under-discussed reason. A managed platform that runs many customers’ traffic can build routing models that learn from aggregated signal across the whole base. No single self-hosted deployment can replicate that — by definition.
• Time to first signal matters. A managed platform gives you a benchmark-based prior on day zero, before you’ve sent a single request. A self-hosted deployment starts from a cold cache and waits for your traffic to fill it.
• You want vendor-handled upgrades. Routing logic improves with research. The honest answer for most teams is that they’d rather inherit improvements automatically than read a changelog and run a migration.
• You need predictable per-request economics. Hosted services price per call; self-hosted has a fixed infrastructure baseline regardless of traffic. Below a certain volume, managed is cheaper end-to-end.

The catch with managed is the data boundary. The honest version: you are trusting a vendor with metadata about your traffic, and in some cases with redacted content used for evaluation and training. Read the data-processing terms carefully. Decide consciously.

The hybrid trap

A common third path — “self-hosted with a managed control plane” — sounds like the best of both worlds. In practice it tends to inherit the costs of both.

You still operate the data plane (so you still need DevOps capacity). You also accept a managed control plane (so metadata still crosses a boundary). The cross-tenant learning benefits don’t materialize because your data plane doesn’t share signal with anyone. The upgrade-control benefits don’t materialize because the control plane upgrades on the vendor’s schedule.

Hybrid setups can be the right answer in narrow regulatory cases — when legal will accept managed control but not managed data plane. Outside those cases, either side of the pure axis is usually a better fit than the middle.

Where Floopy stands, and why

We chose managed SaaS deliberately, and the reason traces directly back to the cross-tenant learning point above.

Floopy’s routing intelligence is built from session-level end-user NPS signal collected across the entire customer base. Free and Pro organizations contribute aggregated signal to a shared model that benefits every Floopy customer. Enterprise organizations can opt out for fully isolated learning. The shared model is the moat — and it is structurally only available in a multi-tenant managed deployment.

A self-hosted Floopy would be a different product. It would have the same gateway, the same firewall, the same per-request observability — and a routing model trained on one customer’s data only. For the right team, that’s exactly what they want. We’re not the right pick for them; TensorZero is closer to the shape they need.

For teams that want managed operations and the benefit of routing intelligence trained across many production workloads, the managed SaaS shape is what makes that possible. That’s the trade we chose, made explicit.

How to decide, in five questions

If you’re trying to pick a side this quarter, the questions that actually settle it are short:

1. Can your data leave your perimeter? If no — self-hosted, full stop.
2. Do you have DevOps headcount to spare? If no — managed.
3. Do you want routing that learns from other teams’ traffic? If yes — managed (with cross-tenant signal). If no — either works, lean self-hosted.
4. Is your traffic volume above or below ~5M requests/month? Below — managed is usually cheaper. Above — run the math both ways.
5. Do you need to fork the routing logic? If yes — self-hosted.

Three or more “managed” answers means a hosted platform fits. Three or more “self-hosted” answers means run it yourself. Mixed answers mean dig deeper before committing to either side.

Try the managed path

If the framework above lands you on the managed side, Floopy is built for that case end-to-end. The Free tier includes the full gateway, the firewall, and the four-signal feedback loop on your own traffic — no commitment required to see how cross-tenant learning behaves on your specific workload.

Start at app.floopy.ai, or read Four Signals, One Loop for the technical detail behind the routing model. If self-hosted is your path, run TensorZero — it’s the right tool for that job.

The architecture choice is upstream of the feature choice. Make it deliberately.