OpenTelemetry Export

Overview

Floopy uses a dual export architecture: every request trace is always stored in ClickHouse for the built-in dashboard, and can optionally be exported in real time to any OpenTelemetry (OTLP) compatible backend. This lets you correlate AI gateway traces with your existing application traces in tools like Datadog, Grafana Tempo, Jaeger, New Relic, or Honeycomb.

How it works:

1. The gateway instruments each request as a set of spans (auth, cache, firewall, provider call, etc.)
2. Spans are batched and flushed to ClickHouse on a regular interval
3. After each ClickHouse flush, if your organization has OTLP export enabled, spans are converted to the OpenTelemetry format and exported in parallel via gRPC or HTTP/protobuf
4. OTLP export is fire-and-forget — failures never affect request processing or ClickHouse storage

Span Schema Reference

Resource Attributes

Resource attributes are set on all spans from your organization:

Span Attributes

Each span carries attributes describing the request:

Additional custom attributes set by your application are passed through as-is.

Span Field Mapping

Floopy’s internal span format is converted to OTLP as follows:

Span Hierarchy Example

A typical request produces a trace with the following span hierarchy:

gateway_request (root span)
├── auth                    # API key validation
├── rate_limit              # Rate limit check
├── cache_lookup            # Exact + semantic cache check
├── firewall                # LLM firewall evaluation
│   ├── verdict_cache       # Qdrant lookup (skipped on cache miss)
│   └── llm_call            # Safety-tuned LLM via BackendRouter
│                            #   span fields: backend, model_ref, attempt_count
├── provider_call           # Upstream AI provider request
│   └── retry_attempt       # Individual retry attempts (if any)
└── post_process            # Logging, metering, cache storage

Setup Guide

Step 1: Enable OTLP Export in Settings

1. Navigate to Settings → Integrations in the Floopy dashboard
2. Enter your OTLP collector endpoint URL
3. Select the protocol: gRPC or HTTP/protobuf
4. Add any required authentication headers (e.g., API keys, bearer tokens)
5. Click Test Connection to verify connectivity
6. Toggle Enable Export to activate

Step 2: Verify Spans Are Arriving

After enabling, send a few test requests through the gateway and check your observability backend for traces with service.name = floopy-gateway.

Integration Guides

Datadog

Grafana Cloud / Tempo

Jaeger

New Relic

Honeycomb

Self-Hosted OpenTelemetry Collector

If you run your own OpenTelemetry Collector, point Floopy directly at it:

Example collector config to receive Floopy spans and export to multiple backends:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 5s

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  otlp/datadog:
    endpoint: https://trace.agent.datadoghq.com
    headers:
      DD-API-KEY: ${DD_API_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/jaeger, otlp/datadog]

Troubleshooting

Connection Refused

• Symptom: Test Connection fails with “connection refused”
• Cause: Endpoint is unreachable from the Floopy gateway
• Fix: Verify the endpoint URL and port. Ensure firewalls allow outbound connections from the gateway. For gRPC, the default port is 4317; for HTTP, it’s 4318.

Authentication Failures

• Symptom: Test Connection fails with 401 or 403
• Cause: Missing or incorrect authentication headers
• Fix: Double-check your API key or token. Headers are stored encrypted — re-enter the value if you suspect it was saved incorrectly. Verify the header name matches your backend’s requirements exactly (e.g., api-key vs Api-Key).

TLS / Certificate Errors

• Symptom: Test Connection fails with TLS handshake error
• Cause: Self-signed certificate or missing CA on the gateway host
• Fix: Use http:// for local/internal collectors without TLS. For production, ensure your collector’s certificate is signed by a trusted CA.

Spans Not Appearing After Config Change

• Symptom: Updated endpoint or headers, but spans still go to old destination
• Cause: OTLP configuration is cached in Redis with a 5-minute TTL
• Fix: Wait up to 5 minutes for the cache to expire, or restart the gateway to clear the cache immediately.

Export Timeout

• Symptom: Gateway logs show “OTLP export timed out” warnings
• Cause: Collector is slow to respond or overloaded
• Fix: OTLP export has a 5-second timeout. Check your collector’s health and resource allocation. Consider deploying the collector closer to the gateway to reduce network latency.

Missing Attributes

• Symptom: Spans arrive but are missing expected attributes
• Cause: Attributes are only set when the corresponding data is available in the request
• Fix: floopy.provider, floopy.model, and floopy.cache_hit are set by the gateway pipeline. If a request fails before reaching the provider stage, provider-specific attributes won’t be present. This is expected behavior.