Running a large language model in production without observability is like flying a plane with no instruments. You don't know your altitude, airspeed, or fuel level — and you won't know anything is wrong until you crash. This is the situation most AI teams find themselves in when they ship their first LLM-powered feature.

I built Observo to solve this problem. Here's the full technical architecture.

What You Actually Need to Observe

Before building any infrastructure, I had to define what "observability" means for an LLM system. It's different from traditional software monitoring. You can't just track error rates and latency — you need to understand the semantic quality of outputs. That means:

  • Trace-level data: Every inference event, including input tokens, output tokens, latency, model version, and timestamp.
  • Evaluation scores: Automated quality signals — faithfulness, relevance, coherence — computed for every response.
  • Drift detection: Statistical signals indicating that model behavior is changing over time, even if individual responses look fine.
  • Cost accounting: Token usage and API costs broken down by feature, user segment, and query type.

The Telemetry Collection Layer

Observo uses OpenTelemetry as its instrumentation standard. This is critical — it means teams don't need to install a proprietary SDK. If you're already using otel, you get Observo compatibility for free.

Every LLM call is instrumented as an OTel span with custom attributes:

from opentelemetry import trace
from opentelemetry.trace import SpanKind

tracer = trace.get_tracer("observo.llm")

def traced_completion(prompt: str, model: str) -> str:
    with tracer.start_as_current_span(
        "llm.completion",
        kind=SpanKind.CLIENT,
        attributes={
            "llm.model": model,
            "llm.prompt.tokens": count_tokens(prompt),
            "llm.prompt.hash": sha256(prompt),
        }
    ) as span:
        response = call_llm(prompt, model)
        
        span.set_attributes({
            "llm.completion.tokens": count_tokens(response),
            "llm.latency_ms": span.elapsed_ms,
            "llm.cost_usd": calculate_cost(model, span),
        })
        
        return response

High-Throughput Storage with ClickHouse

At scale, an LLM system can generate millions of traces per day. Standard relational databases can't handle this write load while maintaining sub-second query performance for the dashboard. I chose ClickHouse as the storage backend for a specific reason: its columnar storage engine allows aggregation queries (like "average faithfulness score over the last 7 days, grouped by model version") to run in milliseconds even over hundreds of millions of rows.

-- Example ClickHouse query Observo runs for dashboard
SELECT
    toStartOfHour(timestamp) AS hour,
    model_version,
    avg(faithfulness_score)  AS avg_faithfulness,
    avg(latency_ms)          AS avg_latency,
    sum(total_tokens)        AS token_volume,
    count()                  AS request_count
FROM llm_traces
WHERE timestamp > now() - INTERVAL 24 HOUR
GROUP BY hour, model_version
ORDER BY hour DESC

Automated Hallucination Scoring

The most technically complex part of Observo is the automated evaluation pipeline. For every response, Observo computes a faithfulness score — a measure of whether the LLM's response is grounded in the provided context. I use a fine-tuned cross-encoder model that takes (question, context, response) triples and outputs a faithfulness score between 0 and 1:

def score_faithfulness(
    question: str,
    context: str,
    response: str,
    model: CrossEncoder
) -> float:
    """
    Returns 0.0 (complete hallucination) to 1.0 (fully grounded).
    Runs asynchronously after each LLM call to avoid adding latency.
    """
    input_pair = f"{question} [SEP] {context} [SEP] {response}"
    score = model.predict(input_pair)
    return float(torch.sigmoid(torch.tensor(score)))

This evaluation runs asynchronously in a background worker pool so it doesn't add any latency to the user-facing LLM call.

Statistical Drift Detection

A model can pass individual quality checks while its overall behavior is slowly degrading. Observo detects this using a Population Stability Index (PSI) calculated over rolling windows of evaluation scores:

def calculate_psi(baseline: np.ndarray, current: np.ndarray, bins: int = 10) -> float:
    """PSI < 0.1: no drift. 0.1-0.25: moderate. > 0.25: major drift."""
    baseline_pct = np.histogram(baseline, bins=bins, density=True)[0]
    current_pct  = np.histogram(current,  bins=bins, density=True)[0]
    
    # Avoid log(0)
    baseline_pct = np.where(baseline_pct == 0, 0.0001, baseline_pct)
    current_pct  = np.where(current_pct  == 0, 0.0001, current_pct)
    
    psi = np.sum((current_pct - baseline_pct) * np.log(current_pct / baseline_pct))
    return psi

When PSI exceeds 0.25, Observo automatically triggers an incident alert and flags the affected time window for manual review.

The Architecture in One Diagram

The full Observo stack is: OTel SDK → FastAPI Collector → ClickHouse → Evaluation Workers → Grafana Dashboards → Alert Manager. Everything runs on standard infrastructure with no proprietary lock-in. The entire stack can be self-hosted in about 20 minutes using the provided Docker Compose file.

The biggest lesson: observability is not a feature you add to an AI system after the fact. It's the foundation you build first, before you ship anything. Without it, you're not operating an AI system — you're operating a black box and hoping for the best.