Observability Overview¶

Audience: SREs wiring Bernstein into an existing observability stack.

Overview¶

Bernstein emits four classes of signal:

• Metrics — Prometheus-format counters, gauges, and histograms exposed at /metrics. Native Datadog APM and OpenTelemetry/OTLP export are also available out of the box.
• Traces — task and agent execution spans via OpenTelemetry, with built-in presets for Jaeger, Grafana Tempo, Datadog, Zipkin, and console.
• Events — structured JSON events on a Server-Sent-Events bus (/events), per-task progress streams, and per-agent log streams.
• Anomaly signals — core/observability/behavior_anomaly.py runs both post-completion and real-time detectors over agent state and emits signals from LOG (informational) all the way to KILL_AGENT (terminate the agent on the next tick).

Everything in this doc is shipped today and exposed via documented endpoints. Wiring it into Prometheus + Grafana takes one scrape config plus one dashboard import; adding Datadog or OTLP export is one config block.

Endpoints catalogue¶

All HTTP. Authentication rules follow the standard middleware (Security and identity) — /metrics is gated by viewer permissions when an auth backend is configured.

Prometheus / Grafana / SLOs¶

Observability detail¶

These power the in-product dashboard and are equally consumable by external tools.

All defined in core/routes/observability.py.

Costs (12 sub-endpoints in core/routes/costs.py)¶

Provider latency¶

A provider+model is flagged degraded: true when its p99 exceeds 2x the 7-day baseline (provider_latency.py:42-44).

Custom metrics + predictive alerts¶

Custom metrics are configured under metrics: in bernstein.yaml, each with a formula, unit, and description (custom_metrics.py:32-39); the evaluator gets tick-level variables (tasks_spawned, errors, active_agents, etc.) plus cumulative totals (custom_metrics.py:44-77).

Health and readiness¶

Prometheus + Grafana wiring¶

Minimal scrape config:

scrape_configs:
  - job_name: bernstein
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets: ['bernstein-host:8052']
    bearer_token: ${BERNSTEIN_AUTH_TOKEN}    # required when auth is on
    scrape_interval: 15s
    scrape_timeout: 10s

Import the bundled dashboard:

curl -H "Authorization: Bearer $BERNSTEIN_AUTH_TOKEN" \
     http://bernstein-host:8052/grafana/dashboard?datasource=Prometheus \
     -o bernstein-dashboard.json
# Then in Grafana UI: Dashboards → Import → upload JSON, pick the
# Prometheus datasource that matches `?datasource=` above.

The dashboard ships as Content-Disposition: attachment so the response is safe to save directly. The ?datasource= query parameter rewrites the datasource references in the JSON (grafana.py:18-30); pass the exact name of your Grafana datasource.

Datadog APM¶

core/observability/apm_integration.py (header at lines 1-44) supports both ddtrace-agent and direct-OTLP-ingest paths.

Environment variables (apm_integration.py:29-37):

Setup is one of:

from bernstein.core.observability.apm_integration import (
    auto_configure_apm, configure_datadog
)

# Auto-pick whatever is available based on env vars
auto_configure_apm()

# Or be explicit
configure_datadog()

What gets sent: task and agent spans (start/end, status, role, model, complexity), gate execution spans, cascade-router escalations, and exception traces. Logs are not piped directly — use the Datadog Agent's log file collection on .sdd/runtime/*.log if you need log correlation.

New Relic is also supported (apm_integration.py:39-44); set NEW_RELIC_LICENSE_KEY (or NEWRELIC_API_KEY) and call configure_newrelic().

OpenTelemetry / OTLP¶

core/observability/telemetry.py provides the canonical OTel surface: tracer, meter, span context manager, and named presets for common backends (telemetry.py:1-22).

from bernstein.core.observability.telemetry import (
    init_telemetry_from_preset, init_telemetry, start_span
)

# Built-in presets (telemetry.py:86-100): "jaeger", "grafana", "datadog",
# "zipkin", "console", and more. Each preset hardcodes endpoint, protocol,
# and TLS expectations.
init_telemetry_from_preset("jaeger")

# Or supply a custom OTLP endpoint
init_telemetry("http://my-collector:4317", protocol="grpc")

# Spans are produced from anywhere in the runtime
with start_span("task.run", {"task.id": task_id}):
    ...

The default protocol is OTLP/gRPC (telemetry.py:50, DEFAULT_PROTOCOL); switch to HTTP/protobuf via the protocol argument or via the preset's _HTTP_PROTOBUF (telemetry.py:53). The default service name is bernstein (telemetry.py:58).

Service maps work out of the box because every span carries service.name as a resource attribute; no extra config required for Tempo or Jaeger to associate them.

SLO tracking¶

core/observability/slo.py defines targets, computes status, and tracks error-budget burn-down (slo.py:1-13). Default targets ship at:

• Task success rate ≥ 90 %
• Merge success rate ≥ 95 %
• P95 task duration < 30 minutes

Each target has current value, target threshold, warning_threshold, and a 1-hour rolling window_seconds (slo.py:80-89). Status is one of green / yellow / red (slo.py:64-69).

Burn-rate snapshots (slo.py:35-61) are kept in a 120-sample ring buffer (~1 hour at 30s intervals). When the rolling burn rate would deplete the error budget, ErrorBudgetAction (slo.py:72-77) signals which remediation policy to apply: REDUCE_AGENTS, UPGRADE_MODEL, or INCREASE_REVIEW.

To define a custom SLO, instantiate SLOTarget and add it to the tracker; expose it via GET /slo. To act on breaches outside the in-process actions, watch GET /slo/budget for is_depleted: true (slo.py:... and routes/slo.py:55-62).

Behavior anomaly detection¶

Source: core/observability/behavior_anomaly.py. Covers two distinct detection modes (behavior_anomaly.py:1-20):

1. Post-completion — BehaviorAnomalyDetector analyses metrics from .sdd/metrics/tasks.jsonl after a task finishes and emits AnomalySignal values.
2. Real-time — RealtimeBehaviorMonitor tracks in-flight session state on every progress update and fires immediately on suspicious activity. On KILL_AGENT severity it writes a structured kill-signal file (.sdd/runtime/{session_id}.kill) so the orchestrator terminates the agent on the next tick — the same mechanism the circuit breaker uses.

Real-time detection dimensions (behavior_anomaly.py:11-19):

Allow-list patterns counter false positives: *.env.example, *.env.template, *.env.sample, tests/*, test/*, docs/* (:80-87).

How to interpret alerts:

• LOG — informational. Often noise during refactors that touch many files at once.
• WARN — investigate the agent log; the agent is misbehaving but not necessarily compromised.
• KILL_AGENT — the runtime has already written a kill signal. The orchestrator will terminate the agent on its next tick and the agent's branch is preserved for human review under .sdd/quarantine/{session_id}.json (see Circuit breakers below).

To export anomaly events to your stack, watch /observability/incidents and /observability/incident-timeline/{id}, and tail .sdd/metrics/kill_audit.jsonl for the same events with full forensic context.

Circuit breakers¶

Two distinct breakers, both producing structured signals.

Agent-level circuit breaker (core/observability/circuit_breaker.py):

Real-time breaker that auto-terminates misbehaving agents (circuit_breaker.py:1-15). Triggers:

• Scope violation — agent edited files outside the task's owned_files.
• Budget violation — agent exceeded a per-session token limit.

On trigger:

1. Writes a structured .sdd/runtime/{session_id}.kill JSON file.
2. Appends to .sdd/metrics/kill_audit.jsonl (circuit_breaker.py:41-72).
3. Writes .sdd/quarantine/{session_id}.json so the agent's git branch is preserved for review (:75-...).

The orchestrator picks up .kill files via check_kill_signals() in the agent_lifecycle module on the next tick.

Service-level cascading-failure circuit breaker (core/observability/cascading_failure_circuit_breaker.py):

Three-state breaker per service (CLOSED / OPEN / HALF_OPEN — cascading_failure_circuit_breaker.py:39-44). Wraps calls to upstream services (LLM providers, task server, git) with independent failure and latency thresholds. Configuration (cascading_failure_circuit_breaker.py:53-69):

How to reset:

• Agent-level: human review of the quarantined branch and resolution via the approvals UI (/approvals/queue/{id}/resolve). The kill signal is consumed by the orchestrator and is not user-replayable.
• Service-level: breakers transition CLOSED automatically after a successful HALF_OPEN probe set. To force a reset, restart the service or call the breaker registry's reset method during incident response.

Provider-specific circuit breakers (core/observability/provider_circuit_breaker.py) extend the same state machine with per-provider rate-limit awareness; they feed into CascadeFallbackManager to drive cross-adapter failover (see Quality pipeline and Model routing).

Code pointers¶