Observability That Actually Pays Off: OTel, SLOs, On-Call

Most engineering orgs don't have an observability problem. They have a signal-to-noise problem. Prometheus is scraping, Grafana is rendering, Datadog is billing — and yet the last three incidents were detected by a customer on Slack. Fixing this in 2025 is less about buying another tool and more about aligning three things: a vendor-neutral instrumentation layer (OpenTelemetry), a service-level contract (SLI/SLO with error budgets), and an on-call practice that treats humans as a finite resource.

Stop instrumenting for a vendor. Instrument for OTel.

OpenTelemetry (OTel) is now the second-most active CNCF project after Kubernetes. Traces are GA in every major language, metrics are stable, and logs reached stable status in the SDK spec during 2024. The practical consequence: there is no longer a good reason to write vendor-specific instrumentation in application code.

A minimal Node.js setup with the OTel SDK looks like this:

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({ url: 'http://otel-collector:4318/v1/traces' }),
  instrumentations: [getNodeAutoInstrumentations()],
  serviceName: 'checkout-api',
});

sdk.start();

The key architectural decision is the OTel Collector sitting between apps and backends. It gives you three things at once: protocol translation (OTLP in, Prometheus/Loki/Tempo/Datadog/whatever out), tail-based sampling, and PII scrubbing before data leaves your perimeter. Switching from Datadog to Grafana Cloud (or vice versa) becomes a config change on the Collector, not a six-month refactor.

The three signals, and where teams get them wrong

| Signal | Good use | Common anti-pattern | |---|---|---| | Metrics | SLIs, capacity planning, alerts | High-cardinality labels (user_id, request_id) blowing up TSDB cost | | Traces | Latency root cause, dependency mapping | 100% sampling in production, then dropping the data anyway | | Logs | Debugging, audit trail | Using logs to compute metrics at query time |

Rule of thumb: *metrics answer is it broken, traces answer where, logs answer why.* If your team routinely greps logs to build a dashboard, you have a metrics gap.

SLIs and SLOs: the contract that makes alerts sane

An SLI is a ratio of good events over total events. An SLO is a target for that ratio over a rolling window. Everything else — error budgets, burn-rate alerts, on-call escalation — derives from those two definitions.

A useful checkout API SLO looks like:

• Availability SLI: sum(rate(http_requests_total{status!~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
• Latency SLI: fraction of requests served under 300 ms at p95
• SLO: 99.9% availability over 30 days → error budget of 43m 12s per month

Once the budget exists, two decisions become mechanical:

1. Alerting: page on burn rate, not on thresholds. A multi-window, multi-burn-rate alert (as documented in the Google SRE workbook) fires when you're consuming budget 14× faster than sustainable over 1h and 5m — catching real incidents without paging on every 500.
2. Release policy: if the budget is exhausted, feature releases pause and reliability work takes priority. This is the part most orgs skip, and it's the part that actually changes engineering behavior.

Tools like Sloth or Nobl9 generate Prometheus recording and alerting rules from a declarative SLO spec. Example with Sloth:

version: "prometheus/v1"
service: "checkout"
slos:
  - name: "availability"
    objective: 99.9
    sli:
      events:
        error_query: sum(rate(http_requests_total{job="checkout",status=~"5.."}[{{.window}}]))
        total_query: sum(rate(http_requests_total{job="checkout"}[{{.window}}]))
    alerting:
      page_alert: { labels: { severity: page } }
      ticket_alert: { labels: { severity: ticket } }

On-call is a system, not a rota

A rotation with a PagerDuty schedule and no operational discipline is how you lose senior engineers. The teams that keep their on-call people happy tend to enforce a few non-negotiables:

• Max 1 week per 6 weeks per engineer. Below that, fatigue compounds silently.
• Every page must be actionable. If an alert fires and the runbook says "check if it recovers", delete the alert or fix the underlying flapping.
• Blameless post-mortems within 5 business days, with at least one action item owned and dated. Track completion rate — under 70% is a red flag.
• On-call compensation, either as time off or additional pay. Free labor at 3 AM is not a culture, it's attrition.
• A weekly "operational review" where the outgoing on-call presents pages received, false positives eliminated, and runbooks updated. This is where signal quality actually improves.

A useful metric to track internally: pages per on-call shift. Healthy teams sit at 0–2. Above 5, you're not doing SRE, you're doing crisis management.

What to build first if you're starting today

1. Deploy an OTel Collector in each environment. Route to your existing backend.
2. Auto-instrument your top 3 services. Don't boil the ocean.
3. Define one SLI and one SLO per service, with the product owner in the room.
4. Replace threshold alerts with burn-rate alerts on those SLOs.
5. Write the runbook before the alert goes live.

Key takeaways

• OpenTelemetry is the default — instrument once, route anywhere via the Collector.
• SLOs turn opinions into contracts. Error budgets align product and engineering on when to slow down.
• Alert on burn rate, not on static thresholds. Multi-window multi-burn-rate is the current best practice.
• On-call health is a leading indicator of reliability. Pages per shift and post-mortem action-item completion are the numbers to watch.
• Cardinality and sampling are the two hidden cost drivers. Design for them from day one, not after the invoice arrives.