The real problem

It’s 1:47 AM. Your phone buzzes. PagerDuty: “checkout-service: 5xx rate > 2%”. You connect, open your dashboard, see a line going up. What did you just do, in 30 seconds, to understand what’s happening?

If the answer involves grep over unstructured logs, you don’t have observability. You have files.

Observability isn’t a tool. It’s the property of a system being able to answer questions you didn’t anticipate about its behavior — without having to stop the system or guess.

The three pillars (and why they don’t suffice alone)

Logs

Discrete events with context. Excellent for understanding an individual transaction in depth.

log.info({
  event: 'ecf.submitted',
  comprobanteId: 'E310000000123',
  rnc: '101099090',
  tipo: '31',
  monto: 15600.00,
  trackingId: 'trk_01HXY...',
  durationMs: 847,
  dgiiEndpoint: 'https://ecf.dgii.gov.do/...'
}, 'e-CF submitted to DGII')

Key point: structured logs. None of console.log('something happened: ' + x). Each log is a JSON object with consistent fields you can filter and aggregate.

Metrics

Numeric values aggregatable over time. Excellent for spotting trends.

ecf.submitted.duration.histogram({ tipo: '31' }).record(847)
ecf.submitted.counter({ tipo: '31', result: 'aceptado' }).inc()

Key point: low cardinality. If your metric has 50 unique labels, you don’t have a metric — you have a log in disguise.

Traces

The full story of a distributed request. Excellent for understanding why something is slow.

trace_id: 8f3d2c9a-7b1e-4a5f
├─ ecf-api (847ms)
│  ├─ validate-xml (12ms)
│  ├─ check-cert (45ms)
│  ├─ dgii.submit (782ms)  ◀── the bottleneck
│  │  └─ network.wait (758ms)
│  └─ build-response (8ms)

Key point: the trace_id propagates. If your trace dies in the second service, it isn’t a trace — it’s a log with hopes.

The common mistake

Teams pick one. “We have Datadog”. “We have Grafana”. “We have OpenTelemetry”. But they use logs without metrics, or metrics without traces. The three pillars are complementary:

• Metrics tell you WHAT is wrong (high 5xx rate).
• Traces tell you WHERE it’s wrong (in the call to DGII).
• Logs tell you WHY it’s wrong (certificate expired).

If you’re missing one, your MTTR (mean time to recovery) goes up.

Rules we apply to every service

1. Context propagates, doesn’t duplicate

Every request has a request_id. If it calls another service, that request_id travels in a header. If the second service logs, that request_id shows up in every log. If everything lives in a trace, everything links via trace_id and span_id.

import { trace, context } from '@opentelemetry/api'

app.use((req, res, next) => {
  const requestId = req.headers['x-request-id'] || crypto.randomUUID()
  res.setHeader('x-request-id', requestId)

  const tracer = trace.getTracer('checkout-service')
  const span = tracer.startSpan('http.request', {
    attributes: { 'http.method': req.method, 'http.url': req.url, 'request.id': requestId }
  })

  context.with(trace.setSpan(context.active(), span), () => {
    res.on('finish', () => {
      span.setAttributes({ 'http.status_code': res.statusCode })
      span.end()
    })
    next()
  })
})

2. Log decisions, not conditions

Bad:

log.info('user.email is ' + user.email)
log.info('user.verified is ' + user.verified)

Good:

log.info({ userId: user.id, verified: user.verified }, 'login.allowed')
// or
log.warn({ userId: user.id, reason: 'unverified' }, 'login.denied')

The second lets you ask “how many logins were denied for unverified” in seconds. The first forces you to process text.

3. SLO before dashboard

Before building 50 panels, define what you care about. Three numbers per critical service:

• Availability (% of successful requests over X time).
• Latency (p95 or p99 below a threshold).
• Error rate (% of requests that fail).

Each has an objective (SLO) and an alert when it’s broken. The rest of the dashboard is exploration — nice to have, not essential.

4. Alerts: few, actionable

An alert that isn’t actionable is noise. If your PagerDuty fires 14 times a day for things that would self-resolve, within a month nobody will pick up.

Simple rule: if the alert doesn’t require immediate human action, it’s not an alert. It’s a metric. It lives in a dashboard, not on your phone.

5. Sample tracing intelligently

100% of traces is expensive and, more importantly, useless: the 999 successful traces tell you nothing. Strategy we use:

• Tail-based sampling: we capture 100% of traces that have errors or are slow (>p95). The rest gets sampled at 5%.
• Head-based sampling with override: if a request enters with header x-debug: 1, that trace captures everything, regardless of outcome.

Typical mistakes we see at clients

“Log everything just in case”

10 GB/day of logs, 99.9% with no value. Storage cost, query cost. Log the important events, not internal variables of every function.

“Metrics with high cardinality”

http_requests_total{user_id="abc123", path="/api/x"}. If you have 1M users, you have 1M series. Your Prometheus implodes.

Better:

http_requests_total{path="/api/x"}  # aggregated
auth_failures_total{reason="bad_password"}  # category, not identity

If you need to know which user failed, that lives in a log or trace.

“Tracing but the spans don’t link”

They forgot to propagate the traceparent header between services. Each span lives alone. It’s not distributed tracing; it’s expensive logging.

“Dashboards nobody looks at”

If your “global performance” dashboard isn’t opened by anyone in a normal week, it’s not a dashboard — it’s a decorative frame. Kill the dashboards you don’t use.

Stack we recommend today

There’s no universal answer. But a reasonable starting point:

Need	Simple option	Enterprise option
Structured logs	Loki	Datadog Logs
Metrics	Prometheus + Grafana	Datadog Metrics
Tracing	Tempo + OpenTelemetry	Datadog APM
Alerts	Grafana + PagerDuty	Datadog Monitors

What matters isn’t the tool — it’s that they’re integrated. If you see a metric rising and can’t jump directly to the corresponding traces, you’ll take 10 minutes where you should take 30 seconds.

Close

Observability is discipline, not budget. A startup with pino + Grafana + OpenTelemetry and clear rules about what to log will debug production faster than an enterprise with Datadog and console.log everywhere.

The question that defines whether your system is observable: “Can I answer in 5 minutes why the last hour looked different from the previous one?” If the answer is no, there’s work to do before the next alert.

— Dawlin