The pillars
Five sources. Different consumers for different purposes. The audit ledger is both an observability source AND the product’s data substrate — it pulls double duty.Logs
Every service uses Pino for structured JSON logging. Configured via@guild/logger.
Shape
Every log line is JSON. A typical line:userId, requestId, path are present on every relevant line.
Request ID propagation
A ULID is generated at the edge for every incoming request and attached to a per-request context (AsyncLocalStorage in Node). Every log line emitted during that request includes the ID. When the request triggers a downstream service call, the ID is forwarded in anx-guild-request-id header. The downstream service picks it up and continues using the same ID. A single user action’s complete trace is recoverable across services by filtering on the request ID.
PII redaction
The logger is configured with a redaction list. Any log line containing a field name in the list has that field’s value replaced with[REDACTED] before being written.
Default redactions: phone, bvn, nin, lastName, password, pin, email, dob, address, cardNumber, cvv, raw body fields on auth endpoints. The list is extensible per service.
A leaked log is a leaked log; redaction is the difference between “an attacker has access to operational events” and “an attacker has access to personally identifying information about specific users.”
Log levels
Where logs go
PM2’s log files initially. Rotated daily. Shipped to a central log store with searchable retention. Archived to R2 at 90 days; kept for compliance for the full retention window.Errors (Sentry)
Every uncaught exception is captured and sent to Sentry via@guild/observability. Errors carry the request ID, user ID (when known), and a stack trace.
Sentry’s grouping logic deduplicates errors. We tag releases so we can correlate a spike in a specific error class to a specific deploy.
Metrics (Prometheus)
Every service exposes a/metrics endpoint with Prometheus-format counters and histograms. Scraped by a monitoring agent on a 15-second interval.
The headline metrics:
The metric cardinality is bounded — we don’t tag per-user. The histograms aggregate to a small set of buckets so the time-series store stays manageable.
Dashboards
A handful of canonical dashboards, each tuned for a specific operational concern.Payment health (primary)
Live view of the payment path’s wellbeing:- OPay call success rate — over the last 5m, 1h, 24h.
- OPay call latency — p50, p95, p99 per method.
- Webhook arrival latency — time from OPay’s event timestamp to our receipt.
- Reconcile guard pass rate — should be 100%; any failures are immediate concerns.
- Wallet motion rate — kobo / second by motion_kind.
- Circuit breaker state — per OPay method, are we open?
Service health
Per-service dashboard:- Request rate and latency by endpoint.
- Error rate (5xx and tracked 4xx by category).
- PM2 process status (alive, memory, uptime).
- Redis connection pool.
- pgbouncer connection pool.
Channel health
How each channel is performing:- Voice calls active / in queue per minute.
- WhatsApp message delivery latency (Meta → us → response).
- SMS gateway outbound queue depth per paired phone.
- SMS delivery success rate per carrier.
ML model health
Per-model dashboard:- Inference latency p50, p95, p99.
- Predictions per minute (volume).
- Boot-time feature-order check result (warning if mismatch).
- Standing tier distribution across active users (should be roughly stable; sudden shifts indicate model drift or data quality issues).
Audit ledger as observability
The audit ledger is a parallel observability channel that’s structurally guaranteed. For compliance investigations or dispute resolution, the audit log is the authoritative source. Application logs can be lost (rotation, retention); the audit ledger is forever. Application logs can be edited (in theory; we don’t); the audit ledger cannot. A typical compliance query: “Show me every action this user took in the last 90 days, sorted by time, with the request ID for tracing.” That’s a SQL query againstchannel_audit. We don’t reach for logs; we reach for the ledger.
Alerting
Most alerts are runbook-backed. The on-call engineer hits a known set of steps; a true unknown promotes to incident response.
Tracing
Distributed tracing across services is via the request-ID propagation described earlier. We don’t run a full OpenTelemetry-style trace collector at v1 — for our service count (5) and call pattern (mostly request-response, not deeply nested), the request-ID-on-every-log approach gives most of the value with very little operational overhead. We’d revisit at higher complexity (more services, more async paths). At our current scale, log filtering on request ID is fast and complete.What the operator sees during an incident
The incident playbook starts with three windows:- The Payment health dashboard. What’s broken?
- Sentry’s release view. Is this correlated to a recent deploy?
- Logs filtered by error and request-ID. What was the user doing?
What good observability looks like (the bar)
Our operational bar:- Every error has a request ID. Tracing a user’s complaint back to the actual error in our system takes one query.
- Every alert has a runbook. The on-call engineer doesn’t improvise during incidents.
- Every deploy is correlated to metrics. A regression in any tracked metric is visible at deploy time.
- Every OPay call is observed. Latency, outcome, retry count — all surfaced.
- Every webhook is logged. Including the ones we reject (for security audit).
- Every user-impacting change is auditable. The ledger sees everything.
What to read next
Resilience & scale
What the alerts catch and what we do about them.
Data flow
The audit ledger that doubles as compliance observability.
Security & trust
The security framing of the observability stack.
Infrastructure
The hosts that all this observability sits on.