Resilience & scale

A payment platform earns trust on the day OPay has a hiccup, NIBSS slows down, or a third-party provider returns 500s for an hour. Guild was built to keep working on those days, and to keep scaling on the days the platform sees a 10x traffic spike. This page is the resilience playbook and the scale path.

​

The layers of defence

Each layer assumes the one below it might fail. Idempotency keys assume retries will happen. Circuit breakers assume OPay will sometimes return 500s. The reconcile guard assumes webhook secrets could leak. The append-only ledger assumes engineers could make mistakes. Demo resilience assumes OPay sandbox could burp on stage. Each layer adds belt to a stack of suspenders.

​

Idempotency: the most important property

Every write operation in Guild is idempotent end to end. Same input, same output, regardless of how many times it fires. Two layers of cache:

1. Redis fast-path. Sub-millisecond lookup. 24-hour TTL.
2. Postgres durable store. Survives Redis flushes. 30-day retention.

The durable store catches retries beyond the Redis window. The fast-path catches everything else with near-zero overhead. The key insight: OPay’s API is also idempotent on the request reference. When we retry a debitMandate with the same reference, OPay recognises it as a retry and returns the original response. So even if our local cache layers fail, the eventual call to OPay will not double-debit.

​

Circuit breakers (opossum)

Every OPay-bound call routes through a circuit breaker. The state machine: When the breaker is open, callers get a TRANSIENT error immediately. The caller’s retry policy backs off appropriately. No thundering herd. No queue of doomed requests piling up. For read-only operations (getTransaction, lookupPayoutAccount), the breaker has a cached-response fallback. If the breaker is open and we have a recent successful response for the same lookup, we serve it. The user gets a slightly stale but still useful response. This is the demo-resilience play — even if OPay sandbox burps during a stage demo, read paths keep working.

​

The reconcile guard, once more

Documented in detail at OPay integration. The short version: every credit-bearing webhook is re-verified against OPay’s authoritative verify endpoint before we ever credit a wallet. Defends against signature-replay attacks even if our webhook secret leaks. This is a small but material design choice. Most platforms verify signatures and credit. We verify signatures, idempotency-check the event ID, AND independently re-verify the transaction. Three checks instead of one.

​

The append-only audit ledger

Detailed at Data flow. Two tables (channel_audit, payments_audit) with BEFORE UPDATE OR DELETE triggers that reject mutation. Once a row is written, it’s permanent. The resilience guarantee: even if every other layer fails — Redis dies, OPay replays, an engineer makes a mistake — the audit history is preserved. Reconstruction of state from the audit log is always possible.

​

Demo-resilience tooling

Stage demos are a specific category of stress test. The platform must work in front of judges at exactly the moment OPay’s sandbox decides to take a coffee break. Three tools we built specifically for this.

​

Operator control panel

A web UI at guild.com.ng/demo. Restricted to a specific role. Buttons for:

• Replay a known-good webhook to re-trigger a payment flow during a demo.
• Play a pre-recorded backup video of the same demo path, in case live execution fails.
• Reset a demo user’s state (clear wallet balance, revoke a mandate) for a clean walkthrough.
• Skip the matching model and pre-seed a specific candidate list (in case the model gives an unexpected ranking).

The control panel is invisible to regular users. It’s a stage-only utility.

​

Webhook replay (pnpm tools/opay-replay <event-id>)

A CLI tool that re-sends a previously captured OPay webhook to our own endpoint. Useful for:

• Replaying a webhook during a stage demo if OPay’s sandbox doesn’t fire it cleanly.
• Reproducing a production issue in dev with the actual production payload.
• Testing webhook handler changes against a real OPay event shape.

The tool reads from a captured-event log we maintain. Every webhook we receive is logged (with signatures redacted) for replay.

​

Pre-recorded backup videos

For each demo path, we have a pre-recorded 25-30 second video of the same flow executing successfully. If the live execution stalls mid-demo, the operator can play the video and narrate over it without missing a beat. The video isn’t a substitute for working code. It’s a backup for the case where OPay’s sandbox is temporarily unreliable in the moment.

​

Failure modes we’ve planned for

​

Health checks

Every service exposes two endpoints:

• /healthz — liveness. Returns 200 if the process is alive and the basic event loop is responsive. Used by PM2 to detect hung processes.
• /readyz — readiness. Returns 200 only if downstream dependencies are healthy (database, Redis, downstream services reachable). Used by future load balancer for routing decisions.

For ml-service, /readyz also verifies model artifacts are loaded and the feature-order sanity check passes. Health checks are checked every 10 seconds. A failure for 30 consecutive seconds (3 checks) triggers an alert.

​

The scale path

The brief asks for 10K to national. Here’s the line.

​

10K → 100K

Same infrastructure. PM2 cluster mode on core-api scales from 2 to N instances (one per vCPU). Postgres handles the load comfortably. Redis stays well below memory pressure. OPay’s rate limits are not a constraint. What changes: more SIM cards in the SMS gateway, more concurrent voice runtime capacity, monitoring thresholds tuned. No architectural change.

​

100K → 1M

Multi-VPS deployment. Load balancer in front of nginx (or DNS-level load balancing via Cloudflare). Postgres gets a dedicated host with replication to read replicas; pgbouncer routes reads to replicas. Redis splits into sessions vs queues vs cache instances. The application is already stateless and horizontally-scalable. The change is operational, not architectural.

​

1M → 10M

Postgres partitioning by user-ID range comes online. Asterisk clusters with a SIP load balancer. Voice runtime adds capacity. ML inference horizontally scales by adding replicas. R2 scales without our intervention. The audit tables, partitioned by month already, continue to age cleanly. The cumulative ledger size at 10M users with average activity is on the order of low TBs — well within standard Postgres operational limits.

​

10M → 40M (theoretical TAM)

This is the point where Guild has effectively become national infrastructure. Multi-region (north, south, east, west) for compute. Multi-region Postgres with read replicas. OPay’s enterprise tier becomes the relationship model. At this scale, the original architecture is mostly intact. The hard work is operational maturity, not redesign.

​

Where we’d swap technologies

A few honest “we’d reconsider at scale” items:

• The single Asterisk instance for SIP would cluster, with a SIP-aware load balancer like Kamailio in front. Documented approach, not novel.
• One Redis instance would shard. Probably Redis Cluster, or a managed service.
• PM2 process supervision would move to a managed orchestrator if we crossed about a dozen VPS hosts. Until then, PM2’s reliability is sufficient.
• Express might give way to a more performance-tuned alternative if profiling shows per-request CPU is the bottleneck. Not anticipated.

​

What we deliberately don’t optimise for

A few “premature optimisation” items we’ve consciously skipped:

• GraphQL or gRPC between services. REST + JSON is enough. Easier to debug.
• Message broker beyond Redis. BullMQ on Redis handles every queue use case at our scale.
• Service mesh / mTLS. HMAC over loopback is the right tool for our topology.
• Eventual-consistency reads. Strong consistency from Postgres is fine at our scale.
• Pre-emptive horizontal scaling. We add capacity when we approach the threshold, not before.

The discipline is “boring at every layer where boring works.” We invest the architecture budget on the things that genuinely matter — the ledger, the OPay integration, the multi-channel surface — and pick the proven option everywhere else.

​

What to read next