Infrastructure

The infrastructure brief was simple. Ship a national-grade financial system, without burning a week on platform plumbing, on a budget a hackathon team can actually carry. Every decision below is a deliberate “pick the boring option” choice.

The VPS layout

One VPS. One Postgres. One Redis. Two sibling processes (Asterisk for SIP, Go for SMS). nginx terminates TLS. PM2 supervises the Node and Python services. UFW blocks everything that doesn’t need to be open.

Why VPS, not Kubernetes

The first question every infrastructure conversation hits.

Option	Why we said no
Kubernetes (EKS / GKE)	Three weeks of platform work for capacity we don’t need at our scale. Pod-to-pod networking adds latency we’d otherwise spend on OPay. Cluster operations require a dedicated SRE we don’t have.
Docker + docker-compose	Adds an abstraction layer with no operational payoff at our scale. Process startup is slower. PM2 reload is faster and more predictable than `docker compose up`.
Serverless (Lambda / Cloud Run)	Cold-start latency is unacceptable on the payment path. Per-request pricing punishes high-frequency webhooks. Stateful workers (BullMQ) need a process, not a function.
Multi-VPS with load balancer	Premature at our scale. One well-provisioned VPS handles five-figure DAU comfortably. Horizontal scaling is a known path when we cross the threshold.
Virtual Private Server, PM2 ✅	Boring. Predictable. Survives a reload without dropping connections. Five-minute deploys. Every team member already knows it.

The decision is reversible. The day Guild crosses the threshold where one host can’t hold the daily traffic, we add a second host behind a load balancer in front of nginx. The application is already stateless; the move is mechanical.

CloudPanel

CloudPanel is a free, open-source server administration panel for Ubuntu. We use it for three concrete things.

1. nginx vhost templating

Every vhost is rendered from a CloudPanel template with substitution placeholders. The reference vhost lives in backend/infra/nginx.conf (checked in for review). CloudPanel substitutes {{ssl_certificate}}, {{ssl_certificate_key}}, {{nginx_access_log}}, and {{app_port}} at render time. After editing, the change is reloaded with nginx -t && systemctl reload nginx (or CloudPanel’s UI reload button). Zero downtime. Active connections drain.

2. Automatic certificate management

Cloudflare DNS validates ownership; CloudPanel renews Origin Certs and Let’s Encrypt certs automatically. No 3 AM page about an expired cert.

3. Audit trail of system changes

Every config change goes through CloudPanel’s audit log. Useful when something stops working at 2 PM and we need to know what changed at 1:55 PM. We don’t use CloudPanel for application deployment — that’s PM2’s job. We use it strictly for the system-edge concerns (nginx, certs, firewall, system users).

nginx configuration

The full vhost lives at backend/infra/nginx.conf. Three things in it worth highlighting.

Loopback-only webhook routes

Each webhook destination is an exact-match location block routed to the specific service’s loopback port.

location = /webhooks/wa {
  proxy_pass http://127.0.0.1:5003/v1/webhook$is_args$args;
  proxy_request_buffering off;
  client_max_body_size 10m;
}

location = /webhooks/voice {
  proxy_pass http://127.0.0.1:5002/v1/webhook$is_args$args;
  proxy_request_buffering off;
  client_max_body_size 10m;
}

location = /webhooks/sms {
  proxy_pass http://127.0.0.1:5001/v1/webhook$is_args$args;
  proxy_request_buffering off;
  client_max_body_size 10m;
}

`proxy_request_buffering off` is non-negotiable

Webhooks sign the raw request body. If nginx buffers and re-concatenates the body, the bytes change and signature verification fails. Setting proxy_request_buffering off makes nginx stream the body through without touching it. This was a real bug we hit. Diagnosed by re-reading OPay’s signature documentation, then re-reading nginx’s buffering documentation. The fix is one line. Knowing to write that line came from understanding both sides.

Everything else → core-api

The catch-all location / block routes anything not webhook-specific to core-api. The proxy timeouts are generous (900s) because some endpoints (file uploads, batch operations) legitimately take longer than the default 60s.

PM2 ecosystem

Every service runs under PM2 with explicit configuration in ecosystem.config.cjs.

// abbreviated
module.exports = {
  apps: [
    {
      name: 'core-api',
      script: './apps/core-api/dist/server.js',
      instances: 2,
      exec_mode: 'cluster',
      max_memory_restart: '500M',
      env: { NODE_ENV: 'production' }
    },
    {
      name: 'whatsapp-svc',
      script: './apps/whatsapp-svc/dist/server.js',
      instances: 1,
      exec_mode: 'fork',
      max_memory_restart: '300M'
    },
    // voice-svc, sms-svc — fork mode
    // ml-service — Python via uvicorn, managed by PM2
  ]
};

Service	PM2 mode	Why
core-api	cluster (2 instances)	Highest traffic, benefits from in-process load balancing across CPU cores.
whatsapp-svc, voice-svc, sms-svc	fork (1 instance each)	Lower per-service traffic; channel-specific webhook signatures don’t benefit from multiple instances at our scale.
ml-service	fork (1 instance)	Python process; model artifacts loaded into memory once at boot. Adding instances doubles memory.

Graceful reload

pm2 reload <service> rolls services one at a time. The new process boots, passes health checks, then PM2 stops the old one. Open connections drain. No interruption visible to users. A full deploy (all services) takes about 90 seconds end to end.

Process supervision

PM2 restarts a crashed process automatically. max_memory_restart triggers a restart if memory exceeds the threshold (defends against slow leaks). Logs go to PM2’s log directory, rotated daily, archived to R2 after 7 days.

System services

Postgres 16 + PostGIS

Installed via apt. PostGIS gives us proper geographic queries (distance between city centroids, point-in-polygon for job location matching). Six schemas, one database:

Schema	Owned by
`identity`	core-api
`jobs`	core-api
`payments`	core-api
`sms`	sms-svc
`voice`	voice-svc
`whatsapp`	whatsapp-svc
`ml`	ml-service

Cross-schema joins are cheap inside one cluster. Each service has its own Prisma schema file declaring @@schema(...). Migrations are owned by the service.

pgbouncer (transaction pool mode)

Connection pooling between application services and Postgres. Transaction mode means a connection from the pool is held only for the duration of one transaction; Prisma’s connection handling plays well with this. Reduces Postgres’s connection-handling overhead at scale.

Redis

One Redis instance. Used for:

Sessions (key prefix sess:)
OTPs (key prefix otp:, short TTL)
Idempotency cache (key prefix idem:)
BullMQ queues (key prefix bull:)
Bandit state (key prefix ml:bandit:)
Read-only OPay cache (key prefix sq:cache:, used by circuit-breaker fallback)

Memory pressure is monitored. We’d shard by purpose (sessions vs queues vs bandit) before we hit a hard ceiling.

asterisk

Sibling process. Listens on TLS 5061 for SIP traffic. Bridges WhatsApp Calling to voice-svc. Full deep dive at SIP infrastructure.

sms-server (Go)

Sibling process. RESTful API on a loopback port. Bridges sms-svc to paired Android devices with Nigerian SIMs. Full deep dive at SMS infrastructure.

ufw

Default-deny firewall.

allow 22/tcp     # SSH (key-only, no password auth)
allow 80/tcp     # HTTP (redirects to HTTPS)
allow 443/tcp    # HTTPS
allow 5061/tcp   # SIP TLS for WhatsApp Calling
deny  *          # everything else

Loopback ports (5000–5004) are not exposed externally. Postgres (5432), Redis (6379), and pgbouncer (6432) all bind to 127.0.0.1.

Cloudflare R2

Object storage for everything that doesn’t belong in Postgres.

Bucket prefix	Contents	Lifecycle
`voice-recordings/`	Tola call audio captures	30-day TTL, signed-URL access only
`documents/`	KYC document uploads	Indefinite, encrypted at rest
`snapshots/`	Partner API snapshots (signed JSON)	90-day TTL, signed-URL access for partners
`exports/`	User data exports (NDPR right-to-portability)	7-day TTL
`logs/`	PM2 log archives	90-day TTL

Egress-free bandwidth to Cloudflare-served viewers — this is the cost advantage that pushed us away from S3 at our access patterns.

DNS and certs

Cloudflare is authoritative DNS. Six A records on Cloudflare orange-cloud mode:

guild.com.ng → VPS IP (Next.js frontend via the BFF)
api.guild.com.ng → VPS IP (core-api via nginx)
wa.guild.com.ng → VPS IP (loopback to whatsapp-svc via nginx)
sip.guild.com.ng → VPS IP (Asterisk directly on 5061)
ml.guild.com.ng → VPS IP (loopback to ml-service via nginx)
docs.guild.com.ng → Mintlify-hosted (CNAME)

Origin Cert (cloudflare-issued, 15-year validity) installed on the VPS for full-strict TLS between Cloudflare and origin. Lets us avoid the certificate validation dance for cloudflare-fronted traffic; cert renewal is on a 15-year cadence, not 90 days.

The deploy pipeline

Capistrano-style symlink release. ~/releases/<commit-sha>/ holds the new build; ~/current/ symlinks to the active release. A bad deploy is rolled back by changing the symlink — ~/releases/ keeps the last five releases. pm2 reload triggers the rolling restart described earlier. No request loss during deploy. Total time from push to live: ~5 minutes for the full pipeline.

Backup and disaster recovery

What	Backup cadence	Recovery target
Postgres	Continuous WAL archive to R2. Daily full backup.	RPO < 5 min, RTO < 30 min from clean snapshot
Redis	RDB snapshot every 6 hours to R2. AOF for in-day replay.	RPO < 6 hours for non-critical; sessions can be re-established
R2 buckets	Cross-region replication enabled	Effectively instantaneous
Application code	Git is the source of truth	Re-deployable in 5 min from any commit
Secrets	Encrypted backup of `shared/.env` in 1Password	Restorable manually in case of host loss

The RTO objective for a worst-case (entire host gone) is under one hour: provision new host, restore database from R2, restore Redis snapshot, redeploy the application from git, restore secrets, point DNS. We test this once a quarter.

Observability summary

(Deeper detail in Observability.)

Logs: Pino JSON, request-ID propagated through every log line, PII-redacted at the logger layer. Shipped to a central log store.
Errors: Sentry. Every uncaught exception surfaces; per-environment quotas.
Metrics: Prometheus exporter via @guild/observability. Scrape-able for any dashboard tool.
Health checks: /healthz (liveness), /readyz (readiness, includes downstream checks).
Audit: the audit ledger itself doubles as a compliance log. Every state change writes a row.

Cost picture

At the current scale (pre-launch), the entire infrastructure stack runs under a few hundred dollars per month. The cost curve scales with daily active users and OPay transaction volume. Cloudflare bandwidth is the rare cost item — R2 egress is free to Cloudflare-served viewers, and Cloudflare’s bandwidth from origin to user is included. The cost line that will move first: OPay transaction fees. They scale with revenue, which is the right direction for a payment platform.

SMS infrastructure

The Go gateway. The paired Android devices. The two-way SMS rail.

SIP infrastructure

Asterisk. WhatsApp Calling. Opus to PCMU transcoding.

Inter-service comms

The HMAC layer that signs every internal call.

Resilience & scale

Circuit breakers, demo resilience, the partition path.

​The VPS layout

​Why VPS, not Kubernetes

​CloudPanel

​1. nginx vhost templating

​2. Automatic certificate management

​3. Audit trail of system changes

​nginx configuration

​Loopback-only webhook routes

​proxy_request_buffering off is non-negotiable

​Everything else → core-api

​PM2 ecosystem

​Graceful reload

​Process supervision

​System services

​Postgres 16 + PostGIS

​pgbouncer (transaction pool mode)

​Redis

​asterisk

​sms-server (Go)

​ufw

​Cloudflare R2

​DNS and certs

​The deploy pipeline

​Backup and disaster recovery

​Observability summary

​Cost picture

​What to read next