Case study · Systems & Infrastructure
RelayDB
Change data capture that can prove what it delivered
The problem
Databases change and everything downstream finds out late: caches drift, search indexes fall behind, invoices wait on batch jobs. Polling misses deletes and double-reads race transactions — and nobody can prove what was delivered, or to whom.
What I built
WAL capture with pgoutput
Reads committed changes from a PostgreSQL publication through pgoutput logical decoding, with TOAST awareness for unchanged large values.
The capture invariant
Capture persists the transaction's normalized events and advances the metadata checkpoint in one transaction. Only after that commit may capture report the flushed LSN back to PostgreSQL — a crash before acknowledgement replays WAL without creating a second durable event identity.
Fenced checkpoints and consumers
Lease generation is the fencing boundary for consumer ownership: an old owner cannot advance a newer owner's cursor. Checkpoints are fenced so a standby capture cannot double-deliver.
Delivery and operations surface
HMAC-signed webhook delivery with an SSRF-guarded dialer, dead letters, replay cursors, a relayctl CLI, REST + gRPC APIs, and a Next.js control room with explicit Live/Demo data modes.
Engineering decisions
At-least-once ingestion, honestly
RelayDB claims at-least-once ingestion, not exactly-once side effects. The safety boundary is the metadata transaction; downstream consumers use their own idempotency keys.
The dashboard never fakes it
Live mode never substitutes fixtures when the API is unavailable or empty — unavailable reads stay visibly unavailable. Demo mode is opt-in deterministic evidence, clearly labeled.
Keys stay server-side
The reader API key lives in the dashboard's server-side BFF proxy; browser code calls same-origin /api/v1/* and never receives the key.
Architecture
Source PostgreSQL → pgoutput WAL → capture persists events + fenced checkpoint in one transaction → flushed LSN acknowledgement → API, control room, and HMAC-signed webhook delivery
Testing & CI
Testcontainers suites prove ordered capture and crash-replay without loss or duplication. Go 1.26, PostgreSQL 16, Buf-managed protobufs; make fmt/vet/lint/test gate CI.
Go / pgoutput / gRPC / HMAC webhooks / Next.js