Self-Healing Reliability Lab: Trading Platform

A trading system is an unforgiving workload for an SRE: stale data costs money, downtime costs more, and an automated action fired at the wrong moment can do more damage than the incident itself. I built this lab to practise failing safely: kill the primary database while orders are flowing, and let the system promote a replica, re-route writes and drain its backlog without a human.

Everything below runs today on Docker Compose: a Python order API, a Kafka event bus, a consumer worker, a MySQL primary/replica pair with GTID replication behind HAProxy, a small incident-control-plane service, and Prometheus with Grafana. Porting this lab to Kubernetes, Terraform and Vault is designed but not built yet; that work lives under Next phase, not mixed in with what runs.

Architecture

Writes and reads take different paths, and the API never waits on the database. The control plane watches the primary directly, so it judges the node rather than the route to it.

Why the API Never Waits for the Database

The order API doesn't write to MySQL. It validates the order, publishes it to a Kafka topic and returns. A consumer worker writes to the primary through HAProxy. That one decision is what makes a database failover invisible to clients: while the primary is down, the API keeps accepting orders and Kafka holds them; the worker retries every two seconds until writes have somewhere to go, then drains the backlog.

• A traffic spike doesn't hit the database directly. Kafka absorbs the burst and the worker writes at its own pace.
• "Accepted" means queued, not stored. The API returns once the order is handed to the Kafka producer, without waiting for the broker's acknowledgement, so the success response is a promise the worker has to keep. Each write runs in a transaction with rollback on failure.
• The worker survives the outage. Its first version gave up after a fixed number of connection retries and died mid-incident; now it holds the in-flight order and retries the connection every two seconds until the database is reachable again. A write that fails after connecting is still logged and dropped; closing that gap is in the next phase.

GTID Replication, and the Bugs That Taught It

Both MySQL nodes run with GTIDs on (gtid_mode, enforce_gtid_consistency, row-based binlogs, and replica updates logged), and replication uses auto-positioning: the replica asks for every transaction it doesn't have, instead of a binlog file name and offset. That is what makes promotion and failback scriptable. There's no position bookkeeping, and the promoted node already keeps the binlog the old primary needs to catch up from.

What broke on the way

• Replicated admin statements. Creating the replication user on the primary minted a GTID; the replica replayed it, hit a duplicate user (MySQL error 1396) and stopped its SQL thread. Admin statements now run with sql_log_bin = 0, so they never enter the replication stream, and the same rule keeps failback free of errant GTIDs.
• A replica that wasn't empty. The container image auto-created the application user and database on the replica too, so the primary's history collided with it. The replica now starts as an empty shell and gets everything from the primary.
• A schema race. The primary created its tables before the replica connected, and GTID replication only ships what happens after the link exists. The setup script now links the pair first, then loads the schema through the primary.

Durability: asynchronous replication and its RPO

Replication is asynchronous. If the primary dies holding transactions it committed but hadn't shipped, the promoted replica won't have them; that window is this lab's RPO. Gating promotion on measured replication lag, so the window is bounded before the system acts, is part of the next phase.

The Incident Control Plane: Automation With Brakes

Fully autonomous remediation is seductive and dangerous: automation that acts on a false positive can do more damage than the original incident. The control plane is deliberately small, so its behaviour is predictable. It checks the primary directly every two seconds with a two-second connect timeout, and it acts on exactly one failure mode.

• Sustained failure, not a blip. One failed check does nothing; three in a row trigger promotion.
• Act once. After a failover the loop latches and stops acting. It never fails back on its own; failback is a deliberate, human-run script.
• Read-only until promoted. The replica runs read_only, so nothing can write to it by accident before promotion.
• Re-route in memory. HAProxy already lists the replica as a backup server for writes. Failover flips server state through HAProxy's Runtime API: no config edit, no reload, and no DNS record change to wait on.

-- on the replica: stop applying
-- the dead primary's stream
STOP REPLICA;
SET GLOBAL read_only = 0;
SET GLOBAL super_read_only = 0;

# HAProxy Runtime API, no reload
disable server <backend>/primary
enable server <backend>/replica

Failure Drill: Kill the Primary Under Load

A fault-injection script force-kills the primary container (docker kill, no graceful shutdown). This timeline comes from the drill's postmortem:

I also ran the drill in the middle of a k6 spike test. The order API kept answering with success throughout, because it never waits on the database. A provisioned Grafana panel plots HAProxy's per-server UP state for the write backend, so the failover is visible as the primary's line falling while the replica's rises.

SLOs and Error Budget

The lab is designed around one SLO: 99.9% availability for the order API, which leaves an error budget of about 43 minutes a month. The budget is the argument for automating this failover at all. A ~10-second automated recovery spends a sliver of it; the same failure waiting for someone to wake up, diagnose it and promote by hand could spend a large share of the month's budget in one night.

The SLIs are chosen: request success rate, p95 and p99 latency, Kafka consumer lag, replication lag and failover duration. Today Prometheus scrapes HAProxy's built-in exporter and the API's order counter every five seconds. Recording rules, burn-rate alerts and an error-budget dashboard are in the next phase, so I quote no measured availability figures here.

Postmortems

Every serious failure got a written postmortem: what happened, why it happened, and the fix. Three so far:

• GTID replication: from two isolated databases to a clean pipeline, through the replicated-user collisions and the schema race described above.
• Kafka event backbone: an API that reported success while orders sat unread in the topic (there was no consumer yet), and misconfigured advertised listeners that black-holed messages until I simplified Kafka to one internal and one external listener.
• Failover drills and self-healing: a worker that gave up mid-outage, HAProxy ignoring the reload signal because it runs as PID 1 in its container (the fix was the Runtime API), and a failover script that sent its commands to its own loopback interface instead of the HAProxy container.

Next Phase: Designed, Not Built

• Kubernetes on GKE and Proxmox: the same services on a managed cluster and a self-hosted one, with PodDisruptionBudgets so a node drain never takes down every replica of a service.
• Terraform for both environments, so the lab is rebuilt from code rather than by hand.
• HashiCorp Vault dynamic database credentials, replacing the static ones in the Compose environment.
• A stricter control plane: GTID-set, replication-lag and replica-health gates before promotion, with alert routing and a runbook attached to every page.
• A chaos suite: a network partition between primary and replica, losing a Kafka broker, draining a node under load, sealing Vault mid-run.
• At-least-once, end to end: wait for the broker's acknowledgement before answering the client, retry failed writes, and commit Kafka offsets only after the MySQL commit, so neither a crash nor a failed write can drop an order.
• Observability and delivery: SLO recording rules, burn-rate alerts, an error-budget dashboard and CI.

Engineering Takeaways

The most important decision in this lab wasn't a technology choice; it was deciding where automation stops. The control plane acts on one well-understood failure (a dead primary, confirmed three times), acts once, and leaves failback to a person. The bugs that would have hurt most, a worker that gave up mid-outage and a reload signal HAProxy silently ignored, only surfaced when I broke the system on purpose, which is the point of a lab.