Restoring production in 10 seconds

Most backup systems report one thing: the job exited zero. That tells you a program finished. It doesn’t tell you whether you can get your business back.

The Sin fleet runs on one rule instead: a backup is not a backup until it has been restored. File count, byte count, hashes, a real extraction and the time it took. An exit code proves nothing. This is the first time we held production to that rule.

The setup

Production runs under a strict read-only mandate: the fleet may look, but it may not write a byte. That rules out the usual shortcuts. Nothing can be installed there, and the vault can’t reach in to pull a copy.

So the backup is streamed out instead, as a single archive, and snapshotted on arrival: encrypted, deduplicated, and measured on the way in. Afterwards every service on production was still running from before the backup started. Nothing was touched.

The drill

The snapshot was restored onto a separate, expendable machine that had never held production data, then checked against the live originals.

• 2.365 GiB restored in 10 seconds.
• 25 files hashed against live production: 0 mismatches.
• Every database opened and queried, not just started.
• A second copy on separate hardware, verified clean.
• The drill machine wiped afterwards.

Trap one: the empty database that looks fine

Point a database at the wrong level of its storage and it doesn’t fail. It quietly starts a brand-new, empty one. Everything comes up, every health check is green, and there’s no data.

A backup report would never catch that. We found it only because the drill asked the database a real question.

Trap two: one runbook, two versions

Two databases that looked alike turned out to be different major versions. A runbook that assumes one version restores one of them and fails the other. At 3 a.m., under pressure, that’s how you lose a day.

The runbook now reads the version from the data itself before it restores anything.

The lie in a green receipt

A receipt that only lists what went right will hide what it forgot. One of ours did, about cleaning up after itself. The fix went into the script, not a sticky note: the drill now proves its own cleanup and prints that line too.

What I take from it

Every failure here looked like success from the outside. The only defense is to finish the job: restore it, open it, count it, hash it, then write down exactly what you proved and nothing more.

That’s the standard I hold my own systems to.