GitLab — rm -rf'd the production database

Casualty stat: 6 hours data lost · 5 of 5 backup methods failed

An on-call engineer trying to fix replication accidentally ran the wipe command on the primary DB instead of the replica. Then came the discovery: none of the five backup mechanisms actually worked. The team live-streamed the recovery on YouTube while restoring from a 6-hour-old snapshot found by accident.