Whoops. A SysAdmin at a high profile Silicon Valley company went to tidy up their database—and accidentally deleted the directory instead.
The employee in question belonged to $32.9 Million start-up GitLab—a virtual workplace for programmers to merge individual projects, similar to GitHub and Atlassian.
The problem started when spammers were hammering the database. This lead to load time and stability problems, forcing an outage.
To fix the problem, the SysAdmin went to the backup database to tidy and use it to fix the problem. Except they accidentally typed the command to delete the directory.
By the time he cancelled the rm -rf command, just 4.5GB remained of their 300GB directory.
The GitLab Incident: Major Backup Strategy Fail
Following the deletion, GitLab immediately took their site down for emergency maintenance. But in a day where everything was already going badly, things didn’t stop there.
When the company went to use it’s five deployed backup/replication techniques, it said “none (were) working reliably or set up in the first place.”
Public opinion on the incident ranged from politely calling GitLab’s backup strategy “sub optimal” to “The world doesn’t contain enough faces and palms to even begin to offer a reaction to that (Gitlab’s) sentence.” The Register called the incident a “Meltdown.”
You do have to wonder about the job security of those in charge of the backups, who not only deleted the backup database, but also failed to either implement or properly implement the five backup strategies decided upon by management.
Luckily for GitLab, they were able to find a backup to restore to from six hours before—however, all the data since then was lost. Thankfully, the data they permanently lost on the database was only comments and bug reports.
GitLab is a code collaboration platform with 100 employees, most of whom work remotely. GitLab attracts investors by having an open source twist on the code repository model, allowing businesses to build on top of their platform.
Don’t Make GitLab’s Mistake!
Make sure your business has a well tested backup plan in place. A good backup scheme can make the difference between losing all your business data, just a few hours, or none at all. Use a 3-2-1 backup scheme to make sure you have your backups stored safely offsite.