Jump to content
RESET Forums (homeservershow.com)

Man accidentally 'deletes his entire company' with one line of bad code


oj88
 Share

Recommended Posts

Server apocalypse!

 

I can almost hear a horde of lawyers banging on his door.

 

 

“I run a small hosting provider with more or less 1535 customers and I use Ansible to automate some operations to be run on all servers,” wrote Marco Marsala. “Last night I accidentally ran, on all servers, a Bash script with a rm -rf {foo}/{bar} with those variables undefined due to a bug in the code above this line.”

 

Mr Marsala confirmed that the code had even deleted all of the backups that he had taken in case of catastrophe. Because the drives that were backing up the computers were mounted to it, the computer managed to wipe all of those, too.

 

“All servers got deleted and the offsite backups too because the remote storage was mounted just before by the same script (that is a backup maintenance script).”

 

 

Source: http://www.independent.co.uk/life-style/gadgets-and-tech/news/man-accidentally-deletes-his-entire-company-with-one-line-of-bad-code-a6984256.html

 

 

Link to comment
Share on other sites

Slow news day for the independent I guess as they're just taking about the server fault thread with no actual communication with the person.

 

The story sounds fake to me and similar to the farfetched stories on dailywtf. The person knows enough to setup a large of servers with Ansible but doesn't have a clue about the dd command and doesn't have proper backups? The only backups he had were a mounted drive or something which he left mounted at the time so they got deleted as well. 1535 customers implies a complex operation with more than just 1 person as well.

Edited by GotNoTime
Link to comment
Share on other sites

I agree that the source is a bit sketchy, but that doesn't mean it can't happen. To me, the take-away here is to test your code on dev environments first before you go out and do it on prod servers.

Link to comment
Share on other sites

Seems that a data center with 1535 customers would be using storage arrays with snapshots. If not, they should have been.

  • Like 2
Link to comment
Share on other sites

Sounds like an actually good backup solution would have saved him. Something not physically connected to the system at the time ....

 

Also, some basic error detection/handling would have helped, as well. 

 

 

As for fake, I would agree if the account was brand new, but the account has been around for a while, and has asked some fairly relavent questions already....

 

Either way, somebody is going out of business.

  • Like 1
Link to comment
Share on other sites

Sounds like an actually good backup solution would have saved him. Something not physically connected to the system at the time ....

 

Also, some basic error detection/handling would have helped, as well. 

 

 

As for fake, I would agree if the account was brand new, but the account has been around for a while, and has asked some fairly relavent questions already....

 

Either way, somebody is going out of business.

Agreed...

Link to comment
Share on other sites

Reminds me of an incident we had here, one of the "Big 4" banks here wiped all their clients, servers, the whole shebang with SCCM one night. I guess someone was looking for a new job after that one

Link to comment
Share on other sites

Seems that a data center with 1535 customers would be using storage arrays with snapshots. If not, they should have been.

 

Our backups are both encrypted volume backups (so others can't open them) but they mount read-only unless explicitly specified, so files cannot be changed or deleted).

 

And I agree with oj88, you never run code in production unless you've tested it on a development setup first.  I'm not a coder, but when I write management scripts, I always have a test box I try them on, before I try them with a small production subset, before I think about deploying larger scale.

It was a marketing hoax. I'm not entirely sure that claiming you were useless enough to nuke all your servers by accident really works for the "any PR is good PR" mantra.

 

I'm thinking "marketing hoax" may be marketing speak for "I'm an idiot, something dumb went down, and in the heat of the moment I said something dumber yet to cover it up".

Edited by LoneWolf
Link to comment
Share on other sites

  • 2 weeks later...

Though I mean I guess this could be possible if you don't have any backup or safety precautions. Linux is one of those OS's where you need to pay attention to what you're doing. 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...