Who, me? What? Monday again? Didn’t we just do one of those last week? Oh well, if we must, dear reader, we must. Welcome once again to Who, Me? in which Reg readers regale us with reports of righteous wrongness.
This week, meet “Leopoldo” who some decades ago landed a stint as a database admin with a certain unspecified national airline. This was his first job as a DBA, so naturally good old Leo did not expect to be handed any serious responsibility – a nice junior position, thanks, and opportunities to learn on the job.
He was assigned to shadow the Senior DBA for a couple of weeks, which seemed fine. Then he was told that at the end of the introductory period he would be taking over the Senior DBA’s duties, as he was leaving the airline. Leo did the career equivalent of making sure he could find the barf bag.
Day one went OK. He went around watching what the Senior did and making notes. The database, which held millions of records, kept track of every reservation, every flight, every passenger’s details – the lot.
Naturally there were two instances of the database: the production instance, on which all the reservations, cancellations, changes etc. were actually made for this global airline; and the dev instance, in which any back-end changes were made, along with maintenance and pruning of unnecessary records.
All good so far.
On day two, the Senior decided that Leopoldo should “reverse shadow” him. That meant he would follow Leo around and tell him what to do, and then watch as he did it.
So the Senior DBA told Leo to open a database, define a set of records, and delete. Leo proceeded to bash in the requisite commands, hit Enter, and wait.
Feeling turbulence increasing, Leo wondered aloud whether it should be taking this long. The Senior replied that no, it should be pretty quick to make a change like that on the dev database. “Wait,” he asked, rather later than he ought, “what instance were you in?”
A review of the last commands typed in confirmed their worst fears: Leopoldo had just dumped a large chunk of the production database.
Right on cue the phones started ringing. In every terminal in every airport around the world, the airline was suddenly unable to create, check or cancel tickets. The manager stormed in, demanding to know what was wrong and insisting that full operation be restored immediately.
And that was when the full horror emerged: the backup server, one of the departing Senior DBA’s responsibilities, hadn’t been operational for weeks.
Thankfully there were some more recent exports of the database around, and with 12 hours of diligent work it was possible to return the airline to something approaching full operation. But a major embarrassment.
Don’t cry for Leopoldo, though. When he returned to work the next day – fully expecting his stint with the airline to be over before it had begun – the manager told him that he understood it wasn’t his fault. He blamed the Senior DBA for poor supervision – not to mention gross dereliction of the backups – and told Leo to get on with the job.
Ever found yourself in Leo’s position, doing the wrong thing because someone higher up the chain was a numbskull? Tell us all about it in an email to Who, Me? and we’ll anonymously make you famous. ®