What does a modern enterprise DBA do? In today's challenging multi-DBMS environments, the answer is "just about everything." The job description apparently also includes keeping the trains running on time. Well, that's the theory at least. This expensive blooper was submitted by Oracle DBA Scott Benitez, who reports that he did indeed find another job after derailing the entire American Midwest.
"About five years ago, I was working for one of the major railroads in the Midwest. A fellow Oracle DBA and I were testing out procedures to move about 12Gb of data from several tables into a consolidated table for reporting purposes. Part of this procedure was to pull over massive amounts of historical data, in addition to real-time data about where all the trains were.
"Unfortunately, I thought it was easier to truncate all the tables via a SQL script that I had stored on my PC when there was an error and load from an export file containing a test subset.
"During the course of our testing, I had to connect to the production database to get some info. And because I was already in a SQL*Plus session and connected from there, the login script that sets my SQL prompt to the correct instance name was never run. Well, we continued and hit an error, so I pulled one of my many sessions from the task bar and proceeded to run the truncate script. This ran for about five minutes before I received a call saying users couldn't find any data. We had unknowingly run it from the session connected to production.
"Because we deleted the real-time data containing train locations, we had to stop the trains -- all of them! -- to re-determine where they were. In addition, we weren't able to bill customers for contractual way-points. We figured the losses in the hundreds of thousands of dollars.
A surprise came, though, while I was updating my resume to start looking for a new job. After I had every major orifice redefined by the CIO, CEO, and President (my standard answer being, "Yes Sir, I realize I'm an idiot"), I was put in charge of the team to determine new security procedures to ensure this never happened again!"
Lessons learned: There is a real difference between test and production environments--keep track! (No pun intended...really.)
Have your own tale of woe to share? Submit your backup/recovery snafus, tuning disasters and ugly upgrades. Stories of good intentions gone bad, over-ambitious and under-trained newbies, clueless consultants, and even more clueless managers will all be accepted. The submitter of the most amusing or wince-inducing blooper of the month will receive a free copy of Craig Mullins' new book Database administration: The complete guide to practices and procedures. Send your bloopers to us today!