Inadvertently deleting important data is surprisingly easy to do. A simple typo and a misplaced double click was all it took for these two DBAs to wipe out entire directories of critical files. They recount their horror in this installment of SearchDatabase.com's series of true DBA bloopers.
Submitted by Tom Vannoy
I had been at my position for about three months when this happened. I had done some really good things in the first couple of months because I was the first on-staff DBA and I was able to take advantage of that to make myself look good.
This day, I was taking the steps necessary to multiplex the redo logs and distribute the groups across different disks for better recovery purposes, which I completed successfully.
Next, I took the old logs offline, and went to delete the old redo logs, which existed in the directory containing all of the data files. I was on Unix and issued the rm command for the logs.
At the last second, I realized that I had an extra space between the rm and the file name, but I had already pressed enter. Immediately thereafter, I sat in horror as I watched all of the data files get deleted.
The saving grace was that it was a Friday at lunch, payroll was already out the door, and only four users were on the system at the time. The little bit of good that came out of it was that I was able to substantiate to them the value of placing the database in archive log mode, which I had done my first week on the job.
The double click
Submitted by an anonymous DBA in Connecticut
I use Oracle's Enterprise Manager to manage my company's Oracle databases. For every database in our company, we have an exact test database where our developers develop and test their code.
Enterprise Manager lists all the databases below one another on the left-hand side panel and from there, you can navigate and do daily administration work. You can store the passwords, so all you have to do to work on a database is double click on it, and it is easy to confuse which one you are on.
One fine day I thought that I was working on my test system, and for a particular set, I had to delete a few schemas and all its objects. So I started dropping schemas one by one until I came across one that wouldn't drop because it was connected. I wondered who in the world would be connected to the test system other than me because I still hadn't even granted access to the developers, then to my horror, I realized that it was in the live production system that I dropped the schemas and all the data in them.
I then spent a couple of hours to do a tablespace point-in-time recovery. Then I faced the task of telling my manager. Guess what: I wasn't fired and I'm still working at the same company!
For more true DBA bloopers, click:
Have your own tale of woe to share? Submit your backup/recovery snafus, tuning disasters and ugly upgrades. Stories of good intentions gone bad, over-ambitious and under-trained newbies, clueless consultants, and even more clueless managers will all be accepted. The submitter of the most amusing or wince-inducing blooper of the month will receive a free copy of Craig Mullins' new book Database administration: The complete guide to practices and procedures. Send your bloopers to us today!