When the call went out for DBA bloopers, Christopher James Thompson sent a whopping five memorable blunders. Is he more accident prone that most DBAs? We doubt it. Thompson is probably just more honest. Here Thompson recounts two of his favorite bloopers for SearchDatabase.com readers.
"I was working for an Australian government body that oversees important financial activities. I was a contractor, an Oracle DBA. Another person, Rowena, worked as a liaison to various bodies within the agency as the Unix representative.
We had installed a Sun box with lots of hard disk space, RAID capabilities and hotswap standby disks.
One day Rowena asked me to help her with a problem. Someone had taken over a disk and repartitioned it, losing some valuable data. Rowena had been investigating for a while, and because there was no other explanation, the fingers were pointed at the two contractors, including me!
I checked the logs and noticed that one of the normal RAID drives had gone AWOL. That meant that the designated hotswap drive would have been brought online. The drive would have been repartitioned and used, except in this case there were processes hanging off the drive that could not be released. The hotswap service had failed but the drive had been repartitioned.
The Unix manager in charge was furious because someone had played with his disc!
The moral: You can do everything right -- and still get blamed!"
Now for Thompson's second act:
"I was working at a university in Sydney, Australia as the Senior Unix Admin/Operations Manager.
I worked closely with an Oracle administrator, Dominic. He came from a VMS background and appreciated the help I gave him with Unix commands.
I had asked Dominic to create a routine that ensured database data was exported out of the database to plain-text formats. I wanted it to be suitable for import into any Oracle database on any platform. If the file sizes dropped for any reason, I would ask: "Did you do a clean up recently?"
I also asked Dominic to import the database text files at least once a month. This would ensure that the data could be imported and allowed Dominic to have a refreshed copy to work with. One other goal was to prove that the data could be imported into a smaller area on the disk set. This involved a fair bit of work for Dominic but it was well worth it because we were planning for any possible disaster. We wanted to be prepared if one of the drives died and we still had to make the payroll deadline.
We performed an upgrade of Oracle 7 at one stage, and a week later Dominic had a problem importing the text data. We couldn't figure out why.
We finally did. There had been a new keyword introduced: "CONSTRAINT". It was used in the export routines but someone at Oracle had failed to cater for encountering the keyword during import.
So Dominic sent off an urgent call to Oracle. Within a week we had a response from a sheepish SE at Oracle. After a small patch, we were finally able to import data. But for a while, we were exposed. Believe me, I had my fingers crossed the entire time.
Who says that the vendor knows everything?"
For more true DBA bloopers, click:
Have your own tale of woe to share? Submit your backup/recovery snafus, tuning disasters and ugly upgrades. Stories of good intentions gone bad, over-ambitious and under-trained newbies, clueless consultants, and even more clueless managers will all be accepted. The submitter of the most amusing or wince-inducing blooper of the month will receive a free copy of Craig Mullins' new book Database administration: The complete guide to practices and procedures. Send your bloopers to us today!