I've heard a lot lately about how data warehouses are not meeting business needs. Sometimes the criticism comes from vendors or consultants who have something to gain by convincing CIOs that their data centers are draining their budgets without fulfilling their potential. But many respectable business managers also echo the sentiment, saying that data warehouses need a drastic overhaul. These voices of discontent claim that data warehouses take too long to build, cost too much and don't adapt quickly enough to meet changing business needs.
Even though I work for The Data Warehousing Institute, I'll admit that there is some truth to the criticism. Data warehousing and business intelligence (BI) professionals could do a better job of delivering cost-effective enhancements more quickly and ensuring adequate query performance.
But that doesn't mean we should dismantle or outsource our data warehouses. Nor should we allow them to suffer from neglect while we pursue supposed better alternatives, as suggested by some. We ask a lot of our data warehouses. They have the nearly impossible task of constantly adapting whenever our business needs change. In fact, the primary reason that data warehouses are so difficult to build is that they support businesses that are often volatile, disorganized and fragmented.
There is one thing about data warehouses that many people won't tell you: We wouldn't need data warehouses if executives ran their organizations
Unfortunately, most executives lost control of their organizations years ago and now spend most of their time -- whether consciously or not -- searching for ways to reassert order and control. Data warehouses are simply one tool to bring cohesiveness to organizations that have fragmented into dozens or hundreds or thousands of disparate business units, each with their own strategies, policies, processes, procedures, systems, applications and data. And data warehouses can't work their magic unless the business achieves some degree of stability.
Enter the CIO
It's the job of the CEO to bring order to the chaos that mergers, acquisitions, competitors and corporate growth can cause. If the CEO exhibits true leadership and provides the business with a clear, coherent strategy and standard rules governing those processes, then there is almost no need for a data warehouse.
In a well-run organization, you can simply point a BI tool at the company's operational systems to obtain most of the information workers might need to do their jobs. The data warehouse would just be another source, one that contains all the historical data that the operational systems don't have room to store.
Consider this: Rohm and Haas, a specialty chemicals manufacturer, decided to replace a hodgepodge of operational systems with a single instance of SAP R/3 worldwide. The implementation was painful because it forced the company to rethink and reintegrate its processes, terminology and rules. But once it reintegrated the company using its operational systems as a lever, it no longer needed its "legacy" data warehouse, which it replaced with SAP's Business Warehouse and BI tools.
"Once the business integrated its operational processes and created a single instance of data across the entire organization, data warehousing became infinitely easier and our data quality improved substantially,'' said Michael Masciandaro, director of business intelligence at Rohm and Haas.
"We now focus less on transforming data to achieve a single version of truth than on helping our users derive real business value from the information."
Putting Humpty Dumpty back together again
Most companies are like Humpty Dumpty -- broken into dozens of pieces. And BI teams are "all the king's horses and all the king's men," who are charged with putting Humpty Dumpty back together again. The king's horses and men didn't cause Humpty Dumpty to fall, so why should they be blamed if they can't glue him back together again? Perhaps poor Humpty Dumpty is cracked beyond repair and even the best and brightest of the king's men can't rehabilitate him.
The moral of the story is this: Don't blame the BI team if it can't deliver a successful data warehouse. Blame the CEO. The BI team is merely a messenger, and the process of creating a data warehouse is the message. A data warehouse reflects the organization. The more fractured and disintegrated the organization, the harder it is to create a robust, highly functional data warehouse.
The good news is that unlike our storybook metaphor, many BI teams do succeed in gluing their companies back together, at least until the next merger, acquisition or other organizational upheaval blasts everything apart. But while it lasts, these unsung heroes should be congratulated and honored, not outsourced, off-shored or reassigned to the IT-equivalent of Siberia.
This column originally appeared on SearchCIO.com.
Wayne W. Eckerson is the director of research and services for The Data Warehousing Institute, a worldwide association of business intelligence and data warehousing professionals that provides education, training, certification and research. He is also the author of Performance Dashboards: Measuring, Monitoring, and Managing Your Business, which will be published in October 2005. He can be reached at email@example.com.