Data warehousing failure is in the eye of the beholder

Douglas Hackney, president of Enterprise Group Ltd., is quoted as saying "70% of all data warehouses are failures". While I think that is quite high (astronomical from my experience), one must look at the basis of what constitutes a data warehouse failure. Since Douglas has worked on many data warehouses, one could only dream about how many of them are failures.

Failures can be defined in many ways with different degrees of failure. Failure to me would mean that the data warehouse added NO value at all, or worse, lost value for the company. It would be pretty hard (although not impossible) to have this degree of failure. On the other hand, one could say that the failure of the data warehouse is where the original calculation of return of investment (ROI) was 15.03% and in fact we only calculated a dismal 14.92% ROI after the first year. To me, this is successful, just not as successful as we planned.

There are many reasons to think that data warehouses are not as successful as we would like them to be. These reasons include, but are not limited to (and are in no real order):

1. Not having Senior Management Support - It is important that the senior management understands the purpose, mission, and value of the data warehouse. When non-technical issues that are difficult to resolve come into view, strong senior management support can help solve those issues.
2. Selecting tools before understanding the problem - It has been my experience that companies want to purchase tools before understanding the problem. This is only natural. I would be an exceedingly rich consultant if I had a dollar every time someone asked me what is the best tool before considering specific problems (or requirements).
3. Knowing you have a data quality problem and not "cleaning" it before moving it into the data warehouse - I am always surprised how many states are in the US (I have seen 51, 56, 59, 65, 90, 113, et. al.). In the data warehouse, we should be precise with a column for state_province and then a separate column for country (at least until the whole world is conquered by the US - ;^} (he says with a fiendish grin!)).
4. Having IT pick the end-user tools that will solve business users' problems. While we can aid in the selection process, the end users need to be intimately involved in the selection process.
5. Having stove pipes of data and being unable to get a "feed" from them on a timely basis - While you organization may have easy access to all the source systems to feed the data warehouse, others do not. Having senior management support can solve this issue.
6. Building the data warehouse without talking to the business users - One of my customers' project manager said that this was one of his proudest accomplishments!
7. Not understanding that the cost/benefit of a data warehouse will occur over time, and not necessarily in the 1st quarter.
8. Designing the complete data model of the enterprise data warehouse before building a single data warehouse subject area - we should learn from the past. We tried this during the early years of data warehousing, not matter what Bill Inmon tried (and is continuing) to tell us!
9. Building stovepipe data marts instead of a data warehouse (unless the concept of conformed dimensions is enforced completely!) - we need to understand that all of these things need to tie together. Without this, we are just building stovepipes of our own.
10. Over-promising what a data warehouse will deliver - No, it will not make better decision-makers. If yours are bad, it will not make them better. It will give them better data to continue to make bad decisions with!).
11. Believing that you can put middleware between the end user tool and the operational systems and that will solve the data-warehousing problem - OK, if you have a real small single source system, then maybe, just maybe, it will work.
12. You start out with a small manageable subject area and allow it to grow wildly without increased funding, time for delivery, and/or people - this is commonly known as "scope creep" and should be considered a major risk factor.

There are many reasons, most of them non-technical, for a data warehouse to succeed less than planned. Having a well defined methodology will help during the implementation, but an organization must understand that the data warehouse will change the way the company does business, and the users will need to change their habits (work flow) to use the data warehouse to its fullest extent. Without this change, I must concede the data warehouse may fail.

Can your data warehouse fail, but still produce a positive ROI? That will depend on your definition of failure. If the promise of the data warehouse is to increase the profitability by 1000%, and you only increased it by 100%, is that a failure with a positive ROI? What is your definition of failure?

About the Author:

Chuck Kelley is president and founder of Excellence In Data, Inc. and an internationally known expert in database technology. He has more than 20 years of experience in designing and implementing operational/production systems and data warehouses. Kelley has worked in some facet of the design and implementation phase of more than 35 data warehouses and data marts. He also teaches seminars, co-authored a book with W. H. Inmon on data warehousing and has been published in many trade magazines on database technology, data warehousing and enterprise data strategies. Please feel free to email him at chuckkelley@usa.net with comments (negative or positive) about this column or ideas for future columns.