Mastering Data Management

Most data warehouses have been implemented using an overly cumbersome and brittle architectural approach, relying on too many tightly-coupled parts and dubious data.

Most data warehouses have been implemented using an overly cumbersome and brittle architectural approach, relying on too many tightly-coupled parts which need to operate on ever-increasing quantities of data of dubious quality.

These ecosystems are incredibly expensive and difficult to change, contributing to the fact that data warehouses in general have failed to deliver on their promises.

In this series of blogs I will discuss the issues with existing data warehouse architectures in the context of today’s business requirements. I will propose that, by adopting services-oriented architecture (SOA) and MDM, an organization can significantly simplify and improve the ability of its data warehouse ecosystem to deliver actionable business intelligence into the hands of decision makers.

So let’s start w/ an overview of the most common business problems we see.  For those of you who’ve read my work, heard me speak, etc. these are common issues.  But it’s critical to set the context of where I’m coming from.

Organizations today are under increased pressure to make better, quicker, less risky decisions than ever before. Decisions that used to take weeks or months are being made in second, minutes, or hours.

Companies are under increased pressures externally, as markets, customers, regulatory and compliance bodies, and competitors push them to deliver more customized solutions in record time, and with less risk. They are also under increased internal pressures to reduce cycle times and costs, and increase predictability, accuracy, transparency, security, and quality.

This is all happening while the number of systems, applications, and services are increasing, not decreasing.  To exacerbate this further, data volumes are growing at an extraordinary pace, and companies are relying on both unstructured as well as structured data – from within their firewalls as well as from within new clouds.

To meet these accelerating needs, data warehouse ecosystems are being pushed beyond their limits to achieve new levels of scalability, availability, reliability, agility, and integrity. However, these increasing demands cannot be supported economically by simply adding more hardware, and network and storage capacity.

Instead organizations need to rethink all aspects of their business intelligence architectures to enable them to provide new levels of integration and responsiveness.

This series of blogs will attempt to propose first a set of issues/problems with current DW architectures.

Then, I’ll propose how SOA and MDM can help transform a data warehouse ecosystem.

Finally, I’ll propose some steps to get there.

There are lots of smart, credible people out there who can help extend this thinking and work – I invite your feedback!