Approaching Operational Data Integration

by Olga Belokurskaya and Alex KhizhniakAugust 8, 2009

Traditionally, data integration is mostly used for data warehousing and BI. However, it is not limited to these business-critical areas.

(Featured image credit: IBM)

Table of Contents

What is operational data integration?

Operational data integration (OpDI) is one of the fastest-growing practices in data integration, as TDWI Research revealed in 2007. In brief, OpDI stands for “implementations, projects, or initiatives commonly described as the consolidation, collocation, migration, upgrade, or synchronization of operational databases,” according to Philip Russom of TDWI.

Philip Russom

In other words, OpDI implies data integration between operational (nonanalytical) applications, “whether in one enterprise or across multiple ones.”

Today, as most organizations make efforts to implement BI and data warehousing initiatives enterprise-wide, there is a growing demand for different integration techniques, including EAI (enterprise application integration), EII (enterprise information integration), ETL (extract, transform, and load), data federation, etc.

“All the techniques and tool types under the broad rubric of data integration operate similarly in that they copy data from a source, merge data coming from multiple sources, and alter the resulting data model to fit the target system that data will be loaded into. Because of the similar operations, industrious users can apply just about any technique (or combination of these) to any data integration implementation, initiative, project, or practice—including those for OpDI.” —Philip Russom, TDWI

The major effects of operational data integration (image credit: TDWI)

Operational data integration is needed to address several challenges data-driven companies face.

Why is it important and hard?

Redundant data in multiple nonstandardized databases across different departments is a serious issue, which leads to increased IT costs and hinders unified visibility into a company’s business processes. A lot of nonstandardized applications throughout an enterprise may also provide redundant data. To fix this, one needs to standardize data and initiate database consolidation, migration (if necessary), and so on.

“But beware, because migration projects are intrusive—even fatal—in that they kill off older systems after their data has been moved to another database platform.” —Philip Russom, TDWI

However, it’s not that straightforward. Very often, you cannot get rid of a legacy system, or it’s not easy. According to Philip, “sometimes it’s best to avoid the risk, cost, and disruption of data migration and leave redundant applications and databases in place.” In this case, an organization may need to implement synchronization workflows and tools. Yet this initiative may face its own challenges, too.

“When each database involved in synchronization is subject to frequent inserts and updates, it’s inevitable that some data values will conflict when multiple systems are compared. For this reason, synchronization technology must include rules for resolving conflicting data values.” —Philip Russom, TDWI

When put in place, according to Philip, data synchronization becomes “a permanent piece of infrastructure that runs daily for years before reaching the end of its useful life.”

As enterprises are heading for more agility and real-time data integration to increase efficiency and accelerate business responsiveness, the demand for operational data integration is only growing. At the same time, Philip notes, they still lack features needed for data quality, MDM, scalability, etc.

Operational versus analytical data integration usage (image credit: TDWI)

“OpDI solutions are in serious need of improvement or replacement. Many are hand-coded legacies that need to be replaced by modern solutions built atop vendor tools.” —Philip Russom, TDWI

For more, see a recent TDWI’s article by Philip covering the main practice areas in OpDI—migration, sync, and B2B data exchange—as well as common pitfalls and best practices involved.