Making an ETL of Your Own—Wise or Not?

by Alena SemeshkoJune 25, 2008

ETL tools. Develop one of your own or buy one from a trusted vendor? What’s best for your company?

Table of Contents

The ETL–migration puzzle

Before jumping into anything or rushing to a popular software vendor, many integration experts analyze the degree to which an out-of-the-box ETL tool will benefit their company or whether it might be wiser to build a custom product from scratch. At the same time, some people confuse ETL with data migration.

John Morris

John Morris’s latest post, where he highlights the difference between data migration and ETL is, in fact, very much to the point. ETL is a much larger concept that needs more effort and time than data migration. Data migration, if properly defined, is a one-way long-term trip of data from one system to another. While ETL not only can be a repeated procedure, but is also much larger in scope, as it includes all sorts of data transformations (from data cleansing and data quality checking to certain elements of business intelligence).

So, if you look closely, the two are considerably different, not even parts of the same process. (Still, the 4 rules of data migration coathored by John are relevant to ETL, as well.)

A homegrown tool vs. proprietary product

Building a custom solution of your own implies hiring technical staff for that purpose, but you have a greater chance of your final solution to be simple and match just your business needs.

Purchasing an ETL tool you are more limited by the market offerings in terms of customization. To add to that, you are likely to face the complexity of educating your in-house staff, which might take a load of your company’s time and resources.

Still, according to the data below, most enterprises resort to building their own solutions. How come? Is it right or wrong? Well, it depends.

The market share for proprietary tools vs. hand-coding (image credit)

Either option considered, try asking yourself the following questions and you might actually come to your own decision.

1. What are your goals, why do you need an ETL solution? Is this a one-time procedure with limited conditions or are you planning to make it a part of your organization’s structure and strategy? What do you want from your ETL tool? Specify priorities.
2. How much can you spend on your solution, what is the budget?
3. How many data sources are you working with, and what kind are they? What functionality do they already have that might be helpful at the extraction/transformation.
4. How much time can you painlessly allow for the transformation process?, for your entire ETL process?
5. How much human resources and time can you dedicate to this project? (don’t forget about education)
6. If you decide to build your own solution, who is going to educate your staff? Are you competent enough in the processes and concepts (such as ETL, data warehousing, etc.)?
7. Just how many ETL experts do you have in your company and how do you estimate their potential and skill? Are they replaceable?

After reading these 7 for starters, check out this opinion about hand-coding or learn about other things to keep in mind when choosing an ETL tool.

Alena Semeshko is Technology Evangelist at Apatar / Altoros. On her blog, she covers ETL, EAI, BI, open-source, and all the aspects of data integration. Alena is focused on creating awareness, fostering a better understanding, and keeping her readers well-informed on the emerging problems and data integration practices of the day. She possesses rich intercultural experience, having lived in several countries across the globe, and is studying Korean in her spare time.