According to a Gartner/Standish group study, 67% of data migration projects suffer from implementation delays. Given the importance of data in the modern enterprise, flawed data migration can have severe repercussions.
A nuanced understanding of the various challenges in Data migration is required to mitigate many of the risks associated with this activity.
Importance of Data Quality
As with most data integration efforts, data quality is one of the biggest challenges in data migration and some of the challenges are:
- Given the non-recurring nature of most data migrations, handling data quality issues in a live post-migration environment can be very challenging.
- Poor data quality imposes significant costs post-migration, with issues ranging from poor business intelligence to delay/disruption in business processes.
- Data quality issues are amplified when migration happens from a legacy system (with poor data quality) to a newer application with a far richer feature set and a stricter data model. This necessitates a lot of planning before the migration process can commence.
Solutions to manage Data Migration effectively
- To ensure good data quality coverage, complete profiling of the source data systems must be conducted. This process must be complemented with the reconciliation of business rules across the source and target systems. Both these activities, when completed with the involvement of all relevant stakeholders will provide the data migration team with a lot of insights into data/rule gaps between the source & target systems and help create data validation/transformation rules to be used in migration. These rules can be further fine-tuned by accounting for deduplication and consolidation if necessary (For instance, when multiple source systems are involved). If the target system is still evolving/being built, strong change management processes need to be put in place to ensure the data migration process keeps up with application changes.
- Choosing the right technology/tool stack is one of the biggest decisions that awaits the migration team. The choices range from a custom-built solution, data integration tools to a hybrid solution. While there are merits and drawbacks to each choice, the flexibility and comprehensive data integration capabilities offered by modern ETL tools make them a compelling choice. Many of these tools offer integrated development environments that speed up the development process and provide plenty of customization capabilities through scripting/reusability. The data migration solution offered by Tavant uses Talend, an open-source tool that provides a robust data integration toolkit and Java/Perl based scripting allowing for significant customization. Other technology challenges include differences between the underlying source/target database systems (mismatches in terms of data types supported, date/time format mismatches etc), encrypted data, and handling of character encoding and numeric precision.
- A good data quality management process and technology solution need to be backed up with a good operational process that can support an iterative data migration validation/ implementation process. Other operational challenges that need to be factored in are the migration of hot data (active business transactions) and the creation of failback provisions. A robust data testing strategy is required to not only ensure that all data within scope is migrated but also that the migrated data is functionally usable on the target system(s).
- Finally, an experienced data migration team can drastically cut down on the learning curve and hit the ground running.
While each data migration project has its own dynamic, a good understanding of challenges and best practices can significantly reduce the chances of running into common data migration roadblocks. Recognition of these challenges will also ensure that the data migration process receives the support it requires.
