Legacy Data Migration – let data know enough about itself

Overview

Despite the fact that the cloud infrastructure is on the rise and there has been a tremendous improvement in infrastructure and platform in past few years, but still mainframes play the critical role in many enterprises, in fact there is hardly any enterprise which doesn’t have any critical application running on mainframe.

Why Migrate?

Running on legacy applications & technology let business continue, but not grow. As a result it is hyper critical for enterprises to migrate from legacy applications to latest technologies, which enables the following benefits.

(A) Business Edge: Ready for a change, business keeps changing or has to bring a change to keep an edge in the market. Having platform & technology flexible enough to make a change is very important for the growth of an organization.

(B) Low Maintenance: Legacy comes with high maintenance cost, and require people with overarching knowledge on various aspects. Migrating to latest technologies will eventually reduce the maintenance cost.

(C) Reduced Risk: Legacy often involves high risk when it comes to change, maintenance and upgrades.

(D) Resource Availability: It is often difficult to find people who are knowledgable in legacy, hence the latest established platforms often doesn’t have this limitation.

Typical Approach

Considering the benefits… every organization wish to migrate to latest platforms, but it comes with several challenges, and most critical challenge is in understanding and migrating the legacy data.

Often times the teams adopt the ETL tools for extracting, transforming and loading the data from source to destinations, and during this process the migration logic or business rules are buried inside the transformation logic, as a result every time they discover an issue in the migration they revisit the code and make required modifications, hence the entire migration goes through several iterations and it consumes lots of time & resources, particularly when you revisit the logic for a change or to understand how you implemented it. And if not that, then at least all the migration steps has to be re executed for every new small change.

What is Missing?

It’s quite typical to use IF’s, Cursors and several layers of Views to build the transformation logic in order to generate the data in final format which is acceptable by destination. Also, there are several GUI workflows available, which helps you sketch the entire flow of the data.

The critical part which is missing is that the REASONING behind the source-logic of specific data elements in the final output. i.e. you need to revisit the original logic or workflows to understand …

(A) why some data is different from other

(B) Why some data is missing for few records, but it is present for others

(C) Why not the same logic is applied to all records and much more…

In short, the data doesn’t know much about itself, as a result you always need to revisit the source/transformation logic.

Recommendation

“Linear Traceable Derivation” is the answer to all the questions, it not only ease the whole migration afford, but also help business to find an opportunity to use the legacy data differently and more effectively.

Nisum Inc is pioneered in legacy applications migration, over a period of 14 years we learned several hard lessons, and the most important is to use right tool at the right time, which implies that one should know the tools first. We are thorough in tools like Black Box Migration, Characterization Testing and Linear Traceable Derivations which not only ensures the successful migrations but also ensures the timely delivery, as often times the legacy application migration takes forever, but Nisum know how to come over it with our great experience in such migrations.

Nisum Inc helped enterprises migrate large-scale financial, retail & commerce applications, which were running on mainframes using Cobol and several other legacy technologies.

Continue reading to learn more about Linear Traceable Derivation.

Linear Traceable Derivation

This approach is built on following principles, there are detail examples on each of this principle to understand it in detail. Not all of these principles may applicable to your migration, you may choose and apply to see what works for you, the examples listed below are just for your understanding, but you don’t have to apply them in the exact manner; based on the technology and tools you are using, you can apply it differently.

The most important aspect to understand here is that “Let the data know enough about itself”

Linear: A linear logic should be adopted for all derivations and content manipulations

Characteristics: Content should contain the metadata which reflects its characteristics

Derivations: All derivations and calculations should be broken into multiple steps, and should be captured at the lowest possible denominator

Traceable: The derivations and transformations should be traceable.

Hints: All transformations should capture hints, which enables the quick understanding without analyzing detail data.

Exception Indicators: The content should contain exception indicators, these enables the record filtration when resolving or fixing exceptions.

Examples