To import or not to import
In today’s society, they say gold is the most valuable commodity, but I have to disagree. The most valuable commodity undoubtedly is “DATA.” It’s treated like the proverbial “Dragon Horde,” ever increasing, while not truly ever knowing what exactly is possessed. It doesn’t matter, it's mine I have it and I’m not going to get rid of it. That’s all well and good but there comes a time in every company’s evolution where they have to embrace the fact that new technology exists and making the step into the future requires a change. This change means leaving an antiquated piece of software behind and a totally new software implementation is on the horizon. This one sentence strikes fear in the hearts and souls of many a system administrators, HR directors, financial directors etc. Pulse racing, sweat forming on brow; is this what a panic attack really feels like? These are the thoughts and emotions of everyone at the initial implementation meeting. Then reality sets in and the natural instinct of “fight or flight kick in”. You have the groups that instantly start protecting their section of the software and the people who want to delay the meeting to “get more information so they can have all the facts”. Inevitably the day comes where the new software consultants meet to discuss the data importation. What are you needing to keep and what is ok to get rid of? The first response, almost ingrained into the very fabric of a person’s being, is “I need all of it, all 30 years worth!” The question is asked by the implementation manager is “Why do you need all 30 years worth?” That is when the fun part begins. Importing data can be a very time and resource consuming endeavor. Both parties have to be open to change and compromise. While you don’t want to lose your valuable data, actually utilizing the information in a newer system may not be efficient or even necessary. You have to weigh the costs vs. the benefits. The costs of importing large amounts of data can easily make what seems like a simple thing “Just put the data that’s now in box “A” into box “B” a monumental task that’s relative to “putting a man on the moon”. Let me tell you right now, “Data importing is almost never a simple task”. There are lots of caveats that come into play that can wreak havoc not only with your budget but with your sanity. Does your current software have data export capability? If so what are the export options available. What are the data import capabilities of the new software? Are the export/import capabilities compatible? What are the legal requirements of keeping old data? What will the performance cost be for importing 30 years of data? Is the current data acceptable to include in a new system? Has there been replication created in the current data set? Do I have personnel capable of doing the import in house or will I need a consultant to handle the import? Benefits of importing all the data is easy access for statistical analysis and report generation. With all your data in one place, it requires less manual work in combining the information to get more information over a longer period of time. So “With great power comes great responsibility” (Thanks Stan Lee) and that has never been more true than for the individual responsible for making the final decision on what will actually be imported. From experience, I can tell you it’s best to streamline and clean what data you actually put into your new system. This is going to not only benefit the import process but improve efficiency and effectiveness of the software and personnel using it. My final piece of advice is to IMMEDIATELY rethink how you’re inputting data into your current system. The old saying that “Bad data in creates bad data coming out” will never ring more true than when you attempt a data export and then import into a new software solution. This will require some time and effort and you may not like what you find, but trust me, it’ll be worth it when that inevitable day comes.