Posts in Category: data

Patterns on Bulk Data Migration

No code in this most, well maybe a bit.

I’ve been working on a number of data migrations over the past few years using a variety of tools, from off the shelf solutions to home grown, I’ve run everything in between.

All of them have their hits and misses and this isn’t to harp on one over the other as I continue to refine my approach in getting data into Dynamics in the most efficient way possible.

My current investigation into how to optimize this process has lead me to consider what the most important pieces for a solution to work are;

  1. Make it As Fast As Possible.
  2. Being able to handle Concurrent Connections (i.e., multi-thread and parallelize it until the cows come home).
  3. Accessible in the environment that the user operates in (read: if you’re

    Read More


File Source Encoding

I’ve been doing some migration from CSV files over the last few months that resulted in some interesting exercises in the cleansing of data before it gets to Dynamics.

For all of this work, the code was written using C# and the Dynamics SDK.

Encoding

I ran into a number of encoding problems in transferring data from the files into Dynamics. Initially, I thought Dynamics could not handle the character set but on closer inspection, it was the code that wasn’t loading it properly.  I tried a number of different Encoding formats but the one that addressed all scenarios as using the Encoding.Default and let the system decide which to use and when.

Therefore, when loading the data, my initial delimited looked as follows;
var data = File.ReadAllLines(file, Encoding.Default).Select(a => a.Split('\n').ToList());

Delimiters

Since all the data

Read More