Patterns on Bulk Data Migration

No code in this most, well maybe a bit.

I’ve been working on a number of data migrations over the past few years using a variety of tools, from off the shelf solutions to home grown, I’ve run everything in between.

All of them have their hits and misses and this isn’t to harp on one over the other as I continue to refine my approach in getting data into Dynamics in the most efficient way possible.

My current investigation into how to optimize this process has lead me to consider what the most important pieces for a solution to work are;

  1. Make it As Fast As Possible.
  2. Being able to handle Concurrent Connections (i.e., multi-thread and parallelize it until the cows come home).
  3. Accessible in the environment that the user operates in (read: if you’re

    Read More

Bulk Delete with Dynamics365

No matter what project you are on, you are at one point or another going to need to delete data.

Your environment will become polluted and you going through each entity or manually queuing up your own delete jobs will not be worth your time.

To bulk delete your entire environment you can make use of the Bulk Delete API with the following lines of code.

QueryExpression bulkquery = new QueryExpression();
bulkquery.EntityName = entity.LogicalName;
bulkquery.Distinct = false;

BulkDeleteRequest request = new BulkDeleteRequest
{
JobName = String.Format("System Purge Requested for Entity [{0}] - All Records", bulkquery.EntityName),
QuerySet = new[] { bulkquery },
StartDateTime = DateTime.Now.ToUniversalTime(),
ToRecipients = new[] { currentUserId },
CCRecipients = new Guid[] { },
SendEmailNotification = false,
RecurrencePattern = String.Empty
};

ExecuteMultipleResponse response = new ExecuteMultipleResponse();
response = (ExecuteMultipleResponse)_svc.CrmService.Execute(multipleRequest);

Of critical importance is the

Read More

Dynamics SDK Fault Errors

Recently I was running some code against the Dynamics SDK that was taking a significant amount of time to run (transferring of data).  In my naivety, I had thought that by constantly doing “something” on that live connection my token would be refreshed.

Not the case.

It seems tokens from the Dynamics SDK are set at the time of connection and kept for the duration of the connection and not updated until you do so again.

To avoid having my connection cut during this migration (which was bad), I added some logic to check when the connection had last logged in, do a quick comparison of X minutes (in my case I chose 90 minutes but it could be more) and triggered a reconnect/refresh of my token.

The code is nothing spectacular, but understanding

Read More

File Source Encoding

I’ve been doing some migration from CSV files over the last few months that resulted in some interesting exercises in the cleansing of data before it gets to Dynamics.

For all of this work, the code was written using C# and the Dynamics SDK.

Encoding

I ran into a number of encoding problems in transferring data from the files into Dynamics. Initially, I thought Dynamics could not handle the character set but on closer inspection, it was the code that wasn’t loading it properly.  I tried a number of different Encoding formats but the one that addressed all scenarios as using the Encoding.Default and let the system decide which to use and when.

Therefore, when loading the data, my initial delimited looked as follows;
var data = File.ReadAllLines(file, Encoding.Default).Select(a => a.Split('\n').ToList());

Delimiters

Since all the data

Read More

Unified Service Desk Customization Files

I’ve been doing some custom hosted control work for the Unified Service Desk lately and wanted to implement the ability to have my controls download automatically on any new deployment without having to copy them over.

The benefits of this approach is that you are not putting your files into your main install directory (and potentially corrupting your USD install) but rather deploying them to your App/Local Data folder where they will be automatically downloaded.

Neil Parkhurst does a great job on the walkthrough there – https://neilparkhurst.com/2017/08/01/usd-customization-files/ – so I’m not going to take away from that.

What I do want to highlight are some other issues I ran into when trying to get things to deploy;

  1. If your zip file is more than 5MB you are going to need to increase your file upload

    Read More