Speeding up small updates to large data using Incremental synchronisation

When we are integrating two different systems we often want to just utilise Simego DS3 to insert or update a small portion of the destinations data. For example we might want to synchronise 100 client records from a source file containing client records to our Dynamics CRM environment containing thousands of records.

Using DS3's normal compare mode, it would download the entire destination records, compare them with the source and present the results

In cases like this we have no interest that there are 24944 records in the destination that are not in the source. We are only interested in the 99 updates but have used 14 seconds to read the entire destination records. If we scale the problem it becomes more obviously significant when reading millions of records from the destination.

In cases like these we can instruct DS3 to reconcile A to A incrementally. This means DS3 will only load the records found in A from B. It can identify that records are missing in B and if they have changed also. It will not detect if there are any to delete but we are not interested in deletion in this scenario. DS3 uses the primary keys in the source to request only the records in parallel from the target. This means that you may have only one primary key for this to work.

To turn on Incremental mode for a project, navigate to the File menu then the project properties. Change the SyncOption from AtoB to AtoBIncremental.

As you can see using the AtoBIncremental setting has reduced the load time from 14.147 seconds to 1.418 and only presented us with the updates rather than deletes.