The number of index values that are stored in the database by a single execution of the bulk import SSIS package can be changed. By default, this is 200,000. Performance is improved when the BulkImportThreshold value is set to a high value and there is still memory to perform the operation without any swapping being necessary, as this harms performance significantly.

The value set for the BulkImportThreshold is not a hard limit; it is a threshold used by DataStoreDSX when calculating how best to divide up a large file.

Reducing the number from 200,000 reduces the amount of memory required by the SSIS package, improving performance; the trade-off is that more SSIS executions are necessary, which then starts to reduce performance again if the threshold is set too low. On a server with a very large amount of memory, increasing the BulkImportThreshold reduces the number of SSIS executions required to store a large file.

The configuration is set in HitecLabs.DataStore.DataStoreService.exe.config, which is located in the same directory as the DataStore service executable.

1. In the <configSections> block at the top of the configuration file, in the <sectionGroup name="applicationSettings" ...> block, look for the following entry.
   (If it does not exist, add it.)

<section name="HitecLabs.DataStore.Services.Database.DataItemSettings" type="System.Configuration.ClientSettingsSection, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" requirePermission="false"/>
 

2. In the <applicationSettings> section, look for the following section. (If it does not exist, add it.)

<HitecLabs.DataStore.Services.Database.DataItemSettings>
...
</HitecLabs.DataStore.Services.Database.DataItemSettings>

3. Within the <HitecLabs.DataStore.Services.Database.DataItemSettings> section, add:

<setting name="BulkImportValueThreshold" serializeAs="String">
    <value>200000</value>
</setting>

replacing 200000 with the threshold that you want to set.

4. Save the file.
5. After modifying this configuration, the service must be restarted for the changes to take effect.