Skip to main content
Mitratech Success Center

Bulk Image Import Performance Improvement

In this chapter we provide some lines of code to improve bulk impage import performance.

The number of index values that are stored in the database by a single execution of the bulk import SSIS package can be changed. By default, this is 200,000. Performance is improved when the BulkImportThreshold value is set to a high value and there is still memory to perform the operation without any swapping being necessary, as this harms performance significantly.

The value set for the BulkImportThreshold is not a hard limit; it is a threshold used by DataStoreDSX when calculating how best to divide up a large file.

Reducing the number from 200,000 reduces the amount of memory required by the SSIS package, improving performance; the trade-off is that more SSIS executions are necessary, which then starts to reduce performance again if the threshold is set too low. On a server with a very large amount of memory, increasing the BulkImportThreshold reduces the number of SSIS executions required to store a large file.

The configuration is set in HitecLabs.DataStore.DataStoreService.exe.config, which is located in the same directory as the DataStore service executable.

  1. In the <configSections> block at the top of the configuration file, in the <sectionGroup name="applicationSettings" ...> block, look for the following entry.
    (If it does not exist, add it.)

<section name="HitecLabs.DataStore.Services.Database.DataItemSettings" type="System.Configuration.ClientSettingsSection, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" requirePermission="false"/>
 

  1. In the <applicationSettings> section, look for the following section. (If it does not exist, add it.)

<HitecLabs.DataStore.Services.Database.DataItemSettings>
...
</HitecLabs.DataStore.Services.Database.DataItemSettings>

  1. Within the <HitecLabs.DataStore.Services.Database.DataItemSettings> section, add:

<setting name="BulkImportValueThreshold" serializeAs="String">
    <value>200000</value>
</setting>

replacing 200000 with the threshold that you want to set.

  1. Save the file.
  2. After modifying this configuration, the service must be restarted for the changes to take effect.
  • Was this article helpful?