Configuring Store and Forward
Store and Forward for High-Speed Buffering​
When configuring the Store and Forward system for high-speed buffering, you are expecting data to come in quick bursts. By buffering the data, the system can accommodate more information than would be possible when going directly against the database.
The key considerations in configuring a buffering system is to avoid expensive operations like storing and reading from the local cache, and setting the memory buffer to be large enough to accommodate the expected burst sizes.
You can leverage the fact that each database connection has its own Store and Forward settings by creating multiple database connections pointing to the same database if you wish to configure multiple Store and Forward engines for different purposes.
Configuring Store and Forward​
To configure the Store and Forward engine for your database, complete the following steps:
- Go to the Platform section on the Gateway Webpage and select System > Store & Forward. Store and Forward metrics for each of your database connections are displayed on the Store and Forward page.
- On the engine you want to edit, click the three dots menu and select Edit to access the Edit panel.
- Once you make the changes you want, click Save Changes at the bottom of the panel to return back to the Store & Forward Engines page.
Store and Forward Settings​
The Store and Forward engine settings define how and when data moves through the system.
Engine Settings​
| Setting | Description |
|---|---|
| Forward Rate | The rate in milliseconds at which data will be considered eligible for forwarding to any attached data sink. Default is 1,000. |
| Schedule Pattern | A comma separated list of time ranges, such as 9:00-15:00, 9pm-5am, or 20.30-04.30. |
Store Settings​
These settings apply to the local disk storage cache.
| Setting | Description |
|---|---|
| Time Threshold | The maximum milliseconds time that data will be allowed to accumulate within memory until written to local disk cache. Default is 30,000. |
| Data Threshold | The maximum amount of data that will be allowed to accumulate within memory until written to the local disk cache. Default is 10,000. |
| Batch Size | The maximum amount of data that will be batched when forwarding to either the disk cache or any attached data sink. Note: Data can be potentially further batched within the data sink when processing incoming data. Default is 10,000. |
Maintenance Settings​
| Setting | Description |
|---|---|
| Primary Store Maintenance Value | The maximum amount of data that will be allowed to accumulate within the primary store until some limiting action is done. If 0, no limit will be enforced. Default is 0. |
| Primary Store Maintenance Type | Sets the maintenance type for the primary store. Options include TIME_DURATION, COUNT, and FILE_SIZE. Default is COUNT. |
| Primary Store Maintenance Action | Determines how data will be handled when the primary store is full. Options include EVICT_OLDEST_DATA and PREVENT_NEW_DATA. Default is PREVENT_NEW_DATA. |
| Secondary Store Maintenance Value | The maximum amount of data that will be allowed to accumulate within the secondary store until some limiting action is done. If 0, no limit will be enforced. Default is 0. |
| Secondary Store Maintenance Type | Sets the maintenance type for the secondary store. Options include TIME_DURATION, COUNT, and FILE_SIZE. Default is COUNT. |
| Secondary Store Maintenance Action | Determines how data will be handled when the secondary store is full. Options include EVICT_OLDEST_DATA and PREVENT_NEW_DATA. Default is PREVENT_NEW_DATA. |
Advanced Settings​
| Setting | Description |
|---|---|
| Forwarding Policy | Sets the forwarding policy to be used: ALL, PRIMARY_ONLY, or SECONDARY_ONLY. Default is ALL. |
| Engine Scan Rate | The rate in milliseconds at which data pipelines are scanned for determining data forwarding. Default is 100. |