Configuring Store and Forward
Store and Forward for High-speed Buffering
When configuring the store-and-forward system for high-speed buffering, you are expecting the case that data will come in quick bursts. By buffering the data, the system can accommodate more information than would be possible going directly against the database.
The key points in configuring a buffering system is to avoid expensive operations like storing and reading from the local cache, and to set the memory buffer large enough to accommodate the expected burst sizes.
Each database connection has its own store and forward settings. Store-and-forward engines are directly correlated to database connections, and are automatically managed so that each connection has an engine defined.
You can create multiple database connections pointing to the same database if you wish to configure multiple store-and-forward engines for different purposes. To configure the Store and Forward engine for your database, do the following steps:
Go to the Gateway Configure section and select Databases > Store and Forward.
The Store and Forward page is displayed and you will see a store-and-forward setting for each of your database connections and you can edit these settings.On the Store and Forward page, look for edit at the far right of the table and click on it to see all the store-and-forward settings.
The settings of a store-and-forward engine define how and when data is moved through the system. You must understand these settings so that you can carefully set them according to your goals.
Store and Forward Settings
Buffer Settings
| Setting | Description |
|---|---|
| Memory Buffer Size | The number of records that can be stored in the memory buffer, the first stage of the store-and-forward chain. Other settings define when the data will move from the memory buffer forward, this setting only determines the maximum size. If the max size is reached, additional data will error out and be discarded. The memory buffer cannot quarantine data, so if there are errors and the disk cache is not enabled, the data will be lost. If set to 0, the memory buffer will not be used. |
Store Settings
These settings apply to the local disk storage cache.
| Setting | Description |
|---|---|
| Disk Cache Enabled | Turns on the hard-disk cache. Data is stored here if it cannot be forwarded in a timely manner. The cache also stores quarantined data (that is, data with errors). |
| Max Records | The maximum size of the cache. After the max is reached, data is backed up into the memory buffer, and once that is full, it is dropped. A 'record' is an insert or update statement. These statements may be batches, thus it is possible for a single 'record' to impact multiple rows. The default value is 25,000. Note: The disk cache is intended to be a short term storage system. It is not recommended to increase the Max Records beyond 50,000. |
| Write Size | The number of records that should be accumulated in the memory store before written to the cache. Writing data in blocks can increase performance, but too large of a size increases the risk of data being lost in the event of a power outage or system failure. |
| Write Time | The max age of records in the memory buffer before they are stored to the cache. This setting is used in combination with the write size in order to give the forwarder the opportunity to retrieve data directly from the memory store and avoid the write to disk entirely. |
Forward Settings
These settings govern when data is forwarded to the database. The data is pulled first from the local cache, and then from the memory store. When no data is present in the cahce, it is pulled directly from the memory store.
| Setting | Description |
|---|---|
| Write Size | The number of records that should be accumulated in the memory store before written to the cache. Writing data in blocks can increase performance, but too large of a size increases the risk of data being lost in the event of a power outage or system failure. |
| Write Time | The max age of records in the memory buffer before they are stored to the cache. This setting is used in combination with the write size in order to give the forwarder the opportunity to retrieve data directly from the memory store and avoid the write to disk entirely. |
| Enable Schedule | If enable schedule is selected, the forward engine will only be enabled during the times specified by the pattern. The pattern can specify specific times and ranges using a simple syntax. |
| Schedule Pattern | The schedule is specified as a comma separated list of times or time ranges. You can use the following formats: 24-hour times, that is 8:00-15:00 (for 8am through 3pm) or 21:00-24:00 (9pm through midnight). 12-hour with am/pm (if not specified, 12 is considered noon): 8am-3pm or 9pm-12am Note: When the time period is over, any queued data will remain cached until the next execution period. That is, the forward engine does not run until all data is forwarded. |
- Once you made the changes you want, click Save Changes at the bottom of the page.
This will take you back to the Store and Forward page.