Data Storage when endpoint unavailable

Temporary data storage

If a connection to an HTTP, Splunk, or Database endpoint can’t be made when session log data is sent from the Blue Prism database to the Data Gateways engine, the data for the associated output is stored temporarily in a dedicated file within the Data Gateways engine. When the endpoint becomes available the data is resent to the specified endpoint and deleted from the file.

This functionality will only work if the initial connection to the Data Gateways engine was made successfully. The temporary storage of data will only be triggered if a connection to the specified endpoint cannot be made during a previously established session. Data that cannot be temporarily stored is added to ‘dead letter queue’ storage, as described below.

The session log data is held indefinitely until the endpoint becomes available and will continue to store additional session log data as subsequent events are written to it.

‘Dead letter queue’ data storage

If the destination endpoint is available but an unexpected error occurs while transferring session log data to it, the log data will be removed from temporary storage and will be written to the dead letter queue along with information about the error. The dead letter queue file is stored by default in the following directory: logstash\data\queue\main on the application server that runs the Data Gateways process. The data is stored in a non-human readable format. The default maximum size for a data storage file in the Data Gateways engine is 1024 MB (1 GB). If a file reaches this limit, a new file is automatically created to store the excess data.

There is no in-built mechanism that will inspect these files and make further attempts to transfer the data to the target system. It is recommended that you implement a system to clear out this data on a regular basis to prevent a build-up of stored data.

Blue Prism output extensions

By default, the output extension BPHTTP is used for HTTP and Splunk, and BPJDBC is used for Database output types. These Blue Prism extensions are integral to the automatic storage of session log data when an endpoint is unavailable.

Upgrading from 6.5

In Blue Prism version 6.6, BPHTTP and BPJDBC output extensions replaced HTTP and JDBC. If upgrading Blue Prism from 6.5, the output types must be updated to use these output extensions to utilize the endpoint unavailable data storage functionality. This can be done by either creating new Data Gateways configuration, or manually editing the existing configuration via advanced or custom configuration and replacing HTTP with BPHTTP , and JDBC with BPJDBC .

File and custom output types

This functionality does not apply to File or custom configured output types. When using these output types, we recommend enabling Write session logs to database to avoid data loss.