Troubleshooting – Application server

Application server configuration

The Windows Service cannot start

If the Windows service will not start or starts and immediately stops, this indicates that there is a problem with the configuration of the server.

When the server service is starting a number of checks occur including, but not limited to, the following:

  • Appropriate access to the SQL database, and expected DB updates have been installed.
  • Encryption scheme keys are held on the server for those records in the database that indicate the key should be held there.
  • The server connection mode supports the Blue Prism authentication mode.
  • The user has appropriate rights to start the listener on the device.
  • Valid license is installed.

In order to identify the cause of issues, the following steps should be followed:

  • Check the profile for warning messages within the BPServer.exe utility – This will highlight issues such as if a server service is not configured for this profile; or if an encryption certificate is required but cannot be found; or if the service logon user does not have appropriate rights to start the listener.

  • Review messages within Event Viewer – This will highlight issues such as if the server service profile cannot be found; if the server cannot authenticate with the database; if an encryption certificate is required but there are issues with it; if expected encryption schemes cannot be found within the service profile; or if the service logon account does not have appropriate rights to start the listener.

  • Attempt to start the service using the BPServer.exe utility – Using this utility in this way is only suitable for troubleshooting purposes as it attempts to start the service under the context of the currently logged in user. If the locally logged on user has different permissions to the service logon account the behavior seen here can differ in comparision to when the service is started from the Services management console. For example, if the service is configured to connect to SQL using windows authentication this will require the currently logged in user to have appropriate minimum rights to the Blue Prism database on the target SQL server.

A valid license could not be detected

Service cannot be started. System.NotSupportedException: A valid license could not be detected.

A valid license must be configured for the environment in order for a Blue Prism server to be able to start.

A new license key can be installed via the Blue Prism user interface. It may be necessary to use a client that has a direct database connection to carry out this action.

Encryption keys could not be resolved

Service cannot be started. BluePrism.BPCoreLib.InvalidStateException: The following encryption keys could not be resolved: 2018 Q2 Encryption Scheme, Default Encryption Scheme

This error indicates that there are encryption scheme keys that are expected to be on the server, but which cannot be found. The error above indicates that it can’t find two schemes which should be defined locally on the Blue Prism Server named: “2018 Q2 Encryption Scheme” and “Default Encryption Scheme”.

It is necessary to review the Encryption Scheme records configured within the database, and ensure that for each with a Key location of Server, that there is an appropriate encryption scheme record created on the Blue Prism Server. An example of comparing the settings within the client against the settings within the Blue Prism server configuration utility.

The account the service is running as does not have the right to create services

Errors such as the following indicate that the account that the service is being run as, does not have appropriate permissions to configure the service to listen on the configured settings:

BluePrism.BPCoreLib.InvalidStateException: An error occurred while trying to start the server. The account the service is running as (AD\bpserverservice001) does not have the right to create services that listen on the server's namespace.

This is a common message when the Blue Prism Server is being started as a user which is not a local administrator; or if the Access Control List (ACL) has not been configured appropriately.

To resolve the issue either:

  • Use the Blue Prism Server configuration utility to set up permissions for the configured user to start the service; or
  • Execute the command provided within the event viewer message.

It is important to ensure that the ACL permission is created specifically for the user that will be starting the service, and that it is configured with either a generic URL if no server binding is specified; or a URL that directly aligns with a specified server binding.

Error checking services

If this error is presented when editing the Blue Prism server profile it indicates that an error has occurred when validating if the currently logged in user is a local administrator.

This is known to occur when a local user account is used to access a device that is a member of an Active Directory Domain, and where a Domain Controller cannot be contacted. It is necessary to ensure that a Domain Controller can be contacted.

Encryption certificate cannot be found

If the certificate used for encryption cannot be accessed or restored the following message displays: The certificate used for encryption of the server configuration cannot be found. Please add the certificate with the correct thumbprint to the certificate store.

In this situation, the user configuring the Blue Prism server will need to recreate the server configuration profiles. To do this, delete the Automate.config file, which is located in: ProgramData\Blue Prism Limited\Automate V3. A new Automate.config file will be automatically created when the BPServer.exe is launched. An encryption scheme and new certificate can then be applied to the new server configuration file.

Service Principal Name (SPN) configuration for Kerberos authentication

Verify that DNS lookup is working

  1. Check the hostname of the configured Blue Prism Server on the Connection Configuration screen in the Blue Prism interactive client.

  2. Perform a forward DNS lookup from the Blue Prism interactive client: nslookup <BP Server hostname>

    This will return the FQDN to use in the SPN registration.

If this is not working, please contact your internal IT Team to troubleshoot further.

Verify that the SPN has been registered correctly

  1. Open Command Prompt as an administrator and run the command setspn -L ACCOUNT_NAME .

    This will list the SPNs registered for the account that you specified during the SPN registration.

  2. Confirm that the returned SPN is the same as the one registered, which should be in the following format: <service class>/<host>:<port>/<service name>

    Where:

    • service class is HTTP. This should NOT be changed to HTTPS.
    • host is the FQDN of Blue Prism application server, for example appserver.bpdomain.local
    • port is the port the Blue Prism Application server is running on.The Default = 8199
    • service name is the BP Server

It is important that the SPN is configured exactly as outlined in SPN configuration to function correctly. SPNs should be unique in the Active Directory forest but under some circumstances duplicates may exist. To check for duplicate entries you can use setspn -F -Q */BPserver.This will list all SPN entries in the forest that contain ‘BPserver’. If there is more than one entry, this can be removed by running the command setspn -D SPN_NAME ACCOUNT_NAME.

Verify Kerberos service tickets

If the SPN is registered to the correct account it should be possible to use klist from a non-elevated command line to check that a Kerberos service ticket can be obtained.

This can be done by:

  • Purging the Kerberos tickets using klist purge.
  • Running klist get SPN_NAME, for example, klist get HTTP/appserver.bpdomain.local:8199/BPServer on the client machine to obtain the output below. If the SPN is in a different realm from your user, you can test by running klist get SPN_NAME@KERBEROS_REALM.

Example Kerberos service ticket:

Copy 
Client: testuser @ BPDOMAIN.LOCAL
Server: HTTP/appserver.bpdomain.local:8199/BPServer @ BPDOMAIN.LOCAL
KerbTicket Encryption Type: RSADSI RC4-HMAC(NT)
Ticket Flags 0x40a10000 -> forwardable renewable pre_authent ok_as_delegate
name_canonicalize
Start Time: 1/11/2022 12:00:00 (local)
End Time:  1/12/2022 22:00:00 (local)
Renew Time: 1/18/2022 12:00:00 (local)
Session Key Type: RSADSI RC4-HMAC(NT)
Cache Flags: 0x200 -> DISABLE-TGT-DELEGATION
Kdc Called: dc1.bpdomain.local

The Kerberos service ticket can be identified by checking the "Server:" section in the above example. If a Kerberos service ticket for the registered SPN is NOT returned, please check that the previous steps have been followed correctly.

If you have followed the steps above, and are still unable to return a Kerberos ticket for the registered SPN, you will need to contact your internal IT Team to investigate further, as Windows Kerberos authentication is not functioning as expected.

If you can successfully return a Kerberos service ticket for the registered SPN, but the issue is not resolved, please see the section below.

RC4 Encryption

In the example above, the Kerberos service ticket has a base Encryption type of RC4 which is considered to be a vulnerable cipher and is not recommended for use.

In some environments, RC4 tickets may be generated, but rules preventing the client from accepting and using such a Kerberos ticket may have been enabled.

If you see RC4 tickets being generated like in the example above, please contact your Internal IT Teams to ensure that the service account’s Kerberos authentication mechanism is able to use RC4, or has AES encryption enabled. The Kerberos authentication type can be identified by checking the KerbTicket Encryption Type section in the example above.

Application server connection

Errors connecting a Blue Prism device to the application server can be caused by a large number of factors, it is strongly recommended that the following are verified:

  • Blue Prism Server Service is started.
  • The address being used for the Server service is resolvable (i.e. via DNS) and that network connectivity is not being prevented. (e.g. verify that firewalls are configured appropriately).
  • The connecting device is configured with the correct settings:
    • The server connection mode, and port match those defined on the server.
    • If the server is configured with an address binding, that the device is connecting using that address.
  • If the server is configured to use transport encryption, the certification authority that issued the server certificate must be trusted by the connecting the device.

Connecting to the database

Review the Database Connectivity troubleshooting section for general connectivity advice.

When troubleshooting, consider that the account being used to authenticate with SQL will depend on SQL authentication mode that has been configured on the connection used by the server:

  • SQL Authentication – The credentials specified on the connection will be used.
  • Windows Authentication – The context of the server service will be used. If starting the service from the Windows Services console, this will be the service logon account; if starting the service directly from BPServer.exe, this will be the currently logged in user.

Database does not exist

Service cannot be started. BluePrism.BPCoreLib.InvalidStateException: Connection not valid: Server is unavailable

Database 'BP_Prod_Native' does not exist

This error indicates that the database cannot be found.

Verify that the database server, and database name are correct. If a Blue Prism database has not yet been created, a user with appropriate SQL permissions can achieve this through use of the in-product Create Database action, or manually through use of a CreateScript.sql.

Incorrect permissions

Service cannot be started. BluePrism.BPCoreLib.InvalidStateException: Connection not valid: Server is unavailable

Cannot open database "BP_Prod_Native" requested by the login. The login failed.

This error indicates that the user used to authenticate against the database does not have permissions to access it.

The user will need to be granted at least SQL permissions on the target database that meet or exceed the minimum permissions.

Incorrect credentials

Service cannot be started. BluePrism.BPCoreLib.InvalidStateException: Connection not valid: Server is unavailable

Unable to determine whether database exists – Login failed for user

This error indicates that the user credentials used to access the database are incorrect (e.g. invalid username or password).

Verify the user credentials being used, and that the user’s SQL permissions on the target database meet or exceed the minimum permissions.