Orchestration

The way Blue Prism Cloud utilizes orchestration is by leveraging the IADA web service within a Blue Prism process called Orchestrator, the Orchestrator process will perform all the logic and intelligent decision making on which process to run; and when to run that process. The Orchestrator process will be running throughout the day, all day (aside from a small automation maintenance window), consuming all your digital workforce or as much as you want to allow it to utilize.

Implementing orchestration will allow you to use more advanced decision making, leveraging Blue Prism Cloud IADA web service and Orchestrator process will ultimately allow you effectively utilize your Blue Prism Cloud Digital Workforce in the best way possible.

What is orchestration?

As we explained, within an organization not all process automations and their associated items have the same value to the organization, some business processes need to be prioritized over others. By default, items will simply be addressed and processed in a First-In-First-Out (FIFO) methodology.

This combined with the fact that there can be multiple queues with the Blue Prism Cloud Digital Workforce that need to be managed, means that work can get stalled in handling an individual queue or stuck processing automations with work items which are of a lower value.

IADA maintains, in effect, its own view, looking across not a single queue but all the queues and assessing assigned values (such as business priorities and SLAs), it will determine the next automation item that needs to run, whilst queuing the next best value item ready for execution by the next available digital worker.

For an automation to be given a priority over another automation, it has to be configured with a business priority and an SLA. There are multiple ways of adding these parameters into your automations and we will show you how in more detail later. This is the key element so when items are added to the work queue, the business metrics are added along with the process.

For clarification, lets define some key terms:

Business priority

The business priority value must be above zero. The lower the value, the higher the business priority (e.g. a business priority value of 001, will be processed before an item with a business priority value of 002). This business priority value will override the SLA value.

IADA accepts a business priority value between 001 to 999. The priority must have three digits.

Business SLA

The business SLA value is a timespan. This should be the amount of time that the work queue item should be completed and processed within. The SLA timespan value will then be used within the IADA algorithm to produce an SLA Datetime.

IADA accepts a business SLA in the format of HH:MM:SS (Hours, Minutes, Seconds) e.g. 043000 = 4 Hours, 30 Minutes, 0 Seconds. The minimum SLA is 1 second, the maximum SLA would be 99 Hours 59 Minutes and 59 Seconds.

Tag

The tag is a combination of the Priority, SLA and Process Name as per the example below, it will be recognized and accounted for by IADA.

In the example the Priority is set to ‘001’, the SLA is ’24 hours, 0 minutes and 0 seconds’ and the Process Name is ‘Testing’.

Orchestration configuration queue

If a digital worker has started running a process automation, how does it notify the remainder of the digital workforce of this action, or how does it set a flag that the remainder of the workforce can check? As with any human workforce, people communicate and talk to each other, sometimes making each other aware of their actions. Your digital workforce needs to do the same.

Unfortunately, there isn’t a concept of global variables that are read/write accessible within the Blue Prism RPA designer. There are a number of different solutions that can be used to create communication channels. For the purpose of this documentation we will use the concept of a separate specific Blue Prism queue containing only one item. The data required will be stored in the ‘Item Data’ of the queue item, though there are other options which could be considered, for example:

  • SQL database table
  • JSON file
  • Custom .config file with parser
  • Excel document
  • XML
  • CSV file

We have chosen to create and maintain a queue item (in the ‘Orchestration Configuration Queue’) that digital workers can read and write to. Now when we want to log an action that the digital workforce needs to be aware of, we can open the item, read back the contents, update the contents and write the updated contents back to the queue item. This is also known as the Orchestration ‘States’ file and you will see the term ‘States’ in the process templates.

We now have a mechanism of logging when we are processing an automation and when an automation has been completed.

In order to ensure that there are no conflicts when reading from or writing to the queue item we will use environment locks.

Environment locks

One of the final pieces of the puzzle to create to enable orchestration to occur is a locking and token release mechanism. We already have this in the form of ‘Environment Locks’. It is recommended that you are familiar with environment locks and understand how they function. We use environment locks quite heavily within orchestration as this concept is key to adding a control mechanism around:

  • Decision making for getting the ‘Next Best Item’ to be worked via IADA;
  • Information gathering – reading from the Orchestration Configuration Queue item; and
  • Writing to the Orchestration Configuration Queue item.

How does orchestration work?

Let us consider the following five digital worker platform as illustrated below.

On each digital worker the Orchestrator process would run. The Orchestrator process runs and operates the same way across the digital workforce environment.

The digital worker will first check to see if there is a scheduled process that needs to be actioned, this is determined by the datetime and when the schedule should occur, ‘1’ in the diagram.

If the process is not scheduled to run it will then go and reassess across all the queues the ‘Get Next Best Item’ to process.

If there was a scheduled task to perform it would initiate that process and complete that task before looking for new work.

In our example the highest business priority is ‘work item 4’ in the Invoice Processing queue, ‘2’ in the diagram.

The first task the Orchestrator process does is write to the Orchestration Configuration Queue to notify the rest of the digital workforce, ‘I am handling this item’.

The digital worker would have either already called its Collector process to retrieve any associated data items, if required, along with the associated process required to be run against the work item, the business priority and the SLA. This information is then used by the Executor process within the Orchestrator process to perform the required automation.

The Collector would not be run if the item was added to the queue using Interact as the associated tag would contain all the necessary information.

This routine is then repeated with the other digital workers, checking for scheduled tasks, before identifying the ‘Next Best Prioritized Item’ for processing, in our example, ‘3’, this is work item 19 in the Leavers Queue.

The other digital workers will source their own work, ‘4’ in the diagram calling their Collectors, if required and Executors to perform the work task on the next business critical process identified.

This also includes items that can be added into the queues on an ad-hoc basis from users using the Interact interface.

It is recommended that the Orchestrator process is run across the whole of the digital workforce.

It is recommended that the digital workers are built identical with each digital worker capable of running every process.

If there is no tag or orchestration status assigned to an item in a queue the Orchestrator process will not process the task and it will be ignored. The Orchestrator process looks across all queues, but only processing items with tags.

What are the benefits of orchestration?

To demonstrate and explain how an Orchestrator process could be designed and in turn show the benefits, we have provided an example use case below.

A company, ‘Xample Mortgages’, have asked Blue Prism Cloud to create nine processes comprised of four Executor processes (referred to as Executors as these will work/process items in a queue), four respective Collector processes (referred to as Collector processes as these are tasked with collecting the data and populating the respective queues to be worked by the Executors) and one monthly reporting process. These process automations are required to be ran as effectively as possible throughout the day. Their parameters and requirements are outlined below:

  1. Redemption Payments Received – Collector:
    • CONTEXT: The data to be collected is only available following money received from Bank transfers on weekdays. All the data for the day will be available by 12:00:00. Only one data dump is received per day so once the Collector has been run successfully it will not need to be run again that day .
    • SCHEDULE: Once a day on Monday / Tuesday / Wednesday / Thursday / Friday.
  2. Redemption Payments Received – Executor:
    • CONTEXT: Each payment received must be applied to the customer’s account within 24 hours. Due to financial regulations this process has a short SLA and must be processed as a higher priority item. If items are not processed within the timeframe then a fine is applicable.
    • SLA: 24 Hours 00 Minutes.
    • PRIORITY: 010.
  3. Mortgage Applications – Collector:
    • CONTEXT: The client is a larger mortgage provider and as such receives thousands of mortgage applications per day. These can and do arrive at any time of day and the automation must account for that. Xample Mortgages aims to respond to the customer application (by providing confirmation/rejection of an offer) within 2 days.
    • SCHEDULE: To be run every day (including weekends) every 30 minutes.
  4. Mortgage Applications – Executor:
    • CONTEXT: The successful mortgage applications that arise from this process are the main source of income for Xample Mortgages. There are no fines associated with late application responses, but the aim is for an internal SLA of 2 days; therefore, it receives 2nd priority for automation.
    • SLA: 48 Hours 00 Minutes.
    • PRIORITY: 020
  5. New Starters – Collector:
    • CONTEXT: The company has several large call centers that have relatively high rates of attrition. There are generally new starters joining the company every week.
    • SCHEDULE: This Collector should be run once in each of the following periods (every weekday): 10:00:00-11:00:00, 13:00:00-14:00:00, 16:00:00-17:00:00, 20:00:00-21:00:00.
  6. New Starters – Executor:
    • CONTEXT: This is an internal process that aims to complete all onboarding activities for new starters to Xample Mortgages. In the past the activities required to onboard new starters (system access, payroll, HR notifications) was disjointed and often completed late. Sometimes employees were unable to begin work on the confirmed start date. This automation aims to speed up the process to ensure that the new joiner will have everything at their disposal once they start working for Xample Mortgages. The new joiner instruction is sent ahead of time and there is an SLA of 2 days associated with this process.
    • SLA: 48 Hours 00 Minutes.
    • PRIORITY: 030.
  7. Leavers – Collector:
    • CONTEXT: As previously mentioned Xample Mortgages has a high rate of attrition and the leaver Collector must be handled in a timely manner (although not as high a priority as the New Starters). So as not to interfere with working week tasks Xample Mortgages would like the Collector for this process to only run on weekends ready for HR to pick up the post automation activities (if any).
    • SCHEDULE: This Collector will only run on Saturday and Sunday from 03:00:00 every 3 hours.
  8. Leavers – Executor:
    • CONTEXT: The leaver related activities have a longer internal SLA and lower priority level.
    • SLA: 72 Hours 00 Minutes.
    • PRIORITY: 040.
  9. Monthly Reporting Process:
    • CONTEXT: On the 1st of each month a summary report will be produced covering a selection of queues and providing Management information. This report must only be run once and should be run in respect of the preceding month.
    • SCHEDULE: Run on the 1st of each month (regardless of day of the week).

If you consider how this may be achieved using a scheduler, you will quickly identify potential issues and limitations. For example:

  • Can you effectively perform all the necessary checks, and deploy the necessary management functionality within the scheduler?
  • Can you confidently provide the business with 100% assurance that the digital workforce can meet and delivery on these requirements?
  • What happens where one schedule processes over runs and a trigger for another key time dependent process is missed or delayed?
  • Can a scheduler determine which items to process next based on business priorities and SLA?
  • Would a controller have to take the following actions (only during their standard work hours):
    • Continually monitor queues?
    • Stop various processes across digital workers in order to free up resources?
    • Diverting said resource to a specific process?
    • Keep an eye on this process and then change all the updated digital workers back to their original schedules?
  • Will items or processes be missed if a human must continually do this throughout the day across 100 digital workers?

The last question is rhetorical, but the above scenarios are a prime example of how orchestration can be implemented to manage and control all process automations to meet the necessary requirements and business metrics.

Designing for orchestration

When implementing orchestration, it is common to feel apprehensive when considering how to approach the design, and it is sometimes thought that complex cognitive systems and Artificial Intelligence (AI) are required to complete the design.

Although cognitive systems like Machine Learning (ML) and AI implementations are important, it should be a secondary consideration after implementing a Logical and Algorithmic design. These technologies need to overlay a base foundation of logical decision making.

Logical decision making

Once you break down the requirements, you can start to understand what checks the process might want to perform when deciding if and when to execute an automation. If we just take one process, the Redemption Payments Received – Collector process automation, the following considerations need to be understood:

  • Has it already been run?
    • This is important and should be one of the first checks that we perform. If the process automation has already ran, then we don’t and should not run this again for that day.
  • Is it after 12pm?
    • Easy enough, we should check the current time and ensure that it is after 12pm. If it is not after 12pm (on the current day) then we should not run this process automation yet.
  • Is today one of the respective days it can be worked on?
    • As previously noted, this Collector process should only be run on weekdays. This means that we need to check the current day and determine if it is a weekday.
  • Is it ‘running’ already by another digital worker?
    • This automation will not be recognized as complete until after the automation has ended. This means that we need to check to see if the process is in progress.
  • What is the current average execution time?
    • This has already been given in the requirements, but we should be careful. Is this an average execution time based on a human completing the process? When was this average execution time noted? Was this noted 6 months ago, and since we have upgraded the process to be more productive and quicker? We could check the latest figures by analyzing previous execution times.
  • Can we delay?
    • Ultimately, we don’t need to run this process at 12PM, or at least there isn’t anything in the requirements above that would suggest we can’t. The only requirement is that it must be completed at some point today after 12:00:00. The answer to this question may be determined by what we think the average execution time of the automation might be.

The same level of understanding and questioning should be performed for all the remaining processes in our example.