Testing for accuracy in Decipher IDP

This guide defines accuracy of data extractions and how to test accuracy of SS&C | Blue Prism® Decipher IDP using a standard set of documents.

  1. Outside of Decipher IDP, collect a “gold data” set of documents and correlating data.
  2. Process that document set as a batch through Decipher IDP twice– first skipping manual verification, and second with manual verification. Comparing the results both sets of results to the “gold data” provides a simple but solid estimate of current accuracy as well as insight into how we can improve batch accuracy.

Define accuracy

In the context of Decipher IDP, we will define the following:

  • TP – True Positive. A value is present, and the correct value is published.
  • FP – False Positive. A value is or is not present, and an incorrect value is published.
  • TN – No value is present, and no value is published.
  • FN – A value is present, but no value is published.

Precision = TP / (TP + FP)

Precision of the values published, what percentage are correct. In this context, FPs are the highest risk to business, and we should focus first on decreasing FPs, which increases precision. Poor machine precision is concerning and should trigger investigation of possible causes, such as misconfiguration of the document form definition (DFD).

Recall = TP / (TP + FN)

Recall of the values present, what percentage are correctly published. We expect recall to be low in early iterations and to increase over time as more documents are observed. As the ML model improves, we should see fewer FNs. This increase in recall means less manual verification (annotation) will be required.

Accuracy = (TP + TN) / (TP + TN + FP + FN)

It is important to differentiate between initial (machine only) precision/recall and the machine and human precision/recall. The manual verification step allows us to keep machine and human accuracy high throughout the Decipher IDP cycle.

Accuracy examples

The following accuracy examples are based on the sample invoice Busted Knuckles Motorsport.

Sample invoice.pdf

True positive example

The document form definition has a field for Grand Total. The total due on the invoice is £2734. When the invoice is processed, Decipher IDP extracts "£2734" for the Grand Total field.

False positive examples

The document form definition has a field for Grand Total. The total due on the invoice is £2734. When the invoice is processed, Decipher IDP extracts "52734" for the Grand Total field.

The document form definition has an optional field for SKU#. This invoice does not have SKU numbers. When the invoice is processed, Decipher IDP extracts the value from the quantity column in the SKU# field.

True negative example

The document form definition has a optional field for SKU#. This invoice does not have SKU numbers. When the invoice is processed, Decipher IDP does not extract a value for SKU#.

False negative example

The document form definition has a field for Grand Total. The total due on the invoice is £2734. When the invoice is processed, Decipher IDP does not extract a value for the Grand Total field.