Smart Vision spy mode

Smart Vision mode uses AI surface automation to improve the spying process, and introduces additional logic when configuring elements to ensure they are uniquely identified.

Currently, Smart Vision supports the following languages for OCR extraction: English, German, French, Spanish, Italian, Portuguese, Swedish, Norwegian, Finnish, Danish, and Polish.

Start spying with Smart Vision mode

1. In Application Modeller, click Identify element.

The Identification tool displays.

2. Press Alt to tab through the available spy modes until the Smart Vision mode displays.



3. Hover the mouse cursor over the application, hold Ctrl, and click the left mouse button to take a screen shot.

After an application is spied, the screenshot is sent to the Smart Vision server. The screenshot is processed by the server and opened in the Smart Vision Editor, where you can start identifying and configuring elements.

To cancel a spying session in progress:

1. Ctrl and right-click anywhere on the screen.

The Identification tool closes and you are returned to Application Modeller.

Open the Blue Prism Smart Vision Editor from Application Modeller

The Smart Vision Editor can be reopened for elements that have been previously spied using Smart Vision mode.

Open existing screenshot

To open the Blue Prism Smart Vision Editor to view and edit an element that has been previously spied using Smart Vision mode:

1. In Application Modeller, select an element in the tree view that has been spied using Smart Vision mode.
2. Click Smart Vision.

The Blue Prism Smart Vision Editor opens, displaying the element overlaying the existing screenshot. You can edit the element, and, if required, you can add more elements to the screenshot, to apply to the parent element.

The bold option in the Smart Vision drop-down is the default option. The Smart Vision button automatically initiates the default option. If you require the non-default option, click the arrow next to Smart Vision and select the required option from the drop-down.

Open new screenshot

Opening a new screenshot replaces the current screenshot. To view the element overlaying a new screenshot:

1. In Application Modeller, select an element in the tree view that has been spied using Smart Vision mode.
2. Click the dropdown arrow next to Smart Vision.
3. Select Open new screenshot.

The Identification tool displays in Smart Vision mode. You cannot change the spy mode.

4. Press Ctrl and click the highlighted application to take a new screenshot.

The Blue Prism Smart Vision Editor opens displaying the element overlaying the new screenshot. You can now collect new attribute information from the element and apply it to the model.

Smart Vision Editor overview

The Smart Vision Editor consists of the following components:

1. Element list – Lists the current Smart Vision parent element and any Smart Vision children elements. The Element list in the Smart Vision Editor does not reflect the Element list in Application Modeller. Only Smart Vision elements are displayed, and no child elements can be nested inside other child elements.

2. Toolbar – Controls the size and display of the canvas, and allows you to delete whole elements from the element list.

3. Canvas – The application you have spied, highlighted with elements that Smart Vision has identified. Select an element to open the details in the Element properties and begin applying anchors to make an element unique.

4. Element properties – Details of the currently selected element. This section is empty if no element is selected. When an element is selected, Deselect and Confirm buttons display at the bottom of the pane, allowing you to cancel, or confirm and save any changes.

Smart Vision basics

The following information walks you through how to adjust your view, select and deselect an element, check if an element is unique, test the robustness of a unique element match, and delete an element.

For information about different types of elements, see Smart Vision element types.

For information about using anchors, see Smart Vision anchoring.

For best practice guidance, see Smart Vision best practice.

Adjust your view

To zoom in on a specific element:

1. Click Zoom in as many times as needed to see the element clearly.
2. Move the scroll bar until the element in central on the canvas.

To reset your view to see the whole screenshot, use one of the following methods:

• Click Zoom out until the whole screenshot is visible.
• Click Zoom to fit.

When the Smart Vision Editor is first opened, the default zoom level is Zoom to fit, with the whole screenshot visible without scrolling, and the window maximized.

Select and deselect an element

When no element is selected, you may see one or more of the following visual overlays:

• Boxes with yellow dotted outline – A field that Smart Vision has identified.

• Green box with solid outline – A previously configured element that is unique.

• Red box with solid outline – A previously configured element that is not unique.

All the elements recognized by Smart Vision are highlighted in dotted yellow borders and can be selected.

To select an element:

1. Hover over any element until its border turns light gray.
2. Click the highlighted element.

When an element is selected, the Element properties area is automatically populated with information about the selected element, and the selected element is indicated on the canvas as follows:

Visual overlay	Indicates	Description
Blue box with solid outline	Selected element	The currently selected element. The element is shown with a blue tag to show the element type. For more information, see Smart Vision element types. A warning tag is shown if the element is not uniquely identified.

3. If the element is unique, click Confirm to add the element to the Element list.

To cancel any changes and deselect an element:

1. Click Cancel.

Check if a selected element is unique

1. Check the message at the top of the Element properties area.

This message tells you if the element is unique, and advises you of your next steps.

2. If the selected element is unique, click Confirm to add it to Application Modeller and return to the element selection screen.

If an element is not identified as unique, you can apply anchors to create a spatial relationship that helps to differentiate the element from any similar elemnts that may be preventing it from being identified as unique. For more information, See Smart Vision anchoring.

Adjusting the text element properties for text elements may also help to make the element unique. See Edit text element properties.

Test that unique elements can be highlighted in Application Modeller

1. Confirm any unique elements, and close Smart Vision Editor to return to Application Modeller.

In the Application Modeller Element List pane, expand the application that you spied using Smart Vision.

2. Select your Smart Vision individual elements from the Elements pane one at a time, and click Highlight element.

A red box should display around each element.

If the element is not successfully highlighted:

1. Click Open editor.

2. You are returned to Smart Vision Editor.

3. Add anchors to create a more robust unique element. For more information, See Smart Vision anchoring.

Delete an element

1. Select an element in the Element list
2. Click Delete element.

You cannot use Delete element in the Smart Vision Editor if a Smart Vision parent (screenshot) is selected. This would need to be performed within Application modeller.