Designer: Component Tree

You can use Alyvix Designer to create selections on the screen. When these regions are detected, whether as images or GUI objects, you can use additional neighboring screen elements to either confirm you have the correct region, or identify other areas that you can then act on, like a nearby button.

The component tree is composed of three types of components:

  • A Root element, which holds a captured screen to serve as a basis for visually recognizing parts of the screen.

  • Groups, which indicate the principal component that needs to be recognized. A maximum of three groups can be created per test case.

  • Components are the additional areas that can be used to confirm what is being recognized (especially when there are multiple similar target objects), such as a text label next to a button, or when the action can be on a separate GUI object, like a slider. A maximum of four components can be assigned to each group.

Structure of the Component Tree

An empty component tree

Alyvix Designer captures the entire screen to use as a basis for locating visual elements. This screen capture is represented in the component tree with the capital letter S as shown above. Initially, before you have added any components, this root element will have no children.

When first started, Designer will default to the first of the three available groups, indicated by the purple/red color. The three color sets (red, green and blue) are only intended to distinguish the three groups, and a group’s elements may in fact change color if you reorder or remove one.

A tree with a single root, group and component

If you now select a region, a copy of that region will appear in red in the component tree. This region represents a potential group of subregions called components, although depending on what you need to do, you may not need to create any components for a group.

Each group and component is one row consisting of a type icon (see Object Types) and an image of the area around the object. A right-click on a row will display a menu of actions for that object.

Object Recognition Types

Alyvix’s visual recognition capability is based on two well-known open source systems:

  • The OpenCV computer vision project, which allows Alyvix to compare screen regions graphically

  • The Tesseract text recognition project, which lets Alyvix find writing in regions and convert it to computer-readable text, regardless of the font and font size used

A tree with a single root, group and component

Alyvix uses these tools to visually detect three types of objects:

  • Image: Given a region of interest on the screen, Alyvix will look for a particular shape and/or color within the matrix of pixels in that RoI. For instance, it may be a particular system or application icon, a shape like a circle regardless of its color, or a box that changes from one color to another. The “photograph” icon above represents an Image object.

  • Rectangle (Rect): Allows Alyvix to find buttons, boxes and windows that have a rectangular shape. The “bounding box” icon represents a Rect object.

  • Text: Looks for text near another located object. Because text recognition over the entire computer screen would require a significant amount of time, a text object cannot be the main component of a group. Alyvix will always look for text near the already-located main component. the letter “A” icon above represents a Text object.

Creating Groups and Components

When the Designer panel is not visible (you haven’t pressed the ESC key), the crosshairs show you where you can create a selection or subselection on the screen. The color will indicate which group any newly selected region will belong to.

You can change to a new group simply by pressing the number associated with it (1, 2 or 3) at which point the crosshairs will change to indicate the color of the new group. Designer will automatically switch to a new group after you add a fourth component.

If the component tree is empty, the first region you select will become the first group in the tree. As mentioned in the section Object Recognition Types, this region cannot be used for text recognition. If a group already exists and is selected, the new region will become a component in that group.

Warning

Although you can add up to add up to four components per group, all components in a given group must match a region on the screen before a match will be declared and any action(s) taken.

In addition to manually selecting the bounds of the region, you can right-click on any point on the screen and Alyvix will try to automatically determine (autocontour) the appropriate selection. You can also resize the bounding edges of any capture region by dragging them with the left mouse button.

You can continue to add components and new groups using the mouse and keyboard shortcuts without showing the Designer panel.

At any point you can switch between selecting regions and the Designer panel. Use the ESC key to open the Designer panel, and the EDIT button at the bottom left corner of the panel to return to the screen capture.

Component Tree Actions

You can modify the component tree at any time by right clicking on a row, which displays a contextual menu showing you the available options (some actions will not be shown if they cannot be used given the current configuration of the tree).

Descriptions of these actions, along with their keyboard shortcuts, can be found in the table below.

Action

Objects

Shortcut

Description

New

All objects

CTRL+N

Create a new component in the current group, in the next group if the current group is full, or a new group if no groups exist yet

Remove

All objects

CTRL-X

Remove the selected component, an entire group, or all groups at once if the root is selected

Duplicate

Group, Component

CTRL-D

Create a duplicate of the selected object if there is space available for more at that level

Detect as Image

Group, Component

CTRL-I

Change the object’s detection type to Image

Detect as Rect

Group, Component

CTRL-R

Change the object’s detection type to Rectangle

Detect as Text

Component

CTRL-T

Change the Component type to Text

Set as Main

Component

CTRL-M

Make the selected component the main group component (the component cannot be a text region)