Visual keywords

Keyword definition

The Alyvix button opens the selector of visual keywords (that has been previously defined by the user, otherwise no objects are listed) for the current test case (e.g. the test case alyvix_website, so that its keyword library would be AlyvixProxyalyvix_website). The selector shows the name and the type of each keyword: [IF] means Image Finder, [RF] means Rect Finder, [TF] means Text Finder, [OF] stays for Object Finder, [TS] means Text Scraper and [OS] stays for Object Scraper.

Note

Imported keywords from other Alyvix test case libraries (e.g. the keyword library of the citrix_outlook test case would be AlyvixProxycitrix_outlook) are not listed in the selector window. You can edit keywords of a certain test case just opening that test case

../_images/ride_05a_2-7-2_keyword_selector.png

The selector of Alyvix Finders allows you to select the type (image, rectangle, text, object) of a new keyword to define. The Delay spin control delays of a given amount of seconds the subsequent selection of components on screen. The graphic definition a visual keyword freezes your screen in order to select components. Increase the delay if you need time to trigger a certain state of the screen (e.g. opening a menu in an app dialog). You can sort and search keywords through the respective bars.

The Edit button brings to the Finder dialog of the selected keyword in order to modify its properties (e.g. component selections and region of interests, detection and interaction settings, performance thresholds). The Remove button deletes the selected keywords or, in case of Object Finders or Object Scrapers, crumbles them into its basic components (IF, RF, TF, TS).

Start defining a visual keyword means selecting its components on screen: one main selection is necessary, but then multiple sub selections are possible. Sub components are selected within their region of interests. The latter ROI are useful to link sub components to the main component. Each Alyvix Finder has a procedure to select its components and ROI. The important reasons for sub components and their ROI follow below. The detection of application transactions can be:

  • more complete: GUI rendering could be element-by-element, so it could be necessary to detect several components to check a completed transaction; moreover, it is possible to interact with several GUI elements and not just with one;
  • more robust: the same GUI element could be used twice, so it could be necessary to define a unique transaction selecting more components;
  • more flexible: the same GUI element could be rendered in a different position, so it is necessary to link sub to main components through ROI; moreover, ROI are also useful to restrict the screen area to process, in order to filter out the rest and to save computation time.

Shortcuts:

CTRL+C keyword selector Copy selected keyword name in Alyvix selector
CTRL+V keyword selector Paste copied keyword name from Alyvix selector
CTRL+D keyword selector Delete selected keywords in Alyvix selector

Image Finder main and subs

../_images/ride_07a_if_main_sub.png

An example GUI shows three identical stars and a possible transaction could be to interact with the third star, which is left to the green sign.

Selecting just the third star as a main component of an Image Finder is not enough. It would be an ambiguous transaction: the three stars have exactly the same aspect. Alyvix detects all the three stars and then interacts with the first one, because of the raster logic of the screen scanner.

To correctly define a unique keyword, it is necessary to select the green sign as a sub component. Remember that sub components of Image Finder are selected within their ROI (region of interests).

Note

The upper left corner of a ROI is linked with the upper left corner of the main component selection.

../_images/ride_07bb_if_main_sub.png
  1. Select the main component on the screen;
  2. select the ROI of the sub component 1 on the screen;
  3. select the sub component 1 within its ROI;
  4. eventually repeat 2. and 3. to add more sub components;
  5. press CTRL+O to set the IF keyword properties.

LEFT CLICK the mouse, hold and release to manually select components, to stretch and move them.

../_images/image_finder_selection_edge_stretching.gif

RIGHT CLICK the mouse to autocontour components.

../_images/image_finder_selection_autocontour.gif

Press SPACE to highlight component candidates and then RIGHT CLICK the mouse to autocontour them.

../_images/image_finder_selection_autocontour_highlight.gif

Other shortcuts:

RIGHT CLICK component ROI edges Push component ROI edges to infinity
CTRL + LEFT CLICK component SELections Reset component ROI edges near around selections inside
CTRL + RIGHT CLICK selected components Remove components both SELections and ROIs
CTRL+Z visual definition Undo visual selections in the visual selection mode
CTRL+Y visual definition Redo visual selections in the visual selection mode

Rect Finder main and subs

../_images/ride_07c_rf_main_sub.png

An example GUI shows three identical text boxes and a possible transaction could be interacting with the third text box, which is left to the OK button.

Selecting just the third text box as a main component of an Rect Finder is not enough. It would be an ambiguous transaction: the three text boxes have exactly the same aspect. Alyvix detects all the three text boxes and then interacts with the first one, because of the raster logic of the screen scanner.

To correctly define a unique keyword, it is necessary to select the OK button as a sub component. Remember that sub components of Rect Finder are selected within their ROI (region of interests).

Note

The upper left corner of a ROI is linked with the upper left corner of the main component selection.

../_images/ride_07db_rf_main_sub.png
  1. Select the main component on the screen;
  2. select the ROI of the sub component 1 on the screen;
  3. select the sub component 1 within its ROI;
  4. eventually repeat 2. and 3. to add more sub components;
  5. press CTRL+O to set the RF keyword properties.

LEFT CLICK the mouse, hold and release to manually select components, to stretch and move them.

../_images/rect_finder_selection_edge_stretching.gif

RIGHT CLICK the mouse to autocontour components.

../_images/rect_finder_selection_autocontour.gif

Press SPACE to highlight component candidates and then RIGHT CLICK the mouse to autocontour them.

../_images/rect_finder_selection_autocontour_highlight.gif

Other shortcuts:

RIGHT CLICK component ROI edges Push component ROI edges to infinity
CTRL + LEFT CLICK component SELections Reset component ROI edges near around selections inside
CTRL + RIGHT CLICK selected components Remove components both SELections and ROIs
CTRL+Z visual definition Undo visual selections in the visual selection mode
CTRL+Y visual definition Redo visual selections in the visual selection mode

Text Finder main and subs

../_images/ride_07e_tf_main_sub.png

An example GUI shows three identical Name text strings and a possible transaction could be interacting with the third Name text string, which is left to the Franz text string.

Selecting just the Name text string as a main component of a Text Finder is not enough. It would be an ambiguous transaction: the three Name text string have exactly the same aspect. Alyvix detects all the three Name text strings and interacts with the first one, because of the raster logic of the screen scanner.

To correctly define a unique keyword, it is necessary to select the Franz text string as a sub component. Remember that all the Text Finder component selections lie in their ROI (region of interests).

Note

In the TF case, also the main component lies in its ROI, which is linked with the upper left corner of the screen. That differs from the Image Finder and Rect Finder: in those cases the main component selection is defined within the entire screen.

Warning

The main component of a Text Finder is going to be searched within its ROI and not on the entire screen. This is the reason why a Text Finder must be only used as sub components of Object Finders.

../_images/ride_07fb_tf_main_sub.png
  1. Select the ROI of the main component on the screen;
  2. select the main component within its ROI;
  3. select the ROI of the sub component 1 on the screen;
  4. select the sub component 1 within its ROI;
  5. eventually repeat 3. and 4. to add more sub components;
  6. press CTRL+O to set the TF keyword properties.

LEFT CLICK the mouse, hold and release to manually select components, to stretch and move them.

../_images/text_finder_selection_edge_stretching.gif

RIGHT CLICK the mouse to autocontour components.

../_images/text_finder_selection_autocontour.gif

Press SPACE to highlight component candidates and then RIGHT CLICK the mouse to autocontour them.

../_images/text_finder_selection_autocontour_highlight.gif

Other shortcuts:

RIGHT CLICK component ROI edges Push component ROI edges to infinity
CTRL + LEFT CLICK component SELections Reset component ROI edges near around selections inside
CTRL + RIGHT CLICK selected components Remove components both SELections and ROIs
CTRL+Z visual definition Undo visual selections in the visual selection mode
CTRL+Y visual definition Redo visual selections in the visual selection mode

Object Finder main and subs

../_images/ride_07g_of_main_sub.png

An example GUI shows three identical stars and a possible transaction could be interacting with the third star, left to the Franz text string.

Selecting just the third star as a main component of an Image Finder is not enough. It would be an ambiguous transaction: the three stars have exactly the same aspect. Alyvix detects all the three stars and then interacts with the first one, because of the raster logic of the screen scanner.

To correctly define a unique keyword, it is necessary to select the Franz text string as a sub component. Firstly, define the star as an Image Finder and the Franz text string as a Text Finder. Secondly, select them as the main and a sub component of an Object Finder defining the ROI (region of interest) around the sub component to link it to the main component.

Note

The most common and useful Object Finder combinations are IF+TF and RF+TF.

../_images/ride_07hb_of_main_sub.png
  1. Select the main component;
  2. select the sub component 1;
  3. select the ROI of the sub component 1 on the screen;
  4. eventually repeat 2. and 3. to add more sub components;
  5. press CTRL+O to set the OF keyword properties.

The ROI of added components is autoselected (also after component editing).

../_images/object_finder_autoselect.gif

LEFT CLICK the mouse, hold and release to manually redraw component ROIs or to stretch it.

../_images/object_finder_roiredraw.gif

Other shortcuts:

RIGHT CLICK component ROI edges Push component ROI edges to infinity
CTRL + LEFT CLICK component SELections Reset component ROI edges near around selections inside
CTRL + RIGHT CLICK selected components Remove components both SELections and ROIs

Name, arguments and components

After having visually define a keyword, the first thing to do is to insert the keyword name in the Name text box.

The best practice is to type it lowercase, with underscores, no spaces and structured as follows: <application name>_<transaction name>_<transaction state> (e.g. citrix_loginform_ready, ax12_dynamicsax_closed). Append the _<keyword type> if you plan to use the keyword in an Object Finder (e.g. citrix_loginform_ready_img, citrix_loginform_ready_rct).

../_images/ride_11a_alyvix_2-4-1_gui_name.png

You can declare a number of argument variables as the keyword inputs. Then you can type arg1, arg2, etc. in the interaction text box of main and sub components. Alyvix will run the keyword taking its argument values from the test case editor using them: in the cells on the right from where you insert the keyword name, you can type the keyword arguments as text strings.

In your test case, type the keyword name and eventually its argument values:

<visual_keyword_name>

or

<visual_keyword_name> <arg1_string> <arg2_string> ..

Example:

  1. Type the keyword name and set the amount of its arguments in the Alyvix Finder dialog:
../_images/ride_11ac_alyvix_2-4-1_gui_arg.png
  1. Type arg1, arg2, etc. in the interaction text box or in the detection text box (in case of a Text Finder):
../_images/ride_11ad_alyvix_2-4-1_gui_arg.png

Warning

Untick Quotes if you use keyword arguments. Moreover, tick Encrypted if you use encrypted keyword arguments. You can encrypt strings with the Alyvix encryption tool.

  1. Type the keyword name (and eventually its arguments) in your test case:
../_images/ride_11ab_alyvix_2-4-1_gui_name.png

In the left pane of keyword dialogs there are all the components previously selected on screen: tick (or untick) them to visualize (or not) their selection and ROI (region of interests) on screen. Click on each component to set its own detection and interaction properties. Tick one or more components to remove them clicking the Remove button: click Select all or Deselect all for a quicker selection.

../_images/ride_11b_alyvix_2-7-0_gui_components.png

Detection settings

In the detection section of each Alyvix Finder, you can select 2 detection modes: Appeared and Disappeared. When a keyword will be executed, its mode affects the way Alyvix will detect it (i.e. the visual elements of a transaction) and report its performances.

../_images/ride_11c_alyvix_2-4-1_gui_detection.png

Selecting Appeared, Alyvix continuously tries and retries to detect the graphic elements on screen at a certain pace (by default this period is 0.5s, but it is customizable thanks to the system keywords Alyvix Config and Set Alyvix Info). That continues until the Timeout threshold will be reached: if the Break option is ticked, then the keyword breaks the test case, otherwise it returns False and lets the test to proceed. The latter option could be useful in case of transactions that not always happen (e.g. to manage popups).

On the other hand, Disappeared allows to detect the disappearance of graphic elements of screen (e.g. disappearance of an hourglass icon at the end of a loading).

The Performance check box enables or disables the performance measurement:

  1. Add Perfdata declares the keyword performance
  2. if the Performance setting is ticked, the keyword outputs its performance (with 0.1s of accuracy and 0.001s of precision)
  3. Print Perfdata collects and prints out all the available performance from the test case

In the Warning and Critical text boxes you can set the amount of seconds of these thresholds.

../_images/performance_thresholds.png

Interaction settings

In the interaction section of each Alyvix Finder, you can set an interaction mode for the selected main or sub component of the keyword that you are defining.

../_images/ride_11d_alyvix_2-7-0_gui_interaction.png

Selecting Click or Right Click, Alyvix will bring the mouse pointer over the detected component and press the left or the right button of the mouse respectively. It is possible to set the number of Clicks and adjust Delays as the amount of milliseconds between clicks.

You can also set an Interaction Point in order to click somewhere else from the center of the component: click on the Interaction Point button, set the point (continuously) clicking on the screen and press CTRL+O to confirm.

../_images/ride_11db_alyvix_2-4-1_gui_interaction.png

Click on the Reset Point button to reset the interaction point at the center of the component.

../_images/ride_11dc_alyvix_2-4-1_gui_interaction.png

Set the amount and direction of Scrolls to scroll at the cursor and/or focus position (e.g. web page).

Hold and Release is useful to drag and drop or slide GUI elements. There are 6 modes that you can select from the drop-down list on the right. The keyword execution will work as follows:

  • Hold: the pointer will press and hold the component;
  • Release: the pointer will release over the component;
../_images/hold_release.gif

Note

To drag and drop a GUI element (look at the animation above) it is necessary to set the Hold mode for a target component (e.g. file icon) and the Release mode for another destination component (e.g. folder icon)

  • Release Up: the pointer will press and hold the component, that will then be released towards up of an amount of pixels (to set in the spin box on the right);
  • Release Down: the pointer will press and hold the component, that will then be released towards down of an amount of pixels (to set in the spin box on the right);
  • Release Left: the pointer will press and hold the component, that will then be released towards left of an amount of pixels (to set in the spin box on the right);
  • Release Right: the pointer will press and hold the component, that will then be released towards right of an amount of pixels (to set in the spin box on the right).
../_images/release_displace.gif

For the latter 4 modes, select the amount of pixel displacement thanks to the spin box on the right: double click and insert the desired number of pixels as the release distance.

../_images/ride_11e_2-7-0_gui_interaction.png

Move will bring the mouse pointer over the detected component without pressing anything.

Selecting None, Alyvix will not interact with the component (the mouse pointer will not move or click anything).

At the bottom of the interaction section, typing settings take place. In the text box can be inserted text strings and shortcuts to send (e.g. bla, most of the time after a Click interaction somewhere).

Regular text strings can be typed together with shortcuts (e.g. bla{enter}). You can find a list of the most useful shortcuts in the description of Send Keys system keyword.

It is also possible to bring the keyword arguments as part of the keystrokes to send.

Warning

Remember to untick Quotes, to add one more Arguments (clicking the up arrow of the spin box) and finally type just an argument variable (e.g. arg1, arg2, etc.).

It is even possible to bring strings, shortcuts and arguments all together (e.g. arg1+"bla{tab}"+arg2+"bla{enter}").

Warning

Also in this case, remember to untick Quotes and to add one or more Arguments.

Delays [ms] sets the sleep intervals (in milliseconds) between keys. Duration [ms] sets how long (in milliseconds) keys are going to be pressed.

Alyvix Finders

The Alyvix Finders are the GUI tools to visually define application transactions. You can build Alyvix visual keywords with the Alyvix Finders. Finally, you can automate application transactions running Alyvix visual keywords in test cases. There are three Alyvix Finders, one for each type of visual element to detect on screen: images, rectangles and text. The Object Finder group together basic Alyvix Finders (IF, RF, TF) in order to detect and interact with different types of visual elements running one single keyword (e.g. a login form made of text boxes, their labels on a side and a brand logo in a corner).

After the selection of an Alyvix Finder and the selection of main and sub components, press CTRL+O to show the setting dialog. The Image, Rect and Text Finders have in common most of their settings:

The Image Finder has its own similarity threshold, the Rect Finder has sizing thresholds and the Text Finder has regular expression filters. The Object Finder dialog is designed to link together the previous basic Alyvix Finders.

Image Finder

Image Finders produce Alyvix visual keywords that are able to detect and interact with images (i.e. pixel matrixes). This is the Image Finder dialog:

../_images/ride_06a_2-7-0_image_finder.png

To define such keywords follow these steps:

  1. select the Image Finder in the selector of visual keywords;

  2. select main and sub components of the selected Finder;

  3. type the keyword name and eventually set the number of arguments;

  4. the Image Finder features a spin box to set the visual likelihood threshold for the selected component. You can set a number between 0 and 1, with two decimal places. The default value is 0.70 and it works just fine most of the cases;

    ../_images/ride_06ac_image_finder.png

Note

To disambiguate graphical elements in a transaction, it is better to add more components instead of increasing the visual likelihood threshold

  1. the Image Finder also features a check box to set the color detection for the selected component. By default, Alyvix does not take into account colors of visual elements, but in some cases that could be crucial (e.g. detect a red sign that becomes green);
  2. set the detection properties;
  3. set the interaction properties;
  4. click the OK button to save the keyword.

Rect Finder

Rect Finders produce Alyvix visual keywords that are able to detect and interact with rectangles (e.g. text boxes, rectangle buttons). This is the Rect Finder dialog:

../_images/ride_08a_2-7-0_rect_finder.png

To define such keywords follow these steps:

  1. select the Rect Finder in the selector of visual keywords;

  2. select main and sub components of the selected Finder;

  3. type the keyword name and eventually set the number of arguments;

  4. the Rect Finder features the controls to set the allowed boundaries of those rectangles that you want to detect:

    1. there are 2 modes to define valid rectangles: Sizing and Tolerance areas. Click on the radio button of one them;
    ../_images/rect_finder_tuning.gif
    1. tick the Show check box of the selected mode to see the ongoing tuning of valid rectangle areas;

    2. type integer numbers, click spin boxes or scroll mouse wheel to tune the violet area for the selected component. The rectangle contours on screen (or in region of interests for sub components) that fit into the allowed boundaries will be taken into account by Alyvix (i.e. during the keyword execution);

      ../_images/ride_08ab_rect_finder.png

    Note

    Sizing mode is generally better for text boxes and Tolerance mode for buttons

  5. set the detection properties;

  6. set the interaction properties;

  7. click the OK button to save the keyword.

Text Finder

Text Finders produce Alyvix visual keywords that are able to detect and interact with text (e.g. button text, icon text). This is the Text Finder dialog:

../_images/ride_09a_2-7-0_text_finder.png

To define such keywords follow these steps:

  1. select the Text Finder in the selector of visual keywords;
  2. select main and sub components of the selected Finder and press CTRL+O;

Warning

The Text Finder is designed to primarly work in the ROI (region of interest) of the main component as a fixed spot and not through out all the screen as the Image and Rect Finders. In fact, unlike the other Finders, you have also to select the ROI of the main component. Through to an Object Finder you have to search text in a position that is related to other graphic elements (i.e. images, rectangles)

  1. type the keyword name and eventually set the number of arguments;

  2. the Text Finder features the properties to set the regular expression for matching the text (e.g. label, chunk) that you want to detect;

    • In the Text box you can insert a case insensitive text string (e.g. name) as well as a regular expression (e.g. .*ame). During the keyword execution, Alyvix will try to properly match that entry with the text within the ROI of the selected component;

    • In the Text box you can also use arguments passing text strings or regular expressions from the keyword arguments. Add one or more arguements, type arg1, arg2, etc. in the Text box and untick Quotes;

    • Click on the Check button to have a preview of what the OCR scanner detects in the ROI of the selected component: CRITICAL means Alyvix is not able to match the regular expression you have provided, EXCELLENT instead means the text component can be found;

    Note

    As a general rule of thumb, try to wrap uniform text with ROI in order to obtain a more accurate character recognition. Sometimes though, you could suffer a poor character recognition (e.g. 8 instead of B, 1 instead of t): in these cases work on the regular expression to correct the detection

    • The characters from ROI pass through the WhiteList filter: the OCR scanner will limit the recognition to that set of characters. Moreover, the OCR scanner will interpret entire words from the language dictionary specified in the Lang box (type eng for English, ita for Italian and deu for German);

      ../_images/ride_09ab_2-7-0_text_finder.png
  3. set the detection properties;

  4. set the interaction properties;

  5. click the OK button to save the keyword.

Object Finder

The Object Finder is probably the most useful Alyvix Finder, because it links together multiple types of Alyvix visual keywords defining complex graphic objects to detect: an Object Finder considers its basic visual keywords as its main and sub components to search, inheriting their arguments as well. This is the Object Finder dialog:

../_images/ride_10a_2-7-0_object_finder.png

To define such keywords follow these steps:

  1. define the Image Finder, Rect Finder and Text Finder keywords that will be components of the Object Finder;

  2. select the Object Finder in the selector of visual keywords;

  3. type the keyword name;

  4. click on Set Main button to select the main component: a list of the defined Finders will appear from where choose an Image Finder [IF] or a Rect Finder [RF];

Note

It is not possible to choose a Text Finder [TF] as the main component of an Object Finder, because it would not make sense: the Text Finder is designed to primarly search text in the fixed ROI (region of interest) area of its main component

  1. click on Add Sub button to select a sub component: a list of the defined Finders will appear from where choose an Image Finders [IF], a Rect Finders [RF] and a Text Finders [TF]; you can add more sub components clicking on Add Sub every time;

Note

Select the main component and click on the Edit button to edit its definition; you can do the same for each sub component as well as redraw its ROI (ROI Redraw button) and remove it (Remove button)

../_images/ride_10b_object_finder.png
  1. set the detection properties;
  2. click the OK button to save the keyword.

You can build, for example, an Object Finder visual keyword made by an IF and a TF (with a regular expression as its argument). It is a powerful object: you can use the same keyword several time to interact with different text elements in the same menu, just changing the TF regex argument.

Also an Object Finder made of a RF and a TF (with its argument) is useful: you can use OF visual keywords to interact with different rectangle buttons, identically shaped, but differently labeled.

../_images/ride_10bb_object_finder.png
../_images/ride_10cb_object_finder.png

Alyvix Scrapers

Text Scraper

Text Scrapers produce Alyvix visual keywords that are able to scrap text (e.g. window titles, text chunks). This is the Text Scraper dialog:

../_images/ride_09b_2-7-0_text_scraper.png

To define such keywords follow these steps:

  1. select the regular Text Finder in the selector of visual keywords;

  2. select the main component of the scraper (i.e. ROI and selection) and press CTRL+O;

  3. type the keyword name and tick Scraper;

  4. the Text Scraper features the properties for scraping the text that you want;

    • Click on the Check button to have a preview of what the OCR scanner will scrap in the ROI of the main component;

    Note

    As a general rule of thumb, try to wrap uniform text with ROI in order to obtain a more accurate character scraping

    • The characters from ROI pass through the WhiteList filter: the OCR scanner will limit the scraping to that set of characters. Moreover, the OCR scanner will interpret entire words from the language dictionary specified in the Lang box (type eng for English, ita for Italian and deu for German);

      ../_images/ride_09bb_2-7-0_text_scraper.png
  5. click the OK button to save the keyword.

Object Scraper

An Object Scraper simply relies on the regular Object Finder to bind Image Finders, Rect Finders, Text Finders together with Text Scrapers.