Test Case Execution

Alyvix test cases you have built with Alyvix Editor can be run using Alyvix Robot. Robot will also execute any individual test case objects you have created with Alyvix Designer along with Alyvix objects you have set up with Alyvix Selector.

In a production environment, the typical use case is to create a set of test cases once, and then repeatedly run those test cases continuously (ideally as quickly as possible). Examples include monitoring the usability of proprietary clients, streamed applications, and web pages on a remote server. A test case may measure responsiveness every 5 minutes, for instance, and send timing and other data to a monitoring system in the output format it expects.

Before you can use Alyvix in a production environment, however, you will first need to iteratively develop and improve your test cases. Here human readable output is more important, and accordingly there is no output format option within Alyvix Editor.

There are two principal modes for interacting with Alyvix Robot to ensure test cases work properly during the development phase:

  • Calling Robot from the Command Prompt, passing it the name of a test case object, using the default --mode option

  • Calling Robot from within Alyvix Editor via the runblue button to run the main script

In either case, one test case is executed at a time.

Launching Alyvix Robot from the Command Prompt

When Robot executes a test case via the command prompt, it first checks to see if one or more test case object names were passed via the -o parameter. If so, Robot will execute those test case objects in sequence. Otherwise, if the -o parameter is not specified and the .alyvix file test case contains a script previously defined by Editor in its script field, then Robot will run that script.

Adding the verbosity (-v) parameter will provide additional information that can help you should you need to debug your test cases (see the section CLI Output Format below).

Alyvix Robot can be run with the following command:

C:\Alyvix\testcases> alyvix_robot

If you used Alyvix Editor to create a file named start-test.alyvix containing a a test case object named start, you can run it with this command:

C:\Alyvix\testcases> alyvix_robot -f start-test -o start

Robot also allows you to execute multiple test case objects in sequence by putting them in order in quotation marks after the --object parameter, as long as all those objects exist in the test case:

C:\Alyvix\testcases> alyvix_robot -f start-test -o "start settings"

The following options are available:

Option

Alias

Argument

Description

--args

-a

<strs>

Supply one or more strings to use in the String field of a test case object in Designer

--filename

-f

<name>

Supply the file name with or without extension

--key

-k

<key>

Supply a private key for use with encryption

--mode

-m

<name>

alyvix — CLI output format for humans (default)

nagios — Nagios output (see below)

nats-influxdb — NATS to InfluxDB (see below)

--object

-o

<name>

Supply the Object name(s)

--verbose

-v

<n>

Set the verbosity level for debugging output ranging from 0 (min) to 2 (max)

0: Records start/stop timestamps, state and time measures for each object (with measure option enabled)

1: Also logs Alyvix actions

2: Also creates screenshots and annotated screenshots as separate .png files in the same directory

What Alyvix Robot Returns

Alyvix uses the following industry-standard return values for monitoring systems:

Error Level

Label

Meaning

0

OK

The service responded and the results were within expectations

1

WARNING

Action should be taken to prevent a likely near-term problem from becoming more serious

2

CRITICAL

A significant incident has already occurred and should be handled immediately

Using the --mode option, you can specify the format for the information returned (defaults to CLI output mode).

CLI Output Format

When run from the command prompt like this:

C:\Alyvix\testcases> alyvix_robot -f start-test

Alyvix Robot will both (a) display a short log describing basic events and timing data, and (b) create a new file based on the original test case, but with more detailed time measures added. A timestamp corresponding to the moment of execution will be appended to the file name:

<filename>_<full-timestamp>.alyvix

For example:

start-test_20191220_145228_UTC+0100.alyvix

When run from the command prompt with the default -m alyvix parameter, Alyvix Robot will return human-readable output like the following when successful:

2019/12/12 18:24:20.405: start starts
2019/12/12 18:24:21.949: open DETECTED in 0.0s (+/-0.060)
2019/12/12 18:24:21.950: start ends OK, taking 1.544s.

If it fails instead, the output will appear like this:

2019/12/12 18:37:41.448: start-test starts
2019/12/12 18:37:51.762: settings FAILED after 10s
2019/12/12 18:37:51.763: start-test ends FAILED because of settings, taking 10.315s.

If you have enabled the break flag in Alyvix Selector for a given test case object, then if that test case object fails, no further test case objects will be executed. If it is not set, then the test case object will instead be skipped after its timeout has expired.

When run from the Windows command prompt, you can access the return value as follows:

C:\Alyvix\testcases> echo %errorlevel%
0

Nagios Output Format

When Alyvix Robot is run from the command prompt, the --mode nagios command option will generate performance data in Nagios message format . The main status result for the monitoring check will be one of the following values:

test_case_output_status

Description

OK

When the test case exits with OK

WARNING

When the test case exits with OK, but the performance of at least 1 object is greater than its warning threshold and still lower than its critical threshold

CRITICAL

When the test case exits with OK, but the performance of at least 1 object is greater than its critical threshold and still lower than its threshold threshold

CRITICAL

Whenever the test case exits with FAILED

The time measurements for Nagios are specified as follows (note that object_timeout_s for SKIPPED must be converted to milliseconds):

Test Case Object Status

Format

JSON Path

DETECTED

<object_performance_ms>ms

object_name>measure>exit>true

SKIPPED

<object_timeout_s>ms

object_name>measure>exit>false

FAILED

ms       (no value)

object_name>measure>exit>fail

NOT EXECUTED

ms       (no value)

object_name>measure>exit>not_executed

First nagios output line:

<test_case_output_status>: <test_case_output_message> | duration=<test_case_duration_s>s;;; <object_01_name>=<object_01_performance_ms>ms;<object_01_warning_s>s;<object_01_critical_s>s;; <object_02_name>=<object_02_performance_ms>ms;<object_02_warning_s>s;<object_02_critical_s>s;;

Second nagios output line:

FAILED transactions (from first to last): <failed_object_01_name>; <failed_object_02_name>

Third nagios output line:

NOT EXECUTED transactions: <not_executed_object_01_name>; <not_executed_object_02_name>

NATS-InfluxDB Output Format

Publish all measures to an InfluxDB database through a NATS channel. You will need the NATS server IP, port, subject name, and measurement name.

alyvix_robot -f vt -m "nats-influxdb <nats_streaming_server_ip>:<port> <influxdb_subject_name> <influxdb_measurement_name>"

Alyvix Cipher for Encryption

You can use a combination of cipher and private key to protect sensitive information such as user names and passwords when for instance you need to log in to a login-protected application or web site.

To encrypt a cipher, supply the text to be encrypted and your private key:

alyvix_cipher -e <text_string_to_encrypt> -k <private_key>

You can also decrypt a cipher as follows:

alyvix_cipher -d <text_string_to_decrypt> -k <private_key>

To use the encrypted text string, put it in the string box of an object component.

You can then run a test case with encrypted strings by supplying the private key:

alyvix_robot -f <test_case_name> -k <private_key>