Campaigns¶
Campaigns shall be launched with the ‘run-campaign.py’ script.
usage: run-campaign.py [-h] [--config-file CONFIG_PATH]
[--config-value KEY VALUE] [--debug-class DEBUG_CLASS]
[--doc-only] [--issue-level-error ISSUE_LEVEL]
[--issue-level-ignored ISSUE_LEVEL]
[--outdir OUTDIR_PATH] [--dt-subdir]
[--extra-info ATTRIBUTE_NAME]
TEST_SUITE_PATH [TEST_SUITE_PATH ...]
Scenario campaign execution.
positional arguments:
TEST_SUITE_PATH Test suite file(s) to execute.
optional arguments:
-h, --help Show this help message and exit.
--config-file CONFIG_PATH
Input configuration file path. This option may be
called several times.
--config-value KEY VALUE
Single configuration value. This option may be called
several times.
--debug-class DEBUG_CLASS
Activate debugging for the given class.
--doc-only Generate documentation without executing the test(s).
--issue-level-error ISSUE_LEVEL
Define the issue level from and above which known
issues should be considered as errors. None by
default, i.e. all known issues are considered as
warnings.
--issue-level-ignored ISSUE_LEVEL
Define the issue level from and under which known
issues should be ignored. None by default, i.e. no
known issue ignored by default.
--outdir OUTDIR_PATH Output directory to store test results into. Defaults
to the current directory.
--dt-subdir Do not store the test results directly in OUTDIR_PATH,
but within a subdirectory named with the current date
and time.
--extra-info ATTRIBUTE_NAME
Scenario attribute to display for extra info when
displaying results. This option may be called several
times to display more info.
Test suite files¶
Test suite files are text files that describe the scenario files to execute, or not to execute.
Example from the demo.suite test suite file:
# This is a sample campaign description file.
# White-list:
+ *.py
# Black-list:
- htmltestlib.py
- run-demo.py
Type of line |
Syntax |
Effects |
|---|---|---|
Comment |
Starts with a ‘#’ character. |
No effect. |
White list |
Starts with a ‘+’ character, or no starter character, followed by a path or a glob-style pattern. |
Describes one (or several) script path(s) of scenario(s) to execute. When the path is relative, it is computed from the test suite file directory. |
Black list |
Starts with a ‘-’ character, followed by a path or a glob-style pattern. |
Describes one (or several) script path(s) to remove from the white list constituted by the preceding lines. When the path is relative, it is computed from the test suite file directory. |
Tip
White-list lines after a black-list line may restore script paths avoided by the latter.
Campaign execution¶
Test suite files are executed one after the others, in the order given by the command line.
A summary of the tests executed is displayed in the end.
$ ./bin/run-campaign.py demo/demo.suite
CAMPAIGN
------------------------------------------------
TEST SUITE 'demo/demo.suite'
----------------------------------------------
Executing 'demo/commutativeaddition.py'
DEBUG Log file: 'out/commutativeaddition.log'
DEBUG JSON report: 'out/commutativeaddition.json'
Executing 'demo/commutativeadditions.py'
DEBUG Log file: 'out/commutativeadditions.log'
DEBUG JSON report: 'out/commutativeadditions.json'
Executing 'demo/handlers.py'
DEBUG Log file: 'out/handlers.log'
DEBUG JSON report: 'out/handlers.json'
Executing 'demo/htmllogin.py'
DEBUG Log file: 'out/htmllogin.log'
DEBUG JSON report: 'out/htmllogin.json'
Executing 'demo/loggingdemo.py'
DEBUG Log file: 'out/loggingdemo.log'
DEBUG JSON report: 'out/loggingdemo.json'
END OF TEST SUITE 'demo/demo.suite'
----------------------------------------------
Number of test cases: 5
Number of tests in error: 0
Number of tests with warnings: 0
Number of steps: 15/15
Number of actions: 39/39
Number of results: 2/2
Time: HH:MM:SS.mmmmmm
END OF CAMPAIGN
------------------------------------------------
Number of test suites: 1
DEBUG JUnit report: 'out/campaign.xml'
------------------------------------------------
INFO TOTAL Status Steps Actions Results Time
INFO 5 tests, 0 failed, 0 with warnings 15/15 39/39 2/2 HH:MM:SS.mmmmmm
------------------------------------------------
INFO demo/commutativeaddition.py SUCCESS 4/4 4/4 1/1 HH:MM:SS.mmmmmm
INFO demo/commutativeadditions.py SUCCESS 3/3 3/3 0/0 HH:MM:SS.mmmmmm
INFO demo/handlers.py SUCCESS 2/2 2/2 0/0 HH:MM:SS.mmmmmm
INFO demo/htmllogin.py SUCCESS 1/1 3/3 1/1 HH:MM:SS.mmmmmm
INFO demo/loggingdemo.py SUCCESS 5/5 27/27 0/0 HH:MM:SS.mmmmmm
Campaign reports¶
The --outdir option specifies the directory where the execution reports should be stored.
--dt-subdir option
In conjonction with it, the --dt-subdir option tells the ‘run-campaign.py’ launcher to create a date/time subdirectory in the output directory.
The date/time subdirectory is formed on the ‘YYYY-MM-DD_HH-MM-SS’ pattern.
For each scenario executed, a JSON report is stored in the output directory.
Eventually, a campaign report is generated in the XML JUnit format.
<?xml version="1.0" encoding="utf-8"?>
<testsuites actions-executed="39" actions-total="39" disabled="0" errors="0" failures="0" results-executed="2" results-total="2" steps-executed="15" steps-total="15" tests="5" time="SSS.mmmmmm">
<testsuite actions-executed="39" actions-total="39" disabled="0" errors="0" failures="0" id="0" name="demo/demo.suite" results-executed="2" results-total="2" skipped="0" steps-executed="15" steps-total="15" tests="5" time="SSS.mmmmmm" timestamp="YYYY-MM-DDTHH:MM:SS.mmmmmm+HH:MM">
<testcase actions-executed="4" actions-total="4" classname="demo/commutativeaddition.py" name="demo/commutativeaddition.py" results-executed="1" results-total="1" status="SUCCESS" steps-executed="4" steps-total="4" time="SSS.mmmmmm">
<link href="out/commutativeaddition.log" rel="log" type="text/plain"/>
<link href="out/commutativeaddition.json" rel="report" type="application/json"/>
<system-out>SCENARIO 'demo/commutativeaddition.py'
------------------------------------------------
STEP#1: Initial conditions (demo/commutativeaddition.py:18:CommutativeAddition.step000)
------------------------------------------------
ACTION: Let a = 1, and b = 3
EVIDENCE: -> a = 1
EVIDENCE: -> b = 3
STEP#2: a + b (demo/commutativeaddition.py:25:CommutativeAddition.step010)
------------------------------------------------
ACTION: Compute (a + b) and store the result as result1.
EVIDENCE: -> result1 = 4
STEP#3: b + a (demo/commutativeaddition.py:32:CommutativeAddition.step020)
------------------------------------------------
ACTION: Compute (b + a) and store the result as result2.
EVIDENCE: -> result2 = 4
STEP#4: Check (demo/commutativeaddition.py:39:CommutativeAddition.step030)
------------------------------------------------
ACTION: Compare result1 and result2.
RESULT: result1 and result2 are the same.
EVIDENCE: -> 4 == 4
END OF 'demo/commutativeaddition.py'
------------------------------------------------
Status: SUCCESS
Number of STEPs: 4/4
Number of ACTIONs: 4/4
Number of RESULTs: 1/1
Time: HH:MM:SS.mmmmmm
</system-out>
</testcase>
<testcase actions-executed="3" actions-total="3" classname="demo/commutativeadditions.py" name="demo/commutativeadditions.py" results-executed="0" results-total="0" status="SUCCESS" steps-executed="3" steps-total="3" time="SSS.mmmmmm">
<link href="out/commutativeadditions.log" rel="log" type="text/plain"/>
<link href="out/commutativeadditions.json" rel="report" type="application/json"/>
<system-out>SCENARIO 'demo/commutativeadditions.py'
------------------------------------------------
STEP#1: Both positive members (demo/commutativeadditions.py:16:CommutativeAdditions.step010)
------------------------------------------------
ACTION: Launch the CommutativeAddition scenario with 4 and 5 for inputs.
| SCENARIO 'demo/commutativeaddition.py'
| ------------------------------------------------
|
| STEP#1: Initial conditions (demo/commutativeaddition.py:18:CommutativeAddition.step000)
| ------------------------------------------------
| ACTION: Let a = 4, and b = 5
| EVIDENCE: -> a = 4
| EVIDENCE: -> b = 5
|
| STEP#2: a + b (demo/commutativeaddition.py:25:CommutativeAddition.step010)
| ------------------------------------------------
| ACTION: Compute (a + b) and store the result as result1.
| EVIDENCE: -> result1 = 9
|
| STEP#3: b + a (demo/commutativeaddition.py:32:CommutativeAddition.step020)
| ------------------------------------------------
| ACTION: Compute (b + a) and store the result as result2.
| EVIDENCE: -> result2 = 9
|
| STEP#4: Check (demo/commutativeaddition.py:39:CommutativeAddition.step030)
| ------------------------------------------------
| ACTION: Compare result1 and result2.
| RESULT: result1 and result2 are the same.
| EVIDENCE: -> 9 == 9
|
| END OF 'demo/commutativeaddition.py'
STEP#2: Positive and negative members (demo/commutativeadditions.py:23:CommutativeAdditions.step020)
------------------------------------------------
ACTION: Launch the CommutativeAddition scenario with -1 and 3 for inputs.
| SCENARIO 'demo/commutativeaddition.py'
| ------------------------------------------------
|
| STEP#1: Initial conditions (demo/commutativeaddition.py:18:CommutativeAddition.step000)
| ------------------------------------------------
| ACTION: Let a = -1, and b = 3
| EVIDENCE: -> a = -1
| EVIDENCE: -> b = 3
|
| STEP#2: a + b (demo/commutativeaddition.py:25:CommutativeAddition.step010)
| ------------------------------------------------
| ACTION: Compute (a + b) and store the result as result1.
| EVIDENCE: -> result1 = 2
|
| STEP#3: b + a (demo/commutativeaddition.py:32:CommutativeAddition.step020)
| ------------------------------------------------
| ACTION: Compute (b + a) and store the result as result2.
| EVIDENCE: -> result2 = 2
|
| STEP#4: Check (demo/commutativeaddition.py:39:CommutativeAddition.step030)
| ------------------------------------------------
| ACTION: Compare result1 and result2.
| RESULT: result1 and result2 are the same.
| EVIDENCE: -> 2 == 2
|
| END OF 'demo/commutativeaddition.py'
STEP#3: Both negative members (demo/commutativeadditions.py:30:CommutativeAdditions.step030)
------------------------------------------------
ACTION: Launch the CommutativeAddition scenario with -1 and -7 for inputs.
| SCENARIO 'demo/commutativeaddition.py'
| ------------------------------------------------
|
| STEP#1: Initial conditions (demo/commutativeaddition.py:18:CommutativeAddition.step000)
| ------------------------------------------------
| ACTION: Let a = -1, and b = -7
| EVIDENCE: -> a = -1
| EVIDENCE: -> b = -7
|
| STEP#2: a + b (demo/commutativeaddition.py:25:CommutativeAddition.step010)
| ------------------------------------------------
| ACTION: Compute (a + b) and store the result as result1.
| EVIDENCE: -> result1 = -8
|
| STEP#3: b + a (demo/commutativeaddition.py:32:CommutativeAddition.step020)
| ------------------------------------------------
| ACTION: Compute (b + a) and store the result as result2.
| EVIDENCE: -> result2 = -8
|
| STEP#4: Check (demo/commutativeaddition.py:39:CommutativeAddition.step030)
| ------------------------------------------------
| ACTION: Compare result1 and result2.
| RESULT: result1 and result2 are the same.
| EVIDENCE: -> -8 == -8
|
| END OF 'demo/commutativeaddition.py'
END OF 'demo/commutativeadditions.py'
------------------------------------------------
Status: SUCCESS
Number of STEPs: 3/3
Number of ACTIONs: 3/3
Number of RESULTs: 0/0
Time: HH:MM:SS.mmmmmm
</system-out>
</testcase>
<testcase actions-executed="2" actions-total="2" classname="demo/handlers.py" name="demo/handlers.py" results-executed="0" results-total="0" status="SUCCESS" steps-executed="2" steps-total="2" time="SSS.mmmmmm">
<link href="out/handlers.log" rel="log" type="text/plain"/>
<link href="out/handlers.json" rel="report" type="application/json"/>
<system-out>SCENARIO 'demo/handlers.py'
------------------------------------------------
DEBUG 'scenario.before-test' handler called with test <Handlers 'demo/handlers.py'>
DEBUG 'scenario.before-step' handler called with step <StepDefinition 'Handlers.step010'>
STEP#1: `UserEvent.FOO` event triggering (demo/handlers.py:55:Handlers.step010)
------------------------------------------------
ACTION: Trigger the `UserEvent.FOO` event, with the following parameters: a=1 and b='bar'.
DEBUG 'foo' handler called with {'a': 1, 'b': 'bar'}
DEBUG 'scenario.after-step' handler called with step <StepDefinition 'Handlers.step010'>
DEBUG 'scenario.before-step' handler called with step <StepDefinition 'Handlers.step020'>
STEP#2: `UserEvent.FOO` event triggering (demo/handlers.py:61:Handlers.step020)
------------------------------------------------
ACTION: Trigger the `UserEvent.FOO` event, with the following parameters: a=2 and b='baz'.
DEBUG 'foo' handler called with {'a': 2, 'b': 'baz'}
DEBUG 'scenario.after-step' handler called with step <StepDefinition 'Handlers.step020'>
DEBUG 'scenario.after-test' handler called with test <Handlers 'demo/handlers.py'>
END OF 'demo/handlers.py'
------------------------------------------------
Status: SUCCESS
Number of STEPs: 2/2
Number of ACTIONs: 2/2
Number of RESULTs: 0/0
Time: HH:MM:SS.mmmmmm
</system-out>
</testcase>
<testcase actions-executed="3" actions-total="3" classname="demo/htmllogin.py" name="demo/htmllogin.py" results-executed="1" results-total="1" status="SUCCESS" steps-executed="1" steps-total="1" time="SSS.mmmmmm">
<link href="out/htmllogin.log" rel="log" type="text/plain"/>
<link href="out/htmllogin.json" rel="report" type="application/json"/>
<system-out>SCENARIO 'demo/htmllogin.py'
------------------------------------------------
STEP#1: Login screen (demo/htmllogin.py:13:TestLoginPage.step010_loginscreen)
------------------------------------------------
ACTION: Type the login.
INFO [edit[@id='login']] Typing text 'john'
ACTION: Type the password.
INFO [edit[@id='password']] Typing text '0000'
ACTION: Click on the OK button.
INFO [edit[@id='submit']] Clicking on the button
RESULT: The login page says hello to the user.
END OF 'demo/htmllogin.py'
------------------------------------------------
Status: SUCCESS
Number of STEPs: 1/1
Number of ACTIONs: 3/3
Number of RESULTs: 1/1
Time: HH:MM:SS.mmmmmm
</system-out>
</testcase>
<testcase actions-executed="27" actions-total="27" classname="demo/loggingdemo.py" name="demo/loggingdemo.py" results-executed="0" results-total="0" status="SUCCESS" steps-executed="5" steps-total="5" time="SSS.mmmmmm">
<link href="out/loggingdemo.log" rel="log" type="text/plain"/>
<link href="out/loggingdemo.json" rel="report" type="application/json"/>
<system-out>SCENARIO 'demo/loggingdemo.py'
------------------------------------------------
STEP#1: Logging with the main logger (demo/loggingdemo.py:22:LoggingScenario.step010)
------------------------------------------------
ACTION: Log messages of different log levels with the main logger.
ERROR This is an error!!!
WARNING This is a warning!
INFO This is information.
DEBUG This is debug.
STEP#2: Logging with the scenario instance (demo/loggingdemo.py:31:LoggingScenario.step020)
------------------------------------------------
ACTION: Log messages of different log levels with the scenario itself.
ERROR [demo/loggingdemo.py] This is an error!!!
WARNING [demo/loggingdemo.py] This is a warning!
INFO [demo/loggingdemo.py] This is information.
DEBUG [demo/loggingdemo.py] This is debug.
STEP#3: Logging with a class logger (demo/loggingdemo.py:40:LoggingScenario.step030)
------------------------------------------------
ACTION: Log messages of different log levels with the class logger instance.
ERROR [My logger] This is an error!!!
WARNING [My logger] This is a warning!
INFO [My logger] This is information.
ACTION: Activate debugging for the class logger instance.
ACTION: Log a debug message again with the class logger instance.
DEBUG [My logger] This is debug again.
STEP#4: Class logger indentation (demo/loggingdemo.py:55:LoggingScenario.step110)
------------------------------------------------
ACTION: Log something with the class logger.
INFO [My logger] Hello
ACTION: Push indentation to the class logger.
ACTION: Log something with the class logger.
INFO [My logger] Hello
ACTION: Push indentation to the class logger.
ACTION: Log something with the class logger.
INFO [My logger] Hello
ACTION: Push indentation to the class logger.
ACTION: Log something with the class logger.
INFO [My logger] Hello
ACTION: Pop indentation from the class logger.
ACTION: Log something with the class logger.
INFO [My logger] Hello
ACTION: Reset the class logger indentation.
ACTION: Log something with the class logger.
INFO [My logger] Hello
STEP#5: Main logger indentation (demo/loggingdemo.py:76:LoggingScenario.step120)
------------------------------------------------
ACTION: Log something with the main logger.
INFO Hello
ACTION: Push indentation to the main logger.
ACTION: Log something with the main logger.
INFO Hello
ACTION: Push indentation to the main logger.
ACTION: Log something with the main logger.
INFO Hello
ACTION: Push indentation to the main logger.
ACTION: Log something with the main logger.
INFO Hello
ACTION: Pop indentation from the main logger.
ACTION: Log something with the main logger.
INFO Hello
ACTION: Reset the main logger indentation.
ACTION: Log something with the main logger.
INFO Hello
END OF 'demo/loggingdemo.py'
------------------------------------------------
Status: SUCCESS
Number of STEPs: 5/5
Number of ACTIONs: 27/27
Number of RESULTs: 0/0
Time: HH:MM:SS.mmmmmm
</system-out>
</testcase>
</testsuite>
</testsuites>
XML JUnit format
A reference documentation could not be found for the XML JUnit format.
In spite of, the [CUBIC] page can be noted as one of the best resources on that topic.