Configuration database

The scenario framework provides a general configuration database.

It is available through the scenario.conf attribute.

Configuration nodes

The database configuration is a tree of sections, sub-sections, sub-sub-sections, … ending with final values.

The ConfigNode class describes a node in the resulting configuration tree, either a section or a final value.

Configuration tree & keys

Configuration keys are a simplified form of JSONPath: they are dot-separated strings, with the ability to index a single list item with a number between square brackets.

With the following sample data:

a:
  b:
    c: 55
x:
  y:
  - z: 100
  - z: 101
  - z: 102
  - z: 104

  • “a.b.c” points to the 55 value,

  • “x.y[2].z” points to the 102 value,

  • “a.b” points to the so-named sub-section (the corresponding data being a dict),

  • “x.y” points to the so-named list (the corresponding data being a list),

  • “” (empty string) points to root node.

Tip

Configuration keys may be passed as strings or string enums.

Loading and setting configurations through the command line

Configuration values are basically set through the command line, with the --config-file and/or --config-value options of test and campaign launchers.

usage: run-test.py [-h] [--config-file CONFIG_PATH] [--config-value KEY VALUE]
                   [--debug-class DEBUG_CLASS] [--doc-only]
                   [--issue-level-error ISSUE_LEVEL]
                   [--issue-level-ignored ISSUE_LEVEL]
                   [--json-report JSON_REPORT_PATH]
                   [--extra-info ATTRIBUTE_NAME]
                   SCENARIO_PATH [SCENARIO_PATH ...]

Scenario test execution.

positional arguments:
  SCENARIO_PATH         Scenario script(s) to execute.

optional arguments:
  -h, --help            Show this help message and exit.
  --config-file CONFIG_PATH
                        Input configuration file path. This option may be
                        called several times.
  --config-value KEY VALUE
                        Single configuration value. This option may be called
                        several times.
  --debug-class DEBUG_CLASS
                        Activate debugging for the given class.
  --doc-only            Generate documentation without executing the test(s).
  --issue-level-error ISSUE_LEVEL
                        Define the issue level from and above which known
                        issues should be considered as errors. None by
                        default, i.e. all known issues are considered as
                        warnings.
  --issue-level-ignored ISSUE_LEVEL
                        Define the issue level from and under which known
                        issues should be ignored. None by default, i.e. no
                        known issue ignored by default.
  --json-report JSON_REPORT_PATH
                        Save the report in the given JSON output file path.
                        Single scenario only.
  --extra-info ATTRIBUTE_NAME
                        Scenario attribute to display for extra info when
                        displaying results. Applicable when executing several
                        tests. This option may be called several times to
                        display more info.

Configuration files can be in one of the following formats:

Configuration file formats

Format

File extensions

INI (as handled by configparser)

.ini, .INI

JSON

.json, .JSON

YAML (requires PyYAML to be installed)

.yaml, .yml, .YAML, .YML

Several files may be loaded consecutively by repeating the --config-file option.

This makes it possible to split configuration files on various purposes:

  • the kind of software / system under test,

  • the environment used to execute the tests,

  • the identity of the person who launches the tests,

  • …

The configuration data from the different files is merged all together in the resulting tree, the values set from the latter files overloading the ones already set by the formers files.

Then, the single values set by the --config-value options finally update the configuration tree.

Boolean value conversions

When configuration values are boolean values, they may be passed as strings in one of the usual forms recognized:

True values:

any non-zero integer or integer string, strings like “True”, “TRUE”, “true”, “Yes, “YES”, “yes”, “Y”, “y”

False values:

0 (zero) or “0”, strings like “False”, “FALSE”, “false”, “No”, “NO”, “no”, “N”, “n”

Manipulating configurations from the code

The code can then access configuration values (resp. ConfigNode instances) through the ConfigDatabase.get() method (resp. ConfigDatabase.getnode()).

# Access a final value (`None` if the value does not exist).
_any = scenario.conf.get("a.b.c")  # type: typing.Optional[typing.Any]
# Access a final value of the given type (`None` if the value does not exist).
_int1 = scenario.conf.get("a.b.c", type=int)  # type: typing.Optional[int]
# Access a final value, or fallback to a default value.
_int2 = scenario.conf.get("a.b.c", type=int, default=100)  # type: int
# Access a whole section as a JSON dictionary (`None` if the section does not exist).
_section = scenario.conf.get("a", type=dict)  # type: typing.Optional[typing.Dict[str, typing.Any]]
# Access a whole list as a JSON list (`None` if the list does not exist).
_list = scenario.conf.get("x.y", type=list)  # type: typing.Optional[typing.List[typing.Any]]

The configuration keys available can be listed with the ConfigDatabase.getkeys() method.

Configuration files can be loaded from the code (see ConfigDatabase.loadfile()).

# Load a configuration file.
scenario.conf.loadfile("demo/conf.yml")

Configuration data can also be set (either sections or lists or single values, see ConfigDatabase.set()).

# Set a single value.
scenario.conf.set("a.b.c", 150)
# Update a whole section (possibly with sub-sections).
scenario.conf.set("a.b", {"c": 200})
scenario.conf.set("a", {"b": {"c": 200}})

Automatic configuration data conversions

When setting data from the code, the configuration database applies the following conversions:

Automatic configuration data conversions

Input data type

Storage

Path-likes

str form of the path, using os.fspath()

enum.EnumMeta

list

enum.IntEnum

int form of the enum value

Other enum.Enum

str form of the enum value

Configuration nodes can be accessed directly from the code, and provide an API that can be used from the user code (see ConfigNode).

# Access a configuration node (`None` if the node does not exist).
_node = scenario.conf.getnode("a.b.c")  # type: typing.Optional[scenario.ConfigNode]

Configuration origins

In case configurations lead to some erroneous situation, the configuration database keeps memory of configuration origins (see ConfigNode.origins and ConfigNode.origin).

This information can help a user fix his/her configuration files when something goes wrong.

For the purpose, the ConfigNode.errmsg() method builds error messages giving the representative origin of the given configuration node.

The ConfigDatabase.show() and ConfigNode.show() methods also display the configuration tree with origins.

$ ./demo/run-demo.py --config-file demo/conf.json --config-value x.y[0].z 0 --show-configs demo/htmllogin.py

INFO     Main path: '/path/to/scenario'
INFO     [scenario.ConfigDatabase]   a:
INFO     [scenario.ConfigDatabase]     a.b:
INFO     [scenario.ConfigDatabase]       a.b.c: 55  # from demo/conf.json
INFO     [scenario.ConfigDatabase]   x:
INFO     [scenario.ConfigDatabase]     x.y[0]:
INFO     [scenario.ConfigDatabase]       x.y[0].z: '0'  # from <args>
INFO     [scenario.ConfigDatabase]     x.y[1]:
INFO     [scenario.ConfigDatabase]       x.y[1].z: 100  # from demo/conf.json
INFO     [scenario.ConfigDatabase]     x.y[2]:
INFO     [scenario.ConfigDatabase]       x.y[2].z: 101  # from demo/conf.json
INFO     [scenario.ConfigDatabase]     x.y[3]:
INFO     [scenario.ConfigDatabase]       x.y[3].z: 102  # from demo/conf.json
INFO     [scenario.ConfigDatabase]     x.y[4]:
INFO     [scenario.ConfigDatabase]       x.y[4].z: 103  # from demo/conf.json

scenario configurable keys and values

The following table describes the scenario configurable keys & values.

Tip

Use the scenario.ConfigKey shortcut to the internal ScenarioConfig.Key enum from scenario user code.

Scenario configurable keys and values

Key

Type

Description

Default

ScenarioConfig.Key.TIMEZONE

scenario.timezone

String

Timezone specification.

Possible values: ‘UTC’, ‘Z’, or numerical forms like ‘+01:00’, ‘-05:00’.

More options when pytz is installed: ‘CET’, ‘US/Pacific’, ‘Japan’, …

Execute the following Python code for the complete list:

import pytz
print("\n".join(pytz.all_timezones))

Not set, i.e. use of the local timezone

ScenarioConfig.Key.LOG_DATETIME

scenario.log_date_time

Boolean

Should the log lines include a timestamp?

Enabled

ScenarioConfig.Key.LOG_CONSOLE

scenario.log_console

Boolean

Should the log lines be displayed in the console?

Enabled

ScenarioConfig.Key.LOG_COLOR_ENABLED

scenario.log_color

Boolean

Should the log lines be colored?

Enabled

ScenarioConfig.Key.LOG_COLOR

scenario.log_%(level)_color, %(level) being one of (error, warning, info, debug)

Integer

Console color code per log level. See Console.Color for a list useful color codes.

scenario.log_error_color: red(91), scenario.log_warning_color: yellow(33), scenario.log_info_color: white(1), scenario.log_debug_color: dark grey(2)

ScenarioConfig.Key.LOG_FILE

scenario.log_file

File path string

Should the log lines be written in a log file?

Not set, i.e. no file logging

ScenarioConfig.Key.DEBUG_CLASSES

scenario.debug_classes

List of strings (or comma-separated string)

Which debug classes to display?

Not set

ScenarioConfig.Key.EXPECTED_ATTRIBUTES

scenario.expected_attributes

List of strings (or comma-separated string)

Expected scenario attributes.

Not set

ScenarioConfig.Key.CONTINUE_ON_ERROR

scenario.continue_on_error

Boolean

Should the scenarios continue on error? If set to True, an error ends the current step, but following steps are still executed. The same behaviour may also be activated scenario by scenario by setting the ScenarioDefinition.continue_on_error attribute.

Disabled

ScenarioConfig.Key.DELAY_BETWEEN_STEPS

scenario.delay_between_steps

Float (in seconds)

Should we wait between two step executions?

0.001 seconds

ScenarioConfig.Key.RUNNER_SCRIPT_PATH

scenario.runner_script_path

File path string

Path of the scenario runner script. Useful when executing campaigns: may be used to make your own launcher script path be called.

‘bin/run-test.py’ provided with the scenario framework

ScenarioConfig.Key.SCENARIO_TIMEOUT

scenario.scenario_timeout

Float (in seconds)

Maximum time for a scenario execution. Useful when executing campaigns.

600.0 seconds, i.e. 10 minutes

ScenarioConfig.Key.RESULTS_EXTRA_INFO

scenario.results_extra_info

List of strings (or comma-separated string)

Scenario attributes to display for extra info when displaying scenario results, after a campaign execution, or when executing several tests in a single command line.

Not set

ScenarioConfig.Key.ISSUE_LEVEL_NAMES

scenario.issue_levels

{str: int} dictionary

Dictionary of names associated with issue level integer values.

Example:

scenario:
  issue_levels:
    SUT: 40
    TEST: 30
    CONTEXT: 20
    PLANNED: 10

Not set

ScenarioConfig.Key.ISSUE_LEVEL_ERROR

scenario.issue_level_error

String or integer.

Issue level from and above which known issues should be considered as errors.

May be set directly as an integer value, or an issue level name if defined (see scenario.issue_levels).

Not set

ScenarioConfig.Key.ISSUE_LEVEL_IGNORED

scenario.issue_level_ignored

String or integer.

Issue level from and under which known issues should be ignored.

May be set directly as an integer value, or an issue level name if defined (see scenario.issue_levels).

Not set