= Writing and running tests for OpenSCAP

This document should help you with writing and running tests for your pull
requests. It is recommended to add a new test if a new functionality is added
or if a code that is not covered by any test is updated. Another recommendation
is to add your test in a separate commit.

All the tests reside in the link:../../tests[tests] directory. It has multiple
subdirectories which should represent various parts of the OpenSCAP library and
its utilities. When you contribute to some part of the OpenSCAP project you
should put a test for your contribution into the corresponding subdirectory
in the link:../../tests[tests] directory. Use your best judgement when deciding
where to put the test for your pull request and if you are not sure don't be
affraid to ask in the pull request, someone will definitely help you with that.

NOTE: OpenSCAP project uses the **CMake** buildsystem which has built-in
testing support through the
link:https://gitlab.kitware.com/cmake/community/wikis/doc/ctest/Testing-With-CTest[CTest]
testing tool.


== Preparing test environment and running tests
To run a specific test or all tests you first need to compile the OpenSCAP
library and then install additional packages required for testing. See the
*Building OpenSCAP on Linux* section in the link:../developer/developer.adoc[OpenSCAP Developer Manual]
for more details.


== Writing a new test
In this guide we will use an example to describe the process of writing a test
for the OpenSCAP project. Let's suppose you want to write a new test for
the Script Check Engine (SCE) to test its basic functionality. SCE allows you
to define your own scripts (usually written in Bash or Python) to extend XCCDF
rule checking capabilities. Custom check scripts can be referenced from
an XCCDF rule using `<check-content-ref>` element, for example:
----
<check system="http://open-scap.org/page/SCE">
    <check-content-ref href="YOUR_BASH_SCRIPT.sh"/>
</check>
----


=== Deciding where to put a new test
In our example, we are testing the SCE module, therefore we will look for
its subdirectory in the link:../../tests[tests] direcotory and we will find it
at the following link: link:../../tests/sce[tests/sce]. We will add our new test
into this subdirectory.


==== Scenario A: There is a suitable directory for my new test
This will happen most of the times. As stated above in our example we will place
our test into the link:../../tests/sce[tests/sce] directory.


==== Scenario B: There is no suitable directory for my new test
This might happen if your test covers a part of the OpenSCAP which has no tests
at all. In this case you would need to add a new directory with suitable name
into the link:../../tests[tests] directory structure and create/update
`CMakeLists.txt` files.

To have an example also for this scenario, let's suppose we want to add the
`foo` subdirectory into the link:../../tests[tests] directory. We need to:

. Create the `foo` subdirectory in the link:../../tests[tests] directory.
. Use the link:https://cmake.org/cmake/help/latest/command/add_subdirectory.html[add_subdirectory]
  command from the link:../../tests/CMakeLists.txt[tests/CMakeLists.txt]
  to add the `foo` subdirectory to the CMake buildsystem.
. Create `CMakeLists.txt` inside the `tests/foo` directory and use the
  `add_oscap_test` function (defined in the
  link:../../tests/CMakeLists.txt[tests/CMakeLists.txt]) to add all your test
  scripts from the `foo` directory.

Please see the
link:https://gitlab.kitware.com/cmake/community/wikis/doc/ctest/Testing-With-CTest[
CTest documentation] for more details.


=== Common test library
When writing tests for OpenSCAP you should use the common test library which is
located at link:../../tests/test_common.sh.in[tests/test_common.sh.in].

NOTE: The `cmake` command will generate the `tests/test_common.sh` file from
the link:../../tests/test_common.sh.in[tests/test_common.sh.in] adding
configuration specific settings.

You will need to source the `tests/test_common.sh` in your test scripts to use
the functions which it provides:
[source,bash]
----
#!/usr/bin/env bash

set -o pipefail

. $builddir/tests/test_common.sh

test1
test2
...
----

NOTE: The `$builddir` variable contains the path to the top level of the build
tree. It is defined in the link:../../tests/CMakeLists.txt[tests/CMakeLists.txt]
as `builddir=${CMAKE_BINARY_DIR}`.


==== Global variables exported by test library
Always use `$OSCAP` variable instead of the plain `oscap` in your tests when
calling the `oscap` command line tool. This is because tests might be run with
`CUSTOM_OSCAP` variable.

You can use `$XMLDIFF` in your tests which will call the
link:../../tests/xmldiff.pl[tests/xmldiff.pl] script.

It is also possible to do XPath queries using `$XPATH` variable, the usage is:
[source,bash]
----
$XPATH FILE 'QUERY'
----


==== Best practices when writing bash tests
It is always good to set `pipefail` option for all your test scripts so you
accidentally don't lose non-zero exit statuses of commands in a pipeline:
[source,bash]
----
set -o pipefail
----

The next option you can consider is `errexit` which exits your script
immediately if a command exits with a non-zero status. You might want to use
this option if tests are somehow dependent and it doesn't make sense to continue
testing after one test fails.
[source,bash]
----
set -e
----

Also consider splitting the tests up into several separate bash scripts if
you test many things at once; long running "all.sh" test should be avoided
if possible, as when these fail it is harder to debug than having more,
smaller tests. However, make sure these independent tests don't have hidden
dependencies: often we'll run the test suite in parallel, meaning in-order
execution of different test scripts cannot be ensured. (However, order
order inside the test script is guaranteed per bash's rules).

Lastly, make sure your tests are architecture, distribution, and operating
system independent. Try to make your tests dependent on local file contents,
not system file contents, and make them independent of listening ports and
installed software as much as possible.

==== test_init function
This function does nothing. Logging is done by the CTest and the log from
testing can be found at `build/Testing/Temporary/LastTest.log`.


==== test_run function
The function is responsible for executing a test script file or a function and
logging its result into the log file.
[source,bash]
----
test_run "DESCRIPTION" TEST_FUNCTION|$srcdir/TEST_SCRIPT_FILE ARG [ARG...]
----

NOTE: The `$srcdir` variable contains the path to the directory with the test
script. It is defined in the link:../../tests/CMakeLists.txt[tests/CMakeLists.txt]
as `srcdir=${CMAKE_CURRENT_SOURCE_DIR}`. The reason is backward compatibility
as originally tests were executed by the GNU automake.

The `test_run` function reports the following results into the log file:

* *PASS* when script/function returns *0*,
* *FAIL* when script/function returns *1*,
* *SKIP* when script/function returns *255*,
* *WARN* when script/function returns none of the above exit statuses.

The result of every test executed by the `test_run` function will be reported
in the log file in a following way:
[source,bash]
----
TEST: DESCRIPTION
<test stdout + stderr output>
RESULT: PASS/FAIL/SKIP/WARN
----


==== test_exit function
The function is responsible for cleaning-up the testing environment. You can
call it without arguments or with one argument -- a script/function which will
do additional clean-up tasks.
[source,bash]
----
test_exit [CLEAN_SCRIPT|CLEAN_FUNCTION]
----


==== require function
Checks if requirements are in the `$PATH`, use it as follows:
[source,bash]
----
require 'program' || return 255
----


==== probecheck function
Checks if probe exists, use it as follows:
[source,bash]
----
probecheck 'probe' || return 255
----


==== verify_results function
Verifies that there is the `COUNT` number of results of selected OVAL `TYPE` in
a `RESULTS_FILE`:
[source,bash]
----
verify_results TYPE CONTENT_FILE RESULTS_FILE COUNT

verify_results "def" test_probe_foo.xml results.xml 13
----

The function extracts the actual test/definition result, and compares it with the respective test/definition comment.
For example, if the test contains `comment="true"`, the test passes only if result of the respective test is `true`,
if `comment="false"`, a `false` result is expected.
If the comment is missing or it has other value, it is assumed that the result should be neither `true` nor `false`.

NOTE: This function expects that the OVAL `TYPE` is numbered from `1` to `COUNT`
in the `RESULTS_FILE`.
`TYPE` is typically `def` or `tst` for definitions and tests respectively.


==== assert_exists function
Does an XPath query to a file specified in the `$result` variable and checks if
number of results matches with an expected number specified as an argument:
[source,bash]
----
result="relative_path_to_file"
assert_exists EXPECTED_NUMBER_OF_RESULTS XPATH_QUERY_STRING
----

For example, let's say you want to check that in the `results.xml` file the
result of the rule `xccdf_com.example.www_rule_test` is fail:
[source,bash]
----
result="./results.xml"
my_rule_="xccdf_com.example.www_rule_test"
assert_exists 1 "//rule-result[@idref=\"$my_rule\"]/result[text()=\"fail\"]"
----


=== Adding test files
Now, as we know where a new test should go and what functions and capabilities
are provided by the common test library, we can add test files which will
contain test scripts and content required for testing.

To sum up, we are adding a tests to check the basic functionality of the Script
Check Engine (SCE) and we have decided that the test will go into the
link:../../tests/sce[tests/sce] directory.

We will add the link:../../tests/sce/test_sce.sh[tests/sce/test_sce.sh]
script which will contain our test and
link:../../tests/sce/sce_xccdf.xml[tests/sce/sce_xccdf.xml], an XML file with
XCCDF rules which are referencing various check scripts (grep the
`check-content-ref` element to see the referenced files). All the referenced
check script files are set to always pass and the
link:../../tests/sce/test_sce.sh[tests/sce/test_sce.sh] script will perform
evaluation of the link:../../tests/sce/sce_xccdf.xml[tests/sce/sce_xccdf.xml]
XCCDF document file and it will check that all rule results are `pass`.


=== Plugging your new test into the test library
You need to plug your test into the test library so it will be run automatically
everytime `make test` is run. To do this, you need to add your test script
into the `CMakeLists.txt`. The `CMakeLists.txt` which you need to modify is
located in the same directory as your test script.

We will demonstrate this on our example with the SCE test. We have prepared our
test script, the XML document file with custom rules and various check scripts
for testing. We placed all our test files into the
link:../../tests/sce[tests/sce] directory. Now we will modify the
link:../../tests/sce/CMakeLists.txt[tests/sce/CMakeLists.txt] and we will add
our test script file using the `add_oscap_test` function which will make sure
that our test will be executed by the `make test`:
----
if(ENABLE_SCE)
	...
	*add_oscap_test("test_sce.sh")*
	...
endif()
----


=== Running your new test
To run your new test you first need to compile the OpenSCAP library. See the
*Building OpenSCAP on Linux* section in the link:../developer/developer.adoc[OpenSCAP Developer Manual]
for more details.
Also you don't need to run all the tests using `make test`, you can run only
the specific test(s). To do so, you need to be in the build directory and
run `ctest -R` from there, for example:
[source,bash]
----
$ cd build/
$ ctest -R sce/test_sce.sh
$ less Testing/Temporary/LastTest.log
----

Results from testing will be printed on the stdout and detailed log file with
your test results can be found in the `Testing/Temporary/LastTest.log` file.

== Running the MITRE tests

The MITRE tests (in `tests/mitre`) are functionality tests for several
key probes.

There are several reasons for putting these tests behind an `ENABLE_MITRE`
CMake flag: they cannot be run in parallel as they race to create and
delete temporary support files; they require port 25 to be open and
listening (`mitre/test_mitre_linux_probes.sh`); there are outdated assumptions
which need to be gated around operating system version checks; and a number of
probes had to be disabled because they're not supported (e.g., SQL and LDAP
checks, etc.).

To run only the MITRE tests, please make sure you've installed and started a
SMTP server that is listening on `127.0.0.1:25`, and that you're running on a
RPM-based distribution. Then:

----
$ cd build/
$ cmake -DENABLE_MITRE=TRUE ..
$ make
$ ctest --output-on-failure -R mitre
----

To run the containerized MITRE tests (which installs a SMTP server and tests
it):

----
$ cd openscap/
$ docker build --tag openscap_mitre_tests:latest -f Dockerfiles/mitre_tests .
$ docker run openscap_mitre_tests:latest
----