Blame tests/writing_tests.md
|
Packit |
01d647 |
## Writing new tests
|
|
Packit |
01d647 |
|
|
Packit |
01d647 |
The test suite is intended to run a binary and compare its standard output,
|
|
Packit |
01d647 |
standard error and return value against provided values. This is implemented
|
|
Packit |
01d647 |
using Python's `unittest` module and thus all test files are Python files.
|
|
Packit |
01d647 |
|
|
Packit |
01d647 |
The simplest test has the following structure:
|
|
Packit |
01d647 |
``` python
|
|
Packit |
01d647 |
# -*- coding: utf-8 -*-
|
|
Packit |
01d647 |
|
|
Packit |
01d647 |
import system_tests
|
|
Packit |
01d647 |
|
|
Packit |
01d647 |
|
|
Packit |
01d647 |
class GoodTestName(metaclass=system_tests.CaseMeta):
|
|
Packit |
01d647 |
|
|
Packit |
01d647 |
filename = "$data_path/test_file"
|
|
Packit |
01d647 |
commands = ["$exiv2 $filename", "$exiv2 $filename" + '_2']
|
|
Packit |
01d647 |
stdout = [""] * 2
|
|
Packit |
01d647 |
stderr = ["""$exiv2_exception_msg $filename:
|
|
Packit |
01d647 |
$kerFailedToReadImageData
|
|
Packit |
01d647 |
"""] * 2
|
|
Packit |
01d647 |
retval = [1] * 2
|
|
Packit |
01d647 |
```
|
|
Packit |
01d647 |
|
|
Packit |
01d647 |
The test suite will run the provided commands in `commands` and compare them to
|
|
Packit |
01d647 |
the output in `stdout` and `stderr` and it will compare the return values.
|
|
Packit |
01d647 |
|
|
Packit |
01d647 |
The strings after a `$` are variables either defined in this test's class or are
|
|
Packit |
01d647 |
taken from the suite's configuration file (see `doc.md` for a complete
|
|
Packit |
01d647 |
explanation).
|
|
Packit |
01d647 |
|
|
Packit |
01d647 |
When creating new tests, follow roughly these steps:
|
|
Packit |
01d647 |
|
|
Packit |
01d647 |
1. Choose an appropriate subdirectory where the test belongs. If none fits
|
|
Packit |
01d647 |
create a new one and put an empty `__init__.py` file there.
|
|
Packit |
01d647 |
|
|
Packit |
01d647 |
2. Create a new file with a name matching `test_*.py`. Copy the class definition
|
|
Packit |
01d647 |
from the above example and choose an appropriate class name.
|
|
Packit |
01d647 |
|
|
Packit |
01d647 |
3. Run the test suite via `python3 runner.py` and ensure that your test case is
|
|
Packit |
01d647 |
actually run! Either run the suite with the `-v` option which will output all
|
|
Packit |
01d647 |
test cases that were run or simply add an error and check if errors occur.
|