## Writing new tests

The test suite is intended to run a binary and compare its standard output,
standard error and return value against provided values. This is implemented
using Python's `unittest` module and thus all test files are Python files.

The simplest test has the following structure:
``` python
# -*- coding: utf-8 -*-

import system_tests


class GoodTestName(metaclass=system_tests.CaseMeta):

    filename = "$data_path/test_file"
    commands = ["$exiv2 $filename", "$exiv2 $filename" + '_2']
    stdout = [""] * 2
    stderr = ["""$exiv2_exception_msg $filename:
$kerFailedToReadImageData
"""] * 2
    retval = [1] * 2
```

The test suite will run the provided commands in `commands` and compare them to
the output in `stdout` and `stderr` and it will compare the return values.

The strings after a `$` are variables either defined in this test's class or are
taken from the suite's configuration file (see `doc.md` for a complete
explanation).

When creating new tests, follow roughly these steps:

1. Choose an appropriate subdirectory where the test belongs. If none fits
   create a new one and put an empty `__init__.py` file there.

2. Create a new file with a name matching `test_*.py`. Copy the class definition
   from the above example and choose an appropriate class name.

3. Run the test suite via `python3 runner.py` and ensure that your test case is
   actually run! Either run the suite with the `-v` option which will output all
   test cases that were run or simply add an error and check if errors occur.