Cockpit-Composer Integration Test

The integration test for Cockpit Composer! It is performed on the application level and tests whether the business requirements are met regardless of app internal architecture, dependencies, data integrity and such. Actually we need to follow the end-user flows and assert they get the intended experience and focus on the behavior of the thing as the user would see it.

The integration tests are powered by Cockpit test framework, which is simple and easy to debug for test development.

Requirement

For testing, the following dependencies are required:

$ sudo dnf install curl expect xz rpm-build chromium-headless \
    libvirt-daemon-kvm libvirt-client python3-libvirt

And chrome-remote-interface and sizzle Javascript libraries need to be installed:

$ npm install

Introduction

Before running the tests, ensure Cockpit-Composer environment has been built:

$ make vm

To run all tests run the following:

$ make check

Alternatively you can run an individual test like this:

$ ./test/verify/check-image

To see more verbose output from the test, use the --verbose and/or --trace flags:

$ ./test/verify/check-image --verbose --trace

In addition if you specify --sit, then the test will wait on failure and allow you to log into cockpit and/or the test instance and diagnose the issue. An address will be printed of the test instance.

$ ./test/verify/check-image --trace --sit

Normally each test starts its own chromium headless browser process on a separate random port. To interactively follow what a test is doing, set environment variable $TEST_SHOW_BROWSER.

$ TEST_SHOW_BROWSER=true ./test/verify/check-image --trace

Test Configuration

You can set these environment variables to configure the test suite:

TEST_OS    The OS to run the tests in.  Currently supported values:
              "fedora-32"
           "fedora-32" is the default

TEST_DATA  Where to find and store test machine images.  The
           default is the same directory that this README file is in.

TEST_CDP_PORT  Attach to an actually running browser that is compatible with
               the Chrome Debug Protocol, on the given port. Don't use this
               with parallel tests.

TEST_BROWSER  What browser should be used for testing. Currently supported values:
                 "chromium"
                 "firefox"
              "chromium" is the default.

TEST_SHOW_BROWSER  Set to run browser interactively. When not specified,
                   browser is run in headless mode.

Guidelines for writing tests

Tests decorated with @nondestructive will all run against the same test machine. The nondestructive test should clean up after itself and restore the state of the machine, such that the next nondestructive test is not impacted.

A fast running test suite is more important than independent, small test cases.

Code coverage from end-to-end tests

Before running the tests the application code must be instrumented with istanbul! Then inside the browser scope all coverage information is available from window.__coverage__ which needs to be passed back to the node scope and made available for the reporting tools to use.

Please include helper method check_coverage() at the end of each test. This helper method will collect coverage result and save result into file .nyc_output/coverage-<HASH>.json. The hash value is the sha256 sum of the coverage report itself. Some cases may have identical coverage so the number of json files will be equal or less to the number of test cases.

Code Style

Python code in this project should follow Flake8.