|
Packit |
63bb0d |
# osbuild-composer testing information
|
|
Packit |
63bb0d |
|
|
Packit Service |
509fd4 |
With the exception of unit tests, all the osbuild-composer tests are shipped
|
|
Packit Service |
509fd4 |
in a dedicated RPM: osbulid-composer-tests, and they are meant to be installed
|
|
Packit Service |
509fd4 |
onto a pristine system and executed from there, rather than executed from a
|
|
Packit Service |
509fd4 |
git checkout.
|
|
Packit Service |
509fd4 |
|
|
Packit Service |
509fd4 |
Test cases are found in `test/cases`. They are intended to be independent of
|
|
Packit Service |
509fd4 |
each other and they are installed into `/usr/libexec/tests/osbuild-composer/`,
|
|
Packit Service |
509fd4 |
with any supporting test-data in `/usr/share/tests/osbuild-composer/`, and
|
|
Packit Service |
509fd4 |
helper binaries in `/usr/libexec/osbuild-composer-test/`.
|
|
Packit Service |
509fd4 |
|
|
Packit Service |
509fd4 |
Provisioning and orchestration of test-systems, as well as configuring what
|
|
Packit Service |
509fd4 |
distros, versions and architectures to test against is out of scope of the
|
|
Packit Service |
509fd4 |
tests themselves. For osbuild-composer CI, schutzbot is used, which can be
|
|
Packit Service |
509fd4 |
found in the directory by the same name.
|
|
Packit Service |
509fd4 |
|
|
Packit Service |
509fd4 |
|
|
Packit Service |
509fd4 |
## Golang tests
|
|
Packit Service |
509fd4 |
|
|
Packit |
63bb0d |
Test binaries, regardless of their scope/type (e.g. unit, API, integration)
|
|
Packit Service |
509fd4 |
that are written in Golang must follow the syntax of the Go
|
|
Packit |
63bb0d |
[testing package](https://golang.org/pkg/testing/), that is implement only
|
|
Packit |
63bb0d |
`TestXxx` functions with their setup/teardown when necessary in a `yyy_test.go`
|
|
Packit |
63bb0d |
file.
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
Test scenario discovery, execution and reporting will be handled by `go test`.
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
Some test files will be executed directly by `go test` during rpm build time
|
|
Packit |
63bb0d |
and/or in CI. These are usually unit tests. Scenarios which require more complex
|
|
Packit |
63bb0d |
setup, e.g. a running osbuild-composer are not intented to be executed directly
|
|
Packit |
63bb0d |
by `go test` at build time. Instead they are intended to be executed as
|
|
Packit |
63bb0d |
stand-alone test binaries on a clean system which has been configured in
|
|
Packit |
63bb0d |
advance (because this is easier/more feasible). These stand-alone test binaries
|
|
Packit |
63bb0d |
are also compiled via `go test -c -o` during rpm build or via `make build`.
|
|
Packit |
63bb0d |
See *Integration testing* for more information.
|
|
Packit |
63bb0d |
|
|
Packit Service |
509fd4 |
When comparing for expected values in test functions you should use the
|
|
Packit Service |
509fd4 |
[testify/assert](https://godoc.org/github.com/stretchr/testify/assert) or
|
|
Packit Service |
509fd4 |
[testify/require](https://godoc.org/github.com/stretchr/testify/require)
|
|
Packit Service |
509fd4 |
packages. Both of them provide an impressive array of assertions with the
|
|
Packit Service |
509fd4 |
possibility to use formatted strings as error messages. For example:
|
|
Packit Service |
509fd4 |
|
|
Packit Service |
509fd4 |
```
|
|
Packit Service |
509fd4 |
assert.Nilf(t, err, "Failed to set up temporary repository: %v", err)
|
|
Packit Service |
509fd4 |
```
|
|
Packit Service |
509fd4 |
|
|
Packit Service |
509fd4 |
If you want to fail immediately, not doing any more of the asserts use the
|
|
Packit Service |
509fd4 |
`require` package instead of the `assert` package, otherwise you'll end up with
|
|
Packit Service |
509fd4 |
panics and nil pointer memory problems.
|
|
Packit Service |
509fd4 |
|
|
Packit Service |
509fd4 |
Stand-alone test binaries also have the `-test.failfast` option.
|
|
Packit Service |
509fd4 |
|
|
Packit Service |
509fd4 |
Code coverage is recorded in
|
|
Packit Service |
509fd4 |
[codecov.io](https://codecov.io/github/osbuild/osbuild-composer).
|
|
Packit Service |
509fd4 |
This information comes only from unit tests and for the time being
|
|
Packit Service |
509fd4 |
we're not concerned with collecting coverage information from integration
|
|
Packit Service |
509fd4 |
tests, see `.github/workflows/tests.yml`.
|
|
Packit Service |
509fd4 |
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
## Image tests
|
|
Packit |
63bb0d |
|
|
Packit Service |
3a6627 |
In the `test/data/manifests` directory, sample image builds and their tests are
|
|
Packit |
63bb0d |
collected for the various distros, architectures, configuration we support.
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
Each test case describes how the image is built, the expected osbuild
|
|
Packit |
63bb0d |
manifest used internally, the expected image-info output and how to
|
|
Packit |
63bb0d |
boot-test the image.
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
To (re)generate these test cases use the tool
|
|
Packit |
63bb0d |
`tools/test-case-generators/generate-test-cases`.
|
|
Packit Service |
3a6627 |
Note that the `generate-test-cases` tool must be run on a host with
|
|
Packit Service |
3a6627 |
the same architecture, as the one intended for the generated test
|
|
Packit Service |
3a6627 |
cases. In other words, you need to generate e.g test cases for `aarch64`
|
|
Packit Service |
3a6627 |
images on an `aarch64` host.
|
|
Packit Service |
3a6627 |
|
|
Packit Service |
3a6627 |
Alternatively to (re)generate test cases for all architectures, or just
|
|
Packit Service |
3a6627 |
the ones different from your host's architecture, you can use the tool
|
|
Packit Service |
3a6627 |
`tools/test-case-generators/generate-all-test-cases`. It creates
|
|
Packit Service |
3a6627 |
an ephemeral virtual machine for each necessary architecture using the
|
|
Packit Service |
3a6627 |
`qemu-system-<arch>` command and generates test cases using the
|
|
Packit Service |
3a6627 |
`generate-test-cases` tool inside the virtual machine. It is important
|
|
Packit Service |
3a6627 |
to note that test case generation in virtual machines may take several
|
|
Packit Service |
3a6627 |
hours. The `generate-all-test-cases` currently does not work with RHEL
|
|
Packit Service |
3a6627 |
images because of missing "9p" filesystem support.
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
### Setting up Azure upload tests
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
By default, the vhd images are run locally using qemu. However, when
|
|
Packit |
63bb0d |
the right set of environment flags is passed to the osbuild-image-tests,
|
|
Packit |
63bb0d |
it uploads the image to Azure, boots it and tries to ssh into it.
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
#### Required flags
|
|
Packit |
63bb0d |
- `AZURE_STORAGE_ACCOUNT`
|
|
Packit |
63bb0d |
- `AZURE_STORAGE_ACCESS_KEY`
|
|
Packit |
63bb0d |
- `AZURE_CONTAINER_NAME`
|
|
Packit |
63bb0d |
- `AZURE_SUBSCRIPTION_ID`
|
|
Packit |
63bb0d |
- `AZURE_CLIENT_ID`
|
|
Packit |
63bb0d |
- `AZURE_CLIENT_SECRET`
|
|
Packit |
63bb0d |
- `AZURE_TENANT_ID`
|
|
Packit |
63bb0d |
- `AZURE_LOCATION`
|
|
Packit |
63bb0d |
- `AZURE_RESOURCE_GROUP`
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
#### Setting up all the required resources
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
1) Firstly, go to *Subscriptions* in the left-side menu. Here you can find
|
|
Packit |
63bb0d |
the `AZURE_SUBSCRIPTION_ID`.
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
2) Now, you need to create a new resource group. In the left-side menu,
|
|
Packit |
63bb0d |
select *Resource groups*. Click on *Add* above the resource group list.
|
|
Packit |
63bb0d |
The name you choose is your `AZURE_RESOURCE_GROUP`. The region you choose
|
|
Packit |
63bb0d |
is your `AZURE_LOCATION`. However, it must be in the "machine-readable
|
|
Packit |
63bb0d |
form". You can list all the locations with their machine-readable names
|
|
Packit |
63bb0d |
using Azure CLI: `az account list-locations -o table`.
|
|
Packit |
63bb0d |
E.g. the machine-readable name of US East location is `eastus`.
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
Note that terms *location* and *region* are synonyms in Azure's context.
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
3) Storage time! Go to Storage accounts in the left-side menu. Click on
|
|
Packit |
63bb0d |
*Add* above the list. Use the resource group you created in
|
|
Packit |
63bb0d |
the previous step. Also, the region should be the same. The name you
|
|
Packit |
63bb0d |
choose is your `AZURE_STORAGE_ACCOUNT`.
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
After the storage account is created, open it.
|
|
Packit |
63bb0d |
Select *Settings > Access keys*. Choose one of the keys, this is your
|
|
Packit |
63bb0d |
`AZURE_STORAGE_ACCESS_KEY`. Select *Blob service > Containers* and create
|
|
Packit |
63bb0d |
a new one. Its name is your `AZURE_CONTAINER_NAME`.
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
4) Now it’s time to create an application. This is needed because Azure uses
|
|
Packit |
63bb0d |
OAuth to do authorization. In the left-side menu, choose *Azure Active
|
|
Packit |
63bb0d |
Directory*. Go to *Manage > App registrations* and register a new
|
|
Packit |
63bb0d |
application.
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
When it’s created, open it. In the overview, you can see
|
|
Packit |
63bb0d |
the Application (client) ID and the Directory (tenant) ID. These are your
|
|
Packit |
63bb0d |
`AZURE_CLIENT_ID` and `AZURE_TENANT_ID`.
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
Now, go to *Manage > Certificates & Secrets* under your new application
|
|
Packit |
63bb0d |
and create a new client secret. The is your `AZURE_CLIENT_SECRET`.
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
5) The last step is to give the new application access to the resource group.
|
|
Packit |
63bb0d |
This step must be done by Azure administrator (@larskarlitski): Go to
|
|
Packit |
63bb0d |
the *Access control (IAM)* section under the newly created resource group.
|
|
Packit |
63bb0d |
Here, add the new application with the *Developer* role.
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
### Setting up OpenStack upload tests
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
The following environment variables are required
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
- `OS_AUTH_URL`
|
|
Packit |
63bb0d |
- `OS_USERNAME`
|
|
Packit |
63bb0d |
- `OS_PASSWORD`
|
|
Packit |
63bb0d |
- `OS_PROJECT_ID`
|
|
Packit |
63bb0d |
- `OS_DOMAIN_NAME`
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
### Setting up VMware vCenter upload tests
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
The following environment variables are required
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
- `GOVMOMI_URL` - vCenter hostname
|
|
Packit |
63bb0d |
- `GOVMOMI_USERNAME`
|
|
Packit |
63bb0d |
- `GOVMOMI_PASSWORD`
|
|
Packit |
63bb0d |
- `GOVMOMI_DATACENTER`
|
|
Packit |
63bb0d |
- `GOVMOMI_CLUSTER`
|
|
Packit |
63bb0d |
- `GOVMOMI_NETWORK`
|
|
Packit |
63bb0d |
- `GOVMOMI_DATASTORE`
|
|
Packit |
63bb0d |
- `GOVMOMI_FOLDER`
|
|
Packit |
63bb0d |
- `GOVMOMI_INSECURE` - value of 1 will skip checking SSL certificates
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
**WARNING:** when configuring the credentials for Schutzbot we've experienced
|
|
Packit |
63bb0d |
an issue where the first line in the credentials file gets lost resulting in
|
|
Packit |
63bb0d |
incomplete credentials. The work-around is to define a dummy ENV variable on
|
|
Packit |
63bb0d |
the first line!
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
## Integration testing
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
This will consume the osbuild-composer API surface via the `composer-cli`
|
|
Packit Service |
509fd4 |
command line interface. Implementation is under
|
|
Packit Service |
509fd4 |
`cmd/osbuild-composer-cli-tests/`.
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
The easiest way to get started with integration testing from a git
|
|
Packit |
63bb0d |
checkout is:
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
* `dnf -y install rpm-build`
|
|
Packit |
63bb0d |
* `dnf -y builddep osbuild-composer.spec`
|
|
Packit |
63bb0d |
* `make rpm` to build the software under test
|
|
Packit |
63bb0d |
* `dnf install rpmbuild/RPMS/x86_64/osbuild-composer-*.rpm` - this will
|
|
Packit |
63bb0d |
install both osbuild-composer, its -debuginfo, -debugsource and -tests packages
|
|
Packit |
63bb0d |
* `systemctl start osbuild-composer`
|
|
Packit Service |
509fd4 |
* `/usr/libexec/tests/osbuild-composer/osbuild-composer-cli-tests` to execute
|
|
Packit Service |
509fd4 |
the test suite.
|
|
Packit Service |
509fd4 |
|
|
Packit Service |
509fd4 |
It is best that you use a fresh system for installing and running the tests!
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
**NOTE:**
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
The easiest way to start osbuild-composer is via systemd because it takes care
|
|
Packit |
63bb0d |
of setting up the UNIX socket for the API server.
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
If you are working on a pull request that adds more integration tests
|
|
Packit |
63bb0d |
(without modifying osbuild-composer itself) then you can execute the test suite
|
|
Packit |
63bb0d |
from the local directory without installing it:
|
|
Packit |
63bb0d |
|
|
Packit |
63bb0d |
* `make build` - will build everything under `cmd/`
|
|
Packit Service |
509fd4 |
* `./osbuild-composer-cli-tests` - will execute the freshly built integration test suite
|
|
Packit Service |
509fd4 |
|
|
Packit Service |
509fd4 |
|
|
Packit Service |
509fd4 |
## Downstream testing notes
|
|
Packit Service |
509fd4 |
|
|
Packit Service |
509fd4 |
To make it easier for us to test & verify downstream builds we are going to
|
|
Packit Service |
509fd4 |
move most of the work upstream and apply the following rules:
|
|
Packit Service |
509fd4 |
|
|
Packit Service |
509fd4 |
1. Preferably the 1st commit of any PR will contain a bug reproducer.
|
|
Packit Service |
509fd4 |
First push a draft PR only containing that commit which will cause CI to FAIL.
|
|
Packit Service |
509fd4 |
2. QE will review and approve the reproducer (can happen in parallel with next item)
|
|
Packit Service |
509fd4 |
3. Subsequent commits provide bug fixes without modifying the reproducer and
|
|
Packit Service |
509fd4 |
CI reports PASS. Push these on top of the approved reproducer.
|
|
Packit Service |
509fd4 |
4. QE has done final review and approved the PR; RHBZ status is set to
|
|
Packit Service |
509fd4 |
`MODIFIED + Verified=Tested`
|
|
Packit Service |
509fd4 |
5. Devel has done final review and approved the PR
|
|
Packit Service |
509fd4 |
|
|
Packit Service |
509fd4 |
**NOTES for devel:**
|
|
Packit Service |
509fd4 |
|
|
Packit Service |
509fd4 |
Pull requests related to new functionality may add their
|
|
Packit Service |
509fd4 |
automated tests together or after commit(s) adding said functionality!
|
|
Packit Service |
509fd4 |
|
|
Packit Service |
509fd4 |
All PRs containing commits referencing `rhbz#` number and/or
|
|
Packit Service |
509fd4 |
all PRs against a dedicated `rhel-` branch should follow the above rules!
|
|
Packit Service |
509fd4 |
|
|
Packit Service |
509fd4 |
**NOTE for QE:**
|
|
Packit Service |
509fd4 |
|
|
Packit Service |
509fd4 |
CI results are also reported against each commit and these
|
|
Packit Service |
509fd4 |
can be used to review the test automation state during a PR lifecycle.
|
|
Packit Service |
509fd4 |
|
|
Packit Service |
509fd4 |
`qa_ack+` on RHBZ will be granted **after** a reproducer has been
|
|
Packit Service |
509fd4 |
identified and with the mutual understanding that PRs related to
|
|
Packit Service |
509fd4 |
that RHBZ must include an automated test reproducer.
|
|
Packit Service |
bcdfb1 |
|
|
Packit Service |
bcdfb1 |
|
|
Packit Service |
15f37d |
## Cron jobs for internal builds testing
|
|
Packit Service |
bcdfb1 |
|
|
Packit Service |
bcdfb1 |
The Schutzbot Pipeline contains conditional sections that facilitate test execution
|
|
Packit Service |
15f37d |
against internal builds. This is achieved by running different preparation steps while
|
|
Packit Service |
bcdfb1 |
the testing stage remains the same. The main difference is that SUT is not compiled
|
|
Packit Service |
bcdfb1 |
locally but installed directly from OS repositories!
|
|
Packit Service |
bcdfb1 |
|
|
Packit Service |
15f37d |
By default we test against latest nightly builds. If you wish to test against other
|
|
Packit Service |
15f37d |
flavors (e.g. rel-eng) specify the `COMPOSE_URL` environment variable to point to
|
|
Packit Service |
15f37d |
the respective URL (stopping before the `/compose/` path). See the image below or
|
|
Packit Service |
15f37d |
`schutzbot/prepare-rhel-internal.sh` for more details.
|
|
Packit Service |
bcdfb1 |
|
|
Packit Service |
15f37d |
!['Define COMPOSE_URL'](./define_compose_url.png)
|
|
Packit Service |
15f37d |
|
|
Packit Service |
15f37d |
### Replay internal Pipeline manually
|
|
Packit Service |
15f37d |
|
|
Packit Service |
15f37d |
If you wish to execute the internal Pipeline by hand, often to verify changes made to it
|
|
Packit Service |
bcdfb1 |
then do the following:
|
|
Packit Service |
bcdfb1 |
|
|
Packit Service |
bcdfb1 |
1. Wait for `schutzbot-psi/pr-head` to report any status on the pull request.
|
|
Packit Service |
bcdfb1 |
This means a regular Pipeline has been started with changes coming from your PR; or
|
|
Packit Service |
bcdfb1 |
1. In Jenkins Blue Ocean UI locate the latest Pipeline execution triggered by timer
|
|
Packit Service |
bcdfb1 |
2. In Jenkins Blue Ocean UI, top-right corner, click ***Go to classic*** button:
|
|
Packit Service |
bcdfb1 |
|
|
Packit Service |
bcdfb1 |
!['Go to classic button'](./go_to_classic.png)
|
|
Packit Service |
bcdfb1 |
|
|
Packit Service |
bcdfb1 |
3. In Jenkins Classic UI, left-hand sidebar click ***Replay*** button. This will allow you to
|
|
Packit Service |
bcdfb1 |
replay a Pipeline with a modified syntax
|
|
Packit Service |
bcdfb1 |
4. Locate the `detect_build_cause()` function near the bottom and modify it so that it
|
|
Packit Service |
bcdfb1 |
will `return "cron"`. This is required because when Pipelines are restarted manually
|
|
Packit Service |
bcdfb1 |
their build cause is **Replayed #xy**. See the images for reference:
|
|
Packit Service |
bcdfb1 |
|
|
Packit Service |
bcdfb1 |
!['Modify build cause'](./pipeline_replay_01.png)
|
|
Packit Service |
bcdfb1 |
|
|
Packit Service |
bcdfb1 |
!['Modify build cause'](./pipeline_replay_02.png)
|
|
Packit Service |
bcdfb1 |
|
|
Packit Service |
bcdfb1 |
5. Click the ***Run*** button - the newly started Pipeline will be forced to take the
|
|
Packit Service |
15f37d |
internal branches instead of the regular ones
|