|
Packit Service |
4d2de5 |
# osbuild-composer testing information
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
Test binaries, regardless of their scope/type (e.g. unit, API, integration)
|
|
Packit Service |
4d2de5 |
must follow the syntax of the Go
|
|
Packit Service |
4d2de5 |
[testing package](https://golang.org/pkg/testing/), that is implement only
|
|
Packit Service |
4d2de5 |
`TestXxx` functions with their setup/teardown when necessary in a `yyy_test.go`
|
|
Packit Service |
4d2de5 |
file.
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
Test scenario discovery, execution and reporting will be handled by `go test`.
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
Some test files will be executed directly by `go test` during rpm build time
|
|
Packit Service |
4d2de5 |
and/or in CI. These are usually unit tests. Scenarios which require more complex
|
|
Packit Service |
4d2de5 |
setup, e.g. a running osbuild-composer are not intented to be executed directly
|
|
Packit Service |
4d2de5 |
by `go test` at build time. Instead they are intended to be executed as
|
|
Packit Service |
4d2de5 |
stand-alone test binaries on a clean system which has been configured in
|
|
Packit Service |
4d2de5 |
advance (because this is easier/more feasible). These stand-alone test binaries
|
|
Packit Service |
4d2de5 |
are also compiled via `go test -c -o` during rpm build or via `make build`.
|
|
Packit Service |
4d2de5 |
See *Integration testing* for more information.
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
## Image tests
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
In the `test/cases` directory, sample image builds and their tests are
|
|
Packit Service |
4d2de5 |
collected for the various distros, architectures, configuration we support.
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
Each test case describes how the image is built, the expected osbuild
|
|
Packit Service |
4d2de5 |
manifest used internally, the expected image-info output and how to
|
|
Packit Service |
4d2de5 |
boot-test the image.
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
To (re)generate these test cases use the tool
|
|
Packit Service |
4d2de5 |
`tools/test-case-generators/generate-test-cases`.
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
### Setting up Azure upload tests
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
By default, the vhd images are run locally using qemu. However, when
|
|
Packit Service |
4d2de5 |
the right set of environment flags is passed to the osbuild-image-tests,
|
|
Packit Service |
4d2de5 |
it uploads the image to Azure, boots it and tries to ssh into it.
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
#### Required flags
|
|
Packit Service |
4d2de5 |
- `AZURE_STORAGE_ACCOUNT`
|
|
Packit Service |
4d2de5 |
- `AZURE_STORAGE_ACCESS_KEY`
|
|
Packit Service |
4d2de5 |
- `AZURE_CONTAINER_NAME`
|
|
Packit Service |
4d2de5 |
- `AZURE_SUBSCRIPTION_ID`
|
|
Packit Service |
4d2de5 |
- `AZURE_CLIENT_ID`
|
|
Packit Service |
4d2de5 |
- `AZURE_CLIENT_SECRET`
|
|
Packit Service |
4d2de5 |
- `AZURE_TENANT_ID`
|
|
Packit Service |
4d2de5 |
- `AZURE_LOCATION`
|
|
Packit Service |
4d2de5 |
- `AZURE_RESOURCE_GROUP`
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
#### Setting up all the required resources
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
1) Firstly, go to *Subscriptions* in the left-side menu. Here you can find
|
|
Packit Service |
4d2de5 |
the `AZURE_SUBSCRIPTION_ID`.
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
2) Now, you need to create a new resource group. In the left-side menu,
|
|
Packit Service |
4d2de5 |
select *Resource groups*. Click on *Add* above the resource group list.
|
|
Packit Service |
4d2de5 |
The name you choose is your `AZURE_RESOURCE_GROUP`. The region you choose
|
|
Packit Service |
4d2de5 |
is your `AZURE_LOCATION`. However, it must be in the "machine-readable
|
|
Packit Service |
4d2de5 |
form". You can list all the locations with their machine-readable names
|
|
Packit Service |
4d2de5 |
using Azure CLI: `az account list-locations -o table`.
|
|
Packit Service |
4d2de5 |
E.g. the machine-readable name of US East location is `eastus`.
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
Note that terms *location* and *region* are synonyms in Azure's context.
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
3) Storage time! Go to Storage accounts in the left-side menu. Click on
|
|
Packit Service |
4d2de5 |
*Add* above the list. Use the resource group you created in
|
|
Packit Service |
4d2de5 |
the previous step. Also, the region should be the same. The name you
|
|
Packit Service |
4d2de5 |
choose is your `AZURE_STORAGE_ACCOUNT`.
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
After the storage account is created, open it.
|
|
Packit Service |
4d2de5 |
Select *Settings > Access keys*. Choose one of the keys, this is your
|
|
Packit Service |
4d2de5 |
`AZURE_STORAGE_ACCESS_KEY`. Select *Blob service > Containers* and create
|
|
Packit Service |
4d2de5 |
a new one. Its name is your `AZURE_CONTAINER_NAME`.
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
4) Now it’s time to create an application. This is needed because Azure uses
|
|
Packit Service |
4d2de5 |
OAuth to do authorization. In the left-side menu, choose *Azure Active
|
|
Packit Service |
4d2de5 |
Directory*. Go to *Manage > App registrations* and register a new
|
|
Packit Service |
4d2de5 |
application.
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
When it’s created, open it. In the overview, you can see
|
|
Packit Service |
4d2de5 |
the Application (client) ID and the Directory (tenant) ID. These are your
|
|
Packit Service |
4d2de5 |
`AZURE_CLIENT_ID` and `AZURE_TENANT_ID`.
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
Now, go to *Manage > Certificates & Secrets* under your new application
|
|
Packit Service |
4d2de5 |
and create a new client secret. The is your `AZURE_CLIENT_SECRET`.
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
5) The last step is to give the new application access to the resource group.
|
|
Packit Service |
4d2de5 |
This step must be done by Azure administrator (@larskarlitski): Go to
|
|
Packit Service |
4d2de5 |
the *Access control (IAM)* section under the newly created resource group.
|
|
Packit Service |
4d2de5 |
Here, add the new application with the *Developer* role.
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
### Setting up OpenStack upload tests
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
The following environment variables are required
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
- `OS_AUTH_URL`
|
|
Packit Service |
4d2de5 |
- `OS_USERNAME`
|
|
Packit Service |
4d2de5 |
- `OS_PASSWORD`
|
|
Packit Service |
4d2de5 |
- `OS_PROJECT_ID`
|
|
Packit Service |
4d2de5 |
- `OS_DOMAIN_NAME`
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
### Setting up VMware vCenter upload tests
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
The following environment variables are required
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
- `GOVMOMI_URL` - vCenter hostname
|
|
Packit Service |
4d2de5 |
- `GOVMOMI_USERNAME`
|
|
Packit Service |
4d2de5 |
- `GOVMOMI_PASSWORD`
|
|
Packit Service |
4d2de5 |
- `GOVMOMI_DATACENTER`
|
|
Packit Service |
4d2de5 |
- `GOVMOMI_CLUSTER`
|
|
Packit Service |
4d2de5 |
- `GOVMOMI_NETWORK`
|
|
Packit Service |
4d2de5 |
- `GOVMOMI_DATASTORE`
|
|
Packit Service |
4d2de5 |
- `GOVMOMI_FOLDER`
|
|
Packit Service |
4d2de5 |
- `GOVMOMI_INSECURE` - value of 1 will skip checking SSL certificates
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
**WARNING:** when configuring the credentials for Schutzbot we've experienced
|
|
Packit Service |
4d2de5 |
an issue where the first line in the credentials file gets lost resulting in
|
|
Packit Service |
4d2de5 |
incomplete credentials. The work-around is to define a dummy ENV variable on
|
|
Packit Service |
4d2de5 |
the first line!
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
## Notes on asserts and comparing expected values
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
When comparing for expected values in test functions you should use the
|
|
Packit Service |
4d2de5 |
[testify/assert](https://godoc.org/github.com/stretchr/testify/assert) or
|
|
Packit Service |
4d2de5 |
[testify/require](https://godoc.org/github.com/stretchr/testify/require)
|
|
Packit Service |
4d2de5 |
packages. Both of them provide an impressive array of assertions with the
|
|
Packit Service |
4d2de5 |
possibility to use formatted strings as error messages. For example:
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
```
|
|
Packit Service |
4d2de5 |
assert.Nilf(t, err, "Failed to set up temporary repository: %v", err)
|
|
Packit Service |
4d2de5 |
```
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
If you want to fail immediately, not doing any more of the asserts use the
|
|
Packit Service |
4d2de5 |
require package instead of the assert package, otherwise you'll end up with
|
|
Packit Service |
4d2de5 |
panics and nil pointer memory problems.
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
Stand-alone test binaries also have the `-test.failfast` option.
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
## Notes on code coverage
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
Code coverage is recorded in
|
|
Packit Service |
4d2de5 |
[codecov.io](https://codecov.io/github/osbuild/osbuild-composer).
|
|
Packit Service |
4d2de5 |
This information comes only from unit tests and for the time being
|
|
Packit Service |
4d2de5 |
we're not concerned with collecting coverage information from integration
|
|
Packit Service |
4d2de5 |
tests, see `.github/workflows/tests.yml`.
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
## Integration testing
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
This will consume the osbuild-composer API surface via the `composer-cli`
|
|
Packit Service |
4d2de5 |
command line interface. Implementation is under `cmd/osbuild-tests/`.
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
The easiest way to get started with integration testing from a git
|
|
Packit Service |
4d2de5 |
checkout is:
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
* `dnf -y install rpm-build`
|
|
Packit Service |
4d2de5 |
* `dnf -y builddep osbuild-composer.spec`
|
|
Packit Service |
4d2de5 |
* `make rpm` to build the software under test
|
|
Packit Service |
4d2de5 |
* `dnf install rpmbuild/RPMS/x86_64/osbuild-composer-*.rpm` - this will
|
|
Packit Service |
4d2de5 |
install both osbuild-composer, its -debuginfo, -debugsource and -tests packages
|
|
Packit Service |
4d2de5 |
* `systemctl start osbuild-composer`
|
|
Packit Service |
4d2de5 |
* `/usr/libexec/tests/osbuild-composer/osbuild-tests` to execute the test suite.
|
|
Packit Service |
4d2de5 |
It is best that you use a fresh system for installing and running the tests!
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
**NOTE:**
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
The easiest way to start osbuild-composer is via systemd because it takes care
|
|
Packit Service |
4d2de5 |
of setting up the UNIX socket for the API server.
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
If you are working on a pull request that adds more integration tests
|
|
Packit Service |
4d2de5 |
(without modifying osbuild-composer itself) then you can execute the test suite
|
|
Packit Service |
4d2de5 |
from the local directory without installing it:
|
|
Packit Service |
4d2de5 |
|
|
Packit Service |
4d2de5 |
* `make build` - will build everything under `cmd/`
|
|
Packit Service |
4d2de5 |
* `./osbuild-tests` - will execute the freshly built integration test suite
|