= OpenSCAP Developer Manual
:oscap_git: https://github.com/OpenSCAP/openscap
:toc: preamble
:numbered:

This part of documentation is meant to serve mainly to developers who want to
contribute to OpenSCAP, help to fix bugs, or take an advantage of
the OpenSCAP library and create own projects on top of it.


== Building OpenSCAP on Linux
If you want to build the `libopenscap` library and the `oscap` tool from
the {oscap_git}[source code] then follow these instructions:

. *Get the source code*
+
Choose *1a* or *1b* depending on whether you want sources from a release tarball or the git repository.

.. Use a release tarball:
+
----
# replace ${version} with the desired version
wget https://github.com/OpenSCAP/openscap/releases/download/${version}/openscap-${version}.tar.gz
tar -xzpf openscap-${version}.tar.gz
cd openscap-${version}
mkdir -p build
----
+
**OR**

.. Use fresh sources from git repository.
+
----
$ git clone --recurse-submodules https://github.com/OpenSCAP/openscap.git
$ cd openscap
----
+

NOTE: We include https://github.com/OpenSCAP/yaml-filter[yaml-filter] library
as a git submodule. To get more information about using git submodules, read
https://git-scm.com/book/en/v2/Git-Tools-Submodules[Git Tools - Submodules].

. *Get the build dependencies*
+
--
To build the library you will also need to install the build dependencies.

Build dependencies may vary depending on enabled features (by the `cmake` command).
Some of the dependencies are optional, if they are not detected, openscap will be compiled
without respective optional features.

On RHEL 7 / Fedora 23 / CentOS 7, the command to install the build dependencies is:

----
sudo yum install \
cmake dbus-devel GConf2-devel libacl-devel libblkid-devel libcap-devel libcurl-devel \
libgcrypt-devel libselinux-devel libxml2-devel libxslt-devel libattr-devel make openldap-devel \
pcre-devel perl-XML-Parser perl-XML-XPath perl-devel python-devel rpm-devel swig \
bzip2-devel gcc-c++ libyaml-devel
----

On Fedora 24+, the command to install the build dependencies is:

----
sudo yum install \
cmake dbus-devel GConf2-devel libacl-devel libblkid-devel libcap-devel libcurl-devel \
libgcrypt-devel libselinux-devel libxml2-devel libxslt-devel libattr-devel make openldap-devel \
pcre-devel perl-XML-Parser perl-XML-XPath perl-devel python3-devel rpm-devel swig \
bzip2-devel gcc-c++ libyaml-devel
----

On RHEL 8 / CentOS 8, the command to install the build dependencies is:

----
sudo yum install \
cmake dbus-devel libacl-devel libblkid-devel libcap-devel libcurl-devel \
libgcrypt-devel libselinux-devel libxml2-devel libxslt-devel libattr-devel make openldap-devel \
pcre-devel perl-XML-Parser perl-XML-XPath perl-devel python36-devel rpm-devel swig \
bzip2-devel gcc-c++ libyaml-devel
----

On Ubuntu 16.04, Debian 8 or Debian 9, the command to install the build dependencies is:

----
sudo apt-get install -y cmake libdbus-1-dev libdbus-glib-1-dev libcurl4-openssl-dev \
libgcrypt20-dev libselinux1-dev libxslt1-dev libgconf2-dev libacl1-dev libblkid-dev \
libcap-dev libxml2-dev libldap2-dev libpcre3-dev python-dev swig libxml-parser-perl \
libxml-xpath-perl libperl-dev libbz2-dev librpm-dev g++ libapt-pkg-dev libyaml-dev
----

When you have all the build dependencies installed you can build the library.
--

. *Build the library*
+
--
Run the following commands to build the library:

----
$ cd build/
$ cmake ../
$ make
----

On Ubuntu 18.04 and potentially other distro, the python3 dist-packages path is wrong.
If the following command:

----
$ python3 -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
----

returns "/usr/local/lib/python3/dist-packages" instead of a path like
"/usr/local/lib/python3.6/dist-packages", you must override this path,
otherwise you will not be able to import openscap_api.

----
$ cmake ../ -DPYTHON_SITE_PACKAGES_INSTALL_DIR=/usr/local/lib/python3.6/dist-packages
----

--

. *Build the HTML documentation*
+
--
It is possible to generate a complete API documentation, User manual,
Developer manual and contribute documents in HTML format.

If you want to build the HTML documentation you will need to install Doxygen
and AsciiDoc.
To install AsciiDoc, you can run `dnf install asciidoc`.
To install Doxygen, you can run `dnf install doxygen`.

Run the following command to build the documentation:

----
$ cmake -DENABLE_DOCS=TRUE ../
$ make docs
----

The resulting documentation is located in `build/docs` directory in these
subdirectories:
--
* `html` - contains the API documentation generated by Doxygen
* `manual` - contains OpenSCAP User Manual
* `developer` - contains OpenSCAP Developer Manual
* `contribute` - contains contribute documents

. *Run the tests*
+
--
After building the library you might want to run library self-checks. To do
that you need to have these additional packages installed:

----
wget lua which procps-ng initscripts chkconfig sendmail bzip2 rpm-build strace
----

On Ubuntu 18.04, also install:

----
rpm-common
----

It is also required to have `sendmail` service running on the system:

----
$ systemctl start sendmail.service
----

Now you can execute the following command to run library self-checks:

----
$ ctest
----

It's also possible to use `ctest` to test any other oscap binary present in the system. You just have to set the path of the binary to the CUSTOM_OSCAP variable:

----
$ export CUSTOM_OSCAP=/usr/bin/oscap; ctest
----

Some tests that use the so-called offline mode of probes need to chroot during the test execution.
Some of those probes use the chroot syscall, which an unprivileged process is not allowed to do.
This is not a problem during the scanning itself, as oscap is usually scanning as root.
However, we don't want to run oscap as root during tests, as the whole test suite would have to use root privileges to clean up.

Instead, build the `oscap-chrootable` target as superuser, or build `oscap-chrootable-nocap` first and then grant the capability manually.
This target creates the binary that the test suite will use for some of those offline tests.
In offline tests, use the `set_offline_test_mode [chroot directory]` and `unset_offline_test_mode` functions from the common test module - those will set variables in such way that the unquoted `$OSCAP` invocation will use the chroot-capable binary, or it will exit with an error code, aborting the test.
Therefore, it is recommended to run

----
$ sudo make oscap-chrootable
----

Not every check tests the oscap tool, however, when the `CUSTOM_OSCAP` variable is set, only the checks which do are executed.

To enable the MITRE tests, use the `ENABLE_MITRE` flag:

----
$ cmake -DENABLE_MITRE=TRUE ..
----

These test require specific features of the environment to function properly; most notably, a MTA needs to be listening on port 25. We suggest using our container `mitre_tests` to test MITRE functionality if possible:

----
$ docker build --tag openscap_mitre_tests:latest -f Dockerfiles/mitre_tests . && docker run openscap_mitre_tests:latest
----

--

. *Install*
+
--
Run the installation procedure by executing the following command:

----
$ make install
----
--

== Running oscap
It is important to use your compiled `libopenscap.so` library with your `oscap` tool.
The easiest way how to achieve that without need to install `libopenscap.so` to the system path, is to use a shell script called *oscap_wrapper* or *run* in the OpenSCAP build directory.

-------------------------------------------------
$ cd build/
$ ./oscap_wrapper xccdf eval ... whatever
$ ./run valgrind utils/oscap xccdf eval ... whatever
-------------------------------------------------

The *run* script is generated at configure time by CMake and it sets the following environment variables:

* *LD_LIBRARY_PATH* - path to `libopenscap.so`
* *OSCAP_SCHEMA_PATH* - path to XCCDF, OVAL, CPE, ... XSD schemas and schematrons
(required for correct SCAP content validation)
* *OSCAP_XSLT_PATH* - path to XSLT transformations. (required if you want
to generate html documents from xml)
* *OSCAP_CPE_PATH* - path to the OpenSCAP CPE dictionary.

The *oscap_wrapper* script is a convenience shortcut for `run utils/oscap` call.


== Debugging
Developers and users who intend to help find and fix possible bugs in OpenSCAP
or possible bugs in their security policies have these possibilities:

=== Verbose mode
The verbose mode provides user additional information about process of system
scanning. The mode is useful for diagnostics of SCAP content evaluation
and also for debugging. It produces a detailed report log with various messages.
The mode is available for `xccdf eval`, `oval eval`, `oval collect`
and `oval analyse` modules.
There is no need to special compilation, the feature is available for all
OpenSCAP users.

To turn the verbose mode on, run `oscap` with this option:

* `--verbose VERBOSITY_LEVEL` - Turn on verbose mode at specified
verbosity level.

The `VERBOSITY_LEVEL` can be one of:

1. *DEVEL* - the most detailed information for developers and bug hunters
2. *INFO* - reports content processing and system scanning
3. *WARNING* - possible failures which OpenSCAP can recover from
4. *ERROR* - shows only serious errors

The verbose messages will be written on standard error output (stderr).
Optionally, you can write the log into a file using
 `--verbose-log-file FILE`.

This is an example describing how to run OpenSCAP in verbose mode:

----
$ oscap oval eval --results results.xml --verbose INFO --verbose-log-file log.txt oval.xml
----

Then see the log using eg.:

----
$ less log.txt
----

=== Debug mode
Debug mode is useful for programmers. You need to build OpenSCAP from source code
with a custom configuration to enable the debug mode. Use this command:

------------------------------------
$ cmake -DCMAKE_BUILD_TYPE=Debug .. && make
------------------------------------

Debug mode provides:

* debug symbols on and optimization off - you can use `gdb`,
every process that was run.
* http://www.gnu.org/software/gawk/manual/html_node/Assert-Function.html[assertions]
are evaluated.


==== Example

----
$ bash ./run gdb --args utils/oscap xccdf eval \
--profile hard --results xccdf-results.xml \
--oval-results my-favourite-xccdf-checklist.xml
----


The `--oval-results` option force `oscap` tool to generate OVAL Result file
for each OVAL session used for evaluation. It's also very useful for
debugging!

=== Environment variables
There are few more environment variables that control `oscap` tool
behaviour.

* *OSCAP_FULL_VALIDATION=1* - validate all exported documents (slower)
* *SEXP_VALIDATE_DISABLE=1* - do not validate SEXP expressions (faster)
* *OSCAP_PCRE_EXEC_RECURSION_LIMIT* - override default recursion limit
  for match in pcre_exec call in textfilecontent(54) probes.



== Generating of code coverage
Code coverage can be usefull during writing of test or performance profiling.
We could separate the process into five phases.

1) *Get dependencies*

----
# dnf install lcov
----

2) *Run CMake & make*

To allow code to generate statistics, we need to compile it with specific flags.

----
$ CFLAGS="--coverage -ftest-coverage -fprofile-arcs" LDFLAGS=-lgcov cmake -DCMAKE_BUILD_TYPE=Debug ../
$ make
----

3) *Run code*

In this phase we should run code. We can run it directly or via test suite.

----
$ bash ./run utils/oscap
----

4) *Generate and browse results*

----
$ lcov -t "OpenSCAP coverage" -o ./coverage.info -c -d .
$ genhtml -o ./coverage ./coverage.info
$ xdg-open ./coverage/index.html # open results in browser
----

5) *Clean stats*

Every run only modify our current statistics and not rewrite them completely.
If we want to generate new statistics, we should remove the old ones.

----
$ lcov --directory ./ --zerocounters ; find ./ -name "*.gcno" | xargs rm
$ rm -rf ./coverage
----

== Building OpenSCAP on Windows using Visual Studio

Prerequisites:

* https://www.visualstudio.com/[Visual Studio]
* https://git-scm.com/[Git]
* https://cmake.org/[CMake]

1) Get dependencies

We will use https://github.com/Microsoft/vcpkg[Vcpkg] to download libraries
that are required to build OpenSCAP.

Click on Start -> Windows System -> Command Prompt.

----
mkdir c:\devel
cd c:\devel
git clone https://github.com/Microsoft/vcpkg.git
cd vcpkg
.\bootstrap-vcpkg.bat
.\vcpkg install curl libxml2 libxslt bzip2 pcre pthreads getopt-win32
.\vcpkg integrate install
----

2) Get OpenSCAP

----
cd c:\devel
git clone -b master https://github.com/OpenSCAP/openscap.git
----

3) Generate Visual Studio Solution

----
cd openscap
cd build
cmake -D ENABLE_PYTHON3=FALSE -D CMAKE_TOOLCHAIN_FILE=c:/devel/vcpkg/scripts/buildsystems/vcpkg.cmake ..
----

4) Open in Visual Studio

1. Launch Visual Studio
2. Click on File -> Open -> Project/Solution...
3. Locate `c:\devel\openscap\build\openscap.sln`

5) Build

1. Select build type (Debug, Release, ...) in the drop-down menu in the top panel.
2. Click on Build -> Build Solution.

Built binaries and their dependencies are now located in `C:\devel\openscap\build\<BUILD_TYPE>\`, eg. `C:\devel\openscap\build\Debug\`

== Building OpenSCAP for Windows on a Linux box (cross-compilation)
Currently it is possible to cross-compile OpenSCAP for Windows only without probes.
The resulting binary is not able to perform scanning.
Instructions for cross-compiling OpenSCAP for Windows:

1) Install the cross-compiler & dependencies

NOTE: mingw32-pthreads needs to be version 5.0 or greater.

-------------------------------------------------------------
# yum install mingw32-gcc mingw32-binutils mingw32-libxml2 \
mingw32-libgcrypt mingw32-pthreads mingw32-libxslt \
mingw32-curl mingw32-pcre \
mingw32-filesystem mingw32-bzip2
-------------------------------------------------------------

2) Checkout the master branch of the OpenSCAP repository

----------------------------------------------------------------------
$ git clone -b master https://github.com/openscap/openscap.git
$ cd openscap
----------------------------------------------------------------------

3) Prepare the build

----------------------------------------------------------------------------------
$ mkdir build-win32
$ cd build-win32
$ mingw32-cmake -D ENABLE_PYTHON3=FALSE -D ENABLE_PROBES=FALSE -D ENABLE_OSCAP_UTIL_DOCKER=FALSE ../
----------------------------------------------------------------------------------

4) Build!

------------------------------
$ make
------------------------------

Resulting `oscap.exe` can be found in the `utils/` directory.


If you would like to send us a patch fixing any Windows
compiling issues, please consult the page about
http://open-scap.org/page/Contribute[contributing to the OpenSCAP
project].


== OpenSCAP Reference Manual
For more information about OpenSCAP library, you can refer to this online
reference manual: http://static.open-scap.org/openscap-1.2/[OpenSCAP
reference manual]. This manual is included in a release tarball and can be
regenerated from project sources by Doxygen documentation system.