diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..1c96873 --- /dev/null +++ b/.gitignore @@ -0,0 +1,8 @@ +*.o +*.so +*.a +*.xml +*.log +*.d +pqos/pqos +rdtset/rdtset diff --git a/.travis.yml b/.travis.yml new file mode 100644 index 0000000..b78f925 --- /dev/null +++ b/.travis.yml @@ -0,0 +1,12 @@ +sudo: required +dist: trusty +before_script: + - sudo apt-get -q update + - sudo apt-get install -y swig cppcheck wget + - wget -P /tmp/ https://raw.githubusercontent.com/torvalds/linux/879be4f378cb412af3a3fe107d35835c99099add/scripts/checkpatch.pl && chmod a+x /tmp/checkpatch.pl + - wget -P /tmp/ https://raw.githubusercontent.com/torvalds/linux/879be4f378cb412af3a3fe107d35835c99099add/scripts/spelling.txt +script: make cppcheck && make style CHECKPATCH=/tmp/checkpatch.pl && make && sudo make install && (cd lib/perl && perl Makefile.PL && make && sudo make install) +language: c +compiler: + - gcc +# - clang diff --git a/ChangeLog b/ChangeLog new file mode 100644 index 0000000..0122662 --- /dev/null +++ b/ChangeLog @@ -0,0 +1,241 @@ +======================================================================= +Release v2.0 2018-06-26 +======================================================================= + +Contributors (alphabetical order): + Colin Ian King + David Williams + Marcel Cornu + Mateusz Starzyk + Michal Aleksinski + Wojciech Andralojc + +Summary: +1. Library + - Added support for resctrl monitoring via OS interface + - Added support for PID group monitoring + - Added support for L2 CDP + - New API added for adding/removing pids to mon groups +2. PQoS Utility + - Added support for PID group monitoring + - Added support for L2 CDP + - Print core values in PID monitoring output +3. rdtset Utility + - Added support for L2 CDP + +====================================================================== +Release v1.2 2017-11-29 +====================================================================== + +Contributors (alphabetical order): + Aaron Hetherington + Brian Dooley + Jessica Bizimungu + Marcel Cornu + Michal Aleksinski + Radoslaw Jablonski + Wojciech Andralojc + +Summary: +1. Library + - Added support for MBA configuration via OS interface +2. PQoS Utility + - Added better feature enumeration functionality + - Added top-pids monitoring functionality + +====================================================================== +Release v1.1 2017-07-19 +====================================================================== + +Contributors (alphabetical order): + Aaron Hetherington + Bernhard M. Wiedemann + Brian Dooley + Jessica Bizimungu + Marcel Cornu + Michal Aleksinski + +Summary: +1. Library + - Added PID support for L2CAT, L3CAT and CDP + - Added global RDT interface enforcement. +2. PQoS Utility + - Added PID support for L2CAT, L3CAT and CDP +3. rdtset Utility + - Added PID support for L2CAT, L3CAT and CDP + - Updated to work with multiple PID's +4. General + - Bug fixes + +====================================================================== +Release v1.0.1 2017-06-06 +====================================================================== + +Contributors (alphabetical order): + Aaron Hetherington + Bernhard M. Wiedemann + Brian Dooley + Marcel Cornu + Michal Aleksinski + +Summary: +1. snmp + - Added OS interface support to Net-SNMP sub-agent +2. General + - Build improvements + - Bug fixes + +====================================================================== +Release v1.0.0 2017-05-16 +====================================================================== + +Contributors (alphabetical order): + Aaron Hetherington + Colin Ian King + Marcel Cornu + Michal Aleksinski + Tomasz Kantecki + +Summary: +1. Library + - Removed NO_PID_API compile time option + - Added OS interface to support L3/L2 CAT & CDP + - Updated support for CMT per PID +2. PQoS Utility + - Added pqos-os & pqos-msr wrapper scripts + - Added option to select OS or MSR interface +3. rdtset Utility + - Added option to select OS or MSR interface +4. Examples + - Support for new OS/MSR interface added to CMT/MBM examples + +====================================================================== +Release v0.1.5-1 2017-02-09 +====================================================================== + +Contributors (alphabetical order): + Aaron Hetherington + Marcel Cornu + Michal Aleksinski + Tomasz Kantecki + Wojciech Andralojc + +Summary: +1. Library + - Perl interface extended to support L2 CAT and CMT + - Logging extended to allow callback and silence functionality + - L2 CAT updated to operate on a per cluster/L2 ID basis + - Support for MBA +2. SNMP + - Added CMT support to Net-SNMP sub-agent +3. PQoS Utility + - L2 CAT updated to operate on a per cluster/L2 ID basis + - Support for MBA +4. rdtset Utility + - L2 CAT updated to operate on a per cluster/L2 ID basis + - Support of MBA + +====================================================================== +Release v0.1.5 2016-09-20 +====================================================================== + +Contributors (alphabetical order): + Aaron Hetherington + Colin Ian King + Fan Du + Marcel Cornu + Pablo Marcos Oltra + Tomasz Kantecki + Wojciech Andralojc + +Summary: +1. General + - Project file layout changed to accommodate extensions + - Support for FreeBSD + - Travis CI +2. Library + - Support for Intel(R) Xeon(R) processor E5 v4 + - DSO built by default + - examples moved to the top folder + - Perl interface created for the library + - PID API no longer compiled by default + - Support for L2 CAT + - Makefile dependencies improved +3. PQoS Utility + - pqos directory created to accommodate the utility files + - Link against DSO + - COS management on a per socket basis + - Support for L2 CAT + - Makefile dependencies improved +4. rdtset Utility + - rdtset directory created to accommodate the utility files + - Link against DSO + - taskset-like functionality and CAT configuration + - Support for L2 CAT + - Makefile dependencies improved +5. Examples + - Copied examples previously located in library directory + - Perl hello world script created +6. SNMP + - Net-SNMP sub-agent created to allow remote CAT configuration +7. SRPM + - Source package file and spec file + +====================================================================== +Release v0.1.4 2016-02-04 +====================================================================== + +Contributors (alphabetical order): + Aaron Hetherington + Colin Ian King + Colm Moore + Jacek Turek + James Hunt + Marcel Cornu + Priya Autee + Tomasz Kantecki + +Summary: +1. Hardware support + Intel(R) Xeon(R) processor E3 v4 support (selected SKU's) +2. Monitoring + - PID/TID monitoring through Linux perf + - IPC (instructions per clock) performance event + - LLC misses performance event + - CSV output + - Monitoring core groups +3. Allocation + - CDP (code data prioritization) detection and management +4. General + - man page + - install and uninstall rules + - cppcheck & coding style rules + +====================================================================== +Release v0.1.3 2015-05-01 +====================================================================== + +Contributors (alphabetical order): + Pandi Maharajan + Priya Autee + Rahul Shah + Tomasz Kantecki + +Summary: +1. Hardware support + Intel(R) Xeon(R) processor E5 v3 support (selected SKU's) + Intel(R) Xeon(R) processor D support +2. Monitoring + - CMT (Cache Monitoring Technology) and + MBM (Memory Bandwidth Monitoring) detection + - XML and text output formats + - output on console or file + - monitoring reset + - top mode - highest LLC occupancy first + - example CMT/MBM application +3. Allocation + - CAT detection & management + - CAT reset + - example CAT application +4. General + - configuration file support diff --git a/INSTALL b/INSTALL new file mode 100644 index 0000000..dfde575 --- /dev/null +++ b/INSTALL @@ -0,0 +1,122 @@ + + +======================================================================== +INSTALLATION Instructions for Intel(R) RDT Software Package + +April 2016 + +======================================================================== + + +Contents +======== + +- Overview +- Installation Steps +- Linux Requirements +- FreeBSD Requirements (experimental) +- Legal Disclaimer + + +Installation Steps +================== + +For installation of the software package untar the gzip image and follow +below instructions. As the results the following components will be built and +installed: +- PQoS DSO and its header file (libpqos.so and pqos.h) +- PQoS utility executable (pqos) +- PQoS utility MSR wrapper script +- PQoS utility OS/Kernel wrapper script +- PQoS man page (pqos.8) +- rdtset tool executable (rdtset) +- rdtset man page (rdtset.8) + +NOTE to FreeBSD users, remember to replace "make" with "gmake" in +the steps described below. + +The following steps are required to compile and install the package: +$ make +$ sudo make install + +"make" compiles all software components of the package. +"sudo make install" installs compiled files into system directories. + +By default, files are installed below /usr/local but it can be changed +with use of PREFIX to install files below /some/where: +$ sudo make install PREFIX=/some/where + +Software package files can be removed but the same PREFIX has to be +used for uninstall and install targets. +To remove files from below default PREFIX: +$ sudo make uninstall +To remove from below /some/where: +$ sudo make uninstall PREFIX=/some/where + +Software package files can be cleaned with "make clean" command. + +NOTE +If you require system wide interface enforcement you can do so by setting the +"RDT_IFACE" environment variable. + +Linux +===== + +CMT, MBM and CAT are configured using Model Specific Registers (MSRs) +to measure occupancy, set up the class of service masks and manage +the association of the cores/logical threads to a class of service. +The pqos software executes in user space, and access to the MSRs is +obtained through a standard Linux* interface. The virtual file system +structure /dev/cpu/CPUNUM/msr provides an interface to read and write +the MSRs. The msr file interface is protected and requires root +privileges. The msr driver might not be auto-loaded and on some +modular kernels the driver may need to be loaded manually: + +$ modprobe msr + +For instructions on package installation please see "Installation Steps" +section. + + +FreeBSD (experimental) +====================== + +CMT, MBM and CAT are configured using Model Specific Registers (MSRs) +to measure occupancy, set up the class of service masks and manage +the association of the cores/logical threads to a class of service. +The pqos software executes in user space, and access to the MSRs is +obtained through a standard FreeBSD* cpuctl driver interface. The virtual +file system structure /dev/cpuctlCPUNUM provides an interface to read +and write the MSR registers. The MSR file interface is protected and +requires root privileges. +The cpuctl driver might not be auto-loaded on some systems. Please follow +cpuctl (4) man page to load cpuctl driver on your system. + +$ man 4 cpuctl + +Please note that all project build scripts have been written for GNU Make so +it is required to install GNU Make on FreeBSD in order to compile the project. + +$ pkg install gmake + +For instructions on package installation please see "Installation Steps" +section. Remember to replace "make" with "gmake" on FreeBSD. + +Currently verified configuration is: +- Intel(R) Xeon(R) processor D +- FreeBSD 9.1 +- GNU Compiler Collection 5 (gcc 5.3.1) +- GNU Make 4.1 + + +Legal Disclaimer +================ + +THIS SOFTWARE IS PROVIDED BY INTEL"AS IS". NO LICENSE, EXPRESS OR +IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS +ARE GRANTED THROUGH USE. EXCEPT AS PROVIDED IN INTEL'S TERMS AND +CONDITIONS OF SALE, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL +DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR +USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO +FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT +OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..90b34a0 --- /dev/null +++ b/LICENSE @@ -0,0 +1,30 @@ +BSD LICENSE + +Copyright(c) 2014-2016 Intel Corporation. All rights reserved. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + * Neither the name of Intel Corporation nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..ad1ec6e --- /dev/null +++ b/Makefile @@ -0,0 +1,95 @@ +############################################################################### +# Makefile script for PQoS sample application +# +# @par +# BSD LICENSE +# +# Copyright(c) 2014-2016 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +# +############################################################################### + +# XXX: modify as desired +PREFIX ?= /usr/local +export PREFIX + +ifdef DEBUG +export DEBUG +endif + +ifdef SHARED +export SHARED +endif + +.PHONY: all clean TAGS install uninstall style cppcheck + +all: + $(MAKE) -C lib + $(MAKE) -C pqos + $(MAKE) -C rdtset + $(MAKE) -C examples/c/CAT + $(MAKE) -C examples/c/CMT_MBM + $(MAKE) -C examples/c/PSEUDO_LOCK + +clean: + $(MAKE) -C lib clean + $(MAKE) -C pqos clean + $(MAKE) -C rdtset clean + $(MAKE) -C examples/c/CAT clean + $(MAKE) -C examples/c/CMT_MBM clean + $(MAKE) -C examples/c/PSEUDO_LOCK clean + +style: + $(MAKE) -C lib style + $(MAKE) -C pqos style + $(MAKE) -C rdtset style + $(MAKE) -C examples/c/CAT style + $(MAKE) -C examples/c/CMT_MBM style + $(MAKE) -C examples/c/PSEUDO_LOCK style + +cppcheck: + $(MAKE) -C lib cppcheck + $(MAKE) -C pqos cppcheck + $(MAKE) -C rdtset cppcheck + $(MAKE) -C examples/c/CAT cppcheck + $(MAKE) -C examples/c/CMT_MBM cppcheck + $(MAKE) -C examples/c/PSEUDO_LOCK cppcheck + +install: + $(MAKE) -C lib install + $(MAKE) -C pqos install + $(MAKE) -C rdtset install + +uninstall: + $(MAKE) -C lib uninstall + $(MAKE) -C pqos uninstall + $(MAKE) -C rdtset uninstall + +TAGS: + find ./ -name "*.[ch]" -print | etags - diff --git a/README b/README new file mode 100644 index 0000000..1a8480b --- /dev/null +++ b/README @@ -0,0 +1,355 @@ +======================================================================== +README for Intel(R) RDT Software Package + +March 2018 + +======================================================================== + + +Contents +======== + +- Overview +- Package Content +- Hardware Support +- OS Support +- Software Compatibility +- Legal Disclaimer + + +Overview +======== + +This software package provides basic support for +Intel(R) Resource Director Technology (Intel(R) RDT) including: +Cache Monitoring Technology (CMT), Memory Bandwidth Monitoring (MBM), +Cache Allocation Technology (CAT), Code and Data Prioritization (CDP) +and Memory Bandwidth Allocation (MBA). + +In principle, the software programs the technologies via +Model Specific Registers (MSR) on a hardware thread basis. +MSR access is arranged via a standard operating system driver: +msr on Linux and cpuctl on FreeBSD. In the most common architectural +implementations, presence of the technologies is detected via the +CPUID instruction. + +In a limited number of special cases where CAT is not architecturally +supported on a particular SKU (but instead a non-architectural +(model-specific) implementation exists) it can be detected via brand +string. This brand string is read from CPUID and compared to a table +of known-supported SKUs. If needed, a final check is to probe the +specific MSR’s to discover hardware capabilities, however it is +recommended that CPUID enumeration should be used wherever possible. + +From software version v1.0.0 the library adds option to use Intel(R) RDT via +available OS interfaces (perf and resctrl on Linux). The library detects +presence of these interfaces and allows to select the preferred one through +a configuration option. +As the result, existing tools like 'pqos' or 'rdtset' can also be used +to manage Intel(R) RDT in an OS compatible way. MSR interface remains +the default option for now. 'pqos' tool wrappers have been added to help with +the interface selection. 'pqos-os' and 'pqos-msr' for OS and MSR interface +operations respectively. + +PID API compile time option has been removed and the APIs are always available. +Note that proper operation of these APIs depends on availability and +selection of OS interface. + +This software package is maintained, updated and developed on +https://github.com/01org/intel-cmt-cat + +https://github.com/01org/intel-cmt-cat/wiki provides FAQ, usage examples and +useful links. + +Please refer to INSTALL file for package installation instructions. + + +Package Content +=============== + +"lib" directory: + Includes software library files providing API's for + technology detection, monitoring and allocation. + Please refer to the library README for more details (lib/README). +“lib/perl” directory: + Includes PQoS library Perl wrapper. + Please refer to the interface README for more details (lib/perl/README). +"pqos" directory: + Includes utility files providing command line access to + Intel(R) RDT. The utility links against the library and programs + the technologies via its API's. + Please refer to the utility README for more details "pqos/README". + Manual page of "pqos" utility also provides information about tool usage: + $ man pqos +"rdtset" directory: + Includes files of utility that provides "taskset"-like functionality and + CAT configuration. + The utility links against the library and programs the technologies + via its API's. + Please refer to the utility README for more details "rdtset/README". + Manual page of "rdtset" utility also provides information about tool usage: + $ man rdtset +"examples" directory: + Includes C and Perl examples of Intel(R) RDT usage via the library API's. + Please refer to README file for more details "examples/README". +"snmp" directory: + Includes Net-SNMP AgentX subagent written in Perl to demonstrate the use of + the PQoS library Perl wrapper API. + Please refer to README file for more details "snmp/README". +"srpm" directory: + Includes *.src.rpm and *.spec files for the software package. +"ChangeLog" file: + Brief description of changes between releases. +"INSTALL" file: + Installation instructions. +"LICENSE" file: + License of the package. + + +Hardware Support +================ + +Table 1. Intel(R) RDT hardware support ++--------------------------------------------------------------------------------------------+ +| | CMT | MBM | L3 CAT | L3 CDP | L2 CAT | MBA | +|--------------------------------------------+-----+-----+--------+--------+--------+--------| +|Intel(R) Xeon(R) processor E5 v3 | Yes | No | Yes (1)| No | No | No | +|--------------------------------------------+-----+-----+--------+--------+--------+--------| +|Intel(R) Xeon(R) processor D | Yes | Yes | Yes (2)| No | No | No | +|--------------------------------------------+-----+-----+--------+--------+--------+--------| +|Intel(R) Xeon(R) processor E3 v4 | No | No | Yes (3)| No | No | No | +|--------------------------------------------+-----+-----+--------+--------+--------+--------| +|Intel(R) Xeon(R) processor E5 v4 | Yes | Yes | Yes (2)| Yes | No | No | +|--------------------------------------------+-----+-----+--------+--------+--------+--------| +|Intel(R) Xeon(R) Scalable Processors (6) | Yes | Yes | Yes (2)| Yes | No | Yes (5)| +|--------------------------------------------+-----+-----+--------+--------+--------+--------| +|Intel(R) Atom(R) processor for Server C3000 | No | No | No | No | Yes (4)| No | ++--------------------------------------------------------------------------------------------+ + +References: +(1) Selected SKU's only: + Intel(R) Xeon(R) processor E5-2658 v3 + Intel(R) Xeon(R) processor E5-2648L v3 + Intel(R) Xeon(R) processor E5-2628L v3 + Intel(R) Xeon(R) processor E5-2618L v3 + Intel(R) Xeon(R) processor E5-2608L v3 + Intel(R) Xeon(R) processor E5-2658A v3 + Four L3 CAT classes of service (CLOS) and a set of pre-defined classes that + should not be changed at run time. + L3 CAT CLOS to hardware thread association can be changed at run time. + +(2) Sixteen L3 CAT classes of service (CLOS). There are no pre-defined + classes of service and they can be changed at run time. + L3 CAT CLOS to hardware thread association can be changed at run time. + +(3) Selected SKU's only: + Intel(R) Xeon(R) processor E3-1258L v4 + Intel(R) Xeon(R) processor E3-1278L v4 + Four L3 CAT classes of service (CLOS) and a set of pre-defined classes that + should not be changed at run time. + L3 CAT CLOS to hardware thread association can be changed at run time. + +(4) Four L2 CAT classes of service (CLOS) per L2 cluster. + L2 CAT CLOS to hardware thread association can be changed at run time. + +(5) Eight MBA classes of service (CLOS). There are no pre-defined + classes of service and they can be changed at run time. + MBA CLOS to hardware thread association can be changed at run time. + +(6) Please find errata for Intel(R) Xeon(R) Processor Scalable Family at: + https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-scalable-spec-update.pdf?asset=14615 + + +For additional Intel(R) RDT details please refer to the Intel(R) +Architecture Software Development Manuals available at: +http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html +Specific information can be found in volume 3, Chapters 17.18 and 17.19 (revision 064). + + +OS Support +========== + +1. Overview +Linux is the primary supported operating system at the moment. There is a +FreeBSD port of the software but due to limited validation scope it is rather +experimental at this stage. Although most modern Linux kernels include support +for Intel(R) RDT, the intel-cmt-cat software package predates these extensions +and can operate with and without kernel support. The intel-cmt-cat software can +detect and leverage these kernel extensions when available to add functionality, +but is also compatible with legacy kernels. + +2. OS Frameworks +Linux kernel support for Intel(R) RDT was originally introduced with Linux perf +system call extensions for CMT and MBM. More recently, the Resctrl interface +added support for CAT, CDP and MBA. On modern Linux kernels, it is advised to +use the kernel/OS interface when available. Details about these interfaces can +be found in intel_rdt_ui.txt. This software package, intel-cmt-cat, remains to +work seamlessly in all Linux kernel versions. + +3. Interfaces +The intel-cmt-cat software library and utilities offer two interfaces to program +Intel(R) RDT technologies, these are the MSR & OS interfaces. + +The MSR interface is used to configure the platform by programming the hardware +(MSR's) directly. This is the legacy interface and requires no kernel support +for Intel(R) RDT but is limited to monitoring and managing resources on a per +core basis. + +The OS interface was later added to the package and when selected, the library +will leverage Linux kernel extensions to program these technologies. This allows +monitoring and managing resources on a per core/process basis and should be used +when available. + +Please see the tables below for more information on when Intel(R) RDT feature +(MSR & OS) support was added to the package. + +Table 2. MSR interface feature support ++---------------+-------------------+----------------+ +| intel-cmt-cat | RDT feature | Kernel version | +| version | enabled | required | ++---------------+-------------------+----------------+ +| 0.1.3 | L3 CAT, CMT, MBM | Any | ++---------------+-------------------+----------------+ +| 0.1.4 | L3 CDP | Any | ++---------------+-------------------+----------------+ +| 0.1.5 | L2 CAT | Any | ++---------------+-------------------+----------------+ +| 1.2.0 | MBA | Any | ++---------------+-------------------+----------------+ +| 2.0.0 | L2 CDP | Any | ++---------------+-------------------+----------------+ + +Table 3. OS interface feature support ++---------------+-------------------+----------------+----------------------------------------------------+ +| intel-cmt-cat | RDT feature | Kernel version | Recommended interface | +| version | enabled | required | | ++---------------+-------------------+----------------+----------------------------------------------------+ +| 0.1.4 | CMT (Perf) | 4.1 | MSR (1) | ++---------------+-------------------+----------------+----------------------------------------------------+ +| 1.0.0 | MBM (Perf) | 4.7 | MSR (1) | ++---------------+-------------------+----------------+----------------------------------------------------+ +| 1.1.0 | L3 CAT, L3 CDP, | 4.10 | OS for allocation only (with the exception of MBA) | +| | L2 CAT (Resctrl) | | MSR for allocation + monitoring (2) | ++---------------+-------------------+----------------+----------------------------------------------------+ +| 1.2.0 | MBA (Resctrl) | 4.12 | OS for allocation only | +| | | | MSR for allocation + monitoring (2) | ++---------------+-------------------+----------------+----------------------------------------------------+ +| 2.0.0 | CMT, MBM (Resctrl)| 4.14 | OS | ++---------------+-------------------+----------------+----------------------------------------------------+ +| 2.0.0 | L2 CDP | 4.16 | OS | ++---------------+-------------------+----------------+----------------------------------------------------+ + +References: +(1) Monitoring with Perf on a per core basis is not supported and returns invalid results. +(2) The MSR and OS interfaces are not compatible. MSR interface is recommended if monitoring and allocation is to be used. + +4. Software dependencies +The only dependencies of intel-cmt-cat is access to C and pthreads libraries and: +- without kernel extensions - 'msr' kernel module +- with kernel extensions - Intel(R) RDT extended Perf system call and Resctrl filesystem + +Enable Intel(R) RDT support in: +- kernel v4.10 - v4.13 with kernel configuration option CONFIG_INTEL_RDT_A +- kernel v4.14+ with kernel configuration option CONFIG_INTEL_RDT + +Note: No kernel configuration options required before v4.10. + + +Software Compatibility +====================== + +In short, using intel-cmt-cat or Intel(R) PCM software together with +Linux perf and cgroup frameworks is not allowed at the moment. + +As disappointing as it is, use of Linux perf for CMT & MBM and +intel-cmt-cat for CAT & CDP is not allowed. This is because +Linux perf overrides existing CAT configuration during its operations. + +There are a number of options to choose from in order to make use of CAT: +- intel-cmt-cat software for CMT/MBM/CAT and CDP (core granularity only) +- use Linux resctrl for CAT and Linux perf for monitoring (kernel 4.10+) +- patch kernel with an out of tree cgroup patch (CAT) and + only use perf for monitoring (CMT kernels 4.1+, MBM kernels 4.6+) + +Table 4. Software interoperability matrix ++-----------------------------------------------------------------------------------------------+ +| | intel-cmt-cat | Intel(R) PCM | Linux perf | Linux cgroup | Linux resctrl | +|------------------+---------------+--------------+--------------+--------------|---------------| +|intel-cmt-cat | Yes(1) | Yes(2) | Yes(5) | No | Yes(5) | +|------------------+---------------+--------------+--------------+--------------|---------------| +|Intel(R) PCM | Yes(2) | Yes | No | No | No | +|------------------+---------------+--------------+--------------+--------------|---------------| +|Linux perf | Yes(5) | No | Yes | Yes(3) | Yes | +|------------------+---------------+--------------+--------------+--------------|---------------| +|Linux cgroup | No | No | Yes | Yes(3) | No | +|------------------+---------------+--------------+--------------+--------------|---------------| +|Linux resctrl (4) | Yes(5) | No | Yes | No | Yes | ++-----------------------------------------------------------------------------------------------+ + +References: +(1) pqos monitoring from intel-cmt-cat can detect other + pqos monitoring processes in the system. + rdtset from intel-cmt-cat detects other processes started with rdtset and + it will not use their CAT/CDP resources. + +(2) pqos from intel-cmt-cat can detect that Intel(R) PCM monitors cores and + it will not attempt to hijack the cores unless forced. + However, if pqos monitoring is started first and then + Intel(R) PCM is started then the latter one will hijack monitoring + infrastructure from pqos for its use. + +(3) Linux cgroup is an out of tree patch + https://github.com/vshiva1/linux/tree/catv15 + +(4) Linux kernel version 4.10 and newer. + A wiki for Intel resctrl is available at: + https://github.com/01org/intel-cmt-cat/wiki/resctrl + +(5) Only with Linux kernel version 4.10 (and newer), + intel-cmt-cat version 1.0.0 (and newer) with selected OS interface + See '-I' option in 'man pqos' or 'pqos-os'. + +Intel(R) PCM is available at +https://software.intel.com/en-us/articles/intel-performance-counter-monitor + + +Table 5. Intel(R) RDT software enabling status. ++---------------------------------------------------------------------------------------+ +| | Core | Task | CMT | MBM | L3 CAT | L3 CDP | L2 CAT | MBA | +|------------------+-------+-------+-------+-------+--------+---------+--------+--------| +|intel-cmt-cat | Yes | Yes(7)| Yes | Yes | Yes | Yes | Yes | Yes | +|------------------+-------+-------+-------+-------+--------+---------+--------+--------| +|Intel(R) PCM | Yes | No | Yes | Yes | No | No | No | No | +|------------------+-------+-------+-------+-------+--------+---------+--------+--------| +|Linux perf | Yes(6)| Yes | Yes(1)| Yes(2)| No(3) | No(3) | No(3) | No | +|------------------+-------+-------+-------+-------+--------+---------+--------+--------| +|Linux cgroup | No | Yes | No | No | Yes(4) | No | No | No | +|------------------+-------+-------+-------+-------+--------+---------+--------+--------| +|Linux resctrl (5) | Yes | Yes | No | No | Yes | Yes | Yes | No | ++---------------------------------------------------------------------------------------+ + +Legend: +Core - use of technology with core granularity +Task - use of technology per task or group of tasks + +References: +(1) Linux kernel version 4.1 and newer +(2) Linux kernel version 4.6 and newer +(3) Linux perf corrupts CAT and CDP configuration even though + it doesn't enable it +(4) This is out of tree patch and relies on Linux perf enabling +(5) Linux kernel version 4.10 and newer +(6) perf API allows for CMT/MBM core monitoring but returned values are incorrect +(7) intel-cmt-cat version 1.0.0 monitoring only and depends on kernel support + +Legal Disclaimer +================ + +THIS SOFTWARE IS PROVIDED BY INTEL"AS IS". NO LICENSE, EXPRESS OR +IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS +ARE GRANTED THROUGH USE. EXCEPT AS PROVIDED IN INTEL'S TERMS AND +CONDITIONS OF SALE, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL +DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR +USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO +FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT +OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. diff --git a/examples/c/CAT/Makefile b/examples/c/CAT/Makefile new file mode 100644 index 0000000..edeb56b --- /dev/null +++ b/examples/c/CAT/Makefile @@ -0,0 +1,87 @@ +############################################################################### +# Makefile script for CAT PQoS sample applications +# +# @par +# BSD LICENSE +# +# Copyright(c) 2014-2016 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +# +############################################################################### + +LIBDIR ?= ../../../lib +CFLAGS =-I$(LIBDIR) \ + -W -Wall -Wextra -Wstrict-prototypes -Wmissing-prototypes \ + -Wmissing-declarations -Wold-style-definition -Wpointer-arith \ + -Wcast-qual -Wundef -Wwrite-strings \ + -Wformat -Wformat-security -fstack-protector -fPIE -D_FORTIFY_SOURCE=2 \ + -Wunreachable-code -Wsign-compare -Wno-endif-labels \ + -g -O2 +ifneq ($(EXTRA_CFLAGS),) +CFLAGS += $(EXTRA_CFLAGS) +endif +LDFLAGS=-L$(LIBDIR) -pie -z noexecstack -z relro -z now +LDLIBS=-lpqos -lpthread + +# ICC and GCC options +ifeq ($(CC),icc) +else +CFLAGS += -Wcast-align \ + -Wnested-externs \ + -Wmissing-noreturn +endif + +# Build targets and dependencies +ALLOCAPP = allocation_app +ASSOCAPP = association_app +RESETAPP = reset_app + +all: $(ALLOCAPP) $(ASSOCAPP) $(RESETAPP) + +$(ALLOCAPP): $(ALLOCAPP).o +$(ASSOCAPP): $(ASSOCAPP).o +$(RESETAPP): $(RESETAPP).o + +.PHONY: clean +clean: + -rm -f $(ALLOCAPP) $(ASSOCAPP) $(RESETAPP) *.o + +CHECKPATCH?=checkpatch.pl +.PHONY: style +style: + $(CHECKPATCH) --no-tree --no-signoff --emacs \ + --ignore CODE_INDENT,INITIALISED_STATIC,LEADING_SPACE,SPLIT_STRING,UNSPECIFIED_INT \ + -f allocation_app.c -f association_app.c -f reset_app.c + +CPPCHECK?=cppcheck +.PHONY: cppcheck +cppcheck: + $(CPPCHECK) --enable=warning,portability,performance,unusedFunction,missingInclude \ + --std=c99 -I$(LIBDIR) --template=gcc \ + allocation_app.c association_app.c reset_app.c diff --git a/examples/c/CAT/README b/examples/c/CAT/README new file mode 100644 index 0000000..ddf975a --- /dev/null +++ b/examples/c/CAT/README @@ -0,0 +1,69 @@ +================================================================================ +README for CAT Sample Code + +April 2016 +================================================================================ + +CONTENTS +======== + +- Overview +- Compilation +- Usage +- Legal Disclaimer + + +OVERVIEW +======== + +This is example code to demonstrate the use of PQoS/Intel(R) Resource Director +Technology (Intel(R) RDT) library APIs to manage Cache Allocation Technology +(CAT). Refer to https://github.com/01org/intel-cmt-cat/blob/master/README table +1 for a list of processors supporting CAT. CAT sample application build will +create three targets as follows: +1. allocation_app - Demonstrates usage of PQoS/Intel(R) RDT library APIs +related to set bit mask for class of service (CLOS) and displaying +class of service (CLOS) and associated bit mask. +2. association_app - Demonstrates usage of PQoS/Intel(R) RDT library APIs +related to association of class of service (CLOS) to cores and displaying +class of service (CLOS) and core association. +3. reset_app - Demonstrates usage of PQoS/Intel(R) RDT library APIs related to +resetting all classes of service to system default bit mask. +All apps operate in user space and use PQoS/Intel(R) RDT and C libraries only. + + +COMPILATION +=========== + +Note: The PQoS/Intel(R) RDT library should be installed before compilation. + +Run "make all" or "make" to compile the programs. If compilation is successful +"allocation_app", "association_app" and "reset_app" binaries should be present +in the directory. + +Run "make clean" to clean the build files. + + +USAGE +===== + +To run: +$ sudo ./allocation_app +$ sudo ./association_app +$ sudo ./reset_app + +For more usage options: +Run the app as stated up above with the "-h" option. + + +LEGAL DISCLAIMER +================ + +THIS SOFTWARE IS PROVIDED BY INTEL"AS IS". NO LICENSE, EXPRESS OR +IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS +ARE GRANTED THROUGH USE. EXCEPT AS PROVIDED IN INTEL'S TERMS AND +CONDITIONS OF SALE, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL +DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR +USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO +FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT +OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. diff --git a/examples/c/CAT/allocation_app.c b/examples/c/CAT/allocation_app.c new file mode 100644 index 0000000..bfa14ee --- /dev/null +++ b/examples/c/CAT/allocation_app.c @@ -0,0 +1,242 @@ +/* + * BSD LICENSE + * + * Copyright(c) 2014-2016 Intel Corporation. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +/** + * @brief Platform QoS sample COS allocation application + * + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include "pqos.h" + +/** + * Maintains number of Class of Services supported by socket for + * L3 cache allocation + */ +static int sel_l3ca_cos_num = 0; +/** + * Maintains table for L3 cache allocation class of service data structure + */ +static struct pqos_l3ca sel_l3ca_cos_tab[PQOS_MAX_L3CA_COS]; +/** + * @brief Converts string into 64-bit unsigned number. + * + * Numbers can be in decimal or hexadecimal format. + * + * On error, this functions causes process to exit with FAILURE code. + * + * @param s string to be converted into 64-bit unsigned number + * + * @return Numeric value of the string representing the number + */ +static uint64_t +strtouint64(const char *s) +{ + const char *str = s; + int base = 10; + uint64_t n = 0; + char *endptr = NULL; + + if (strncasecmp(s, "0x", 2) == 0) { + base = 16; + s += 2; + } + n = strtoull(s, &endptr, base); + if (endptr != NULL && *endptr != '\0' && !isspace(*endptr)) { + printf("Error converting '%s' to unsigned number!\n", str); + exit(EXIT_FAILURE); + } + return n; +} +/** + * @brief Verifies and translates definition of single + * allocation class of service + * from args into internal configuration. + * + * @param argc Number of arguments in input command + * @param argv Input arguments for COS allocation + */ +static void +allocation_get_input(int argc, char *argv[]) +{ + uint64_t mask = 0; + + if (argc < 2) + sel_l3ca_cos_num = 0; + else if (!strcmp(argv[1], "-h") || !strcmp(argv[1], "-H")) { + printf("Usage: %s [ ]\n", argv[0]); + printf("Example: %s 1 0xff\n\n", argv[0]); + sel_l3ca_cos_num = 0; + } else { + sel_l3ca_cos_tab[0].class_id = (unsigned)atoi(argv[1]); + mask = strtouint64(argv[2]); + sel_l3ca_cos_tab[0].u.ways_mask = mask; + sel_l3ca_cos_num = 1; + } +} +/** + * @brief Sets up allocation classes of service on selected CPU sockets + * + * @param sock_count number of CPU sockets + * @param sockets arrays with CPU socket id's + * + * @return Number of classes of service set + * @retval 0 no class of service set (nor selected) + * @retval negative error + * @retval positive success + */ +static int +set_allocation_class(unsigned sock_count, + const unsigned *sockets) +{ + int ret; + + while (sock_count > 0 && sel_l3ca_cos_num > 0) { + ret = pqos_l3ca_set(*sockets, + sel_l3ca_cos_num, + sel_l3ca_cos_tab); + if (ret != PQOS_RETVAL_OK) { + printf("Setting up cache allocation class of " + "service failed!\n"); + return -1; + } + sock_count--; + sockets++; + } + return sel_l3ca_cos_num; +} +/** + * @brief Prints allocation configuration + * @param sock_count number of CPU sockets + * @param sockets arrays with CPU socket id's + * + * @return PQOS_RETVAL_OK on success + * @return error value on failure + */ +static int +print_allocation_config(const unsigned sock_count, + const unsigned *sockets) +{ + int ret = PQOS_RETVAL_OK; + unsigned i; + + for (i = 0; i < sock_count; i++) { + struct pqos_l3ca tab[PQOS_MAX_L3CA_COS]; + unsigned num = 0; + + ret = pqos_l3ca_get(sockets[i], PQOS_MAX_L3CA_COS, + &num, tab); + if (ret == PQOS_RETVAL_OK) { + unsigned n = 0; + + printf("L3CA COS definitions for Socket %u:\n", + sockets[i]); + for (n = 0; n < num; n++) { + printf(" L3CA COS%u => MASK 0x%llx\n", + tab[n].class_id, + (unsigned long long)tab[n].u.ways_mask); + } + } else { + printf("Error:%d", ret); + return ret; + } + } + return ret; +} +int main(int argc, char *argv[]) +{ + struct pqos_config cfg; + const struct pqos_cpuinfo *p_cpu = NULL; + const struct pqos_cap *p_cap = NULL; + unsigned sock_count, *p_sockets = NULL; + int ret, exit_val = EXIT_SUCCESS; + + memset(&cfg, 0, sizeof(cfg)); + cfg.fd_log = STDOUT_FILENO; + cfg.verbose = 0; + /* PQoS Initialization - Check and initialize CAT and CMT capability */ + ret = pqos_init(&cfg); + if (ret != PQOS_RETVAL_OK) { + printf("Error initializing PQoS library!\n"); + exit_val = EXIT_FAILURE; + goto error_exit; + } + /* Get CMT capability and CPU info pointer */ + ret = pqos_cap_get(&p_cap, &p_cpu); + if (ret != PQOS_RETVAL_OK) { + printf("Error retrieving PQoS capabilities!\n"); + exit_val = EXIT_FAILURE; + goto error_exit; + } + /* Get CPU socket information to set COS */ + p_sockets = pqos_cpu_get_sockets(p_cpu, &sock_count); + if (p_sockets == NULL) { + printf("Error retrieving CPU socket information!\n"); + exit_val = EXIT_FAILURE; + goto error_exit; + } + /* Get input from user */ + allocation_get_input(argc, argv); + if (sel_l3ca_cos_num != 0) { + /* Set bit mask for COS allocation */ + ret = set_allocation_class(sock_count, p_sockets); + if (ret < 0) { + printf("Allocation configuration error!\n"); + goto error_exit; + } + printf("Allocation configuration altered.\n"); + } + /* Print COS and associated bit mask */ + ret = print_allocation_config(sock_count, p_sockets); + if (ret != PQOS_RETVAL_OK) { + printf("Allocation capability not detected!\n"); + exit_val = EXIT_FAILURE; + goto error_exit; + } + error_exit: + /* reset and deallocate all the resources */ + ret = pqos_fini(); + if (ret != PQOS_RETVAL_OK) + printf("Error shutting down PQoS library!\n"); + if (p_sockets != NULL) + free(p_sockets); + return exit_val; +} diff --git a/examples/c/CAT/association_app.c b/examples/c/CAT/association_app.c new file mode 100644 index 0000000..9a13407 --- /dev/null +++ b/examples/c/CAT/association_app.c @@ -0,0 +1,213 @@ +/* + * BSD LICENSE + * + * Copyright(c) 2014-2016 Intel Corporation. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +/** + * @brief Platform QoS sample COS association application + * + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include "pqos.h" + +/** + * Defines + */ +#define PQOS_MAX_CORES (1024) + +/** + * Number of cores selected for cache allocation association + */ +static int sel_l3ca_assoc_num = 0; + +/** + * Maintains a table of core and class_id that are selected in config string for + * setting up allocation policy per core + */ +static struct { + unsigned core; + unsigned class_id; +} sel_l3ca_assoc_tab[PQOS_MAX_CORES]; + +/** + * @brief Verifies and translates association config string into + * internal configuration. + * + * @param argc Number of arguments in input command + * @param argv Input arguments for COS association + */ +static void +enforcement_get_input(int argc, char *argv[]) +{ + int i = 0; + + if (argc < 2) + sel_l3ca_assoc_num = 0; + else if (!strcmp(argv[1], "-h") || !strcmp(argv[1], "-H")) { + printf("Usage: %s [ ...]\n", + argv[0]); + printf("Eg : %s 1 1 3 6\n\n", argv[0]); + sel_l3ca_assoc_num = 0; + } else { + for (i = 0; i < argc-2; i++) { + sel_l3ca_assoc_tab[i].class_id = + (unsigned) atoi(argv[1]); + sel_l3ca_assoc_tab[i].core = + (unsigned) atoi(argv[i+2]); + } + sel_l3ca_assoc_num = (int) i; + } +} + +/** + * @brief Prints information about cache allocation settings in the system + */ +static void +print_allocation_config(void) +{ + int ret; + unsigned i; + unsigned sock_count, *sockets = NULL; + const struct pqos_cpuinfo *p_cpu = NULL; + const struct pqos_cap *p_cap = NULL; + + /* Get CMT capability and CPU info pointer */ + ret = pqos_cap_get(&p_cap, &p_cpu); + if (ret != PQOS_RETVAL_OK) { + printf("Error retrieving PQoS capabilities!\n"); + return; + } + /* Get CPU socket information to set COS */ + sockets = pqos_cpu_get_sockets(p_cpu, &sock_count); + if (sockets == NULL) { + printf("Error retrieving CPU socket information!\n"); + return; + } + for (i = 0; i < sock_count; i++) { + unsigned *lcores = NULL; + unsigned lcount = 0, n = 0; + + lcores = pqos_cpu_get_cores(p_cpu, sockets[i], &lcount); + if (lcores == NULL || lcount == 0) { + printf("Error retrieving core information!\n"); + free(lcores); + free(sockets); + return; + } + printf("Core information for socket %u:\n", + sockets[i]); + for (n = 0; n < lcount; n++) { + unsigned class_id = 0; + + ret = pqos_alloc_assoc_get(lcores[n], &class_id); + if (ret == PQOS_RETVAL_OK) + printf(" Core %u => COS%u\n", + lcores[n], class_id); + else + printf(" Core %u => ERROR\n", + lcores[n]); + } + free(lcores); + } + free(sockets); +} +/** + * @brief Sets up association between cores and allocation classes of service + * + * @return Number of associations made + * @retval 0 no association made (nor requested) + * @retval negative error + * @retval positive sucess + */ +static int +set_allocation_assoc(void) +{ + int i; + + for (i = 0; i < sel_l3ca_assoc_num; i++) { + int ret; + + ret = pqos_alloc_assoc_set(sel_l3ca_assoc_tab[i].core, + sel_l3ca_assoc_tab[i].class_id); + if (ret != PQOS_RETVAL_OK) { + printf("Setting allocation class of service " + "association failed!\n"); + return -1; + } + } + return sel_l3ca_assoc_num; +} + +int main(int argc, char *argv[]) +{ + struct pqos_config cfg; + int ret, exit_val = EXIT_SUCCESS; + + memset(&cfg, 0, sizeof(cfg)); + cfg.fd_log = STDOUT_FILENO; + cfg.verbose = 0; + /* PQoS Initialization - Check and initialize CAT and CMT capability */ + ret = pqos_init(&cfg); + if (ret != PQOS_RETVAL_OK) { + printf("Error initializing PQoS library!\n"); + exit_val = EXIT_FAILURE; + goto error_exit; + } + + /* Get input from user */ + enforcement_get_input(argc, argv); + if (sel_l3ca_assoc_num) { + /* Enforce COS to the associated cores */ + ret = set_allocation_assoc(); + if (ret < 0) { + printf("CAT association error!\n"); + goto error_exit; + } + printf("Allocation configuration altered.\n"); + } + /* Print COS and associated cores */ + print_allocation_config(); +error_exit: + /* reset and deallocate all the resources */ + ret = pqos_fini(); + if (ret != PQOS_RETVAL_OK) + printf("Error shutting down PQoS library!\n"); + return exit_val; +} diff --git a/examples/c/CAT/reset_app.c b/examples/c/CAT/reset_app.c new file mode 100644 index 0000000..864def0 --- /dev/null +++ b/examples/c/CAT/reset_app.c @@ -0,0 +1,172 @@ +/* + * BSD LICENSE + * + * Copyright(c) 2014-2018 Intel Corporation. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +/** + * @brief Platform QoS sample COS reset application + * + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include "pqos.h" + +/** + * @brief Prints information about cache allocation settings in the system + * + * @param sock_count number of detected CPU sockets + * @param sockets arrays with detected CPU socket id's + * @param cpu_info cpu information structure + */ +static void +print_allocation_config(const struct pqos_capability *cap_l3ca, + const unsigned sock_count, + const unsigned *sockets, + const struct pqos_cpuinfo *cpu_info) +{ + int ret; + unsigned i; + + if (cap_l3ca == NULL) + return; + + for (i = 0; i < sock_count; i++) { + struct pqos_l3ca tab[PQOS_MAX_L3CA_COS]; + unsigned num = 0; + + ret = pqos_l3ca_get(sockets[i], PQOS_MAX_L3CA_COS, + &num, tab); + if (ret == PQOS_RETVAL_OK) { + unsigned n = 0; + + printf("L3CA COS definitions for Socket %u:\n", + sockets[i]); + for (n = 0; n < num; n++) { + printf(" L3CA COS%u => MASK 0x%llx\n", + tab[n].class_id, + (unsigned long long)tab[n].u.ways_mask); + } + } + } + + for (i = 0; i < sock_count; i++) { + unsigned *lcores = NULL; + unsigned lcount = 0, n = 0; + + lcores = pqos_cpu_get_cores(cpu_info, sockets[i], &lcount); + if (lcores == NULL || lcount == 0) { + printf("Error retrieving core information!\n"); + free(lcores); + return; + } + printf("Core information for socket %u:\n", + sockets[i]); + for (n = 0; n < lcount; n++) { + unsigned class_id = 0; + int ret1 = PQOS_RETVAL_OK; + + if (cap_l3ca != NULL) + ret1 = pqos_alloc_assoc_get(lcores[n], + &class_id); + if (ret1 == PQOS_RETVAL_OK) + printf(" Core %u => COS%u\n", + lcores[n], class_id); + else + printf(" Core %u => ERROR\n", + lcores[n]); + } + free(lcores); + } +} + +int main(int argc, char *argv[]) +{ + struct pqos_config cfg; + const struct pqos_cpuinfo *p_cpu = NULL; + const struct pqos_cap *p_cap = NULL; + const struct pqos_capability *cap_l3ca = NULL; + unsigned sock_count, *sockets = NULL; + int ret, exit_val = EXIT_SUCCESS; + + memset(&cfg, 0, sizeof(cfg)); + cfg.fd_log = STDOUT_FILENO; + cfg.verbose = 0; + /* PQoS Initialization - Check and initialize CAT and CMT capability */ + ret = pqos_init(&cfg); + if (ret != PQOS_RETVAL_OK) { + printf("Error initializing PQoS library!\n"); + exit_val = EXIT_FAILURE; + goto error_exit; + } + /* Get CMT capability and CPU info pointer */ + ret = pqos_cap_get(&p_cap, &p_cpu); + if (ret != PQOS_RETVAL_OK) { + printf("Error retrieving PQoS capabilities!\n"); + exit_val = EXIT_FAILURE; + goto error_exit; + } + if (argc == 2 && (!strcmp(argv[1], "-h") || !strcmp(argv[1], "-H"))) { + printf("Usage: %s\n\n", argv[0]); + goto error_exit; + } + /* Reset Api */ + ret = pqos_alloc_reset(PQOS_REQUIRE_CDP_ANY, PQOS_REQUIRE_CDP_ANY); + if (ret != PQOS_RETVAL_OK) + printf("CAT reset failed!\n"); + else + printf("CAT reset successful\n"); + /* Get CPU socket information to set COS */ + sockets = pqos_cpu_get_sockets(p_cpu, &sock_count); + if (sockets == NULL) { + printf("Error retrieving CPU socket information!\n"); + exit_val = EXIT_FAILURE; + goto error_exit; + } + (void) pqos_cap_get_type(p_cap, PQOS_CAP_TYPE_L3CA, &cap_l3ca); + /* Print COS and associated cores */ + print_allocation_config(cap_l3ca, sock_count, sockets, p_cpu); + error_exit: + /* reset and deallocate all the resources */ + ret = pqos_fini(); + if (ret != PQOS_RETVAL_OK) + printf("Error shutting down PQoS library!\n"); + if (sockets != NULL) + free(sockets); + return exit_val; +} diff --git a/examples/c/CMT_MBM/Makefile b/examples/c/CMT_MBM/Makefile new file mode 100644 index 0000000..ed37607 --- /dev/null +++ b/examples/c/CMT_MBM/Makefile @@ -0,0 +1,83 @@ +############################################################################### +# Makefile script for PQoS sample application +# +# @par +# BSD LICENSE +# +# Copyright(c) 2014-2016 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +# +############################################################################### + +LIBDIR ?= ../../../lib +CFLAGS =-I$(LIBDIR) -pthread \ + -W -Wall -Wextra -Wstrict-prototypes -Wmissing-prototypes \ + -Wmissing-declarations -Wold-style-definition -Wpointer-arith \ + -Wcast-qual -Wundef -Wwrite-strings \ + -Wformat -Wformat-security -fstack-protector -fPIE -D_FORTIFY_SOURCE=2 \ + -Wunreachable-code -Wsign-compare -Wno-endif-labels \ + -g -O2 +ifneq ($(EXTRA_CFLAGS),) +CFLAGS += $(EXTRA_CFLAGS) +endif +LDFLAGS=-L$(LIBDIR) -pie -z noexecstack -z relro -z now +LDLIBS=-lpqos -lpthread + +# ICC and GCC options +ifeq ($(CC),icc) +else +CFLAGS += -Wcast-align \ + -Wnested-externs \ + -Wmissing-noreturn +endif + +# Build targets and dependencies +MONITORAPP = monitor_app + +all: $(MONITORAPP) + +$(MONITORAPP): $(MONITORAPP).o + +.PHONY: clean +clean: + -rm -f $(MONITORAPP) ./*.o + +CHECKPATCH?=checkpatch.pl +.PHONY: style +style: + $(CHECKPATCH) --no-tree --no-signoff --emacs \ + --ignore CODE_INDENT,INITIALISED_STATIC,LEADING_SPACE,SPLIT_STRING,UNSPECIFIED_INT \ + -f monitor_app.c + +CPPCHECK?=cppcheck +.PHONY: cppcheck +cppcheck: + $(CPPCHECK) --enable=warning,portability,performance,unusedFunction,missingInclude \ + --std=c99 -I$(LIBDIR) --template=gcc \ + monitor_app.c diff --git a/examples/c/CMT_MBM/README b/examples/c/CMT_MBM/README new file mode 100644 index 0000000..fa37f25 --- /dev/null +++ b/examples/c/CMT_MBM/README @@ -0,0 +1,63 @@ +================================================================================ +README for CMT MBM Sample Code + +April 2016 +================================================================================ + +CONTENTS +======== + +- Overview +- Compilation +- Usage +- Legal Disclaimer + + +OVERVIEW +======== + +This is example code to demonstrate the use of PQoS/Intel(R) Resource Director +Technology (Intel(R) RDT) library APIs to manage Cache Monitoring Technology +(CMT), Memory Bandwidth Monitoring (MBM). Refer to +https://github.com/01org/intel-cmt-cat/blob/master/README table 1 for a list +of processors supporting CMT and MBM. +CMT_MBM sample application build will create one target as follows: +1. monitor_app - Demonstrates usage of PQoS/Intel(R) RDT library APIs related +to monitoring events like cache occupancy, local memory bandwidth usage and +remote memory bandwidth usage. It operates in user space and uses PQoS/Intel(R) +RDT and C libraries only. +Note: Monitor events are hardware dependant. Only supported event will be shown. + + +COMPILATION +=========== + +Note: The PQoS/Intel(R) RDT library should be installed before compilation. + +Run "make all" or "make" to compile the program. If compilation is successful +"monitor_app" binary should be present in the directory. + +Run "make clean" to clean the build files. + + +USAGE +===== + +To run: +$ sudo ./monitor_app + +For more usage options: +$ sudo ./monitor_app -h + + +LEGAL DISCLAIMER +================ + +THIS SOFTWARE IS PROVIDED BY INTEL"AS IS". NO LICENSE, EXPRESS OR +IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS +ARE GRANTED THROUGH USE. EXCEPT AS PROVIDED IN INTEL'S TERMS AND +CONDITIONS OF SALE, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL +DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR +USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO +FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT +OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. diff --git a/examples/c/CMT_MBM/monitor_app.c b/examples/c/CMT_MBM/monitor_app.c new file mode 100644 index 0000000..dedc6f1 --- /dev/null +++ b/examples/c/CMT_MBM/monitor_app.c @@ -0,0 +1,422 @@ +/* + * BSD LICENSE + * + * Copyright(c) 2014-2018 Intel Corporation. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +/** + * @brief Platform QoS sample LLC occupancy monitoring application + * + */ + +#include +#include +#include +#include +#include +#include +#include +#include "pqos.h" + +/** + * Defines + */ +#define PQOS_MAX_CORES 1024 +#define PQOS_MAX_PIDS 16 +#define PQOS_MAX_MON_EVENTS 1 + +/** + * Number of cores that are selected in config string + * for monitoring LLC occupancy + */ +static int sel_monitor_num = 0; + +/** + * The mask to tell which events to display + */ +static enum pqos_mon_event sel_events_max = (enum pqos_mon_event)0; + +/** + * Maintains a table of core, event, number of events that are + * selected in config string for monitoring + */ +static struct { + unsigned core; + struct pqos_mon_data *pgrp; + enum pqos_mon_event events; +} sel_monitor_core_tab[PQOS_MAX_CORES]; +static struct pqos_mon_data *m_mon_grps[PQOS_MAX_CORES]; + +/** + * Maintains a table of process id, event, number of events that are selected + * in config string for monitoring LLC occupancy + */ +static struct { + pid_t pid; + struct pqos_mon_data *pgrp; + enum pqos_mon_event events; +} sel_monitor_pid_tab[PQOS_MAX_PIDS]; + +/** + * Maintains the number of process id's you want to track + */ +static int sel_process_num = 0; + +/** + * Flag to determine which library interface to use + */ +static enum pqos_interface interface = PQOS_INTER_MSR; + +static void stop_monitoring(void); + +/** + * @brief CTRL-C handler for infinite monitoring loop + * + * @param [in] signo signal number + */ +static void __attribute__((noreturn)) monitoring_ctrlc(int signo) +{ + printf("\nExiting[%d]...\n", signo); + stop_monitoring(); + if (pqos_fini() != PQOS_RETVAL_OK) { + printf("Error shutting down PQoS library!\n"); + exit(EXIT_FAILURE); + } + exit(EXIT_SUCCESS); +} + +/** + * @brief Scale byte value up to KB + * + * @param [in] bytes value to be scaled up + * @return scaled up value in KB's + */ +static inline double bytes_to_kb(const double bytes) +{ + return bytes / 1024.0; +} + +/** + * @brief Scale byte value up to MB + * + * @param [in] bytes value to be scaled up + * @return scaled up value in MB's + */ +static inline double bytes_to_mb(const double bytes) +{ + return bytes / (1024.0 * 1024.0); +} + +/** + * @brief Check to determine if processes or cores are monitored + * + * @return Process monitoring mode status + * @retval 0 monitoring cores + * @retval 1 monitoring processes + */ +static inline int process_mode(void) +{ + return (sel_process_num <= 0) ? 0 : 1; +} + +/** + * @brief Verifies and translates monitoring config string into + * internal monitoring configuration. + * + * @param [in] argc Number of arguments in input command + * @param [in] argv Input arguments for LLC occupancy monitoring + */ +static void +monitoring_get_input(int argc, char *argv[]) +{ + int num_args, num_opts = 1, i = 0, sel_pid = 0, help = 0; + + for (i = 0; i < argc; i++) { + if (!strcmp(argv[i], "-p")) { + sel_pid = 1; + num_opts++; + } else if (!strcmp(argv[i], "-I")) { + interface = PQOS_INTER_OS; + num_opts++; + } else if (!strcmp(argv[i], "-H") || !strcmp(argv[i], "-h")) { + help = 1; + num_opts++; + } + } + /* Ensure OS interface selected if monitoring tasks */ + if (sel_pid && interface == PQOS_INTER_MSR) { + printf("Error: PID monitoring requires OS interface " + "selection!\nPlease use the -I option.\n"); + help = 1; + } + num_args = (argc - num_opts); + if (help) { + printf("Usage: %s [ ...]\n" + " %s -I -p [ ...]\n", + argv[0], argv[0]); + printf("Eg : %s 1 2 6\n " + "%s -I -p 3564 7638 356\n" + "Notes:\n " + "-h help\n " + "-I select library OS interface\n " + "-p select process ID's to monitor LLC occupancy" + "\n\n", argv[0], argv[0]); + exit(EXIT_SUCCESS); + } else if (num_args == 0) { + sel_monitor_num = 0; + } else { + if (sel_pid) { + if (num_args > PQOS_MAX_PIDS) + num_args = PQOS_MAX_PIDS; + for (i = 0; i < num_args; i++) { + m_mon_grps[i] = malloc(sizeof(**m_mon_grps)); + sel_monitor_pid_tab[i].pgrp = m_mon_grps[i]; + sel_monitor_pid_tab[i].pid = + (unsigned) atoi(argv[num_opts + i]); + } + sel_process_num = (int) num_args; + } else { + if (num_args > PQOS_MAX_CORES) + num_args = PQOS_MAX_CORES; + for (i = 0; i < num_args; i++) { + m_mon_grps[i] = malloc(sizeof(**m_mon_grps)); + sel_monitor_core_tab[i].pgrp = m_mon_grps[i]; + sel_monitor_core_tab[i].core = + (unsigned) atoi(argv[num_opts + i]); + } + sel_monitor_num = (int) num_args; + } + } +} + +/** + * @brief Starts monitoring on selected cores/PIDs + * + * @param [in] cpu_info cpu information structure + * @param [in] cap_mon monitoring capabilities structure + * + * @return Operation status + * @retval 0 OK + * @retval -1 error + */ +static int +setup_monitoring(const struct pqos_cpuinfo *cpu_info, + const struct pqos_capability * const cap_mon) +{ + unsigned i; + const enum pqos_mon_event perf_events = (enum pqos_mon_event) + (PQOS_PERF_EVENT_IPC | PQOS_PERF_EVENT_LLC_MISS); + + for (i = 0; (unsigned)i < cap_mon->u.mon->num_events; i++) + sel_events_max |= (cap_mon->u.mon->events[i].type); + + /* Remove perf events IPC and LLC MISSES */ + sel_events_max &= ~perf_events; + if (sel_monitor_num == 0 && sel_process_num == 0) { + for (i = 0; i < cpu_info->num_cores; i++) { + unsigned lcore = cpu_info->cores[i].lcore; + + sel_monitor_core_tab[sel_monitor_num].core = lcore; + sel_monitor_core_tab[sel_monitor_num].events = + sel_events_max; + m_mon_grps[sel_monitor_num] = + malloc(sizeof(**m_mon_grps)); + sel_monitor_core_tab[sel_monitor_num].pgrp = + m_mon_grps[sel_monitor_num]; + sel_monitor_num++; + } + } + if (!process_mode()) { + for (i = 0; i < (unsigned) sel_monitor_num; i++) { + unsigned lcore = sel_monitor_core_tab[i].core; + int ret; + + ret = pqos_mon_start(1, &lcore, + sel_events_max, + NULL, + sel_monitor_core_tab[i].pgrp); + if (ret != PQOS_RETVAL_OK) { + printf("Monitoring start error on core %u," + "status %d\n", lcore, ret); + return ret; + } + } + } else { + for (i = 0; i < (unsigned) sel_process_num; i++) { + pid_t pid = sel_monitor_pid_tab[i].pid; + int ret; + + ret = pqos_mon_start_pids(1, &pid, + PQOS_MON_EVENT_L3_OCCUP, + NULL, + sel_monitor_pid_tab[i].pgrp); + if (ret != PQOS_RETVAL_OK) { + printf("Monitoring start error on pid %u," + "status %d\n", pid, ret); + return ret; + } + } + } + return PQOS_RETVAL_OK; +} + +/** + * @brief Stops monitoring on selected cores + * + */ +static void stop_monitoring(void) +{ + unsigned i, mon_number = 0; + + if (!process_mode()) + mon_number = (unsigned) sel_monitor_num; + else + mon_number = (unsigned) sel_process_num; + + for (i = 0; i < mon_number; i++) { + int ret; + + ret = pqos_mon_stop(m_mon_grps[i]); + if (ret != PQOS_RETVAL_OK) + printf("Monitoring stop error!\n"); + free(m_mon_grps[i]); + } +} + +/** + * @brief Reads monitoring event data + */ +static void monitoring_loop(void) +{ + unsigned mon_number = 0; + int ret = PQOS_RETVAL_OK; + int i = 0; + + if (signal(SIGINT, monitoring_ctrlc) == SIG_ERR) + printf("Failed to catch SIGINT!\n"); + + if (!process_mode()) + mon_number = (unsigned) sel_monitor_num; + else + mon_number = (unsigned) sel_process_num; + + while (1) { + ret = pqos_mon_poll(m_mon_grps, (unsigned)mon_number); + if (ret != PQOS_RETVAL_OK) { + printf("Failed to poll monitoring data!\n"); + return; + } + if (!process_mode()) { + printf(" CORE RMID LLC[KB]" + " MBL[MB] MBR[MB]\n"); + for (i = 0; i < sel_monitor_num; i++) { + const struct pqos_event_values *pv = + &m_mon_grps[i]->values; + double llc = bytes_to_kb(pv->llc); + double mbr = bytes_to_mb(pv->mbm_remote_delta); + double mbl = bytes_to_mb(pv->mbm_local_delta); + + if (interface == PQOS_INTER_OS) + printf("%8u %s %10.1f %10.1f %10.1f\n", + m_mon_grps[i]->cores[0], + " N/A", llc, mbl, mbr); + else + printf("%8u %8u %10.1f %10.1f %10.1f\n", + m_mon_grps[i]->cores[0], + m_mon_grps[i]->poll_ctx[0].rmid, + llc, mbl, mbr); + } + } else { + printf("PID LLC[KB]\n"); + for (i = 0; i < sel_process_num; i++) { + const struct pqos_event_values *pv = + &m_mon_grps[i]->values; + double llc = bytes_to_kb(pv->llc); + + printf("%6d %10.1f\n", + m_mon_grps[i]->pids[0], llc); + } + } + printf("\nPress Enter to continue or Ctrl+c to exit"); + if (getchar() != '\n') + break; + printf("\e[1;1H\e[2J"); + } +} + +int main(int argc, char *argv[]) +{ + struct pqos_config config; + const struct pqos_cpuinfo *p_cpu = NULL; + const struct pqos_cap *p_cap = NULL; + int ret, exit_val = EXIT_SUCCESS; + const struct pqos_capability *cap_mon = NULL; + + /* Get input from user */ + monitoring_get_input(argc, argv); + + memset(&config, 0, sizeof(config)); + config.fd_log = STDOUT_FILENO; + config.verbose = 0; + config.interface = interface; + + /* PQoS Initialization - Check and initialize CAT and CMT capability */ + ret = pqos_init(&config); + if (ret != PQOS_RETVAL_OK) { + printf("Error initializing PQoS library!\n"); + exit_val = EXIT_FAILURE; + goto error_exit; + } + /* Get CMT capability and CPU info pointer */ + ret = pqos_cap_get(&p_cap, &p_cpu); + if (ret != PQOS_RETVAL_OK) { + printf("Error retrieving PQoS capabilities!\n"); + exit_val = EXIT_FAILURE; + goto error_exit; + } + (void) pqos_cap_get_type(p_cap, PQOS_CAP_TYPE_MON, &cap_mon); + /* Setup the monitoring resources */ + ret = setup_monitoring(p_cpu, cap_mon); + if (ret != PQOS_RETVAL_OK) { + printf("Error Setting up monitoring!\n"); + exit_val = EXIT_FAILURE; + goto error_exit; + } + /* Start Monitoring */ + monitoring_loop(); + /* Stop Monitoring */ + stop_monitoring(); + error_exit: + ret = pqos_fini(); + if (ret != PQOS_RETVAL_OK) + printf("Error shutting down PQoS library!\n"); + return exit_val; +} diff --git a/examples/c/PSEUDO_LOCK/Makefile b/examples/c/PSEUDO_LOCK/Makefile new file mode 100644 index 0000000..05557f3 --- /dev/null +++ b/examples/c/PSEUDO_LOCK/Makefile @@ -0,0 +1,85 @@ +############################################################################### +# Makefile script for CAT PQoS sample applications +# +# @par +# BSD LICENSE +# +# Copyright(c) 2016 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +# +############################################################################### + +LIBDIR ?= ../../../lib +LDFLAGS = -L$(LIBDIR) -pie -z noexecstack -z relro -z now +LDLIBS = -lpqos -lrt -lpthread +CFLAGS = -I$(LIBDIR) \ + -W -Wall -Wextra -Wstrict-prototypes -Wmissing-prototypes \ + -Wmissing-declarations -Wold-style-definition -Wpointer-arith \ + -Wcast-qual -Wundef -Wwrite-strings \ + -Wformat -Wformat-security -fstack-protector -fPIE -D_FORTIFY_SOURCE=2 \ + -Wunreachable-code -Wsign-compare -Wno-endif-labels \ + -g -O2 +ifneq ($(EXTRA_CFLAGS),) +CFLAGS += $(EXTRA_CFLAGS) +endif + +# ICC and GCC options +ifeq ($(CC),icc) +else +CFLAGS += -Wcast-align \ + -Wnested-externs \ + -Wmissing-noreturn +endif + +# Build targets and dependencies +APP = pseudo_lock +OBJS = pseudo_lock.o dlock.o tsc.o + +all: $(APP) + +$(APP): $(OBJS) + +.PHONY: clean +clean: + -rm -f $(APP) $(OBJS) *.o + +CHECKPATCH?=checkpatch.pl +.PHONY: style +style: + $(CHECKPATCH) --no-tree --no-signoff --emacs \ + --ignore CODE_INDENT,INITIALISED_STATIC,LEADING_SPACE,SPLIT_STRING,UNSPECIFIED_INT,\ + ARRAY_SIZE,BLOCK_COMMENT_STYLE \ + -f pseudo_lock.c -f tsc.c -f tsc.h -f dlock.c -f dlock.h + +CPPCHECK?=cppcheck +.PHONY: cppcheck +cppcheck: + $(CPPCHECK) --enable=warning,portability,performance,unusedFunction,missingInclude \ + --std=c99 -I$(LIBDIR) --template=gcc \ + pseudo_lock.c dlock.c dlock.h tsc.c tsc.h diff --git a/examples/c/PSEUDO_LOCK/README b/examples/c/PSEUDO_LOCK/README new file mode 100644 index 0000000..a31084f --- /dev/null +++ b/examples/c/PSEUDO_LOCK/README @@ -0,0 +1,126 @@ +======================================================================== +README for PSEUDO LOCK Sample Code + +January 2016 +======================================================================== + +CONTENTS +======== + +- Overview +- Compilation +- Usage +- Design +- Examples +- FAQ +- Legal Disclaimer + +OVERVIEW +======== + +This is just an example code demonstrating concept of data pseudo locking in +LLC. It operates in user space and uses PQoS and C libraries only. It is far from +being perfect and it may not function as expected at every execution. + +The concept is to use CAT to bring latency sensitive data into selected LLC cache +ways and then exclude these cache ways from use so that this data +doesn't get evicted. Data locking results in better determinism of the code that +depends on accesses to sensitive data. +This method is called "pseudo locking" as there are numerous flows that +can evict "locked" data and no guarantees can be given. +Just to mention some of the flows: +- instructions like WBINVD or INVD will evict "locked" data +- CLFLUSH instruction will evict the data +- PCI devices may be allowed to fill into "locked" cache ways resulting in + eviction of the data +- power management may cause data eviction +- weakly ordered or uncachable access to "locked" memory will evict the data +- some fragments of data may not get locked due to data layout in physical memory +- finally, this approach is not architecturally supported and it may not work on + future CPU models + +COMPILATION +=========== + +Note: The PQoS/Intel(R) RDT library should be installed before compilation. + +Run "make all" or "make" to compile the program. If compilation is successful +"pseudo_lock" binary should be present in the directory. + +Run "make clean" to clean the build files. + +USAGE +===== + +-bash-4.3$ sudo ./pseudo_lock +Usage: ./pseudo_lock + +Where: + core_id - identifies CPU to be used to lock the data + lock|nolock - apply data pseudo locking or not + +As the result of execution program prints cycle cost statistics of the timer +handler procedure. + +DESIGN +====== + +Design of the program is very simple. It is split into three modules: + tsc - provides API to run cycle cost statistics + dlock - provides API to pseudo lock the data (uses PQoS) + pseudo_lock - main module with application control logic. This is where + timer handler and main thread workloads are implemented. + Timer handler kicks in fixed intervals and does arithmetic + operations on random memory locations on memory block A. + Main thread runs memory intensive workload on memory block B + - this normally results in evicting data from block A used by + the timer handler. + +EXAMPLES +======== + +With data pseudo locking: + +-bash-4.3$ sudo ./pseudo_lock 1 lock +main_thread() started. please wait ... +main_thread() has finished. +[Timer Handler] work items 145; cycles per work item: avg=343999.724 min=326052.000 max=730548.000 jitter=404496.000 +-bash-4.3$ + +Without data pseudo locking: + +-bash-4.3$ sudo ./pseudo_lock 1 nolock +main_thread() started. please wait ... +main_thread() has finished. +[Timer Handler] work items 143; cycles per work item: avg=607408.643 min=588868.000 max=1016704.000 jitter=427836.000 +-bash-4.3$ + +Typically data pseudo-locking code delivers lower avg, min and max cycle costs. This may also result in lower jitter. + +FAQ +=== + +Q: How do I know pseudo locking works? +A: With pseudo locking in place "avg", "min" and "max" cycle costs + should be lower for "lock" run vs "nolock" one. Due to code imperfections + data may not get locked as expected every time and "max" may be higher for + "lock" case. + +Q: "max" cycle cost is higher for "lock" than for "nolock". What is wrong? +A: This is user space code demonstrating concept of data pseudo locking. + This procedure is far from perfect and ideally pseudo locking should be + done kernel space. When all timer data is not locked properly then extra + timer handler may pay big cost of accessing the data from RAM. + + +LEGAL DISCLAIMER +================ + +THIS SOFTWARE IS PROVIDED BY INTEL"AS IS". NO LICENSE, EXPRESS OR +IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS +ARE GRANTED THROUGH USE. EXCEPT AS PROVIDED IN INTEL'S TERMS AND +CONDITIONS OF SALE, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL +DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR +USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO +FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT +OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. diff --git a/examples/c/PSEUDO_LOCK/dlock.c b/examples/c/PSEUDO_LOCK/dlock.c new file mode 100644 index 0000000..598b077 --- /dev/null +++ b/examples/c/PSEUDO_LOCK/dlock.c @@ -0,0 +1,477 @@ +/* + * BSD LICENSE + * + * Copyright(c) 2016 Intel Corporation. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#define _GNU_SOURCE +#include /* uint64_t etc. */ +#include /* malloc() */ +#include /* memcpy() */ +#include /* ASSERT() */ +#include +#ifdef __linux__ +#include /* sched_setaffinity() */ +#endif +#ifdef __FreeBSD__ +#include /* sched affinity */ +#include /* sched affinity */ +#endif +#include "dlock.h" + +#define MAX_SOCK_NUM 16 +#define DIM(x) (sizeof(x)/sizeof(x[0])) + +static int m_is_chunk_allocated = 0; +static char *m_chunk_start = NULL; +static size_t m_chunk_size = 0; +static unsigned m_num_clos = 0; +static struct { + unsigned id; + struct pqos_l3ca *cos_tab; +} m_socket_cos[MAX_SOCK_NUM]; + +/** + * @brief Removes the memory block from cache hierarchy + * + * @param p pointer to memory block + * @param s size of memory block in bytes + */ +static void mem_flush(const void *p, const size_t s) +{ + const size_t cache_line = 64; + const char *cp = (const char *)p; + size_t i = 0; + + if (p == NULL || s <= 0) + return; + + for (i = 0; i < s; i += cache_line) { + asm volatile("clflush (%0)\n\t" + : + : "r"(&cp[i]) + : "memory"); + } + + asm volatile("sfence\n\t" + : + : + : "memory"); +} + +/** + * @brief Reads the memory block which places it in cache hierarchy + * + * @param p pointer to memory block + * @param s size of memory block in bytes + */ +static void mem_read(const void *p, const size_t s) +{ + register size_t i; + + if (p == NULL || s <= 0) + return; + + for (i = 0; i < (s / sizeof(uint32_t)); i++) { + asm volatile("xor (%0,%1,4), %%eax\n\t" + : + : "r" (p), "r" (i) + : "%eax", "memory"); + } + + for (i = s & (~(sizeof(uint32_t) - 1)); i < s; i++) { + asm volatile("xorb (%0,%1,1), %%al\n\t" + : + : "r" (p), "r" (i) + : "%al", "memory"); + } +} + +/** + * @brief Initializes the memory block with random data + * + * This is to avoid any page faults or copy-on-write exceptions later on. + * + * @param p pointer to memory block + * @param s size of memory block in bytes + */ +static void mem_init(void *p, const size_t s) +{ + char *cp = (char *)p; + size_t i; + + if (p == NULL || s <= 0) + return; + + for (i = 0; i < s; i++) + cp[i] = (char) rand(); +} + +/** + * @brief Calculates number of cache ways required to fit a number of \a bytes + * + * @param cat_cap pointer to L3CA PQoS capability structure + * @param bytes number of bytes + * @param ways pointer to store number of required cache ways + * + * @return Operation status + * @retval 0 OK + * @retval <0 error + */ +static int bytes_to_cache_ways(const struct pqos_capability *cat_cap, + const size_t bytes, size_t *ways) +{ + size_t llc_size = 0, num_ways = 0; + const struct pqos_cap_l3ca *cat = NULL; + + if (cat_cap == NULL || ways == NULL) + return -1; + + if (cat_cap->type != PQOS_CAP_TYPE_L3CA) + return -2; + + cat = cat_cap->u.l3ca; + llc_size = cat->way_size * cat->num_ways; + if (bytes > llc_size) + return -3; + + num_ways = (bytes + cat->way_size - 1) / cat->way_size; + if (num_ways >= cat->num_ways) + return -4; + + if (num_ways < 2) + num_ways = 2; + + *ways = num_ways; + return 0; +} + +int dlock_init(void *ptr, const size_t size, const int clos, const int cpuid) +{ + const struct pqos_cpuinfo *p_cpu = NULL; + const struct pqos_cap *p_cap = NULL; + const struct pqos_capability *p_l3ca_cap = NULL; + unsigned *sockets = NULL; + unsigned socket_count = 0, i = 0; + int ret = 0, res = 0; + size_t num_cache_ways = 0; + unsigned clos_save = 0; + +#ifdef __linux__ + cpu_set_t cpuset_save, cpuset; +#endif +#ifdef __FreeBSD__ + cpuset_t cpuset_save, cpuset; +#endif + + if (m_chunk_start != NULL) + return -1; + + if (size <= 0) + return -2; + + if (ptr != NULL) { + m_chunk_start = ptr; + m_is_chunk_allocated = 0; + } else { + /** + * For best results allocated memory should be physically + * contiguous. Yet this would require allocating memory in + * kernel space or using huge pages. + * Let's use malloc() and 4K pages for simplicity. + */ + m_chunk_start = malloc(size); + if (m_chunk_start == NULL) + return -3; + m_is_chunk_allocated = 1; + mem_init(m_chunk_start, size); + } + m_chunk_size = size; + + /** + * Get task affinity to restore it later + */ +#ifdef __linux__ + res = sched_getaffinity(0, sizeof(cpuset_save), &cpuset_save); +#endif +#ifdef __FreeBSD__ + res = cpuset_getaffinity(CPU_LEVEL_WHICH, CPU_WHICH_TID, -1, + sizeof(cpuset_save), &cpuset_save); +#endif + if (res != 0) { + perror("dlock_init() error"); + ret = -4; + goto dlock_init_error1; + } + + /** + * Set task affinity to cpuid for data locking phase + */ + CPU_ZERO(&cpuset); + CPU_SET(cpuid, &cpuset); +#ifdef __linux__ + res = sched_setaffinity(0, sizeof(cpuset), &cpuset); +#endif +#ifdef __FreeBSD__ + res = cpuset_setaffinity(CPU_LEVEL_WHICH, CPU_WHICH_TID, -1, + sizeof(cpuset), &cpuset); +#endif + if (res != 0) { + perror("dlock_init() error"); + ret = -4; + goto dlock_init_error1; + } + + /** + * Clear table for restoring CAT configuration + */ + for (i = 0; i < DIM(m_socket_cos); i++) { + m_socket_cos[i].id = 0; + m_socket_cos[i].cos_tab = NULL; + } + + /** + * Retrieve CPU topology and PQoS capabilities + */ + res = pqos_cap_get(&p_cap, &p_cpu); + if (res != PQOS_RETVAL_OK) { + ret = -5; + goto dlock_init_error2; + } + + /** + * Retrieve list of CPU sockets + */ + sockets = pqos_cpu_get_sockets(p_cpu, &socket_count); + if (sockets == NULL) { + ret = -6; + goto dlock_init_error2; + } + + /** + * Get CAT capability structure + */ + res = pqos_cap_get_type(p_cap, PQOS_CAP_TYPE_L3CA, &p_l3ca_cap); + if (res != PQOS_RETVAL_OK) { + ret = -7; + goto dlock_init_error2; + } + + /** + * Compute number of cache ways required for the data + */ + res = bytes_to_cache_ways(p_l3ca_cap, size, &num_cache_ways); + if (res != 0) { + ret = -8; + goto dlock_init_error2; + } + + /** + * Compute class bit mask for data lock and + * retrieve number of classes of service + */ + m_num_clos = p_l3ca_cap->u.l3ca->num_classes; + + for (i = 0; i < socket_count; i++) { + /** + * This would be enough to run the below code for the socket + * corresponding to \a cpuid. Yet it is safer to keep CLOS + * definitions coherent across sockets. + */ + const uint64_t dlock_mask = (1ULL << num_cache_ways) - 1ULL; + + ASSERT(m_num_clos > 0); + struct pqos_l3ca cos[m_num_clos]; + unsigned num = 0, j; + + /* get current CAT classes on this socket */ + res = pqos_l3ca_get(sockets[i], m_num_clos, &num, &cos[0]); + if (res != PQOS_RETVAL_OK) { + printf("pqos_l3ca_get() error!\n"); + ret = -9; + goto dlock_init_error2; + } + + /* paranoia check */ + if (m_num_clos != num) { + printf("CLOS number mismatch!\n"); + ret = -9; + goto dlock_init_error2; + } + + /* save CAT classes to restore it later */ + m_socket_cos[i].id = sockets[i]; + m_socket_cos[i].cos_tab = malloc(m_num_clos * sizeof(cos[0])); + if (m_socket_cos[i].cos_tab == NULL) { + printf("malloc() error!\n"); + ret = -9; + goto dlock_init_error2; + } + memcpy(m_socket_cos[i].cos_tab, cos, + m_num_clos * sizeof(cos[0])); + + /** + * Modify the classes in the following way: + * if class_id == clos then + * set class mask so that it has exclusive access to + * \a num_cache_ways + * else + * exclude class from accessing \a num_cache_ways + */ + for (j = 0; j < m_num_clos; j++) { + if (cos[j].cdp) { + if (cos[j].class_id == (unsigned)clos) { + cos[j].u.s.code_mask = dlock_mask; + cos[j].u.s.data_mask = dlock_mask; + } else { + cos[j].u.s.code_mask &= ~dlock_mask; + cos[j].u.s.data_mask &= ~dlock_mask; + } + } else { + if (cos[j].class_id == (unsigned)clos) + cos[j].u.ways_mask = dlock_mask; + else + cos[j].u.ways_mask &= ~dlock_mask; + } + } + + res = pqos_l3ca_set(sockets[i], m_num_clos, &cos[0]); + if (res != PQOS_RETVAL_OK) { + printf("pqos_l3ca_set() error!\n"); + ret = -10; + goto dlock_init_error2; + } + } + + /** + * Read current cpuid CLOS association and set the new one + */ + res = pqos_alloc_assoc_get(cpuid, &clos_save); + if (res != PQOS_RETVAL_OK) { + printf("pqos_alloc_assoc_get() error!\n"); + ret = -11; + goto dlock_init_error2; + } + res = pqos_alloc_assoc_set(cpuid, clos); + if (res != PQOS_RETVAL_OK) { + printf("pqos_alloc_assoc_set() error!\n"); + ret = -12; + goto dlock_init_error2; + } + + /** + * Remove buffer data from cache hierarchy and read it back into + * selected cache ways. + * WBINVD is another option to remove data from cache but it is + * privileged instruction and as such has to be done in kernel space. + */ + mem_flush(m_chunk_start, m_chunk_size); + + /** + * Read the data couple of times. This may help as this is ran in + * user space and code can be interrupted and data removed + * from cache hierarchy. + * Ideally all locking should be done at privileged level with + * full system control. + */ + for (i = 0; i < 10; i++) + mem_read(m_chunk_start, m_chunk_size); + + /** + * Restore cpuid clos association + */ + res = pqos_alloc_assoc_set(cpuid, clos_save); + if (res != PQOS_RETVAL_OK) { + printf("pqos_alloc_assoc_set() error (revert)!\n"); + ret = -13; + goto dlock_init_error2; + } + + dlock_init_error2: + for (i = 0; (i < DIM(m_socket_cos)) && (ret != 0); i++) + if (m_socket_cos[i].cos_tab != NULL) + free(m_socket_cos[i].cos_tab); + +#ifdef __linux__ + res = sched_setaffinity(0, sizeof(cpuset_save), &cpuset_save); +#endif +#ifdef __FreeBSD__ + res = cpuset_setaffinity(CPU_LEVEL_WHICH, CPU_WHICH_TID, -1, + sizeof(cpuset_save), &cpuset_save); +#endif + if (res != 0) + perror("dlock_init() error restoring affinity"); + + dlock_init_error1: + if (m_is_chunk_allocated && ret != 0) + free(m_chunk_start); + + if (ret != 0) { + m_chunk_start = NULL; + m_chunk_size = 0; + m_is_chunk_allocated = 0; + } + + if (sockets != NULL) + free(sockets); + + return ret; +} + +int dlock_exit(void) +{ + int ret = 0; + unsigned i; + + if (m_chunk_start == NULL) + return -1; + + for (i = 0; i < DIM(m_socket_cos); i++) { + if (m_socket_cos[i].cos_tab != NULL) { + int res = pqos_l3ca_set(m_socket_cos[i].id, m_num_clos, + m_socket_cos[i].cos_tab); + + if (res != PQOS_RETVAL_OK) + ret = -2; + } + free(m_socket_cos[i].cos_tab); + m_socket_cos[i].cos_tab = NULL; + m_socket_cos[i].id = 0; + } + + if (m_is_chunk_allocated) + free(m_chunk_start); + + m_chunk_start = NULL; + m_chunk_size = 0; + m_is_chunk_allocated = 0; + + return ret; +} diff --git a/examples/c/PSEUDO_LOCK/dlock.h b/examples/c/PSEUDO_LOCK/dlock.h new file mode 100644 index 0000000..a93d93a --- /dev/null +++ b/examples/c/PSEUDO_LOCK/dlock.h @@ -0,0 +1,84 @@ +/* + * BSD LICENSE + * + * Copyright(c) 2016 Intel Corporation. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +/** + * @brief Data pseudo locking module + */ + +#ifndef __DLOCK_H__ +#define __DLOCK_H__ + +#ifdef __cplusplus +extern "C" { +#endif + +/** + * @brief Initializes data pseudo lock module + * + * @note It is assumed that \a clos is not associated to any CPU. + * @note It is assumed PQoS library is already initialized. + * @note Function modifies CAT classes on all sockets. + * This configuration is restored at dlock_exit(). + * @note Data will be locked in ways corresponding to least significant bits of + * the bit mask. + * @note It is not allowed to initialize the module multiple times for + * different memory blocks. + * + * @param ptr pointer to memory block to be locked. + * If NULL then memory block is allocated. + * @param size size of memory block to be locked + * @param clos CAT class of service to be used for data locking + * @param cpuid CPU ID to be used for data locking + * + * @return Operation status + * @retval 0 OK + * @retval <0 error + */ +int dlock_init(void *ptr, const size_t size, const int clos, const int cpuid); + +/** + * @brief Shuts down data pseudo lock module + * + * @note CAT configuration modified at dlock_init() is restored here. + * + * @return Operation status + * @retval 0 OK + * @retval <0 error + */ +int dlock_exit(void); + +#ifdef __cplusplus +} +#endif + +#endif /* __DLOCK_H__ */ diff --git a/examples/c/PSEUDO_LOCK/pseudo_lock.c b/examples/c/PSEUDO_LOCK/pseudo_lock.c new file mode 100644 index 0000000..42f8fcb --- /dev/null +++ b/examples/c/PSEUDO_LOCK/pseudo_lock.c @@ -0,0 +1,422 @@ +/* + * BSD LICENSE + * + * Copyright(c) 2016-2018 Intel Corporation. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include "dlock.h" +#include "tsc.h" + +#define DIM(x) (sizeof(x)/sizeof(x[0])) +#define MB (1024 * 1024) + +static void *timer_data_ptr = NULL; +static const size_t timer_data_size = 2 * MB; + +static void *main_data_ptr = NULL; +static const size_t main_data_size = 96 * MB; + +const long long freq_ms = 100; + +static struct tsc_prof timer_prof; +static timer_t timerid; + +/** + * @brief Allocates memory block and initializes it with random data + * + * This is to avoid any page faults or copy-on-write exceptions later on + * when measuring cycles. + * + * For simplicity, malloc() is used to allocate memory. Ideally special + * allocator should be used that allocates physically contiguous memory block. + * + * @param sz size of memory block in bytes + * + * @return Pointer to allocated memory block + */ +static void *init_memory(const size_t sz) +{ + char *p = NULL; + size_t i; + + if (sz <= 0) + return NULL; + + p = (char *) malloc(sz); + if (p == NULL) + return NULL; + + for (i = 0; i < sz; i += 32) + p[i] = (char) rand(); + + return (void *)p; +} + +/** + * @brief Generates random number for the timer handler procedure + * + * This is required as rand() is not thread safe. This dummy implementation + * computes large number of random numbers, stores them in a table and + * re-uses them all over again. This is good enough for the purpose of this + * application. + * + * @return Random number value + */ +static int timer_rand(void) +{ + static int _rand_tab[8192]; /* size has to be power of 2 */ + static int _rand_idx = -1; + int ret; + + /* generate bunch of random numbers */ + if (_rand_idx == -1) { + unsigned i; + + for (i = 0; i < DIM(_rand_tab); i++) + _rand_tab[i] = rand(); + _rand_idx = 0; + } + + ret = _rand_tab[_rand_idx]; + _rand_idx = (_rand_idx + 1) & (DIM(_rand_tab) - 1); + return ret; +} + +/** + * @brief Timer handler procedure + * + * This is not a realistic workload and it is a demonstration code only. + * + * It runs couple thousand of iterations and each iteration is randomizing + * memory locations to run a number of arithmetic operations on them. + * + * @param sig UNUSED + * @param si UNUSED + * @param uc UNUSED + */ +static void timer_handler(int sig, siginfo_t *si, void *uc) +{ + const int num_iterations = 5000; + int *p = (int *) timer_data_ptr; + const size_t sz = timer_data_size / sizeof(int); + int m; + + (void) (sig); + (void) (si); + (void) (uc); + + tsc_start(&timer_prof); + /* START - "latency sensitive" code */ + for (m = 0; m < num_iterations; m++) { + const size_t stride = 5; + const int idx0 = timer_rand() % (sz - stride); + const int idx1 = timer_rand() % (sz - stride); + size_t n; + + for (n = 0; n < stride; n++) + p[idx0 + n] = 2 * p[idx1 + n] + p[idx0 + n]; + } + /* END - "latency sensitive" code */ + tsc_end(&timer_prof, 1); +} + +/** + * @brief Set up the timer + * + * The timer expiration is delivered as a signal, + * the signal handler is timer_handler() above. + * + * @param freq_nanosecs timer frequency in nanoseconds + * + * @return Operation status + * @retval 0 OK + * @retval <0 error + */ +static int init_timer(const long long freq_nanosecs) +{ + sigset_t mask; + struct sigaction sa; + struct sigevent sev; + struct itimerspec its; + + /* this will initialize the table with random numbers */ + (void) timer_rand(); + + /* Block timer signal temporarily */ + sigemptyset(&mask); + sigaddset(&mask, SIGRTMIN); + if (sigprocmask(SIG_SETMASK, &mask, NULL) == -1) { + printf("Error masking signal!\n"); + return -1; + } + + /* set signal handler */ + sa.sa_flags = SA_SIGINFO; + sa.sa_sigaction = timer_handler; + sigemptyset(&sa.sa_mask); + if (sigaction(SIGRTMIN, &sa, NULL) == -1) { + printf("Error setting signal handler!\n"); + return -1; + } + + /* Create the timer */ + sev.sigev_notify = SIGEV_SIGNAL; + sev.sigev_signo = SIGRTMIN; + sev.sigev_value.sival_ptr = &timerid; + if (timer_create(CLOCK_REALTIME, &sev, &timerid) == -1) { + printf("Error creating the timer!\n"); + return -1; + } + + /* Start the timer */ + its.it_value.tv_sec = freq_nanosecs / 1000000000; + its.it_value.tv_nsec = freq_nanosecs % 1000000000; + its.it_interval.tv_sec = freq_nanosecs / 1000000000; + its.it_interval.tv_nsec = freq_nanosecs % 1000000000; + + if (timer_settime(timerid, 0, &its, NULL) == -1) { + printf("Error starting the timer!\n"); + return -1; + } + + /* Unlock the timer signal */ + if (sigprocmask(SIG_UNBLOCK, &mask, NULL) == -1) { + printf("Error unmasking signal!\n"); + return -1; + } + + return 0; +} + +/** + * @brief Stop & close the timer + * + * @return Operation status + * @retval 0 OK + * @retval <0 error + */ +static int close_timer(void) +{ + if (timer_delete(timerid) == -1) { + printf("Error deleting the timer!\n"); + return -1; + } + return 0; +} + +/** + * @brief Initializes PQoS library + * + * To satisfy dlock_init() requirements CAT is reset here. + * More sophisticated solution would be to look for unused CLOS here and + * pass it on to dlock_init(). + * + * @return Operation status + * @retval 0 OK + * @retval <0 error + */ +static int init_pqos(void) +{ + const struct pqos_cpuinfo *p_cpu = NULL; + const struct pqos_cap *p_cap = NULL; + struct pqos_config cfg; + int ret; + + memset(&cfg, 0, sizeof(cfg)); + cfg.fd_log = STDOUT_FILENO; + cfg.verbose = 0; + ret = pqos_init(&cfg); + if (ret != PQOS_RETVAL_OK) { + printf("Error initializing PQoS library!\n"); + return -1; + } + + /* Get CMT capability and CPU info pointer */ + ret = pqos_cap_get(&p_cap, &p_cpu); + if (ret != PQOS_RETVAL_OK) { + pqos_fini(); + printf("Error retrieving PQoS capabilities!\n"); + return -1; + } + + /* Reset CAT */ + ret = pqos_alloc_reset(PQOS_REQUIRE_CDP_ANY, PQOS_REQUIRE_CDP_ANY); + if (ret != PQOS_RETVAL_OK) { + pqos_fini(); + printf("Error resetting CAT!\n"); + return -1; + } + + return 0; +} + +/** + * @brief Closes PQoS library + * + * @return Operation status + * @retval 0 OK + * @retval <0 error + */ +static int close_pqos(void) +{ + int ret_val = 0; + + if (pqos_fini() != PQOS_RETVAL_OK) { + printf("Error shutting down PQoS library!\n"); + ret_val = -1; + } + + return ret_val; +} + +/** + * @brief Implements memory intensive workload on random locations + * + * This is plain memcpy() on random locations and random sizes. + * + * @param p pointer to memory block on which the workload is to be run + * @param size size of the memory block + */ +static void main_thread(char *p, const size_t size) +{ + const size_t half_size = size / 2; + const unsigned loop_iter = 10000000; + unsigned i; + + printf("%s() started. please wait ...\n", __func__); + + for (i = 0; i < loop_iter; i++) { + const size_t copy_size = 6 * 1024; + const int rnd1 = rand(); + const int rnd2 = rand(); + const size_t si = half_size + rnd1 % (half_size - copy_size); + const size_t di = rnd2 % (half_size - copy_size); + + memcpy(&p[di], &p[si], copy_size); + } + + printf("%s() has finished.\n", __func__); +} + +/** + * @brief Parses command line options and implements application logic + * + * @param argc number of arguments in the command line + * @param argv table with command line argument strings + * + * @return Process exit code + */ +int main(int argc, char *argv[]) +{ + const long long freq_nanosecs = freq_ms * 1000LL * 1000LL; + int core_id, lock_data = 1, exit_val = EXIT_SUCCESS; + + if (argc < 3) { + printf("Usage: %s \n", argv[0]); + exit(EXIT_FAILURE); + } + + if (strcasecmp(argv[2], "nolock") != 0 && + strcasecmp(argv[2], "lock") != 0) { + printf("Invalid data lock setting '%s'!\n", argv[2]); + printf("Usage: %s \n", argv[0]); + exit(EXIT_FAILURE); + } + + core_id = atoi(argv[1]); + lock_data = (strcasecmp(argv[2], "nolock") == 0) ? 0 : 1; + + /* allocate memory blocks */ + main_data_ptr = init_memory(main_data_size); + timer_data_ptr = init_memory(timer_data_size); + if (main_data_ptr == NULL || timer_data_ptr == NULL) { + exit_val = EXIT_FAILURE; + goto error_exit1; + } + + if (lock_data) { + /* initialize PQoS and lock the data */ + if (init_pqos() != 0) { + exit_val = EXIT_FAILURE; + goto error_exit1; + } + + /* lock the timer data */ + if (dlock_init(timer_data_ptr, + timer_data_size, 1 /* CLOS */, core_id) != 0) { + printf("Pseudo data lock error!\n"); + exit_val = EXIT_FAILURE; + goto error_exit1; + } + } + + tsc_init(&timer_prof, "Timer Handler"); + + if (init_timer(freq_nanosecs) != 0) { + printf("Timer start error!\n"); + exit_val = EXIT_FAILURE; + goto error_exit2; + } + + main_thread((char *)main_data_ptr, main_data_size); + + (void) close_timer(); + + tsc_print(&timer_prof); + + error_exit2: + if (lock_data) + dlock_exit(); + + error_exit1: + if (lock_data) + (void) close_pqos(); + + if (main_data_ptr != NULL) + free(main_data_ptr); + if (timer_data_ptr != NULL) + free(timer_data_ptr); + return exit_val; +} diff --git a/examples/c/PSEUDO_LOCK/tsc.c b/examples/c/PSEUDO_LOCK/tsc.c new file mode 100644 index 0000000..e13d9c8 --- /dev/null +++ b/examples/c/PSEUDO_LOCK/tsc.c @@ -0,0 +1,67 @@ +/* + * BSD LICENSE + * + * Copyright(c) 2016 Intel Corporation. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include /* vsnprintf() */ +#include /* va_start(), va_end() */ +#include /* ULONG_MAX */ +#include /* memset() */ +#include "tsc.h" + +static const double __measurement_cost = 0; + +void tsc_init(struct tsc_prof *p, const char *name, ...) +{ + va_list ap; + + va_start(ap, name); + memset(p->name, 0, sizeof(p->name)); + vsnprintf(p->name, sizeof(p->name) - 1, name, ap); + va_end(ap); + + p->clk_avg = 0; + p->clk_avgc = 0; + p->clk_result = 0.0; + p->clk_max = 0.0; + p->clk_min = (double) ULONG_MAX; + p->cost = __measurement_cost; +} + +void tsc_print(struct tsc_prof *p) +{ + tsc_get_avg(p); + + printf("[%s] work items %llu; cycles per work item: " + "avg=%.3f min=%.3f max=%.3f jitter=%.3f\n", + p->name, (unsigned long long)p->clk_avgc, + p->clk_result, p->clk_min, p->clk_max, p->clk_max - p->clk_min); +} diff --git a/examples/c/PSEUDO_LOCK/tsc.h b/examples/c/PSEUDO_LOCK/tsc.h new file mode 100644 index 0000000..5e43d5e --- /dev/null +++ b/examples/c/PSEUDO_LOCK/tsc.h @@ -0,0 +1,217 @@ +/* + * BSD LICENSE + * + * Copyright(c) 2016 Intel Corporation. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include + +#ifndef __TSC_H__ +#define __TSC_H__ + +#ifdef __cplusplus +extern "C" { +#endif + +/** + * TSC profile structure + */ +struct tsc_prof { + uint64_t clk_start; /**< start TSC of an iteration */ + uint64_t clk_avgc; /**< count to calculate an average */ + double clk_min; /**< min cycle cost recorded */ + double clk_max; /**< max cycle cost recorded */ + double clk_avg; /**< cumulative sum to + calculate an average */ + double clk_result; /**< avg cycles cost */ + double cost; /**< cost of measurement */ + char name[128]; +}; + +/** + * @brief Get TSC value for the start of measured block of code + * + * This function prevents out of order execution before reading TSC. + * LFENCE instruction is used for it: + * - no OOO + * - load buffers are empty after lfence + * - no deliberate restrictions on store buffers, some stores may drain though + * Another options to prevent OOO are: + * - cpuid; affects LB and SB (both get emptied) + * - forced branch miss-prediction; no effect on LB or SB but + * loads/stores may drain + * When measured code has very high cycle cost preventing OOO + * may not be required and RDTSCP instruction may be enough. + * + * @return TSC value + */ +static __attribute__((always_inline)) inline uint64_t __tsc_start(void) +{ + uint32_t cycles_high, cycles_low; + +#ifdef __x86_64__ + asm volatile("lfence\n\t" + "rdtscp\n\t" + "mov %%edx, %0\n\t" + "mov %%eax, %1\n\t" + : "=r" (cycles_high), "=r" (cycles_low) + : : "%rax", "%rdx"); +#else + asm volatile("lfence\n\t" + "rdtscp\n\t" + "mov %%edx, %0\n\t" + "mov %%eax, %1\n\t" + : "=r" (cycles_high), "=r" (cycles_low) + : : "%eax", "%edx"); +#endif + return(((uint64_t)cycles_high << 32) | cycles_low); +} + +/** + * @brief Get TSC value for the end of measured block of code + * + * No OOO prevention required. RDTSCP is used here which makes sure + * all previous instructions retire before reading TSC. + * + * @return TSC value + */ +static __attribute__((always_inline)) inline uint64_t __tsc_end(void) +{ + uint32_t cycles_high, cycles_low; + +#ifdef __x86_64__ + asm volatile("rdtscp\n\t" + "mov %%edx, %0\n\t" + "mov %%eax, %1\n\t" + : "=r" (cycles_high), "=r" (cycles_low) + : : "%rax", "%rdx"); +#else + asm volatile("rdtscp\n\t" + "mov %%edx, %0\n\t" + "mov %%eax, %1\n\t" + : "=r" (cycles_high), "=r" (cycles_low) + : : "%eax", "%edx"); +#endif + return ((uint64_t)cycles_high << 32) | cycles_low; +} + +/** + * @brief Starts cycle measurement of code iteration + * + * tsc_start() and tsc_end() or tsc_end_ex() can be called multiple times. + * + * @param p pointer to TSC profile structure + */ +static __attribute__((always_inline)) inline void +tsc_start(struct tsc_prof *p) +{ + p->clk_start = __tsc_start(); +} + +/** + * @brief Stops cycle measurement of code iteration + * + * @param p pointer to TSC profile structure + * @param inc number of items processed within the iteration. + * This allows code to calculate average cycle cost per work item even + * though number of code iterations may be different. + * @param clk_start start TSC value. This is useful when using one + * start TSC reading for multiple different TSC profiles. + */ +static __attribute__((always_inline)) inline void +tsc_end_ex(struct tsc_prof *p, const unsigned inc, const uint64_t clk_start) +{ + double clk_diff = (double) (__tsc_end() - clk_start); + + p->clk_avgc += inc; + p->clk_avg += (clk_diff - p->cost); + + clk_diff = clk_diff / (double) inc; + + if (clk_diff < p->clk_min) + p->clk_min = clk_diff; + if (clk_diff > p->clk_max) + p->clk_max = clk_diff; +} + +/** + * @brief Stops cycle measurement of code iteration + * + * @param p pointer to TSC profile structure + * @param inc number of items processed within the iteration + */ +static __attribute__((always_inline)) inline void +tsc_end(struct tsc_prof *p, const unsigned inc) +{ + tsc_end_ex(p, inc, p->clk_start); +} + +/** + * @brief Calculates an average cycle cost per item + * + * Calculated average cycle cost is also stored in TSC profile structure. + * + * @param p pointer to TSC profile structure + * + * @return Calculated average cycle cost per work item + * @retval NAN if no code measurement done so far + */ +static __attribute__((always_inline)) inline +double tsc_get_avg(struct tsc_prof *p) +{ + double avg_c = 0.0; + + if (p->clk_avgc > 0) + avg_c = (p->clk_avg / ((double) p->clk_avgc)); + p->clk_result = avg_c; + return avg_c; +} + +/** + * @brief Initializes TSC profile structure + * + * @param p pointer to TSC profile structure + * @param name string describing the measurement in printf() format + * @param ... variadic arguments depending on \a name format + */ +void tsc_init(struct tsc_prof *p, const char *name, ...); + +/** + * @brief Prints measured TSC profile data + * + * @param p pointer to TSC profile structure + */ +void tsc_print(struct tsc_prof *p); + +#ifdef __cplusplus +} +#endif + +#endif /* __TSC_H__ */ diff --git a/examples/perl/Makefile b/examples/perl/Makefile new file mode 100644 index 0000000..040c000 --- /dev/null +++ b/examples/perl/Makefile @@ -0,0 +1,69 @@ +############################################################################### +# Makefile script for PQoS hello_world.pl Perl script +# +# @par +# BSD LICENSE +# +# Copyright(c) 2016 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +############################################################################### + +PERLTIDY?=perltidy + +PERLTIDY_STYLE=-bar -ce -pt=2 -sbt=2 -bt=2 -bbt=2 -et=4 -baao -nsfs +# http://perltidy.sourceforge.net/stylekey.html +# http://perltidy.sourceforge.net/perltidy.html +# -bar - keep the brace on the right even for multiple-line test expressions +# -ce - Cuddled Else +# -pt=2 - parentheses horizontal tightness - space is never used +# -sbt=2 - square brackets horizontal tightness - space is never used +# -bbt=2 - code block curly braces horizontal tightness - space is never used +# -et=4 - entab leading whitespace with one tab character for each 4 spaces +# -baao - "break after all operators" +# -nsfs - no space before a semicolon in a for statement + +# List of Perl scripts to be processed, space separated list +PERL_SCRIPTS_LIST=hello_world.pl + +# To check if required style is followed, perltidy is called and +# formatted output is compared to actual file, +# if there are no differences, it means that file meets the requirements +.PHONY: style +style: + @bash -c 'for i in $(PERL_SCRIPTS_LIST); do \ + $(PERLTIDY) $(PERLTIDY_STYLE) ./$${i} -st |\ + diff - ./$${i} && echo "$${i} style OK" ||\ + { echo "$${i} style check failed!"; exit 1; }; \ + done' + +.PHONY: style-fix +style-fix: + @bash -c 'for i in $(PERL_SCRIPTS_LIST); do \ + $(PERLTIDY) $(PERLTIDY_STYLE) -b ./$${i}; \ + done' diff --git a/examples/perl/README b/examples/perl/README new file mode 100644 index 0000000..1444739 --- /dev/null +++ b/examples/perl/README @@ -0,0 +1,52 @@ + + +======================================================================== +README for hello_world.pl Perl script + +April 2016 +======================================================================== + + +Contents +======== + +- Overview +- Requirements +- Usage +- Legal Disclaimer + + +Overview +======== + +hello_world.pl is a simple Perl script to demonstrate the use of the +PQoS/Intel(R) RDT library Perl wrapper API. This script shows how to +initialize the library, read COS bitmasks and associations and shutdown +the library. + +Requirements +============ + +Tested with SWIG Version 3.0.5 and Perl v5.18.4. + +The hello_world.pl script depends on the pqos.pm Perl module generated +by SWIG. For more information about this module and how to build it, +please refer to the README found in the "lib/perl" directory. + +Usage +===== + +To run the hello_world.pl script: + "sudo ./hello_world.pl" + +Legal Disclaimer +================ + +THIS SOFTWARE IS PROVIDED BY INTEL"AS IS". NO LICENSE, EXPRESS OR +IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS +ARE GRANTED THROUGH USE. EXCEPT AS PROVIDED IN INTEL'S TERMS AND +CONDITIONS OF SALE, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL +DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR +USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO +FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT +OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. \ No newline at end of file diff --git a/examples/perl/hello_world.pl b/examples/perl/hello_world.pl new file mode 100755 index 0000000..4f996d0 --- /dev/null +++ b/examples/perl/hello_world.pl @@ -0,0 +1,89 @@ +#!/usr/bin/perl + +################################################################################ +# BSD LICENSE +# +# Copyright(c) 2016 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +################################################################################ + +use strict; +use warnings; +use pqos; + +=begin +Script to test PQoS Perl interface +=cut + +my $cfg = pqos::pqos_config->new(); +my $l3ca = pqos::pqos_l3ca->new(); + +# Setup config +$cfg->{verbose} = 0; +$cfg->{fd_log} = 1; + +# Initialize the library +if (0 != pqos::pqos_init($cfg)) { + print "Error initializing PQoS library!\n"; + exit 0; +} +print "Hello, World!\n"; +print "\t\t\t\tAssociation\tWay Mask\n"; + +# Get number of cores +my $cpuinfo_p = pqos::get_cpuinfo(); +my $cpu_num = pqos::cpuinfo_p_value($cpuinfo_p)->{num_cores}; + +# Print L3CA info for each core +for (my $i = 0; $i < $cpu_num; $i++) { + + # Get core association + (my $result, my $cos) = pqos::pqos_alloc_assoc_get($i); + if (0 != $result) { + next; + } + + # Get socket ID for this core + ($result, my $socket_id) = pqos::pqos_cpu_get_socketid($cpuinfo_p, $i); + if (0 != $result) { + next; + } + + # Get way mask info + if (0 != pqos::get_l3ca($l3ca, $socket_id, $cos)) { + next; + } + + # Print info + printf("Hello from core %d on socket %d\tCOS %d \t\t0x%x\n", + $i, $socket_id, $cos, $l3ca->{u}->{ways_mask}); +} + +# Shutdown the library +pqos::pqos_fini(); diff --git a/lib/Makefile b/lib/Makefile new file mode 100644 index 0000000..4d99548 --- /dev/null +++ b/lib/Makefile @@ -0,0 +1,221 @@ +############################################################################### +# Makefile script for PQoS library and sample application +# +# @par +# BSD LICENSE +# +# Copyright(c) 2014-2017 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +############################################################################### + +LIB = libpqos +VERSION = 2.0.0 +SO_VERSION = 2 +SHARED ?= y +LDFLAGS = -L. -lpthread -z noexecstack -z relro -z now +CFLAGS = -pthread -I./ -D_GNU_SOURCE \ + -W -Wall -Wextra -Wstrict-prototypes -Wmissing-prototypes \ + -Wmissing-declarations -Wold-style-definition -Wpointer-arith \ + -Wcast-qual -Wundef -Wwrite-strings \ + -Wformat -Wformat-security -fstack-protector -D_FORTIFY_SOURCE=2\ + -Wunreachable-code -Wsign-compare -Wno-endif-labels +ifneq ($(EXTRA_CFLAGS),) +CFLAGS += $(EXTRA_CFLAGS) +endif + +DOXY_DIRS = doc_api doc_lib + +# ICC and GCC options +ifeq ($(CC),icc) +else +CFLAGS += -Wcast-align \ + -Wnested-externs \ + -Wmissing-noreturn +endif + +# so or static build +ifeq ($(SHARED),y) +CFLAGS += -fPIC +LIBNAME = $(LIB).so.$(VERSION) +LIBPERM = 0755 +else +CFLAGS += -fPIE +LIBNAME = $(LIB).a +LIBPERM = 0644 +endif + +# DEBUG build +ifeq ($(DEBUG),y) +CFLAGS += -g -ggdb -O0 -DDEBUG +else +CFLAGS += -g -O2 +endif + +# Build targets and dependencies +SRCS = $(sort $(wildcard *.c)) +OBJS = $(SRCS:.c=.o) +CLEAN_OBJS = $(SRCS:.c=.o) + +# On FreeBSD build with no OS support +ifeq ($(shell uname), FreeBSD) +OBJS := $(filter-out perf.o \ + os_allocation.o \ + os_monitoring.o \ + resctrl.o \ + resctrl_alloc.o \ + resctrl_monitoring.o \ + perf_monitoring.o,$(OBJS)) +endif + +HDR = pqos.h +PREFIX ?= /usr/local +LIB_INSTALL_DIR ?= $(PREFIX)/lib +HDR_DIR ?= $(PREFIX)/include +DEPFILE = $(LIB).dep +NOLDCONFIG ?= n + +all: $(LIBNAME) + +$(LIBNAME): $(OBJS) +ifeq ($(SHARED),y) + $(CC) -shared -Wl,-soname,$(LIB).so.$(SO_VERSION) -o $(LIBNAME) $^ -lc + ln -f -s $(LIBNAME) $(LIB).so.$(SO_VERSION) + ln -f -s $(LIB).so.$(SO_VERSION) $(LIB).so +else + $(AR) crvsD $@ $^ +endif + +install: $(LIBNAME) +# Install on FreeBSD +ifeq ($(shell uname), FreeBSD) + install -d $(LIB_INSTALL_DIR) + install -m $(LIBPERM) $(LIBNAME) $(LIB_INSTALL_DIR) + install -m 0644 $(HDR) $(HDR_DIR) +# Install on Linux +else + install -m $(LIBPERM) $(LIBNAME) -D $(LIB_INSTALL_DIR)/$(LIBNAME) + install -m 0644 $(HDR) -D $(HDR_DIR)/$(HDR) +endif + +# Create symlinks to DSO +ifeq ($(SHARED),y) + cd $(LIB_INSTALL_DIR); \ + ln -f -s $(LIB).so.$(VERSION) $(LIB).so.$(SO_VERSION); \ + ln -f -s $(LIB).so.$(SO_VERSION) $(LIB).so +ifneq ($(NOLDCONFIG),y) + ldconfig +endif +endif + +uninstall: + -rm $(LIB_INSTALL_DIR)/libpqos* +ifeq ($(SHARED),y) +ifneq ($(NOLDCONFIG),y) + ldconfig +endif +endif + +DEPFILES = $(SRCS:.c=.d) + +%.o: %.c %.d + +%.d: %.c + $(CC) -MM -MP -MF $@ $(CFLAGS) $< + cat $@ | sed 's/$(@:.d=.o)/$@/' >> $@ + +.PHONY: clean clobber doxy help + +help: + @echo "PQoS library make targets:" + @echo " make - build shared library" + @echo " make SHARED=n - build static library" + @echo " make DEBUG=y - build shared library for debugging" + @echo " make install - install library (accepts PREFIX=/some/where)" + @echo " make uninstall - uninstall library (accepts PREFIX=/some/where)" + @echo " make clean - remove files produced normally by make" + @echo " make clobber - clean and remove documentation" + @echo " make doxy - make doxygen documentation" + @echo " make style - coding style check (accepts CHECKPATCH=/some/where/checkpatch.pl)" + @echo " make cppcheck - static code analysis (accepts CPPCHECK=/some/where/cppcheck)" + +doxy: + doxygen api_doxygen.cfg + doxygen lib_doxygen.cfg + +clean: + -rm -f $(CLEAN_OBJS) $(LIB)* $(DEPFILES) + +clobber: + -rm -f $(CLEAN_OBJS) $(LIB)* $(DEPFILES) + -rm -rf $(DOXY_DIRS) + +CHECKPATCH?=checkpatch.pl +.PHONY: style +style: + $(CHECKPATCH) --no-tree --no-signoff --emacs \ + --ignore CODE_INDENT,INITIALISED_STATIC,LEADING_SPACE,SPLIT_STRING,\ + UNSPECIFIED_INT,ARRAY_SIZE,BLOCK_COMMENT_STYLE,GLOBAL_INITIALISERS \ + -f api.c -f api.h -f cap.c -f cap.h -f allocation.c -f perf.c \ + -f perf.h -f allocation.h -f monitoring.c -f monitoring.h \ + -f log.c -f log.h -f types.h -f machine.c -f machine.h \ + -f utils.c -f utils.h \ + -f cpuinfo.h -f os_allocation.h -f os_allocation.c \ + -f os_monitoring.h os_monitoring.c \ + -f resctrl_alloc.h -f resctrl_alloc.c + $(CHECKPATCH) --no-tree --no-signoff --emacs \ + --ignore CODE_INDENT,INITIALISED_STATIC,LEADING_SPACE,SPLIT_STRING,\ + NEW_TYPEDEFS,UNSPECIFIED_INT,BLOCK_COMMENT_STYLE \ + -f pqos.h -f cpuinfo.c \ + -f perf_monitoring.h -f perf_monitoring.c \ + -f resctrl.h -f resctrl.c \ + -f resctrl_monitoring.h -f resctrl_monitoring.c + +CPPCHECK?=cppcheck +.PHONY: cppcheck +cppcheck: + $(CPPCHECK) \ + --enable=warning,portability,performance,missingInclude \ + --std=c99 --template=gcc \ + api.c api.h cap.c cap.h allocation.c perf.c perf.h \ + allocation.h monitoring.c monitoring.h \ + log.c log.h types.h machine.c machine.h \ + utils.c utils.h \ + cpuinfo.c cpuinfo.h os_allocation.h os_allocation.c \ + os_monitoring.h os_monitoring.c \ + perf_monitoring.h perf_monitoring.c \ + resctrl_alloc.h resctrl_alloc.c \ + resctrl.h resctrl.c \ + resctrl_monitoring.h resctrl_monitoring.c + +# if target not clean or rinse then make dependencies +ifneq ($(MAKECMDGOALS),clean) +ifneq ($(MAKECMDGOALS),clobber) +-include $(DEPFILES) +endif +endif diff --git a/lib/README b/lib/README new file mode 100644 index 0000000..1aa34a9 --- /dev/null +++ b/lib/README @@ -0,0 +1,71 @@ + + +======================================================================== +README for PQoS/Intel(R) RDT library + +April 2016 +======================================================================== + + +Contents +======== + +- Overview +- Installation +- Legal Disclaimer + + +Overview +======== + +PQoS library provides API to detect and configure Intel(R) RDT including: +Cache Monitoring Technology (CMT), Memory Bandwidth Monitoring (MBM), +Cache Allocation Technology (CAT), Code and Data Prioritization (CDP) Technology. +For more information about Intel(R) RDT please see top level README. + + +Installation +============ + +NOTE to FreeBSD users, remember to replace "make" with "gmake" in +the steps described below. + +The following steps are required to compile and install library: +$ make +$ sudo make install + +By default make builds shared library. + +"make" accepts extra options e.g.: +"SHARED=n" - for static library +"DEBUG=y" - for library for debugging + +"sudo make install" installs compiled library into system directories. + +By default, library files are installed in "lib" directory below "/usr/local" +but it can be changed with use of PREFIX to install files below "/some/where": +$ sudo make install PREFIX=/some/where + +Library files can be removed but the same PREFIX has to be +used for uninstall and install targets. + +To remove files from below default PREFIX: +$ sudo make uninstall + +To remove from below /some/where: +$ sudo make uninstall PREFIX=/some/where + +For more info about make targets, please run "make help" command. + + +Legal Disclaimer +================ + +THIS SOFTWARE IS PROVIDED BY INTEL"AS IS". NO LICENSE, EXPRESS OR +IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS +ARE GRANTED THROUGH USE. EXCEPT AS PROVIDED IN INTEL'S TERMS AND +CONDITIONS OF SALE, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL +DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR +USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO +FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT +OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. \ No newline at end of file diff --git a/lib/allocation.c b/lib/allocation.c new file mode 100644 index 0000000..00f3f63 --- /dev/null +++ b/lib/allocation.c @@ -0,0 +1,1561 @@ +/* + * BSD LICENSE + * + * Copyright(c) 2014-2018 Intel Corporation. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +/** + * @brief Implementation of CAT related PQoS API + * + * CPUID and MSR operations are done on the 'local'/host system. + * Module operate directly on CAT registers. + */ + +#include +#include +#include + +#include "pqos.h" + +#include "cap.h" +#include "allocation.h" +#include "os_allocation.h" + +#include "machine.h" +#include "types.h" +#include "log.h" + +/** + * --------------------------------------- + * Local macros + * --------------------------------------- + */ + +/** + * Allocation & Monitoring association MSR register + * - bits [63..32] QE COS + * - bits [31..10] Reserved + * - bits [9..0] RMID + */ +#define PQOS_MSR_ASSOC 0xC8F +#define PQOS_MSR_ASSOC_QECOS_SHIFT 32 +#define PQOS_MSR_ASSOC_QECOS_MASK 0xffffffff00000000ULL + +/** + * Allocation class of service (COS) MSR registers + */ +#define PQOS_MSR_L3CA_MASK_START 0xC90 +#define PQOS_MSR_L3CA_MASK_END 0xD0F +#define PQOS_MSR_L3CA_MASK_NUMOF \ + (PQOS_MSR_L3CA_MASK_END - PQOS_MSR_L3CA_MASK_START + 1) + +#define PQOS_MSR_L2CA_MASK_START 0xD10 +#define PQOS_MSR_MBA_MASK_START 0xD50 + +#define PQOS_MSR_L3_QOS_CFG 0xC81 /**< L3 CAT config register */ +#define PQOS_MSR_L3_QOS_CFG_CDP_EN 1ULL /**< L3 CDP enable bit */ + +#define PQOS_MSR_L2_QOS_CFG 0xC82 /**< L2 CAT config register */ +#define PQOS_MSR_L2_QOS_CFG_CDP_EN 1ULL /**< L2 CDP enable bit */ + +/** + * MBA linear max value + */ +#define PQOS_MBA_LINEAR_MAX 100 + +/** + * --------------------------------------- + * Local data types + * --------------------------------------- + */ + +/** + * --------------------------------------- + * Local data structures + * --------------------------------------- + */ +static const struct pqos_cap *m_cap = NULL; +static const struct pqos_cpuinfo *m_cpu = NULL; +static int m_interface = PQOS_INTER_MSR; +/** + * --------------------------------------- + * External data + * --------------------------------------- + */ + +/** + * --------------------------------------- + * Internal functions + * --------------------------------------- + */ + +/** + * @brief Gets highest COS id which could be used to configure set technologies + * + * @param [in] technology technologies bitmask to get highest common COS id for + * @param [out] hi_class_id highest common COS id + * + * @return Operation status + */ +static int +get_hi_cos_id(const unsigned technology, + unsigned *hi_class_id) +{ + const int l2_req = ((technology & (1 << PQOS_CAP_TYPE_L2CA)) != 0); + const int l3_req = ((technology & (1 << PQOS_CAP_TYPE_L3CA)) != 0); + const int mba_req = ((technology & (1 << PQOS_CAP_TYPE_MBA)) != 0); + unsigned num_l2_cos = 0, num_l3_cos = 0, num_mba_cos = 0, num_cos = 0; + int ret; + + ASSERT(l2_req || l3_req || mba_req); + if (hi_class_id == NULL) + return PQOS_RETVAL_PARAM; + + ASSERT(m_cap != NULL); + + if (l3_req) { + ret = pqos_l3ca_get_cos_num(m_cap, &num_l3_cos); + if (ret != PQOS_RETVAL_OK && ret != PQOS_RETVAL_RESOURCE) + return ret; + + if (num_l3_cos == 0) + return PQOS_RETVAL_ERROR; + + num_cos = num_l3_cos; + } + + if (l2_req) { + ret = pqos_l2ca_get_cos_num(m_cap, &num_l2_cos); + if (ret != PQOS_RETVAL_OK && ret != PQOS_RETVAL_RESOURCE) + return ret; + if (num_l2_cos == 0) + return PQOS_RETVAL_ERROR; + + if (num_cos == 0 || num_l2_cos < num_cos) + num_cos = num_l2_cos; + } + + if (mba_req) { + ret = pqos_mba_get_cos_num(m_cap, &num_mba_cos); + if (ret != PQOS_RETVAL_OK && ret != PQOS_RETVAL_RESOURCE) + return ret; + + if (num_mba_cos == 0) + return PQOS_RETVAL_ERROR; + + if (num_cos == 0 || num_mba_cos < num_cos) + num_cos = num_mba_cos; + } + + *hi_class_id = num_cos - 1; + + return PQOS_RETVAL_OK; +} + +/** + * @brief Gets COS associated to \a lcore + * + * @param [in] lcore lcore to read COS association from + * @param [out] class_id associated COS + * + * @return Operation status + */ +static int +cos_assoc_get(const unsigned lcore, unsigned *class_id) +{ + const uint32_t reg = PQOS_MSR_ASSOC; + uint64_t val = 0; + + if (class_id == NULL) + return PQOS_RETVAL_PARAM; + + if (msr_read(lcore, reg, &val) != MACHINE_RETVAL_OK) + return PQOS_RETVAL_ERROR; + + val >>= PQOS_MSR_ASSOC_QECOS_SHIFT; + *class_id = (unsigned) val; + + return PQOS_RETVAL_OK; +} + +/** + * @brief Gets unused COS on a socket or L2 cluster + * + * The lowest acceptable COS is 1, as 0 is a default one + * + * @param [in] id socket or L2 cache ID to search for unused COS on + * @param [in] technology selection of allocation technologies + * @param [out] class_id unused COS + * + * @return Operation status + */ +static int +get_unused_cos(const unsigned id, + const unsigned technology, + unsigned *class_id) +{ + const int l2_req = ((technology & (1 << PQOS_CAP_TYPE_L2CA)) != 0); + unsigned used_classes[PQOS_MAX_L3CA_COS]; + unsigned i, cos; + unsigned hi_class_id; + int ret; + + if (class_id == NULL) + return PQOS_RETVAL_PARAM; + + /* obtain highest class id for all requested technologies */ + ret = get_hi_cos_id(technology, &hi_class_id); + if (ret != PQOS_RETVAL_OK) + return ret; + + memset(used_classes, 0, sizeof(used_classes)); + + /* Create a list of COS used on socket/L2 cluster */ + for (i = 0; i < m_cpu->num_cores; i++) { + + if (l2_req) { + /* L2 requested so looking in L2 cluster scope */ + if (m_cpu->cores[i].l2_id != id) + continue; + } else { + /* L2 not requested so looking at socket scope */ + if (m_cpu->cores[i].socket != id) + continue; + } + + ret = cos_assoc_get(m_cpu->cores[i].lcore, &cos); + if (ret != PQOS_RETVAL_OK) + return ret; + + if (cos > hi_class_id) + continue; + + /* Mark as used */ + used_classes[cos] = 1; + } + + /* Find unused COS */ + for (cos = hi_class_id; cos != 0; cos--) { + if (used_classes[cos] == 0) { + *class_id = cos; + return PQOS_RETVAL_OK; + } + } + + return PQOS_RETVAL_RESOURCE; +} + +/** + * ======================================= + * ======================================= + * + * initialize and shutdown + * + * ======================================= + * ======================================= + */ + +int +pqos_alloc_init(const struct pqos_cpuinfo *cpu, + const struct pqos_cap *cap, + const struct pqos_config *cfg) +{ + int ret = PQOS_RETVAL_OK; + + m_cap = cap; + m_cpu = cpu; + if (cfg == NULL) + m_interface = PQOS_INTER_MSR; + else if (cfg->interface == PQOS_INTER_OS_RESCTRL_MON) + m_interface = PQOS_INTER_OS; + else + m_interface = cfg->interface; +#ifdef __linux__ + if (m_interface == PQOS_INTER_OS) + ret = os_alloc_init(cpu, cap); +#endif + return ret; +} + +int +pqos_alloc_fini(void) +{ + int ret = PQOS_RETVAL_OK; + + m_cap = NULL; + m_cpu = NULL; +#ifdef __linux__ + if (m_interface == PQOS_INTER_OS) + ret = os_alloc_fini(); +#endif + return ret; +} + +/** + * ======================================= + * L3 cache allocation + * ======================================= + */ + +int +hw_l3ca_set(const unsigned socket, + const unsigned num_ca, + const struct pqos_l3ca *ca) +{ + int ret = PQOS_RETVAL_OK; + unsigned i = 0, count = 0, core = 0; + int cdp_enabled = 0; + + ASSERT(ca != NULL); + ASSERT(num_ca != 0); + ASSERT(m_cap != NULL); + + ret = pqos_l3ca_get_cos_num(m_cap, &count); + if (ret != PQOS_RETVAL_OK) + return ret; /**< perhaps no L3CA capability */ + + if (num_ca > count) + return PQOS_RETVAL_ERROR; + + ret = pqos_l3ca_cdp_enabled(m_cap, NULL, &cdp_enabled); + if (ret != PQOS_RETVAL_OK) + return ret; + + ASSERT(m_cpu != NULL); + ret = pqos_cpu_get_one_core(m_cpu, socket, &core); + if (ret != PQOS_RETVAL_OK) + return ret; + + if (cdp_enabled) { + for (i = 0; i < num_ca; i++) { + uint32_t reg = + (ca[i].class_id*2) + PQOS_MSR_L3CA_MASK_START; + int retval = MACHINE_RETVAL_OK; + uint64_t cmask = 0, dmask = 0; + + if (ca[i].cdp) { + dmask = ca[i].u.s.data_mask; + cmask = ca[i].u.s.code_mask; + } else { + dmask = ca[i].u.ways_mask; + cmask = ca[i].u.ways_mask; + } + + retval = msr_write(core, reg, dmask); + if (retval != MACHINE_RETVAL_OK) + return PQOS_RETVAL_ERROR; + + retval = msr_write(core, reg+1, cmask); + if (retval != MACHINE_RETVAL_OK) + return PQOS_RETVAL_ERROR; + } + } else { + for (i = 0; i < num_ca; i++) { + uint32_t reg = + ca[i].class_id + PQOS_MSR_L3CA_MASK_START; + uint64_t val = ca[i].u.ways_mask; + int retval = MACHINE_RETVAL_OK; + + if (ca[i].cdp) { + LOG_ERROR("Attempting to set CDP COS " + "while L3 CDP is disabled!\n"); + return PQOS_RETVAL_ERROR; + } + + retval = msr_write(core, reg, val); + if (retval != MACHINE_RETVAL_OK) + return PQOS_RETVAL_ERROR; + } + } + return ret; +} + +int +hw_l3ca_get(const unsigned socket, + const unsigned max_num_ca, + unsigned *num_ca, + struct pqos_l3ca *ca) +{ + int ret = PQOS_RETVAL_OK; + unsigned i = 0, count = 0, core = 0; + uint32_t reg = 0; + uint64_t val = 0; + int retval = MACHINE_RETVAL_OK; + int cdp_enabled = 0; + + ASSERT(num_ca != NULL); + ASSERT(ca != NULL); + ASSERT(max_num_ca != 0); + ASSERT(m_cap != NULL); + ret = pqos_l3ca_get_cos_num(m_cap, &count); + if (ret != PQOS_RETVAL_OK) + return ret; /**< perhaps no L3CA capability */ + + ret = pqos_l3ca_cdp_enabled(m_cap, NULL, &cdp_enabled); + if (ret != PQOS_RETVAL_OK) + return ret; + + if (count > max_num_ca) + return PQOS_RETVAL_ERROR; + + ASSERT(m_cpu != NULL); + ret = pqos_cpu_get_one_core(m_cpu, socket, &core); + if (ret != PQOS_RETVAL_OK) + return ret; + + if (cdp_enabled) { + for (i = 0, reg = PQOS_MSR_L3CA_MASK_START; + i < count; i++, reg += 2) { + ca[i].cdp = 1; + ca[i].class_id = i; + + retval = msr_read(core, reg, &val); + if (retval != MACHINE_RETVAL_OK) + return PQOS_RETVAL_ERROR; + + ca[i].u.s.data_mask = val; + + retval = msr_read(core, reg + 1, &val); + if (retval != MACHINE_RETVAL_OK) + return PQOS_RETVAL_ERROR; + + ca[i].u.s.code_mask = val; + } + } else { + for (i = 0, reg = PQOS_MSR_L3CA_MASK_START; + i < count; i++, reg++) { + retval = msr_read(core, reg, &val); + if (retval != MACHINE_RETVAL_OK) + return PQOS_RETVAL_ERROR; + + ca[i].cdp = 0; + ca[i].class_id = i; + ca[i].u.ways_mask = val; + } + } + *num_ca = count; + + return ret; +} + +int +hw_l3ca_get_min_cbm_bits(unsigned *min_cbm_bits) +{ + int ret; + unsigned *sockets, socket_id, sockets_num; + unsigned class_id, l3ca_num, ways, i; + int technology = 1 << PQOS_CAP_TYPE_L3CA; + const struct pqos_capability *l3_cap = NULL; + struct pqos_l3ca l3ca_config[PQOS_MAX_L3CA_COS]; + + ASSERT(m_cap != NULL); + ASSERT(m_cpu != NULL); + ASSERT(min_cbm_bits != NULL); + + /** + * Get L3 CAT capabilities + */ + ret = pqos_cap_get_type(m_cap, PQOS_CAP_TYPE_L3CA, &l3_cap); + if (ret != PQOS_RETVAL_OK) + return PQOS_RETVAL_RESOURCE; /* L3 CAT not supported */ + + /** + * Get number & list of sockets in the system + */ + sockets = pqos_cpu_get_sockets(m_cpu, &sockets_num); + if (sockets == NULL || sockets_num == 0) { + ret = PQOS_RETVAL_ERROR; + goto pqos_l3ca_get_min_cbm_bits_exit; + } + + /** + * Find free COS + */ + for (socket_id = 0; socket_id < sockets_num; socket_id++) { + ret = get_unused_cos(socket_id, technology, &class_id); + if (ret == PQOS_RETVAL_OK) + break; + + if (ret != PQOS_RETVAL_RESOURCE) + goto pqos_l3ca_get_min_cbm_bits_exit; + } + + if (ret == PQOS_RETVAL_RESOURCE) { + LOG_INFO("No free L3 COS available. " + "Unable to determine minimum L3 CBM bits\n"); + goto pqos_l3ca_get_min_cbm_bits_exit; + } + + /** + * Get current configuration + */ + ret = hw_l3ca_get(socket_id, PQOS_MAX_L3CA_COS, &l3ca_num, l3ca_config); + if (ret != PQOS_RETVAL_OK) + goto pqos_l3ca_get_min_cbm_bits_exit; + + /** + * Probe for min cbm bits + */ + for (ways = 1; ways <= l3_cap->u.l3ca->num_ways; ways++) { + struct pqos_l3ca l3ca_tab[PQOS_MAX_L3CA_COS]; + unsigned num_ca; + uint64_t mask = (1 << ways) - 1; + + memset(l3ca_tab, 0, sizeof(struct pqos_l3ca)); + l3ca_tab[0].class_id = class_id; + l3ca_tab[0].u.ways_mask = mask; + + /** + * Try to set mask + */ + ret = hw_l3ca_set(socket_id, 1, l3ca_tab); + if (ret != PQOS_RETVAL_OK) + continue; + + /** + * Validate if mask was correctly set + */ + ret = hw_l3ca_get(socket_id, PQOS_MAX_L3CA_COS, &num_ca, + l3ca_tab); + if (ret != PQOS_RETVAL_OK) + goto pqos_l3ca_get_min_cbm_bits_restore; + + for (i = 0; i < num_ca; i++) { + struct pqos_l3ca *l3ca = &(l3ca_tab[i]); + + if (l3ca->class_id != class_id) + continue; + + if ((l3ca->cdp && + l3ca->u.s.data_mask == mask && + l3ca->u.s.code_mask == mask) || + (!l3ca->cdp && l3ca->u.ways_mask == mask)) { + *min_cbm_bits = ways; + ret = PQOS_RETVAL_OK; + goto pqos_l3ca_get_min_cbm_bits_restore; + } + } + } + + /** + * Restore old settings + */ + pqos_l3ca_get_min_cbm_bits_restore: + for (i = 0; i < l3ca_num; i++) { + int ret_val; + + if (l3ca_config[i].class_id != class_id) + continue; + + ret_val = hw_l3ca_set(socket_id, 1, &(l3ca_config[i])); + if (ret_val != PQOS_RETVAL_OK) { + LOG_ERROR("Failed to restore CAT configuration. CAT" + " configuration has been altered!\n"); + ret = ret_val; + break; + } + } + + pqos_l3ca_get_min_cbm_bits_exit: + if (sockets != NULL) + free(sockets); + + return ret; +} + +int +hw_l2ca_set(const unsigned l2id, + const unsigned num_ca, + const struct pqos_l2ca *ca) +{ + int ret = PQOS_RETVAL_OK; + unsigned i = 0, count = 0, core = 0; + int cdp_enabled = 0; + + ASSERT(ca != NULL); + ASSERT(num_ca != 0); + ASSERT(m_cap != NULL); + ASSERT(m_cpu != NULL); + + /* + * Check if L2 CAT is supported + */ + ret = pqos_l2ca_get_cos_num(m_cap, &count); + if (ret != PQOS_RETVAL_OK) + return PQOS_RETVAL_RESOURCE; /* L2 CAT not supported */ + + /** + * Check if class id's are within allowed range. + */ + for (i = 0; i < num_ca; i++) { + if (ca[i].class_id >= count) { + LOG_ERROR("L2 COS%u is out of range (COS%u is max)!\n", + ca[i].class_id, count - 1); + return PQOS_RETVAL_PARAM; + } + } + + ret = pqos_l2ca_cdp_enabled(m_cap, NULL, &cdp_enabled); + if (ret != PQOS_RETVAL_OK) + return ret; + + /** + * Pick one core from the L2 cluster and + * perform MSR writes to COS registers on the cluster. + */ + ret = pqos_cpu_get_one_by_l2id(m_cpu, l2id, &core); + if (ret != PQOS_RETVAL_OK) + return ret; + + for (i = 0; i < num_ca; i++) { + if (cdp_enabled) { + uint32_t reg = + (ca[i].class_id * 2) + PQOS_MSR_L2CA_MASK_START; + int retval = MACHINE_RETVAL_OK; + uint64_t cmask = 0, dmask = 0; + + if (ca[i].cdp) { + dmask = ca[i].u.s.data_mask; + cmask = ca[i].u.s.code_mask; + } else { + dmask = ca[i].u.ways_mask; + cmask = ca[i].u.ways_mask; + } + + retval = msr_write(core, reg, dmask); + if (retval != MACHINE_RETVAL_OK) + return PQOS_RETVAL_ERROR; + + retval = msr_write(core, reg + 1, cmask); + if (retval != MACHINE_RETVAL_OK) + return PQOS_RETVAL_ERROR; + } else { + uint32_t reg = + ca[i].class_id + PQOS_MSR_L2CA_MASK_START; + uint64_t val = ca[i].u.ways_mask; + int retval; + + if (ca[i].cdp) { + LOG_ERROR("Attempting to set CDP COS " + "while L2 CDP is disabled!\n"); + return PQOS_RETVAL_ERROR; + } + + retval = msr_write(core, reg, val); + if (retval != MACHINE_RETVAL_OK) + return PQOS_RETVAL_ERROR; + } + } + + return ret; +} + +int +hw_l2ca_get(const unsigned l2id, + const unsigned max_num_ca, + unsigned *num_ca, + struct pqos_l2ca *ca) +{ + int ret = PQOS_RETVAL_OK; + unsigned i = 0, count = 0; + unsigned core = 0; + int cdp_enabled = 0; + + ASSERT(num_ca != NULL); + ASSERT(ca != NULL); + ASSERT(max_num_ca != 0); + ASSERT(m_cap != NULL); + ret = pqos_l2ca_get_cos_num(m_cap, &count); + if (ret != PQOS_RETVAL_OK) + return PQOS_RETVAL_RESOURCE; /* L2 CAT not supported */ + + ret = pqos_l2ca_cdp_enabled(m_cap, NULL, &cdp_enabled); + if (ret != PQOS_RETVAL_OK) + return ret; + + if (max_num_ca < count) + /* Not enough space to store the classes */ + return PQOS_RETVAL_PARAM; + + ASSERT(m_cpu != NULL); + ret = pqos_cpu_get_one_by_l2id(m_cpu, l2id, &core); + if (ret != PQOS_RETVAL_OK) + return ret; + + for (i = 0; i < count; i++) { + int retval; + uint64_t val; + + ca[i].class_id = i; + ca[i].cdp = cdp_enabled; + + if (cdp_enabled) { + const uint32_t reg = PQOS_MSR_L2CA_MASK_START + i * 2; + + retval = msr_read(core, reg, &val); + if (retval != MACHINE_RETVAL_OK) + return PQOS_RETVAL_ERROR; + + ca[i].u.s.data_mask = val; + + retval = msr_read(core, reg + 1, &val); + if (retval != MACHINE_RETVAL_OK) + return PQOS_RETVAL_ERROR; + + ca[i].u.s.code_mask = val; + + } else { + const uint32_t reg = PQOS_MSR_L2CA_MASK_START + i; + + retval = msr_read(core, reg, &val); + if (retval != MACHINE_RETVAL_OK) + return PQOS_RETVAL_ERROR; + + ca[i].u.ways_mask = val; + } + } + *num_ca = count; + + return ret; +} + +int +hw_l2ca_get_min_cbm_bits(unsigned *min_cbm_bits) +{ + int ret; + unsigned *l2ids = NULL, l2id_num = 0, l2id; + unsigned class_id, l2ca_num, ways, i; + int technology = 1 << PQOS_CAP_TYPE_L2CA; + const struct pqos_capability *l2_cap = NULL; + struct pqos_l2ca l2ca_config[PQOS_MAX_L2CA_COS]; + + ASSERT(m_cap != NULL); + ASSERT(m_cpu != NULL); + ASSERT(min_cbm_bits != NULL); + + /** + * Get L2 CAT capabilities + */ + ret = pqos_cap_get_type(m_cap, PQOS_CAP_TYPE_L2CA, &l2_cap); + if (ret != PQOS_RETVAL_OK) + return PQOS_RETVAL_RESOURCE; /* L2 CAT not supported */ + + /** + * Get number & list of L2ids in the system + */ + l2ids = pqos_cpu_get_l2ids(m_cpu, &l2id_num); + if (l2ids == NULL || l2id_num == 0) { + ret = PQOS_RETVAL_ERROR; + goto hw_l2ca_get_min_cbm_bits_exit; + } + + /** + * Find free COS + */ + for (l2id = 0; l2id < l2id_num; l2id++) { + ret = get_unused_cos(l2id, technology, &class_id); + if (ret == PQOS_RETVAL_OK) + break; + if (ret != PQOS_RETVAL_RESOURCE) + goto hw_l2ca_get_min_cbm_bits_exit; + } + + if (ret == PQOS_RETVAL_RESOURCE) { + LOG_INFO("No free L2 COS available. " + "Unable to determine minimum L2 CBM bits\n"); + goto hw_l2ca_get_min_cbm_bits_exit; + } + + /** + * Get current configuration + */ + ret = hw_l2ca_get(l2id, PQOS_MAX_L2CA_COS, &l2ca_num, l2ca_config); + if (ret != PQOS_RETVAL_OK) + goto hw_l2ca_get_min_cbm_bits_exit; + + /** + * Probe for min cbm bits + */ + for (ways = 1; ways <= l2_cap->u.l2ca->num_ways; ways++) { + struct pqos_l2ca l2ca_tab[PQOS_MAX_L2CA_COS]; + unsigned num_ca; + uint64_t mask = (1 << ways) - 1; + + memset(l2ca_tab, 0, sizeof(struct pqos_l2ca)); + l2ca_tab[0].class_id = class_id; + l2ca_tab[0].u.ways_mask = mask; + + /** + * Try to set mask + */ + ret = hw_l2ca_set(l2id, 1, l2ca_tab); + if (ret != PQOS_RETVAL_OK) + continue; + + /** + * Validate if mask was correctly set + */ + ret = hw_l2ca_get(l2id, PQOS_MAX_L2CA_COS, &num_ca, l2ca_tab); + if (ret != PQOS_RETVAL_OK) + goto hw_l2ca_get_min_cbm_bits_restore; + + for (i = 0; i < num_ca; i++) { + struct pqos_l2ca *l2ca = &(l2ca_tab[i]); + + if (l2ca->class_id != class_id) + continue; + + if ((l2ca->cdp && + l2ca->u.s.data_mask == mask && + l2ca->u.s.code_mask == mask) || + (!l2ca->cdp && l2ca->u.ways_mask == mask)) { + *min_cbm_bits = ways; + ret = PQOS_RETVAL_OK; + goto hw_l2ca_get_min_cbm_bits_restore; + } + } + } + + /** + * Restore old settings + */ + hw_l2ca_get_min_cbm_bits_restore: + for (i = 0; i < l2ca_num; i++) { + int ret_val; + + if (l2ca_config[i].class_id != class_id) + continue; + + ret_val = hw_l2ca_set(l2id, 1, &(l2ca_config[i])); + if (ret_val != PQOS_RETVAL_OK) { + LOG_ERROR("Failed to restore CAT configuration. CAT" + " configuration has been altered!\n"); + ret = ret_val; + break; + } + } + + hw_l2ca_get_min_cbm_bits_exit: + if (l2ids != NULL) + free(l2ids); + + return ret; +} + +int +hw_mba_set(const unsigned socket, + const unsigned num_cos, + const struct pqos_mba *requested, + struct pqos_mba *actual) +{ + int ret = PQOS_RETVAL_OK; + unsigned i = 0, count = 0, core = 0, step = 0; + const struct pqos_capability *mba_cap = NULL; + + ASSERT(requested != NULL); + ASSERT(num_cos != 0); + + /** + * Check if MBA is supported + */ + ASSERT(m_cap != NULL); + ret = pqos_cap_get_type(m_cap, PQOS_CAP_TYPE_MBA, &mba_cap); + if (ret != PQOS_RETVAL_OK) + return PQOS_RETVAL_RESOURCE; /* MBA not supported */ + + count = mba_cap->u.mba->num_classes; + step = mba_cap->u.mba->throttle_step; + + /** + * Non-linear mode not currently supported + */ + if (!mba_cap->u.mba->is_linear) { + LOG_ERROR("MBA non-linear mode not currently supported!\n"); + return PQOS_RETVAL_RESOURCE; + } + /** + * Check if class id's are within allowed range. + */ + for (i = 0; i < num_cos; i++) + if (requested[i].class_id >= count) { + LOG_ERROR("MBA COS%u is out of range (COS%u is max)!\n", + requested[i].class_id, count - 1); + return PQOS_RETVAL_PARAM; + } + + ASSERT(m_cpu != NULL); + ret = pqos_cpu_get_one_core(m_cpu, socket, &core); + if (ret != PQOS_RETVAL_OK) + return ret; + + for (i = 0; i < num_cos; i++) { + const uint32_t reg = + requested[i].class_id + PQOS_MSR_MBA_MASK_START; + uint64_t val = PQOS_MBA_LINEAR_MAX - + (((requested[i].mb_rate + (step/2)) / step) * step); + int retval = MACHINE_RETVAL_OK; + + if (val > mba_cap->u.mba->throttle_max) + val = mba_cap->u.mba->throttle_max; + + retval = msr_write(core, reg, val); + if (retval != MACHINE_RETVAL_OK) + return PQOS_RETVAL_ERROR; + + /** + * If table to store actual values set is passed, + * read MSR values and store in table + */ + if (actual == NULL) + continue; + + retval = msr_read(core, reg, &val); + if (retval != MACHINE_RETVAL_OK) + return PQOS_RETVAL_ERROR; + + actual[i] = requested[i]; + actual[i].mb_rate = (PQOS_MBA_LINEAR_MAX - val); + } + + return ret; +} + +int +hw_mba_get(const unsigned socket, + const unsigned max_num_cos, + unsigned *num_cos, + struct pqos_mba *mba_tab) +{ + int ret = PQOS_RETVAL_OK; + unsigned i = 0, count = 0, core = 0; + + ASSERT(num_cos != NULL); + ASSERT(mba_tab != NULL); + ASSERT(max_num_cos != 0); + ASSERT(m_cap != NULL); + ret = pqos_mba_get_cos_num(m_cap, &count); + if (ret != PQOS_RETVAL_OK) + return ret; /**< no MBA capability */ + + if (count > max_num_cos) + return PQOS_RETVAL_ERROR; + + ASSERT(m_cpu != NULL); + ret = pqos_cpu_get_one_core(m_cpu, socket, &core); + if (ret != PQOS_RETVAL_OK) + return ret; + + for (i = 0; i < count; i++) { + const uint32_t reg = PQOS_MSR_MBA_MASK_START + i; + uint64_t val = 0; + int retval = msr_read(core, reg, &val); + + if (retval != MACHINE_RETVAL_OK) + return PQOS_RETVAL_ERROR; + + mba_tab[i].class_id = i; + mba_tab[i].mb_rate = (unsigned) PQOS_MBA_LINEAR_MAX - val; + } + *num_cos = count; + + return ret; +} + +/** + * @brief Sets COS associated to \a lcore + * + * @param [in] lcore lcore to set COS association + * @param [in] class_id COS to associate lcore to + * + * @return Operation status + */ +static int +cos_assoc_set(const unsigned lcore, const unsigned class_id) +{ + const uint32_t reg = PQOS_MSR_ASSOC; + uint64_t val = 0; + int ret; + + ret = msr_read(lcore, reg, &val); + if (ret != MACHINE_RETVAL_OK) + return PQOS_RETVAL_ERROR; + + val &= (~PQOS_MSR_ASSOC_QECOS_MASK); + val |= (((uint64_t) class_id) << PQOS_MSR_ASSOC_QECOS_SHIFT); + + ret = msr_write(lcore, reg, val); + if (ret != MACHINE_RETVAL_OK) + return PQOS_RETVAL_ERROR; + + return PQOS_RETVAL_OK; +} + +int +hw_alloc_assoc_set(const unsigned lcore, + const unsigned class_id) +{ + int ret = PQOS_RETVAL_OK; + unsigned num_l2_cos = 0, num_l3_cos = 0; + + ASSERT(m_cpu != NULL); + ret = pqos_cpu_check_core(m_cpu, lcore); + if (ret != PQOS_RETVAL_OK) + return PQOS_RETVAL_PARAM; + + ASSERT(m_cap != NULL); + ret = pqos_l3ca_get_cos_num(m_cap, &num_l3_cos); + if (ret != PQOS_RETVAL_OK && ret != PQOS_RETVAL_RESOURCE) + return ret; + + ret = pqos_l2ca_get_cos_num(m_cap, &num_l2_cos); + if (ret != PQOS_RETVAL_OK && ret != PQOS_RETVAL_RESOURCE) + return ret; + + if (class_id >= num_l3_cos && class_id >= num_l2_cos) + /* class_id is out of bounds */ + return PQOS_RETVAL_PARAM; + + ret = cos_assoc_set(lcore, class_id); + + return ret; +} + +int +hw_alloc_assoc_get(const unsigned lcore, + unsigned *class_id) +{ + const struct pqos_capability *l3_cap = NULL; + const struct pqos_capability *l2_cap = NULL; + int ret = PQOS_RETVAL_OK; + + ASSERT(class_id != NULL); + ASSERT(m_cpu != NULL); + ret = pqos_cpu_check_core(m_cpu, lcore); + if (ret != PQOS_RETVAL_OK) + return PQOS_RETVAL_PARAM; + + ASSERT(m_cap != NULL); + ret = pqos_cap_get_type(m_cap, PQOS_CAP_TYPE_L3CA, &l3_cap); + if (ret != PQOS_RETVAL_OK && ret != PQOS_RETVAL_RESOURCE) + return ret; + + ret = pqos_cap_get_type(m_cap, PQOS_CAP_TYPE_L2CA, &l2_cap); + if (ret != PQOS_RETVAL_OK && ret != PQOS_RETVAL_RESOURCE) + return ret; + + if (l2_cap == NULL && l3_cap == NULL) + /* no L2/L3 CAT detected */ + return PQOS_RETVAL_RESOURCE; + + ret = cos_assoc_get(lcore, class_id); + + return ret; +} + + +int +hw_alloc_assign(const unsigned technology, + const unsigned *core_array, + const unsigned core_num, + unsigned *class_id) +{ + const int l2_req = ((technology & (1 << PQOS_CAP_TYPE_L2CA)) != 0); + unsigned i; + unsigned socket = 0, l2id = 0; + int ret; + + ASSERT(core_num > 0); + ASSERT(core_array != NULL); + ASSERT(class_id != NULL); + ASSERT(technology != 0); + + /* Check if core belongs to one resource entity */ + for (i = 0; i < core_num; i++) { + const struct pqos_coreinfo *pi = NULL; + + pi = pqos_cpu_get_core_info(m_cpu, core_array[i]); + if (pi == NULL) { + ret = PQOS_RETVAL_PARAM; + goto pqos_alloc_assign_exit; + } + + if (l2_req) { + /* L2 is requested + * The smallest manageable entity is L2 cluster + */ + if (i != 0 && l2id != pi->l2_id) { + ret = PQOS_RETVAL_PARAM; + goto pqos_alloc_assign_exit; + } + l2id = pi->l2_id; + } else { + if (i != 0 && socket != pi->socket) { + ret = PQOS_RETVAL_PARAM; + goto pqos_alloc_assign_exit; + } + socket = pi->socket; + } + } + + /* find an unused class from highest down */ + if (!l2_req) + ret = get_unused_cos(socket, technology, class_id); + else + ret = get_unused_cos(l2id, technology, class_id); + + if (ret != PQOS_RETVAL_OK) + goto pqos_alloc_assign_exit; + + /* assign cores to the unused class */ + for (i = 0; i < core_num; i++) { + ret = cos_assoc_set(core_array[i], *class_id); + if (ret != PQOS_RETVAL_OK) + goto pqos_alloc_assign_exit; + } + + pqos_alloc_assign_exit: + return ret; +} + +int +hw_alloc_release(const unsigned *core_array, + const unsigned core_num) +{ + unsigned i; + int ret = PQOS_RETVAL_OK; + + ASSERT(core_num > 0 && core_array != NULL); + + for (i = 0; i < core_num; i++) + if (cos_assoc_set(core_array[i], 0) != PQOS_RETVAL_OK) + ret = PQOS_RETVAL_ERROR; + + return ret; +} + +/** + * @brief Enables or disables CDP across selected CPU sockets + * + * @param [in] sockets_num dimension of \a sockets array + * @param [in] sockets array with socket ids to change CDP config on + * @param [in] enable CDP enable/disable flag, 1 - enable, 0 - disable + * + * @return Operations status + * @retval PQOS_RETVAL_OK on success + * @retval PQOS_RETVAL_ERROR on failure, MSR read/write error + */ +static int +l3cdp_enable(const unsigned sockets_num, + const unsigned *sockets, + const int enable) +{ + unsigned j = 0; + + ASSERT(sockets_num > 0 && sockets != NULL); + + LOG_INFO("%s L3 CDP across sockets...\n", + (enable) ? "Enabling" : "Disabling"); + + for (j = 0; j < sockets_num; j++) { + uint64_t reg = 0; + unsigned core = 0; + int ret = PQOS_RETVAL_OK; + + ret = pqos_cpu_get_one_core(m_cpu, sockets[j], &core); + if (ret != PQOS_RETVAL_OK) + return ret; + + ret = msr_read(core, PQOS_MSR_L3_QOS_CFG, ®); + if (ret != MACHINE_RETVAL_OK) + return PQOS_RETVAL_ERROR; + + if (enable) + reg |= PQOS_MSR_L3_QOS_CFG_CDP_EN; + else + reg &= ~PQOS_MSR_L3_QOS_CFG_CDP_EN; + + ret = msr_write(core, PQOS_MSR_L3_QOS_CFG, reg); + if (ret != MACHINE_RETVAL_OK) + return PQOS_RETVAL_ERROR; + } + + return PQOS_RETVAL_OK; +} + +/** + * @brief Enables or disables CDP across selected CPU clusters + * + * @param [in] l2id_num dimension of \a l2ids array + * @param [in] l2ids array with clusters ids to change CDP config on + * @param [in] enable CDP enable/disable flag, 1 - enable, 0 - disable + * + * @return Operations status + * @retval PQOS_RETVAL_OK on success + * @retval PQOS_RETVAL_ERROR on failure, MSR read/write error + */ +static int +l2cdp_enable(const unsigned l2id_num, + const unsigned *l2ids, + const int enable) +{ + unsigned i = 0; + int ret; + + ASSERT(l2id_num > 0 && l2ids != NULL); + + LOG_INFO("%s L2 CDP across clusters...\n", + (enable) ? "Enabling" : "Disabling"); + + for (i = 0; i < l2id_num; i++) { + uint64_t reg = 0; + unsigned core = 0; + + ret = pqos_cpu_get_one_by_l2id(m_cpu, l2ids[i], &core); + if (ret != PQOS_RETVAL_OK) + return ret; + + ret = msr_read(core, PQOS_MSR_L2_QOS_CFG, ®); + if (ret != PQOS_RETVAL_OK) + return PQOS_RETVAL_ERROR; + + if (enable) + reg |= PQOS_MSR_L2_QOS_CFG_CDP_EN; + else + reg &= ~PQOS_MSR_L2_QOS_CFG_CDP_EN; + + ret = msr_write(core, PQOS_MSR_L2_QOS_CFG, reg); + if (ret != MACHINE_RETVAL_OK) + return PQOS_RETVAL_ERROR; + } + + return PQOS_RETVAL_OK; +} + +/** + * @brief Writes range of MBA/CAT COS MSR's with \a msr_val value + * + * Used as part of CAT/MBA reset process. + * + * @param [in] msr_start First MSR to be written + * @param [in] msr_num Number of MSR's to be written + * @param [in] coreid Core ID to be used for MSR write operations + * @param [in] msr_val Value to be written to MSR's + * + * @return Operation status + * @retval PQOS_RETVAL_OK on success + * @retval PQOS_RETVAL_ERROR on MSR write error + */ +static int +alloc_cos_reset(const unsigned msr_start, + const unsigned msr_num, + const unsigned coreid, + const uint64_t msr_val) +{ + int ret = PQOS_RETVAL_OK; + unsigned i; + + for (i = 0; i < msr_num; i++) { + int retval = msr_write(coreid, msr_start + i, msr_val); + + if (retval != MACHINE_RETVAL_OK) + ret = PQOS_RETVAL_ERROR; + } + + return ret; +} + +/** + * @brief Associates each of the cores with COS0 + * + * Operates on m_cpu structure. + * + * @return Operation status + * @retval PQOS_RETVAL_OK on success + * @retval PQOS_RETVAL_ERROR on MSR write error + */ +static int +alloc_assoc_reset(void) +{ + int ret = PQOS_RETVAL_OK; + unsigned i; + + for (i = 0; i < m_cpu->num_cores; i++) + if (cos_assoc_set(m_cpu->cores[i].lcore, 0) != PQOS_RETVAL_OK) + ret = PQOS_RETVAL_ERROR; + + return ret; +} + +int +hw_alloc_reset(const enum pqos_cdp_config l3_cdp_cfg, + const enum pqos_cdp_config l2_cdp_cfg) +{ + unsigned *sockets = NULL; + unsigned sockets_num = 0; + unsigned *l2ids = NULL; + unsigned l2id_num = 0; + const struct pqos_capability *alloc_cap = NULL; + const struct pqos_cap_l3ca *l3_cap = NULL; + const struct pqos_cap_l2ca *l2_cap = NULL; + const struct pqos_cap_mba *mba_cap = NULL; + int ret = PQOS_RETVAL_OK; + unsigned max_l3_cos = 0; + unsigned max_l2_cos = 0; + unsigned j; + int cdp_supported; + + ASSERT(l3_cdp_cfg == PQOS_REQUIRE_CDP_ON || + l3_cdp_cfg == PQOS_REQUIRE_CDP_OFF || + l3_cdp_cfg == PQOS_REQUIRE_CDP_ANY); + + ASSERT(l2_cdp_cfg == PQOS_REQUIRE_CDP_ON || + l2_cdp_cfg == PQOS_REQUIRE_CDP_OFF || + l2_cdp_cfg == PQOS_REQUIRE_CDP_ANY); + + /* Get L3 CAT capabilities */ + (void) pqos_cap_get_type(m_cap, PQOS_CAP_TYPE_L3CA, &alloc_cap); + if (alloc_cap != NULL) + l3_cap = alloc_cap->u.l3ca; + + /* Get L2 CAT capabilities */ + alloc_cap = NULL; + (void) pqos_cap_get_type(m_cap, PQOS_CAP_TYPE_L2CA, &alloc_cap); + if (alloc_cap != NULL) + l2_cap = alloc_cap->u.l2ca; + + /* Get MBA capabilities */ + alloc_cap = NULL; + (void) pqos_cap_get_type(m_cap, PQOS_CAP_TYPE_MBA, &alloc_cap); + if (alloc_cap != NULL) + mba_cap = alloc_cap->u.mba; + + /* Check if either L2 CAT, L3 CAT or MBA is supported */ + if (l2_cap == NULL && l3_cap == NULL && mba_cap == NULL) { + LOG_ERROR("L2 CAT/L3 CAT/MBA not present!\n"); + ret = PQOS_RETVAL_RESOURCE; /* no L2/L3 CAT present */ + goto pqos_alloc_reset_exit; + } + /* Check L3 CDP requested while not present */ + if (l3_cap == NULL && l3_cdp_cfg != PQOS_REQUIRE_CDP_ANY) { + LOG_ERROR("L3 CDP setting requested but no L3 CAT present!\n"); + ret = PQOS_RETVAL_RESOURCE; + goto pqos_alloc_reset_exit; + } + /* Check L2 CDP requested while not present */ + if (l2_cap == NULL && l2_cdp_cfg != PQOS_REQUIRE_CDP_ANY) { + LOG_ERROR("L2 CDP setting requested but no L2 CAT present!\n"); + ret = PQOS_RETVAL_RESOURCE; + goto pqos_alloc_reset_exit; + } + if (l3_cap != NULL) { + ret = pqos_l3ca_cdp_enabled(m_cap, &cdp_supported, NULL); + if (ret != PQOS_RETVAL_OK) + goto pqos_alloc_reset_exit; + + /* Check against erroneous L3 CDP request */ + if (l3_cdp_cfg == PQOS_REQUIRE_CDP_ON && !cdp_supported) { + LOG_ERROR("L3 CAT/CDP requested but not supported by " + "the platform!\n"); + ret = PQOS_RETVAL_PARAM; + goto pqos_alloc_reset_exit; + } + + /* Get maximum number of L3 CAT classes */ + max_l3_cos = l3_cap->num_classes; + if (l3_cap->cdp && l3_cap->cdp_on) + max_l3_cos = max_l3_cos * 2; + } + if (l2_cap != NULL) { + ret = pqos_l2ca_cdp_enabled(m_cap, &cdp_supported, NULL); + if (ret != PQOS_RETVAL_OK) + goto pqos_alloc_reset_exit; + + /* Check against erroneous L2 CDP request */ + if (l2_cdp_cfg == PQOS_REQUIRE_CDP_ON && !cdp_supported) { + LOG_ERROR("L2 CAT/CDP requested but not supported by " + "the platform!\n"); + ret = PQOS_RETVAL_PARAM; + goto pqos_alloc_reset_exit; + } + + /* Get maximum number of L2 CAT classes */ + max_l2_cos = l2_cap->num_classes; + if (l2_cap->cdp && l2_cap->cdp_on) + max_l2_cos = max_l2_cos * 2; + } + + /** + * Get number & list of sockets in the system + */ + sockets = pqos_cpu_get_sockets(m_cpu, &sockets_num); + if (sockets == NULL || sockets_num == 0) + goto pqos_alloc_reset_exit; + + if (l3_cap != NULL) { + /** + * Change L3 COS definition on all sockets + * so that each COS allows for access to all cache ways + */ + for (j = 0; j < sockets_num; j++) { + unsigned core = 0; + + ret = pqos_cpu_get_one_core(m_cpu, sockets[j], &core); + if (ret != PQOS_RETVAL_OK) + goto pqos_alloc_reset_exit; + + const uint64_t ways_mask = + (1ULL << l3_cap->num_ways) - 1ULL; + + ret = alloc_cos_reset(PQOS_MSR_L3CA_MASK_START, + max_l3_cos, core, ways_mask); + if (ret != PQOS_RETVAL_OK) + goto pqos_alloc_reset_exit; + } + } + + if (l2_cap != NULL) { + /** + * Get number & list of L2ids in the system + * Then go through all L2 ids and reset L2 classes on them + */ + l2ids = pqos_cpu_get_l2ids(m_cpu, &l2id_num); + if (l2ids == NULL || l2id_num == 0) + goto pqos_alloc_reset_exit; + + for (j = 0; j < l2id_num; j++) { + const uint64_t ways_mask = + (1ULL << l2_cap->num_ways) - 1ULL; + unsigned core = 0; + + ret = pqos_cpu_get_one_by_l2id(m_cpu, l2ids[j], &core); + if (ret != PQOS_RETVAL_OK) + goto pqos_alloc_reset_exit; + + ret = alloc_cos_reset(PQOS_MSR_L2CA_MASK_START, + max_l2_cos, core, ways_mask); + if (ret != PQOS_RETVAL_OK) + goto pqos_alloc_reset_exit; + } + } + + if (mba_cap != NULL) { + /** + * Go through all sockets and reset MBA class defintions + * 0 is the default MBA COS value in linear mode. + */ + for (j = 0; j < sockets_num; j++) { + unsigned core = 0; + + ret = pqos_cpu_get_one_core(m_cpu, sockets[j], &core); + if (ret != PQOS_RETVAL_OK) + goto pqos_alloc_reset_exit; + + ret = alloc_cos_reset(PQOS_MSR_MBA_MASK_START, + mba_cap->num_classes, core, 0); + if (ret != PQOS_RETVAL_OK) + goto pqos_alloc_reset_exit; + } + } + + /** + * Associate all cores with COS0 + */ + ret = alloc_assoc_reset(); + if (ret != PQOS_RETVAL_OK) + goto pqos_alloc_reset_exit; + + /** + * Turn L3 CDP ON or OFF upon the request + */ + if (l3_cap != NULL) { + if (l3_cdp_cfg == PQOS_REQUIRE_CDP_ON && !l3_cap->cdp_on) { + /** + * Turn on L3 CDP + */ + LOG_INFO("Turning L3 CDP ON ...\n"); + ret = l3cdp_enable(sockets_num, sockets, 1); + if (ret != PQOS_RETVAL_OK) { + LOG_ERROR("L3 CDP enable error!\n"); + goto pqos_alloc_reset_exit; + } + } + + if (l3_cdp_cfg == PQOS_REQUIRE_CDP_OFF && l3_cap->cdp_on) { + /** + * Turn off L3 CDP + */ + LOG_INFO("Turning L3 CDP OFF ...\n"); + ret = l3cdp_enable(sockets_num, sockets, 0); + if (ret != PQOS_RETVAL_OK) { + LOG_ERROR("L3 CDP disable error!\n"); + goto pqos_alloc_reset_exit; + } + } + _pqos_cap_l3cdp_change(l3_cdp_cfg); + } + + /** + * Turn L2 CDP ON or OFF upon the request + */ + if (l2_cap != NULL) { + if (l2_cdp_cfg == PQOS_REQUIRE_CDP_ON && !l2_cap->cdp_on) { + /** + * Turn on L2 CDP + */ + LOG_INFO("Turning L2 CDP ON ...\n"); + ret = l2cdp_enable(l2id_num, l2ids, 1); + if (ret != PQOS_RETVAL_OK) { + LOG_ERROR("L2 CDP enable error!\n"); + goto pqos_alloc_reset_exit; + } + } + + if (l2_cdp_cfg == PQOS_REQUIRE_CDP_OFF && l2_cap->cdp_on) { + /** + * Turn off L2 CDP + */ + LOG_INFO("Turning L2 CDP OFF ...\n"); + ret = l2cdp_enable(l2id_num, l2ids, 0); + if (ret != PQOS_RETVAL_OK) { + LOG_ERROR("L2 CDP disable error!\n"); + goto pqos_alloc_reset_exit; + } + } + _pqos_cap_l2cdp_change(l2_cdp_cfg); + } + + pqos_alloc_reset_exit: + if (sockets != NULL) + free(sockets); + if (l2ids != NULL) + free(l2ids); + return ret; +} diff --git a/lib/allocation.h b/lib/allocation.h new file mode 100644 index 0000000..15bb4ef --- /dev/null +++ b/lib/allocation.h @@ -0,0 +1,280 @@ +/* + * BSD LICENSE + * + * Copyright(c) 2014-2018 Intel Corporation. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +/** + * @brief Internal header file to PQoS allocation initialization + */ + +#ifndef __PQOS_ALLOC_H__ +#define __PQOS_ALLOC_H__ + +#ifdef __cplusplus +extern "C" { +#endif + +/** + * @brief Initializes allocation sub-module of PQoS library + * + * @param cpu cpu topology structure + * @param cap capabilities structure + * @param cfg library configuration structure + * + * @return Operation status + * @retval PQOS_RETVAL_OK success + */ +int pqos_alloc_init(const struct pqos_cpuinfo *cpu, + const struct pqos_cap *cap, + const struct pqos_config *cfg); + +/** + * @brief Shuts down allocation sub-module of PQoS library + * + * @return Operation status + * @retval PQOS_RETVAL_OK success + */ +int pqos_alloc_fini(void); + +/** + * @brief Hardware interface to associate \a lcore + * with given class of service + * + * @param [in] lcore CPU logical core id + * @param [in] class_id class of service + * + * @return Operations status + */ +int hw_alloc_assoc_set(const unsigned lcore, + const unsigned class_id); + +/** + * @brief Hardware interface to read association + * of \a lcore with class of service + * + * @param [in] lcore CPU logical core id + * @param [out] class_id class of service + * + * @return Operations status + * @retval PQOS_RETVAL_OK on success + */ +int hw_alloc_assoc_get(const unsigned lcore, + unsigned *class_id); + +/** + * @brief Hardware interface to assign first available + * COS to cores in \a core_array + * + * While searching for available COS take technologies it is intended to use + * with into account. + * Note on \a technology and \a core_array selection: + * - if L2 CAT technology is requested then cores need to belong to + * one L2 cluster (same L2ID) + * - if only L3 CAT is requested then cores need to belong to one socket + * - if only MBA is selected then cores need to belong to one socket + * + * @param [in] technology bit mask selecting technologies + * (1 << enum pqos_cap_type) + * @param [in] core_array list of core ids + * @param [in] core_num number of core ids in the \a core_array + * @param [out] class_id place to store reserved COS id + * + * @return Operations status + */ +int hw_alloc_assign(const unsigned technology, + const unsigned *core_array, + const unsigned core_num, + unsigned *class_id); + +/** + * @brief Hardware interface to reassign cores + * in \a core_array to default COS#0 + * + * @param [in] core_array list of core ids + * @param [in] core_num number of core ids in the \a core_array + * + * @return Operations status + */ +int hw_alloc_release(const unsigned *core_array, + const unsigned core_num); + +/** + * @brief Hardware interface to reset configuration + * of allocation technologies + * + * Reverts allocation state to the one after reset: + * - all cores associated with COS0 + * - all COS are set to give access to entire resource + * + * As part of allocation reset CDP reconfiguration can be performed. + * This can be requested via \a l3_cdp_cfg or \a l2_cdp_cfg. + * + * @param [in] l3_cdp_cfg requested L3 CAT CDP config + * @param [in] l2_cdp_cfg requested L2 CAT CDP config + * + * @return Operation status + * @retval PQOS_RETVAL_OK on success + */ +int hw_alloc_reset(const enum pqos_cdp_config l3_cdp_cfg, + const enum pqos_cdp_config l2_cdp_cfg); + +/** + * @brief Hardware interface to set classes of service + * defined by \a ca on \a socket + * + * @param [in] socket CPU socket id + * @param [in] num_ca number of classes of service at \a ca + * @param [in] ca table with class of service definitions + * + * @return Operations status + * @retval PQOS_RETVAL_OK on success + */ +int hw_l3ca_set(const unsigned socket, + const unsigned num_ca, + const struct pqos_l3ca *ca); + +/** + * @brief Hardware interface to read classes of service from \a socket + * + * @param [in] socket CPU socket id + * @param [in] max_num_ca maximum number of classes of service + * that can be accommodated at \a ca + * @param [out] num_ca number of classes of service read into \a ca + * @param [out] ca table with read classes of service + * + * @return Operations status + * @retval PQOS_RETVAL_OK on success + */ +int hw_l3ca_get(const unsigned socket, + const unsigned max_num_ca, + unsigned *num_ca, + struct pqos_l3ca *ca); + +/** + * @brief Probe hardware for minimum number of bits that must be set + * + * @note Uses free COS to determine lowest number of bits accepted + * @note If no free COS is available PQOS_RETVAL_RESOURCE will be returned + * + * @param [out] min_cbm_bits minimum number of bits that must be set + * + * @return Operational status + * @retval PQOS_RETVAL_OK on success + * @retval PQOS_RETVAL_RESOURCE when no free COS found + */ +int hw_l3ca_get_min_cbm_bits(unsigned *min_cbm_bits); + +/** + * @brief Hardware interface to set classes of + * service defined by \a ca on \a l2id + * + * @param [in] l2id unique L2 cache identifier + * @param [in] num_cos number of classes of service at \a ca + * @param [in] ca table with class of service definitions + * + * @return Operations status + * @retval PQOS_RETVAL_OK on success + */ +int hw_l2ca_set(const unsigned l2id, + const unsigned num_cos, + const struct pqos_l2ca *ca); + +/** + * @brief Hardware interface to read classes of service from \a l2id + * + * @param [in] l2id unique L2 cache identifier + * @param [in] max_num_ca maximum number of classes of service + * that can be accommodated at \a ca + * @param [out] num_ca number of classes of service read into \a ca + * @param [out] ca table with read classes of service + * + * @return Operations status + * @retval PQOS_RETVAL_OK on success + */ +int hw_l2ca_get(const unsigned l2id, + const unsigned max_num_ca, + unsigned *num_ca, + struct pqos_l2ca *ca); + +/** + * @brief Probe hardware for minimum number of bits that must be set + * + * @note Uses free COS to determine lowest number of bits accepted + * @note If no free COS is available PQOS_RETVAL_RESOURCE will be returned + * + * @param [out] min_cbm_bits minimum number of bits that must be set + * + * @return Operational status + * @retval PQOS_RETVAL_OK on success + * @retval PQOS_RETVAL_RESOURCE when no free COS found + */ +int hw_l2ca_get_min_cbm_bits(unsigned *min_cbm_bits); + +/** + * @brief Hardware interface to set classes of service + * defined by \a mba on \a socket + * + * @param [in] socket CPU socket id + * @param [in] num_cos number of classes of service at \a ca + * @param [in] requested table with class of service definitions + * @param [out] actual table with class of service definitions + * + * @return Operations status + * @retval PQOS_RETVAL_OK on success + */ +int hw_mba_set(const unsigned socket, + const unsigned num_cos, + const struct pqos_mba *requested, + struct pqos_mba *actual); + +/** + * @brief Hardware interface to read MBA from \a socket + * + * @param [in] socket CPU socket id + * @param [in] max_num_cos maximum number of classes of service + * that can be accommodated at \a mba_tab + * @param [out] num_cos number of classes of service read into \a mba_tab + * @param [out] mba_tab table with read classes of service + * + * @return Operations status + * @retval PQOS_RETVAL_OK on success + */ +int hw_mba_get(const unsigned socket, + const unsigned max_num_cos, + unsigned *num_cos, + struct pqos_mba *mba_tab); + +#ifdef __cplusplus +} +#endif + +#endif /* __PQOS_ALLOC_H__ */ diff --git a/lib/api.c b/lib/api.c new file mode 100644 index 0000000..c25d6bb --- /dev/null +++ b/lib/api.c @@ -0,0 +1,1151 @@ +/* + * BSD LICENSE + * + * Copyright(c) 2017-2018 Intel Corporation. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#include + +#include "pqos.h" +#include "api.h" +#include "allocation.h" +#include "os_allocation.h" +#include "os_monitoring.h" +#include "monitoring.h" +#include "os_monitoring.h" +#include "cap.h" +#include "log.h" +#include "types.h" + +/** + * Value marking monitoring group structure as "valid". + * Group becomes "valid" after successful pqos_mon_start() or + * pqos_mon_start_pid() call. + */ +#define GROUP_VALID_MARKER (0x00DEAD00) + +/** + * Flag used to determine what interface to use: + * - MSR is 0 + * - OS is 1 + */ +static int m_interface = PQOS_INTER_MSR; + +/* + * ======================================= + * Init module + * ======================================= + */ +int +api_init(int interface) +{ + if (interface != PQOS_INTER_MSR && interface != PQOS_INTER_OS && + interface != PQOS_INTER_OS_RESCTRL_MON) + return PQOS_RETVAL_PARAM; + + m_interface = interface; + + return PQOS_RETVAL_OK; +} + +/* + * ======================================= + * Allocation Technology + * ======================================= + */ +int +pqos_alloc_assoc_set(const unsigned lcore, + const unsigned class_id) +{ + int ret; + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + + if (m_interface == PQOS_INTER_MSR) + ret = hw_alloc_assoc_set(lcore, class_id); + else { +#ifdef __linux__ + ret = os_alloc_assoc_set(lcore, class_id); +#else + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + } + _pqos_api_unlock(); + + return ret; +} + +int +pqos_alloc_assoc_get(const unsigned lcore, + unsigned *class_id) +{ + int ret; + + if (class_id == NULL) + return PQOS_RETVAL_PARAM; + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + + if (m_interface == PQOS_INTER_MSR) + ret = hw_alloc_assoc_get(lcore, class_id); + else { +#ifdef __linux__ + ret = os_alloc_assoc_get(lcore, class_id); +#else + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + } + _pqos_api_unlock(); + + return ret; +} + +int +pqos_alloc_assoc_set_pid(const pid_t task, + const unsigned class_id) +{ + int ret; + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + + if (m_interface != PQOS_INTER_OS && + m_interface != PQOS_INTER_OS_RESCTRL_MON) { + LOG_ERROR("Incompatible interface " + "selected for task association!\n"); + _pqos_api_unlock(); + return PQOS_RETVAL_ERROR; + } + +#ifdef __linux__ + ret = os_alloc_assoc_set_pid(task, class_id); +#else + UNUSED_PARAM(task); + UNUSED_PARAM(class_id); + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + _pqos_api_unlock(); + + return ret; + +} + +int +pqos_alloc_assoc_get_pid(const pid_t task, + unsigned *class_id) +{ + int ret; + + if (class_id == NULL) + return PQOS_RETVAL_PARAM; + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + + if (m_interface != PQOS_INTER_OS && + m_interface != PQOS_INTER_OS_RESCTRL_MON) { + LOG_ERROR("Incompatible interface " + "selected for task association!\n"); + _pqos_api_unlock(); + return PQOS_RETVAL_ERROR; + } + +#ifdef __linux__ + ret = os_alloc_assoc_get_pid(task, class_id); +#else + UNUSED_PARAM(task); + UNUSED_PARAM(class_id); + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + _pqos_api_unlock(); + + return ret; +} + +int +pqos_alloc_assign(const unsigned technology, + const unsigned *core_array, + const unsigned core_num, + unsigned *class_id) +{ + int ret; + const int l2_req = ((technology & (1 << PQOS_CAP_TYPE_L2CA)) != 0); + const int l3_req = ((technology & (1 << PQOS_CAP_TYPE_L3CA)) != 0); + const int mba_req = ((technology & (1 << PQOS_CAP_TYPE_MBA)) != 0); + + if (core_num == 0 || core_array == NULL || class_id == NULL || + !(l2_req || l3_req || mba_req)) + return PQOS_RETVAL_PARAM; + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + if (m_interface == PQOS_INTER_MSR) + ret = hw_alloc_assign(technology, core_array, + core_num, class_id); + else { +#ifdef __linux__ + ret = os_alloc_assign(technology, core_array, core_num, + class_id); +#else + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + } + _pqos_api_unlock(); + + return ret; +} + +int +pqos_alloc_release(const unsigned *core_array, + const unsigned core_num) +{ + int ret; + + if (core_num == 0 || core_array == NULL) + return PQOS_RETVAL_PARAM; + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + + if (m_interface == PQOS_INTER_MSR) + ret = hw_alloc_release(core_array, core_num); + else { +#ifdef __linux__ + ret = os_alloc_release(core_array, core_num); +#else + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + } + _pqos_api_unlock(); + + return ret; +} + +int +pqos_alloc_assign_pid(const unsigned technology, + const pid_t *task_array, + const unsigned task_num, + unsigned *class_id) +{ + int ret; + + if (task_array == NULL || task_num == 0 || class_id == NULL) + return PQOS_RETVAL_PARAM; + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + + if (m_interface != PQOS_INTER_OS && + m_interface != PQOS_INTER_OS_RESCTRL_MON) { + LOG_ERROR("Incompatible interface " + "selected for task association!\n"); + _pqos_api_unlock(); + return PQOS_RETVAL_ERROR; + } + +#ifdef __linux__ + ret = os_alloc_assign_pid(technology, task_array, task_num, class_id); +#else + UNUSED_PARAM(technology); + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + _pqos_api_unlock(); + + return ret; +} + +int +pqos_alloc_release_pid(const pid_t *task_array, + const unsigned task_num) +{ + int ret; + + if (task_array == NULL || task_num == 0) + return PQOS_RETVAL_PARAM; + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + + if (m_interface != PQOS_INTER_OS && + m_interface != PQOS_INTER_OS_RESCTRL_MON) { + LOG_ERROR("Incompatible interface " + "selected for task association!\n"); + _pqos_api_unlock(); + return PQOS_RETVAL_ERROR; + } + +#ifdef __linux__ + ret = os_alloc_release_pid(task_array, task_num); +#else + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + _pqos_api_unlock(); + + return ret; +} + +int +pqos_alloc_reset(const enum pqos_cdp_config l3_cdp_cfg, + const enum pqos_cdp_config l2_cdp_cfg) +{ + int ret; + + if (l3_cdp_cfg != PQOS_REQUIRE_CDP_ON && + l3_cdp_cfg != PQOS_REQUIRE_CDP_OFF && + l3_cdp_cfg != PQOS_REQUIRE_CDP_ANY) { + LOG_ERROR("Unrecognized L3 CDP configuration setting %d!\n", + l3_cdp_cfg); + return PQOS_RETVAL_PARAM; + } + + if (l2_cdp_cfg != PQOS_REQUIRE_CDP_ON && + l2_cdp_cfg != PQOS_REQUIRE_CDP_OFF && + l2_cdp_cfg != PQOS_REQUIRE_CDP_ANY) { + LOG_ERROR("Unrecognized L2 CDP configuration setting %d!\n", + l2_cdp_cfg); + return PQOS_RETVAL_PARAM; + } + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + + if (m_interface == PQOS_INTER_MSR) + ret = hw_alloc_reset(l3_cdp_cfg, l2_cdp_cfg); + else { +#ifdef __linux__ + ret = os_alloc_reset(l3_cdp_cfg, l2_cdp_cfg); +#else + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + } + _pqos_api_unlock(); + + return ret; +} + +unsigned * +pqos_pid_get_pid_assoc(const unsigned class_id, unsigned *count) +{ + unsigned *tasks = NULL; + int ret; + + if (count == NULL) + return NULL; + + if (m_interface != PQOS_INTER_OS && + m_interface != PQOS_INTER_OS_RESCTRL_MON) { + LOG_ERROR("Incompatible interface " + "selected for task association!\n"); + return NULL; + } + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return NULL; + } + +#ifdef __linux__ + tasks = os_pid_get_pid_assoc(class_id, count); + if (tasks == NULL) + LOG_ERROR("Error retrieving task information!\n"); +#else + UNUSED_PARAM(class_id); + LOG_INFO("OS interface not supported!\n"); +#endif + + _pqos_api_unlock(); + + return tasks; +} + +/* + * ======================================= + * L3 cache allocation + * ======================================= + */ + +/** + * @brief Tests if \a bitmask is contiguous + * + * Zero bit mask is regarded as not contiguous. + * + * The function shifts out first group of contiguous 1's in the bit mask. + * Next it checks remaining bitmask content to make a decision. + * + * @param bitmask bit mask to be validated for contiguity + * + * @return Bit mask contiguity check result + * @retval 0 not contiguous + * @retval 1 contiguous + */ +static int +is_contiguous(uint64_t bitmask) +{ + if (bitmask == 0) + return 0; + + while ((bitmask & 1) == 0) /**< Shift until 1 found at position 0 */ + bitmask >>= 1; + + while ((bitmask & 1) != 0) /**< Shift until 0 found at position 0 */ + bitmask >>= 1; + + return (bitmask) ? 0 : 1; /**< non-zero bitmask is not contiguous */ +} + +int +pqos_l3ca_set(const unsigned socket, + const unsigned num_cos, + const struct pqos_l3ca *ca) +{ + int ret; + unsigned i; + + if (ca == NULL || num_cos == 0) + return PQOS_RETVAL_PARAM; + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + + /** + * Check if class bitmasks are contiguous. + */ + for (i = 0; i < num_cos; i++) { + int is_contig = 0; + + if (ca[i].cdp) { + is_contig = is_contiguous(ca[i].u.s.data_mask) && + is_contiguous(ca[i].u.s.code_mask); + } else + is_contig = is_contiguous(ca[i].u.ways_mask); + + if (!is_contig) { + LOG_ERROR("L3 COS%u bit mask is not contiguous!\n", + ca[i].class_id); + _pqos_api_unlock(); + return PQOS_RETVAL_PARAM; + } + } + + if (m_interface == PQOS_INTER_MSR) + ret = hw_l3ca_set(socket, num_cos, ca); + else { +#ifdef __linux__ + ret = os_l3ca_set(socket, num_cos, ca); +#else + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + } + _pqos_api_unlock(); + + return ret; +} + +int +pqos_l3ca_get(const unsigned socket, + const unsigned max_num_ca, + unsigned *num_ca, + struct pqos_l3ca *ca) +{ + int ret; + + if (num_ca == NULL || ca == NULL || max_num_ca == 0) + return PQOS_RETVAL_PARAM; + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + if (m_interface == PQOS_INTER_MSR) + ret = hw_l3ca_get(socket, max_num_ca, num_ca, ca); + else { +#ifdef __linux__ + ret = os_l3ca_get(socket, max_num_ca, num_ca, ca); +#else + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + } + _pqos_api_unlock(); + + return ret; +} + +int +pqos_l3ca_get_min_cbm_bits(unsigned *min_cbm_bits) +{ + int ret; + + if (min_cbm_bits == NULL) + return PQOS_RETVAL_PARAM; + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + + if (m_interface == PQOS_INTER_MSR) + ret = hw_l3ca_get_min_cbm_bits(min_cbm_bits); + else { +#ifdef __linux__ + ret = os_l3ca_get_min_cbm_bits(min_cbm_bits); +#else + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + } + + _pqos_api_unlock(); + + return ret; +} + +/* + * ======================================= + * L2 cache allocation + * ======================================= + */ + +int +pqos_l2ca_set(const unsigned l2id, + const unsigned num_cos, + const struct pqos_l2ca *ca) +{ + int ret; + unsigned i; + + if (ca == NULL || num_cos == 0) + return PQOS_RETVAL_PARAM; + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + + /** + * Check if class bitmasks are contiguous + */ + for (i = 0; i < num_cos; i++) { + int is_contig = 0; + + if (ca[i].cdp) { + is_contig = is_contiguous(ca[i].u.s.data_mask) && + is_contiguous(ca[i].u.s.code_mask); + } else + is_contig = is_contiguous(ca[i].u.ways_mask); + + if (!is_contig) { + LOG_ERROR("L2 COS%u bit mask is not contiguous!\n", + ca[i].class_id); + _pqos_api_unlock(); + return PQOS_RETVAL_PARAM; + } + } + if (m_interface == PQOS_INTER_MSR) + ret = hw_l2ca_set(l2id, num_cos, ca); + else { +#ifdef __linux__ + ret = os_l2ca_set(l2id, num_cos, ca); +#else + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + } + _pqos_api_unlock(); + + return ret; +} + +int +pqos_l2ca_get(const unsigned l2id, + const unsigned max_num_ca, + unsigned *num_ca, + struct pqos_l2ca *ca) +{ + int ret; + + if (num_ca == NULL || ca == NULL || max_num_ca == 0) + return PQOS_RETVAL_PARAM; + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + + if (m_interface == PQOS_INTER_MSR) + ret = hw_l2ca_get(l2id, max_num_ca, num_ca, ca); + else { +#ifdef __linux__ + ret = os_l2ca_get(l2id, max_num_ca, num_ca, ca); +#else + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + } + _pqos_api_unlock(); + + return ret; +} + +int +pqos_l2ca_get_min_cbm_bits(unsigned *min_cbm_bits) +{ + int ret; + + if (min_cbm_bits == NULL) + return PQOS_RETVAL_PARAM; + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + + if (m_interface == PQOS_INTER_MSR) + ret = hw_l2ca_get_min_cbm_bits(min_cbm_bits); + else { +#ifdef __linux__ + ret = os_l2ca_get_min_cbm_bits(min_cbm_bits); +#else + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + } + + _pqos_api_unlock(); + + return ret; +} + +/* + * ======================================= + * Memory Bandwidth Allocation + * ======================================= + */ + +int +pqos_mba_set(const unsigned socket, + const unsigned num_cos, + const struct pqos_mba *requested, + struct pqos_mba *actual) +{ + int ret; + unsigned i; + + if (requested == NULL || num_cos == 0) + return PQOS_RETVAL_PARAM; + + /** + * Check if MBA rate is within allowed range + */ + for (i = 0; i < num_cos; i++) + if (requested[i].mb_rate == 0 || requested[i].mb_rate > 100) { + LOG_ERROR("MBA COS%u rate out of range (from 1-100)!\n", + requested[i].class_id); + return PQOS_RETVAL_PARAM; + } + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + + if (m_interface == PQOS_INTER_MSR) + ret = hw_mba_set(socket, num_cos, requested, actual); + else { +#ifdef __linux__ + ret = os_mba_set(socket, num_cos, requested, actual); +#else + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + } + + _pqos_api_unlock(); + + return ret; + +} + +int +pqos_mba_get(const unsigned socket, + const unsigned max_num_cos, + unsigned *num_cos, + struct pqos_mba *mba_tab) +{ + int ret; + + if (num_cos == NULL || mba_tab == NULL || max_num_cos == 0) + return PQOS_RETVAL_PARAM; + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + + if (m_interface == PQOS_INTER_MSR) + ret = hw_mba_get(socket, max_num_cos, num_cos, mba_tab); + else { +#ifdef __linux__ + ret = os_mba_get(socket, max_num_cos, num_cos, mba_tab); +#else + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + } + + _pqos_api_unlock(); + + return ret; +} + +/* + * ======================================= + * Monitoring + * ======================================= + */ + +int +pqos_mon_reset(void) +{ + int ret; + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + + if (m_interface == PQOS_INTER_MSR) + ret = hw_mon_reset(); + else { +#ifdef __linux__ + ret = os_mon_reset(); +#else + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + } + + _pqos_api_unlock(); + + return ret; +} + +int +pqos_mon_assoc_get(const unsigned lcore, + pqos_rmid_t *rmid) +{ + int ret; + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + + if (m_interface == PQOS_INTER_MSR) + ret = hw_mon_assoc_get(lcore, rmid); + else { + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; + } + + _pqos_api_unlock(); + + return ret; +} + +int +pqos_mon_start(const unsigned num_cores, + const unsigned *cores, + const enum pqos_mon_event event, + void *context, + struct pqos_mon_data *group) +{ + int ret; + + if (group == NULL || cores == NULL || num_cores == 0 || event == 0) + return PQOS_RETVAL_PARAM; + + if (group->valid == GROUP_VALID_MARKER) + return PQOS_RETVAL_PARAM; + + /** + * Validate event parameter + * - only combinations of events allowed + * - do not allow non-PQoS events to be monitored on its own + */ + if (event & (~(PQOS_MON_EVENT_L3_OCCUP | PQOS_MON_EVENT_LMEM_BW | + PQOS_MON_EVENT_TMEM_BW | PQOS_MON_EVENT_RMEM_BW | + PQOS_PERF_EVENT_IPC | PQOS_PERF_EVENT_LLC_MISS))) + return PQOS_RETVAL_PARAM; + + if ((event & (PQOS_MON_EVENT_L3_OCCUP | PQOS_MON_EVENT_LMEM_BW | + PQOS_MON_EVENT_TMEM_BW | PQOS_MON_EVENT_RMEM_BW)) == 0 && + (event & (PQOS_PERF_EVENT_IPC | PQOS_PERF_EVENT_LLC_MISS)) != 0) + return PQOS_RETVAL_PARAM; + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + + if (m_interface == PQOS_INTER_MSR) + ret = hw_mon_start(num_cores, cores, event, context, group); + else { +#ifdef __linux__ + ret = os_mon_start(num_cores, cores, event, context, group); +#else + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + } + if (ret == PQOS_RETVAL_OK) + group->valid = GROUP_VALID_MARKER; + + _pqos_api_unlock(); + + return ret; +} + +int +pqos_mon_stop(struct pqos_mon_data *group) +{ + int ret; + + if (group == NULL) + return PQOS_RETVAL_PARAM; + + if (group->valid != GROUP_VALID_MARKER) + return PQOS_RETVAL_PARAM; + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + + if (m_interface == PQOS_INTER_MSR) + ret = hw_mon_stop(group); + else { +#ifdef __linux__ + ret = os_mon_stop(group); +#else + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + } + _pqos_api_unlock(); + + return ret; +} + +int +pqos_mon_poll(struct pqos_mon_data **groups, + const unsigned num_groups) +{ + int ret; + unsigned i; + + if (groups == NULL || num_groups == 0 || *groups == NULL) + return PQOS_RETVAL_PARAM; + + for (i = 0; i < num_groups; i++) { + if (groups[i] == NULL) + return PQOS_RETVAL_PARAM; + if (groups[i]->valid != GROUP_VALID_MARKER) + return PQOS_RETVAL_PARAM; + } + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + + if (m_interface == PQOS_INTER_MSR) + ret = hw_mon_poll(groups, num_groups); + else { +#ifdef __linux__ + ret = os_mon_poll(groups, num_groups); +#else + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + } + _pqos_api_unlock(); + + return ret; +} + +int +pqos_mon_start_pid(const pid_t pid, + const enum pqos_mon_event event, + void *context, + struct pqos_mon_data *group) +{ + return pqos_mon_start_pids(1, &pid, event, context, group); +} + +int +pqos_mon_start_pids(const unsigned num_pids, + const pid_t *pids, + const enum pqos_mon_event event, + void *context, + struct pqos_mon_data *group) +{ + int ret; + + if (num_pids == 0 || pids == NULL || group == NULL || event == 0) + return PQOS_RETVAL_PARAM; + + if (group->valid == GROUP_VALID_MARKER) + return PQOS_RETVAL_PARAM; + + if (m_interface != PQOS_INTER_OS && + m_interface != PQOS_INTER_OS_RESCTRL_MON) { + LOG_ERROR("Incompatible interface " + "selected for task monitoring!\n"); + return PQOS_RETVAL_ERROR; + } + + /** + * Validate event parameter + * - only combinations of events allowed + * - do not allow non-PQoS events to be monitored on its own + */ + if (event & (~(PQOS_MON_EVENT_L3_OCCUP | PQOS_MON_EVENT_LMEM_BW | + PQOS_MON_EVENT_TMEM_BW | PQOS_MON_EVENT_RMEM_BW | + PQOS_PERF_EVENT_IPC | PQOS_PERF_EVENT_LLC_MISS))) + return PQOS_RETVAL_PARAM; + + if ((event & (PQOS_MON_EVENT_L3_OCCUP | PQOS_MON_EVENT_LMEM_BW | + PQOS_MON_EVENT_TMEM_BW | PQOS_MON_EVENT_RMEM_BW)) == 0 && + (event & (PQOS_PERF_EVENT_IPC | PQOS_PERF_EVENT_LLC_MISS)) != 0) + return PQOS_RETVAL_PARAM; + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + +#ifdef __linux__ + ret = os_mon_start_pids(num_pids, pids, event, context, group); +#else + UNUSED_PARAM(context); + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + + if (ret == PQOS_RETVAL_OK) + group->valid = GROUP_VALID_MARKER; + + _pqos_api_unlock(); + + return ret; +} + +int pqos_mon_add_pids(const unsigned num_pids, + const pid_t *pids, + struct pqos_mon_data *group) +{ + int ret; + + if (num_pids == 0 || pids == NULL || group == NULL) + return PQOS_RETVAL_PARAM; + + if (group->valid != GROUP_VALID_MARKER) + return PQOS_RETVAL_PARAM; + + if (m_interface != PQOS_INTER_OS && + m_interface != PQOS_INTER_OS_RESCTRL_MON) { + LOG_ERROR("Incompatible interface " + "selected for task monitoring!\n"); + return PQOS_RETVAL_ERROR; + } + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + +#ifdef __linux__ + ret = os_mon_add_pids(num_pids, pids, group); +#else + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + + _pqos_api_unlock(); + + return ret; +} + +int pqos_mon_remove_pids(const unsigned num_pids, + const pid_t *pids, + struct pqos_mon_data *group) +{ + int ret; + + if (num_pids == 0 || pids == NULL || group == NULL) + return PQOS_RETVAL_PARAM; + + if (group->valid != GROUP_VALID_MARKER) + return PQOS_RETVAL_PARAM; + + if (m_interface != PQOS_INTER_OS && + m_interface != PQOS_INTER_OS_RESCTRL_MON) { + LOG_ERROR("Incompatible interface " + "selected for task monitoring!\n"); + return PQOS_RETVAL_ERROR; + } + + _pqos_api_lock(); + + ret = _pqos_check_init(1); + if (ret != PQOS_RETVAL_OK) { + _pqos_api_unlock(); + return ret; + } + +#ifdef __linux__ + ret = os_mon_remove_pids(num_pids, pids, group); +#else + LOG_INFO("OS interface not supported!\n"); + ret = PQOS_RETVAL_RESOURCE; +#endif + + _pqos_api_unlock(); + + return ret; +} diff --git a/lib/api.h b/lib/api.h new file mode 100644 index 0000000..9604b9e --- /dev/null +++ b/lib/api.h @@ -0,0 +1,57 @@ +/* + * BSD LICENSE + * + * Copyright(c) 2017 Intel Corporation. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#ifndef API_H +#define API_H + +#ifdef __cplusplus +extern "C" { +#endif + +/** + * @brief Initializes api module + * + * @param interface option, MSR or OS + * + * @return Operational status + * @retval PQOS_RETVAL_OK success + */ +int api_init(int interface); + +#ifdef __cplusplus +} +#endif + +#endif /* API_H */ + diff --git a/lib/api_doxygen.cfg b/lib/api_doxygen.cfg new file mode 100644 index 0000000..cf55c03 --- /dev/null +++ b/lib/api_doxygen.cfg @@ -0,0 +1,2460 @@ +# +# BSD LICENSE +# +# Copyright(c) 2014-2016 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +# + +# Doxyfile 1.8.11 + +# This file describes the settings to be used by the documentation system +# doxygen (www.doxygen.org) for a project. +# +# All text after a double hash (##) is considered a comment and is placed in +# front of the TAG it is preceding. +# +# All text after a single hash (#) is considered a comment and will be ignored. +# The format is: +# TAG = value [value, ...] +# For lists, items can also be appended using: +# TAG += value [value, ...] +# Values that contain spaces should be placed between quotes (\" \"). + +#--------------------------------------------------------------------------- +# Project related configuration options +#--------------------------------------------------------------------------- + +# This tag specifies the encoding used for all characters in the config file +# that follow. The default is UTF-8 which is also the encoding used for all text +# before the first occurrence of this tag. Doxygen uses libiconv (or the iconv +# built into libc) for the transcoding. See http://www.gnu.org/software/libiconv +# for the list of possible encodings. +# The default value is: UTF-8. + +DOXYFILE_ENCODING = UTF-8 + +# The PROJECT_NAME tag is a single word (or a sequence of words surrounded by +# double-quotes, unless you are using Doxywizard) that should identify the +# project for which the documentation is generated. This name is used in the +# title of most generated pages and in a few other places. +# The default value is: My Project. + +PROJECT_NAME = "PQoS Library API" + +# The PROJECT_NUMBER tag can be used to enter a project or revision number. This +# could be handy for archiving the generated documentation or if some version +# control system is used. + +PROJECT_NUMBER = + +# Using the PROJECT_BRIEF tag one can provide an optional one line description +# for a project that appears at the top of each page and should give viewer a +# quick idea about the purpose of the project. Keep the description short. + +PROJECT_BRIEF = + +# With the PROJECT_LOGO tag one can specify a logo or an icon that is included +# in the documentation. The maximum height of the logo should not exceed 55 +# pixels and the maximum width should not exceed 200 pixels. Doxygen will copy +# the logo to the output directory. + +PROJECT_LOGO = + +# The OUTPUT_DIRECTORY tag is used to specify the (relative or absolute) path +# into which the generated documentation will be written. If a relative path is +# entered, it will be relative to the location where doxygen was started. If +# left blank the current directory will be used. + +OUTPUT_DIRECTORY = doc_api + +# If the CREATE_SUBDIRS tag is set to YES then doxygen will create 4096 sub- +# directories (in 2 levels) under the output directory of each output format and +# will distribute the generated files over these directories. Enabling this +# option can be useful when feeding doxygen a huge amount of source files, where +# putting all generated files in the same directory would otherwise causes +# performance problems for the file system. +# The default value is: NO. + +CREATE_SUBDIRS = NO + +# If the ALLOW_UNICODE_NAMES tag is set to YES, doxygen will allow non-ASCII +# characters to appear in the names of generated files. If set to NO, non-ASCII +# characters will be escaped, for example _xE3_x81_x84 will be used for Unicode +# U+3044. +# The default value is: NO. + +ALLOW_UNICODE_NAMES = NO + +# The OUTPUT_LANGUAGE tag is used to specify the language in which all +# documentation generated by doxygen is written. Doxygen will use this +# information to generate all constant output in the proper language. +# Possible values are: Afrikaans, Arabic, Armenian, Brazilian, Catalan, Chinese, +# Chinese-Traditional, Croatian, Czech, Danish, Dutch, English (United States), +# Esperanto, Farsi (Persian), Finnish, French, German, Greek, Hungarian, +# Indonesian, Italian, Japanese, Japanese-en (Japanese with English messages), +# Korean, Korean-en (Korean with English messages), Latvian, Lithuanian, +# Macedonian, Norwegian, Persian (Farsi), Polish, Portuguese, Romanian, Russian, +# Serbian, Serbian-Cyrillic, Slovak, Slovene, Spanish, Swedish, Turkish, +# Ukrainian and Vietnamese. +# The default value is: English. + +OUTPUT_LANGUAGE = English + +# If the BRIEF_MEMBER_DESC tag is set to YES, doxygen will include brief member +# descriptions after the members that are listed in the file and class +# documentation (similar to Javadoc). Set to NO to disable this. +# The default value is: YES. + +BRIEF_MEMBER_DESC = YES + +# If the REPEAT_BRIEF tag is set to YES, doxygen will prepend the brief +# description of a member or function before the detailed description +# +# Note: If both HIDE_UNDOC_MEMBERS and BRIEF_MEMBER_DESC are set to NO, the +# brief descriptions will be completely suppressed. +# The default value is: YES. + +REPEAT_BRIEF = YES + +# This tag implements a quasi-intelligent brief description abbreviator that is +# used to form the text in various listings. Each string in this list, if found +# as the leading text of the brief description, will be stripped from the text +# and the result, after processing the whole list, is used as the annotated +# text. Otherwise, the brief description is used as-is. If left blank, the +# following values are used ($name is automatically replaced with the name of +# the entity):The $name class, The $name widget, The $name file, is, provides, +# specifies, contains, represents, a, an and the. + +ABBREVIATE_BRIEF = + +# If the ALWAYS_DETAILED_SEC and REPEAT_BRIEF tags are both set to YES then +# doxygen will generate a detailed section even if there is only a brief +# description. +# The default value is: NO. + +ALWAYS_DETAILED_SEC = NO + +# If the INLINE_INHERITED_MEMB tag is set to YES, doxygen will show all +# inherited members of a class in the documentation of that class as if those +# members were ordinary class members. Constructors, destructors and assignment +# operators of the base classes will not be shown. +# The default value is: NO. + +INLINE_INHERITED_MEMB = NO + +# If the FULL_PATH_NAMES tag is set to YES, doxygen will prepend the full path +# before files name in the file list and in the header files. If set to NO the +# shortest path that makes the file name unique will be used +# The default value is: YES. + +FULL_PATH_NAMES = YES + +# The STRIP_FROM_PATH tag can be used to strip a user-defined part of the path. +# Stripping is only done if one of the specified strings matches the left-hand +# part of the path. The tag can be used to show relative paths in the file list. +# If left blank the directory from which doxygen is run is used as the path to +# strip. +# +# Note that you can specify absolute paths here, but also relative paths, which +# will be relative from the directory where doxygen is started. +# This tag requires that the tag FULL_PATH_NAMES is set to YES. + +STRIP_FROM_PATH = + +# The STRIP_FROM_INC_PATH tag can be used to strip a user-defined part of the +# path mentioned in the documentation of a class, which tells the reader which +# header file to include in order to use a class. If left blank only the name of +# the header file containing the class definition is used. Otherwise one should +# specify the list of include paths that are normally passed to the compiler +# using the -I flag. + +STRIP_FROM_INC_PATH = + +# If the SHORT_NAMES tag is set to YES, doxygen will generate much shorter (but +# less readable) file names. This can be useful is your file systems doesn't +# support long names like on DOS, Mac, or CD-ROM. +# The default value is: NO. + +SHORT_NAMES = NO + +# If the JAVADOC_AUTOBRIEF tag is set to YES then doxygen will interpret the +# first line (until the first dot) of a Javadoc-style comment as the brief +# description. If set to NO, the Javadoc-style will behave just like regular Qt- +# style comments (thus requiring an explicit @brief command for a brief +# description.) +# The default value is: NO. + +JAVADOC_AUTOBRIEF = NO + +# If the QT_AUTOBRIEF tag is set to YES then doxygen will interpret the first +# line (until the first dot) of a Qt-style comment as the brief description. If +# set to NO, the Qt-style will behave just like regular Qt-style comments (thus +# requiring an explicit \brief command for a brief description.) +# The default value is: NO. + +QT_AUTOBRIEF = NO + +# The MULTILINE_CPP_IS_BRIEF tag can be set to YES to make doxygen treat a +# multi-line C++ special comment block (i.e. a block of //! or /// comments) as +# a brief description. This used to be the default behavior. The new default is +# to treat a multi-line C++ comment block as a detailed description. Set this +# tag to YES if you prefer the old behavior instead. +# +# Note that setting this tag to YES also means that rational rose comments are +# not recognized any more. +# The default value is: NO. + +MULTILINE_CPP_IS_BRIEF = NO + +# If the INHERIT_DOCS tag is set to YES then an undocumented member inherits the +# documentation from any documented member that it re-implements. +# The default value is: YES. + +INHERIT_DOCS = YES + +# If the SEPARATE_MEMBER_PAGES tag is set to YES then doxygen will produce a new +# page for each member. If set to NO, the documentation of a member will be part +# of the file/class/namespace that contains it. +# The default value is: NO. + +SEPARATE_MEMBER_PAGES = NO + +# The TAB_SIZE tag can be used to set the number of spaces in a tab. Doxygen +# uses this value to replace tabs by spaces in code fragments. +# Minimum value: 1, maximum value: 16, default value: 4. + +TAB_SIZE = 8 + +# This tag can be used to specify a number of aliases that act as commands in +# the documentation. An alias has the form: +# name=value +# For example adding +# "sideeffect=@par Side Effects:\n" +# will allow you to put the command \sideeffect (or @sideeffect) in the +# documentation, which will result in a user-defined paragraph with heading +# "Side Effects:". You can put \n's in the value part of an alias to insert +# newlines. + +ALIASES = + +# This tag can be used to specify a number of word-keyword mappings (TCL only). +# A mapping has the form "name=value". For example adding "class=itcl::class" +# will allow you to use the command class in the itcl::class meaning. + +TCL_SUBST = + +# Set the OPTIMIZE_OUTPUT_FOR_C tag to YES if your project consists of C sources +# only. Doxygen will then generate output that is more tailored for C. For +# instance, some of the names that are used will be different. The list of all +# members will be omitted, etc. +# The default value is: NO. + +OPTIMIZE_OUTPUT_FOR_C = YES + +# Set the OPTIMIZE_OUTPUT_JAVA tag to YES if your project consists of Java or +# Python sources only. Doxygen will then generate output that is more tailored +# for that language. For instance, namespaces will be presented as packages, +# qualified scopes will look different, etc. +# The default value is: NO. + +OPTIMIZE_OUTPUT_JAVA = NO + +# Set the OPTIMIZE_FOR_FORTRAN tag to YES if your project consists of Fortran +# sources. Doxygen will then generate output that is tailored for Fortran. +# The default value is: NO. + +OPTIMIZE_FOR_FORTRAN = NO + +# Set the OPTIMIZE_OUTPUT_VHDL tag to YES if your project consists of VHDL +# sources. Doxygen will then generate output that is tailored for VHDL. +# The default value is: NO. + +OPTIMIZE_OUTPUT_VHDL = NO + +# Doxygen selects the parser to use depending on the extension of the files it +# parses. With this tag you can assign which parser to use for a given +# extension. Doxygen has a built-in mapping, but you can override or extend it +# using this tag. The format is ext=language, where ext is a file extension, and +# language is one of the parsers supported by doxygen: IDL, Java, Javascript, +# C#, C, C++, D, PHP, Objective-C, Python, Fortran (fixed format Fortran: +# FortranFixed, free formatted Fortran: FortranFree, unknown formatted Fortran: +# Fortran. In the later case the parser tries to guess whether the code is fixed +# or free formatted code, this is the default for Fortran type files), VHDL. For +# instance to make doxygen treat .inc files as Fortran files (default is PHP), +# and .f files as C (default is Fortran), use: inc=Fortran f=C. +# +# Note: For files without extension you can use no_extension as a placeholder. +# +# Note that for custom extensions you also need to set FILE_PATTERNS otherwise +# the files are not read by doxygen. + +EXTENSION_MAPPING = + +# If the MARKDOWN_SUPPORT tag is enabled then doxygen pre-processes all comments +# according to the Markdown format, which allows for more readable +# documentation. See http://daringfireball.net/projects/markdown/ for details. +# The output of markdown processing is further processed by doxygen, so you can +# mix doxygen, HTML, and XML commands with Markdown formatting. Disable only in +# case of backward compatibilities issues. +# The default value is: YES. + +MARKDOWN_SUPPORT = YES + +# When enabled doxygen tries to link words that correspond to documented +# classes, or namespaces to their corresponding documentation. Such a link can +# be prevented in individual cases by putting a % sign in front of the word or +# globally by setting AUTOLINK_SUPPORT to NO. +# The default value is: YES. + +AUTOLINK_SUPPORT = YES + +# If you use STL classes (i.e. std::string, std::vector, etc.) but do not want +# to include (a tag file for) the STL sources as input, then you should set this +# tag to YES in order to let doxygen match functions declarations and +# definitions whose arguments contain STL classes (e.g. func(std::string); +# versus func(std::string) {}). This also make the inheritance and collaboration +# diagrams that involve STL classes more complete and accurate. +# The default value is: NO. + +BUILTIN_STL_SUPPORT = NO + +# If you use Microsoft's C++/CLI language, you should set this option to YES to +# enable parsing support. +# The default value is: NO. + +CPP_CLI_SUPPORT = NO + +# Set the SIP_SUPPORT tag to YES if your project consists of sip (see: +# http://www.riverbankcomputing.co.uk/software/sip/intro) sources only. Doxygen +# will parse them like normal C++ but will assume all classes use public instead +# of private inheritance when no explicit protection keyword is present. +# The default value is: NO. + +SIP_SUPPORT = NO + +# For Microsoft's IDL there are propget and propput attributes to indicate +# getter and setter methods for a property. Setting this option to YES will make +# doxygen to replace the get and set methods by a property in the documentation. +# This will only work if the methods are indeed getting or setting a simple +# type. If this is not the case, or you want to show the methods anyway, you +# should set this option to NO. +# The default value is: YES. + +IDL_PROPERTY_SUPPORT = YES + +# If member grouping is used in the documentation and the DISTRIBUTE_GROUP_DOC +# tag is set to YES then doxygen will reuse the documentation of the first +# member in the group (if any) for the other members of the group. By default +# all members of a group must be documented explicitly. +# The default value is: NO. + +DISTRIBUTE_GROUP_DOC = NO + +# If one adds a struct or class to a group and this option is enabled, then also +# any nested class or struct is added to the same group. By default this option +# is disabled and one has to add nested compounds explicitly via \ingroup. +# The default value is: NO. + +GROUP_NESTED_COMPOUNDS = NO + +# Set the SUBGROUPING tag to YES to allow class member groups of the same type +# (for instance a group of public functions) to be put as a subgroup of that +# type (e.g. under the Public Functions section). Set it to NO to prevent +# subgrouping. Alternatively, this can be done per class using the +# \nosubgrouping command. +# The default value is: YES. + +SUBGROUPING = YES + +# When the INLINE_GROUPED_CLASSES tag is set to YES, classes, structs and unions +# are shown inside the group in which they are included (e.g. using \ingroup) +# instead of on a separate page (for HTML and Man pages) or section (for LaTeX +# and RTF). +# +# Note that this feature does not work in combination with +# SEPARATE_MEMBER_PAGES. +# The default value is: NO. + +INLINE_GROUPED_CLASSES = NO + +# When the INLINE_SIMPLE_STRUCTS tag is set to YES, structs, classes, and unions +# with only public data fields or simple typedef fields will be shown inline in +# the documentation of the scope in which they are defined (i.e. file, +# namespace, or group documentation), provided this scope is documented. If set +# to NO, structs, classes, and unions are shown on a separate page (for HTML and +# Man pages) or section (for LaTeX and RTF). +# The default value is: NO. + +INLINE_SIMPLE_STRUCTS = NO + +# When TYPEDEF_HIDES_STRUCT tag is enabled, a typedef of a struct, union, or +# enum is documented as struct, union, or enum with the name of the typedef. So +# typedef struct TypeS {} TypeT, will appear in the documentation as a struct +# with name TypeT. When disabled the typedef will appear as a member of a file, +# namespace, or class. And the struct will be named TypeS. This can typically be +# useful for C code in case the coding convention dictates that all compound +# types are typedef'ed and only the typedef is referenced, never the tag name. +# The default value is: NO. + +TYPEDEF_HIDES_STRUCT = NO + +# The size of the symbol lookup cache can be set using LOOKUP_CACHE_SIZE. This +# cache is used to resolve symbols given their name and scope. Since this can be +# an expensive process and often the same symbol appears multiple times in the +# code, doxygen keeps a cache of pre-resolved symbols. If the cache is too small +# doxygen will become slower. If the cache is too large, memory is wasted. The +# cache size is given by this formula: 2^(16+LOOKUP_CACHE_SIZE). The valid range +# is 0..9, the default is 0, corresponding to a cache size of 2^16=65536 +# symbols. At the end of a run doxygen will report the cache usage and suggest +# the optimal cache size from a speed point of view. +# Minimum value: 0, maximum value: 9, default value: 0. + +LOOKUP_CACHE_SIZE = 0 + +#--------------------------------------------------------------------------- +# Build related configuration options +#--------------------------------------------------------------------------- + +# If the EXTRACT_ALL tag is set to YES, doxygen will assume all entities in +# documentation are documented, even if no documentation was available. Private +# class members and static file members will be hidden unless the +# EXTRACT_PRIVATE respectively EXTRACT_STATIC tags are set to YES. +# Note: This will also disable the warnings about undocumented members that are +# normally produced when WARNINGS is set to YES. +# The default value is: NO. + +EXTRACT_ALL = YES + +# If the EXTRACT_PRIVATE tag is set to YES, all private members of a class will +# be included in the documentation. +# The default value is: NO. + +EXTRACT_PRIVATE = NO + +# If the EXTRACT_PACKAGE tag is set to YES, all members with package or internal +# scope will be included in the documentation. +# The default value is: NO. + +EXTRACT_PACKAGE = NO + +# If the EXTRACT_STATIC tag is set to YES, all static members of a file will be +# included in the documentation. +# The default value is: NO. + +EXTRACT_STATIC = NO + +# If the EXTRACT_LOCAL_CLASSES tag is set to YES, classes (and structs) defined +# locally in source files will be included in the documentation. If set to NO, +# only classes defined in header files are included. Does not have any effect +# for Java sources. +# The default value is: YES. + +EXTRACT_LOCAL_CLASSES = YES + +# This flag is only useful for Objective-C code. If set to YES, local methods, +# which are defined in the implementation section but not in the interface are +# included in the documentation. If set to NO, only methods in the interface are +# included. +# The default value is: NO. + +EXTRACT_LOCAL_METHODS = NO + +# If this flag is set to YES, the members of anonymous namespaces will be +# extracted and appear in the documentation as a namespace called +# 'anonymous_namespace{file}', where file will be replaced with the base name of +# the file that contains the anonymous namespace. By default anonymous namespace +# are hidden. +# The default value is: NO. + +EXTRACT_ANON_NSPACES = NO + +# If the HIDE_UNDOC_MEMBERS tag is set to YES, doxygen will hide all +# undocumented members inside documented classes or files. If set to NO these +# members will be included in the various overviews, but no documentation +# section is generated. This option has no effect if EXTRACT_ALL is enabled. +# The default value is: NO. + +HIDE_UNDOC_MEMBERS = NO + +# If the HIDE_UNDOC_CLASSES tag is set to YES, doxygen will hide all +# undocumented classes that are normally visible in the class hierarchy. If set +# to NO, these classes will be included in the various overviews. This option +# has no effect if EXTRACT_ALL is enabled. +# The default value is: NO. + +HIDE_UNDOC_CLASSES = NO + +# If the HIDE_FRIEND_COMPOUNDS tag is set to YES, doxygen will hide all friend +# (class|struct|union) declarations. If set to NO, these declarations will be +# included in the documentation. +# The default value is: NO. + +HIDE_FRIEND_COMPOUNDS = NO + +# If the HIDE_IN_BODY_DOCS tag is set to YES, doxygen will hide any +# documentation blocks found inside the body of a function. If set to NO, these +# blocks will be appended to the function's detailed documentation block. +# The default value is: NO. + +HIDE_IN_BODY_DOCS = NO + +# The INTERNAL_DOCS tag determines if documentation that is typed after a +# \internal command is included. If the tag is set to NO then the documentation +# will be excluded. Set it to YES to include the internal documentation. +# The default value is: NO. + +INTERNAL_DOCS = NO + +# If the CASE_SENSE_NAMES tag is set to NO then doxygen will only generate file +# names in lower-case letters. If set to YES, upper-case letters are also +# allowed. This is useful if you have classes or files whose names only differ +# in case and if your file system supports case sensitive file names. Windows +# and Mac users are advised to set this option to NO. +# The default value is: system dependent. + +CASE_SENSE_NAMES = YES + +# If the HIDE_SCOPE_NAMES tag is set to NO then doxygen will show members with +# their full class and namespace scopes in the documentation. If set to YES, the +# scope will be hidden. +# The default value is: NO. + +HIDE_SCOPE_NAMES = NO + +# If the HIDE_COMPOUND_REFERENCE tag is set to NO (default) then doxygen will +# append additional text to a page's title, such as Class Reference. If set to +# YES the compound reference will be hidden. +# The default value is: NO. + +HIDE_COMPOUND_REFERENCE= NO + +# If the SHOW_INCLUDE_FILES tag is set to YES then doxygen will put a list of +# the files that are included by a file in the documentation of that file. +# The default value is: YES. + +SHOW_INCLUDE_FILES = YES + +# If the SHOW_GROUPED_MEMB_INC tag is set to YES then Doxygen will add for each +# grouped member an include statement to the documentation, telling the reader +# which file to include in order to use the member. +# The default value is: NO. + +SHOW_GROUPED_MEMB_INC = NO + +# If the FORCE_LOCAL_INCLUDES tag is set to YES then doxygen will list include +# files with double quotes in the documentation rather than with sharp brackets. +# The default value is: NO. + +FORCE_LOCAL_INCLUDES = NO + +# If the INLINE_INFO tag is set to YES then a tag [inline] is inserted in the +# documentation for inline members. +# The default value is: YES. + +INLINE_INFO = YES + +# If the SORT_MEMBER_DOCS tag is set to YES then doxygen will sort the +# (detailed) documentation of file and class members alphabetically by member +# name. If set to NO, the members will appear in declaration order. +# The default value is: YES. + +SORT_MEMBER_DOCS = YES + +# If the SORT_BRIEF_DOCS tag is set to YES then doxygen will sort the brief +# descriptions of file, namespace and class members alphabetically by member +# name. If set to NO, the members will appear in declaration order. Note that +# this will also influence the order of the classes in the class list. +# The default value is: NO. + +SORT_BRIEF_DOCS = NO + +# If the SORT_MEMBERS_CTORS_1ST tag is set to YES then doxygen will sort the +# (brief and detailed) documentation of class members so that constructors and +# destructors are listed first. If set to NO the constructors will appear in the +# respective orders defined by SORT_BRIEF_DOCS and SORT_MEMBER_DOCS. +# Note: If SORT_BRIEF_DOCS is set to NO this option is ignored for sorting brief +# member documentation. +# Note: If SORT_MEMBER_DOCS is set to NO this option is ignored for sorting +# detailed member documentation. +# The default value is: NO. + +SORT_MEMBERS_CTORS_1ST = NO + +# If the SORT_GROUP_NAMES tag is set to YES then doxygen will sort the hierarchy +# of group names into alphabetical order. If set to NO the group names will +# appear in their defined order. +# The default value is: NO. + +SORT_GROUP_NAMES = NO + +# If the SORT_BY_SCOPE_NAME tag is set to YES, the class list will be sorted by +# fully-qualified names, including namespaces. If set to NO, the class list will +# be sorted only by class name, not including the namespace part. +# Note: This option is not very useful if HIDE_SCOPE_NAMES is set to YES. +# Note: This option applies only to the class list, not to the alphabetical +# list. +# The default value is: NO. + +SORT_BY_SCOPE_NAME = NO + +# If the STRICT_PROTO_MATCHING option is enabled and doxygen fails to do proper +# type resolution of all parameters of a function it will reject a match between +# the prototype and the implementation of a member function even if there is +# only one candidate or it is obvious which candidate to choose by doing a +# simple string match. By disabling STRICT_PROTO_MATCHING doxygen will still +# accept a match between prototype and implementation in such cases. +# The default value is: NO. + +STRICT_PROTO_MATCHING = NO + +# The GENERATE_TODOLIST tag can be used to enable (YES) or disable (NO) the todo +# list. This list is created by putting \todo commands in the documentation. +# The default value is: YES. + +GENERATE_TODOLIST = YES + +# The GENERATE_TESTLIST tag can be used to enable (YES) or disable (NO) the test +# list. This list is created by putting \test commands in the documentation. +# The default value is: YES. + +GENERATE_TESTLIST = YES + +# The GENERATE_BUGLIST tag can be used to enable (YES) or disable (NO) the bug +# list. This list is created by putting \bug commands in the documentation. +# The default value is: YES. + +GENERATE_BUGLIST = YES + +# The GENERATE_DEPRECATEDLIST tag can be used to enable (YES) or disable (NO) +# the deprecated list. This list is created by putting \deprecated commands in +# the documentation. +# The default value is: YES. + +GENERATE_DEPRECATEDLIST= YES + +# The ENABLED_SECTIONS tag can be used to enable conditional documentation +# sections, marked by \if ... \endif and \cond +# ... \endcond blocks. + +ENABLED_SECTIONS = + +# The MAX_INITIALIZER_LINES tag determines the maximum number of lines that the +# initial value of a variable or macro / define can have for it to appear in the +# documentation. If the initializer consists of more lines than specified here +# it will be hidden. Use a value of 0 to hide initializers completely. The +# appearance of the value of individual variables and macros / defines can be +# controlled using \showinitializer or \hideinitializer command in the +# documentation regardless of this setting. +# Minimum value: 0, maximum value: 10000, default value: 30. + +MAX_INITIALIZER_LINES = 30 + +# Set the SHOW_USED_FILES tag to NO to disable the list of files generated at +# the bottom of the documentation of classes and structs. If set to YES, the +# list will mention the files that were used to generate the documentation. +# The default value is: YES. + +SHOW_USED_FILES = YES + +# Set the SHOW_FILES tag to NO to disable the generation of the Files page. This +# will remove the Files entry from the Quick Index and from the Folder Tree View +# (if specified). +# The default value is: YES. + +SHOW_FILES = YES + +# Set the SHOW_NAMESPACES tag to NO to disable the generation of the Namespaces +# page. This will remove the Namespaces entry from the Quick Index and from the +# Folder Tree View (if specified). +# The default value is: YES. + +SHOW_NAMESPACES = YES + +# The FILE_VERSION_FILTER tag can be used to specify a program or script that +# doxygen should invoke to get the current version for each file (typically from +# the version control system). Doxygen will invoke the program by executing (via +# popen()) the command command input-file, where command is the value of the +# FILE_VERSION_FILTER tag, and input-file is the name of an input file provided +# by doxygen. Whatever the program writes to standard output is used as the file +# version. For an example see the documentation. + +FILE_VERSION_FILTER = + +# The LAYOUT_FILE tag can be used to specify a layout file which will be parsed +# by doxygen. The layout file controls the global structure of the generated +# output files in an output format independent way. To create the layout file +# that represents doxygen's defaults, run doxygen with the -l option. You can +# optionally specify a file name after the option, if omitted DoxygenLayout.xml +# will be used as the name of the layout file. +# +# Note that if you run doxygen from a directory containing a file called +# DoxygenLayout.xml, doxygen will parse it automatically even if the LAYOUT_FILE +# tag is left empty. + +LAYOUT_FILE = + +# The CITE_BIB_FILES tag can be used to specify one or more bib files containing +# the reference definitions. This must be a list of .bib files. The .bib +# extension is automatically appended if omitted. This requires the bibtex tool +# to be installed. See also http://en.wikipedia.org/wiki/BibTeX for more info. +# For LaTeX the style of the bibliography can be controlled using +# LATEX_BIB_STYLE. To use this feature you need bibtex and perl available in the +# search path. See also \cite for info how to create references. + +CITE_BIB_FILES = + +#--------------------------------------------------------------------------- +# Configuration options related to warning and progress messages +#--------------------------------------------------------------------------- + +# The QUIET tag can be used to turn on/off the messages that are generated to +# standard output by doxygen. If QUIET is set to YES this implies that the +# messages are off. +# The default value is: NO. + +QUIET = NO + +# The WARNINGS tag can be used to turn on/off the warning messages that are +# generated to standard error (stderr) by doxygen. If WARNINGS is set to YES +# this implies that the warnings are on. +# +# Tip: Turn warnings on while writing the documentation. +# The default value is: YES. + +WARNINGS = YES + +# If the WARN_IF_UNDOCUMENTED tag is set to YES then doxygen will generate +# warnings for undocumented members. If EXTRACT_ALL is set to YES then this flag +# will automatically be disabled. +# The default value is: YES. + +WARN_IF_UNDOCUMENTED = YES + +# If the WARN_IF_DOC_ERROR tag is set to YES, doxygen will generate warnings for +# potential errors in the documentation, such as not documenting some parameters +# in a documented function, or documenting parameters that don't exist or using +# markup commands wrongly. +# The default value is: YES. + +WARN_IF_DOC_ERROR = YES + +# This WARN_NO_PARAMDOC option can be enabled to get warnings for functions that +# are documented, but have no documentation for their parameters or return +# value. If set to NO, doxygen will only warn about wrong or incomplete +# parameter documentation, but not about the absence of documentation. +# The default value is: NO. + +WARN_NO_PARAMDOC = NO + +# If the WARN_AS_ERROR tag is set to YES then doxygen will immediately stop when +# a warning is encountered. +# The default value is: NO. + +WARN_AS_ERROR = NO + +# The WARN_FORMAT tag determines the format of the warning messages that doxygen +# can produce. The string should contain the $file, $line, and $text tags, which +# will be replaced by the file and line number from which the warning originated +# and the warning text. Optionally the format may contain $version, which will +# be replaced by the version of the file (if it could be obtained via +# FILE_VERSION_FILTER) +# The default value is: $file:$line: $text. + +WARN_FORMAT = "$file:$line: $text" + +# The WARN_LOGFILE tag can be used to specify a file to which warning and error +# messages should be written. If left blank the output is written to standard +# error (stderr). + +WARN_LOGFILE = + +#--------------------------------------------------------------------------- +# Configuration options related to the input files +#--------------------------------------------------------------------------- + +# The INPUT tag is used to specify the files and/or directories that contain +# documented source files. You may enter file names like myfile.cpp or +# directories like /usr/src/myproject. Separate the files or directories with +# spaces. See also FILE_PATTERNS and EXTENSION_MAPPING +# Note: If this tag is empty the current directory is searched. + +INPUT = pqos.h + +# This tag can be used to specify the character encoding of the source files +# that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses +# libiconv (or the iconv built into libc) for the transcoding. See the libiconv +# documentation (see: http://www.gnu.org/software/libiconv) for the list of +# possible encodings. +# The default value is: UTF-8. + +INPUT_ENCODING = UTF-8 + +# If the value of the INPUT tag contains directories, you can use the +# FILE_PATTERNS tag to specify one or more wildcard patterns (like *.cpp and +# *.h) to filter out the source-files in the directories. +# +# Note that for custom extensions or not directly supported extensions you also +# need to set EXTENSION_MAPPING for the extension otherwise the files are not +# read by doxygen. +# +# If left blank the following patterns are tested:*.c, *.cc, *.cxx, *.cpp, +# *.c++, *.java, *.ii, *.ixx, *.ipp, *.i++, *.inl, *.idl, *.ddl, *.odl, *.h, +# *.hh, *.hxx, *.hpp, *.h++, *.cs, *.d, *.php, *.php4, *.php5, *.phtml, *.inc, +# *.m, *.markdown, *.md, *.mm, *.dox, *.py, *.pyw, *.f90, *.f, *.for, *.tcl, +# *.vhd, *.vhdl, *.ucf, *.qsf, *.as and *.js. + +FILE_PATTERNS = + +# The RECURSIVE tag can be used to specify whether or not subdirectories should +# be searched for input files as well. +# The default value is: NO. + +RECURSIVE = NO + +# The EXCLUDE tag can be used to specify files and/or directories that should be +# excluded from the INPUT source files. This way you can easily exclude a +# subdirectory from a directory tree whose root is specified with the INPUT tag. +# +# Note that relative paths are relative to the directory from which doxygen is +# run. + +EXCLUDE = + +# The EXCLUDE_SYMLINKS tag can be used to select whether or not files or +# directories that are symbolic links (a Unix file system feature) are excluded +# from the input. +# The default value is: NO. + +EXCLUDE_SYMLINKS = NO + +# If the value of the INPUT tag contains directories, you can use the +# EXCLUDE_PATTERNS tag to specify one or more wildcard patterns to exclude +# certain files from those directories. +# +# Note that the wildcards are matched against the file with absolute path, so to +# exclude all test directories for example use the pattern */test/* + +EXCLUDE_PATTERNS = + +# The EXCLUDE_SYMBOLS tag can be used to specify one or more symbol names +# (namespaces, classes, functions, etc.) that should be excluded from the +# output. The symbol name can be a fully qualified name, a word, or if the +# wildcard * is used, a substring. Examples: ANamespace, AClass, +# AClass::ANamespace, ANamespace::*Test +# +# Note that the wildcards are matched against the file with absolute path, so to +# exclude all test directories use the pattern */test/* + +EXCLUDE_SYMBOLS = + +# The EXAMPLE_PATH tag can be used to specify one or more files or directories +# that contain example code fragments that are included (see the \include +# command). + +EXAMPLE_PATH = + +# If the value of the EXAMPLE_PATH tag contains directories, you can use the +# EXAMPLE_PATTERNS tag to specify one or more wildcard pattern (like *.cpp and +# *.h) to filter out the source-files in the directories. If left blank all +# files are included. + +EXAMPLE_PATTERNS = + +# If the EXAMPLE_RECURSIVE tag is set to YES then subdirectories will be +# searched for input files to be used with the \include or \dontinclude commands +# irrespective of the value of the RECURSIVE tag. +# The default value is: NO. + +EXAMPLE_RECURSIVE = NO + +# The IMAGE_PATH tag can be used to specify one or more files or directories +# that contain images that are to be included in the documentation (see the +# \image command). + +IMAGE_PATH = + +# The INPUT_FILTER tag can be used to specify a program that doxygen should +# invoke to filter for each input file. Doxygen will invoke the filter program +# by executing (via popen()) the command: +# +# +# +# where is the value of the INPUT_FILTER tag, and is the +# name of an input file. Doxygen will then use the output that the filter +# program writes to standard output. If FILTER_PATTERNS is specified, this tag +# will be ignored. +# +# Note that the filter must not add or remove lines; it is applied before the +# code is scanned, but not when the output code is generated. If lines are added +# or removed, the anchors will not be placed correctly. +# +# Note that for custom extensions or not directly supported extensions you also +# need to set EXTENSION_MAPPING for the extension otherwise the files are not +# properly processed by doxygen. + +INPUT_FILTER = + +# The FILTER_PATTERNS tag can be used to specify filters on a per file pattern +# basis. Doxygen will compare the file name with each pattern and apply the +# filter if there is a match. The filters are a list of the form: pattern=filter +# (like *.cpp=my_cpp_filter). See INPUT_FILTER for further information on how +# filters are used. If the FILTER_PATTERNS tag is empty or if none of the +# patterns match the file name, INPUT_FILTER is applied. +# +# Note that for custom extensions or not directly supported extensions you also +# need to set EXTENSION_MAPPING for the extension otherwise the files are not +# properly processed by doxygen. + +FILTER_PATTERNS = + +# If the FILTER_SOURCE_FILES tag is set to YES, the input filter (if set using +# INPUT_FILTER) will also be used to filter the input files that are used for +# producing the source files to browse (i.e. when SOURCE_BROWSER is set to YES). +# The default value is: NO. + +FILTER_SOURCE_FILES = NO + +# The FILTER_SOURCE_PATTERNS tag can be used to specify source filters per file +# pattern. A pattern will override the setting for FILTER_PATTERN (if any) and +# it is also possible to disable source filtering for a specific pattern using +# *.ext= (so without naming a filter). +# This tag requires that the tag FILTER_SOURCE_FILES is set to YES. + +FILTER_SOURCE_PATTERNS = + +# If the USE_MDFILE_AS_MAINPAGE tag refers to the name of a markdown file that +# is part of the input, its contents will be placed on the main page +# (index.html). This can be useful if you have a project on for instance GitHub +# and want to reuse the introduction page also for the doxygen output. + +USE_MDFILE_AS_MAINPAGE = + +#--------------------------------------------------------------------------- +# Configuration options related to source browsing +#--------------------------------------------------------------------------- + +# If the SOURCE_BROWSER tag is set to YES then a list of source files will be +# generated. Documented entities will be cross-referenced with these sources. +# +# Note: To get rid of all source code in the generated output, make sure that +# also VERBATIM_HEADERS is set to NO. +# The default value is: NO. + +SOURCE_BROWSER = NO + +# Setting the INLINE_SOURCES tag to YES will include the body of functions, +# classes and enums directly into the documentation. +# The default value is: NO. + +INLINE_SOURCES = NO + +# Setting the STRIP_CODE_COMMENTS tag to YES will instruct doxygen to hide any +# special comment blocks from generated source code fragments. Normal C, C++ and +# Fortran comments will always remain visible. +# The default value is: YES. + +STRIP_CODE_COMMENTS = YES + +# If the REFERENCED_BY_RELATION tag is set to YES then for each documented +# function all documented functions referencing it will be listed. +# The default value is: NO. + +REFERENCED_BY_RELATION = NO + +# If the REFERENCES_RELATION tag is set to YES then for each documented function +# all documented entities called/used by that function will be listed. +# The default value is: NO. + +REFERENCES_RELATION = NO + +# If the REFERENCES_LINK_SOURCE tag is set to YES and SOURCE_BROWSER tag is set +# to YES then the hyperlinks from functions in REFERENCES_RELATION and +# REFERENCED_BY_RELATION lists will link to the source code. Otherwise they will +# link to the documentation. +# The default value is: YES. + +REFERENCES_LINK_SOURCE = YES + +# If SOURCE_TOOLTIPS is enabled (the default) then hovering a hyperlink in the +# source code will show a tooltip with additional information such as prototype, +# brief description and links to the definition and documentation. Since this +# will make the HTML file larger and loading of large files a bit slower, you +# can opt to disable this feature. +# The default value is: YES. +# This tag requires that the tag SOURCE_BROWSER is set to YES. + +SOURCE_TOOLTIPS = YES + +# If the USE_HTAGS tag is set to YES then the references to source code will +# point to the HTML generated by the htags(1) tool instead of doxygen built-in +# source browser. The htags tool is part of GNU's global source tagging system +# (see http://www.gnu.org/software/global/global.html). You will need version +# 4.8.6 or higher. +# +# To use it do the following: +# - Install the latest version of global +# - Enable SOURCE_BROWSER and USE_HTAGS in the config file +# - Make sure the INPUT points to the root of the source tree +# - Run doxygen as normal +# +# Doxygen will invoke htags (and that will in turn invoke gtags), so these +# tools must be available from the command line (i.e. in the search path). +# +# The result: instead of the source browser generated by doxygen, the links to +# source code will now point to the output of htags. +# The default value is: NO. +# This tag requires that the tag SOURCE_BROWSER is set to YES. + +USE_HTAGS = NO + +# If the VERBATIM_HEADERS tag is set the YES then doxygen will generate a +# verbatim copy of the header file for each class for which an include is +# specified. Set to NO to disable this. +# See also: Section \class. +# The default value is: YES. + +VERBATIM_HEADERS = YES + +# If the CLANG_ASSISTED_PARSING tag is set to YES then doxygen will use the +# clang parser (see: http://clang.llvm.org/) for more accurate parsing at the +# cost of reduced performance. This can be particularly helpful with template +# rich C++ code for which doxygen's built-in parser lacks the necessary type +# information. +# Note: The availability of this option depends on whether or not doxygen was +# generated with the -Duse-libclang=ON option for CMake. +# The default value is: NO. + +CLANG_ASSISTED_PARSING = NO + +# If clang assisted parsing is enabled you can provide the compiler with command +# line options that you would normally use when invoking the compiler. Note that +# the include paths will already be set by doxygen for the files and directories +# specified with INPUT and INCLUDE_PATH. +# This tag requires that the tag CLANG_ASSISTED_PARSING is set to YES. + +CLANG_OPTIONS = + +#--------------------------------------------------------------------------- +# Configuration options related to the alphabetical class index +#--------------------------------------------------------------------------- + +# If the ALPHABETICAL_INDEX tag is set to YES, an alphabetical index of all +# compounds will be generated. Enable this if the project contains a lot of +# classes, structs, unions or interfaces. +# The default value is: YES. + +ALPHABETICAL_INDEX = YES + +# The COLS_IN_ALPHA_INDEX tag can be used to specify the number of columns in +# which the alphabetical index list will be split. +# Minimum value: 1, maximum value: 20, default value: 5. +# This tag requires that the tag ALPHABETICAL_INDEX is set to YES. + +COLS_IN_ALPHA_INDEX = 5 + +# In case all classes in a project start with a common prefix, all classes will +# be put under the same header in the alphabetical index. The IGNORE_PREFIX tag +# can be used to specify a prefix (or a list of prefixes) that should be ignored +# while generating the index headers. +# This tag requires that the tag ALPHABETICAL_INDEX is set to YES. + +IGNORE_PREFIX = + +#--------------------------------------------------------------------------- +# Configuration options related to the HTML output +#--------------------------------------------------------------------------- + +# If the GENERATE_HTML tag is set to YES, doxygen will generate HTML output +# The default value is: YES. + +GENERATE_HTML = YES + +# The HTML_OUTPUT tag is used to specify where the HTML docs will be put. If a +# relative path is entered the value of OUTPUT_DIRECTORY will be put in front of +# it. +# The default directory is: html. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_OUTPUT = html + +# The HTML_FILE_EXTENSION tag can be used to specify the file extension for each +# generated HTML page (for example: .htm, .php, .asp). +# The default value is: .html. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_FILE_EXTENSION = .html + +# The HTML_HEADER tag can be used to specify a user-defined HTML header file for +# each generated HTML page. If the tag is left blank doxygen will generate a +# standard header. +# +# To get valid HTML the header file that includes any scripts and style sheets +# that doxygen needs, which is dependent on the configuration options used (e.g. +# the setting GENERATE_TREEVIEW). It is highly recommended to start with a +# default header using +# doxygen -w html new_header.html new_footer.html new_stylesheet.css +# YourConfigFile +# and then modify the file new_header.html. See also section "Doxygen usage" +# for information on how to generate the default header that doxygen normally +# uses. +# Note: The header is subject to change so you typically have to regenerate the +# default header when upgrading to a newer version of doxygen. For a description +# of the possible markers and block names see the documentation. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_HEADER = + +# The HTML_FOOTER tag can be used to specify a user-defined HTML footer for each +# generated HTML page. If the tag is left blank doxygen will generate a standard +# footer. See HTML_HEADER for more information on how to generate a default +# footer and what special commands can be used inside the footer. See also +# section "Doxygen usage" for information on how to generate the default footer +# that doxygen normally uses. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_FOOTER = + +# The HTML_STYLESHEET tag can be used to specify a user-defined cascading style +# sheet that is used by each HTML page. It can be used to fine-tune the look of +# the HTML output. If left blank doxygen will generate a default style sheet. +# See also section "Doxygen usage" for information on how to generate the style +# sheet that doxygen normally uses. +# Note: It is recommended to use HTML_EXTRA_STYLESHEET instead of this tag, as +# it is more robust and this tag (HTML_STYLESHEET) will in the future become +# obsolete. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_STYLESHEET = + +# The HTML_EXTRA_STYLESHEET tag can be used to specify additional user-defined +# cascading style sheets that are included after the standard style sheets +# created by doxygen. Using this option one can overrule certain style aspects. +# This is preferred over using HTML_STYLESHEET since it does not replace the +# standard style sheet and is therefore more robust against future updates. +# Doxygen will copy the style sheet files to the output directory. +# Note: The order of the extra style sheet files is of importance (e.g. the last +# style sheet in the list overrules the setting of the previous ones in the +# list). For an example see the documentation. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_EXTRA_STYLESHEET = + +# The HTML_EXTRA_FILES tag can be used to specify one or more extra images or +# other source files which should be copied to the HTML output directory. Note +# that these files will be copied to the base HTML output directory. Use the +# $relpath^ marker in the HTML_HEADER and/or HTML_FOOTER files to load these +# files. In the HTML_STYLESHEET file, use the file name only. Also note that the +# files will be copied as-is; there are no commands or markers available. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_EXTRA_FILES = + +# The HTML_COLORSTYLE_HUE tag controls the color of the HTML output. Doxygen +# will adjust the colors in the style sheet and background images according to +# this color. Hue is specified as an angle on a colorwheel, see +# http://en.wikipedia.org/wiki/Hue for more information. For instance the value +# 0 represents red, 60 is yellow, 120 is green, 180 is cyan, 240 is blue, 300 +# purple, and 360 is red again. +# Minimum value: 0, maximum value: 359, default value: 220. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_COLORSTYLE_HUE = 220 + +# The HTML_COLORSTYLE_SAT tag controls the purity (or saturation) of the colors +# in the HTML output. For a value of 0 the output will use grayscales only. A +# value of 255 will produce the most vivid colors. +# Minimum value: 0, maximum value: 255, default value: 100. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_COLORSTYLE_SAT = 100 + +# The HTML_COLORSTYLE_GAMMA tag controls the gamma correction applied to the +# luminance component of the colors in the HTML output. Values below 100 +# gradually make the output lighter, whereas values above 100 make the output +# darker. The value divided by 100 is the actual gamma applied, so 80 represents +# a gamma of 0.8, The value 220 represents a gamma of 2.2, and 100 does not +# change the gamma. +# Minimum value: 40, maximum value: 240, default value: 80. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_COLORSTYLE_GAMMA = 80 + +# If the HTML_TIMESTAMP tag is set to YES then the footer of each generated HTML +# page will contain the date and time when the page was generated. Setting this +# to YES can help to show when doxygen was last run and thus if the +# documentation is up to date. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_TIMESTAMP = NO + +# If the HTML_DYNAMIC_SECTIONS tag is set to YES then the generated HTML +# documentation will contain sections that can be hidden and shown after the +# page has loaded. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_DYNAMIC_SECTIONS = NO + +# With HTML_INDEX_NUM_ENTRIES one can control the preferred number of entries +# shown in the various tree structured indices initially; the user can expand +# and collapse entries dynamically later on. Doxygen will expand the tree to +# such a level that at most the specified number of entries are visible (unless +# a fully collapsed tree already exceeds this amount). So setting the number of +# entries 1 will produce a full collapsed tree by default. 0 is a special value +# representing an infinite number of entries and will result in a full expanded +# tree by default. +# Minimum value: 0, maximum value: 9999, default value: 100. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_INDEX_NUM_ENTRIES = 100 + +# If the GENERATE_DOCSET tag is set to YES, additional index files will be +# generated that can be used as input for Apple's Xcode 3 integrated development +# environment (see: http://developer.apple.com/tools/xcode/), introduced with +# OSX 10.5 (Leopard). To create a documentation set, doxygen will generate a +# Makefile in the HTML output directory. Running make will produce the docset in +# that directory and running make install will install the docset in +# ~/Library/Developer/Shared/Documentation/DocSets so that Xcode will find it at +# startup. See http://developer.apple.com/tools/creatingdocsetswithdoxygen.html +# for more information. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +GENERATE_DOCSET = NO + +# This tag determines the name of the docset feed. A documentation feed provides +# an umbrella under which multiple documentation sets from a single provider +# (such as a company or product suite) can be grouped. +# The default value is: Doxygen generated docs. +# This tag requires that the tag GENERATE_DOCSET is set to YES. + +DOCSET_FEEDNAME = "Doxygen generated docs" + +# This tag specifies a string that should uniquely identify the documentation +# set bundle. This should be a reverse domain-name style string, e.g. +# com.mycompany.MyDocSet. Doxygen will append .docset to the name. +# The default value is: org.doxygen.Project. +# This tag requires that the tag GENERATE_DOCSET is set to YES. + +DOCSET_BUNDLE_ID = org.doxygen.Project + +# The DOCSET_PUBLISHER_ID tag specifies a string that should uniquely identify +# the documentation publisher. This should be a reverse domain-name style +# string, e.g. com.mycompany.MyDocSet.documentation. +# The default value is: org.doxygen.Publisher. +# This tag requires that the tag GENERATE_DOCSET is set to YES. + +DOCSET_PUBLISHER_ID = org.doxygen.Publisher + +# The DOCSET_PUBLISHER_NAME tag identifies the documentation publisher. +# The default value is: Publisher. +# This tag requires that the tag GENERATE_DOCSET is set to YES. + +DOCSET_PUBLISHER_NAME = Publisher + +# If the GENERATE_HTMLHELP tag is set to YES then doxygen generates three +# additional HTML index files: index.hhp, index.hhc, and index.hhk. The +# index.hhp is a project file that can be read by Microsoft's HTML Help Workshop +# (see: http://www.microsoft.com/en-us/download/details.aspx?id=21138) on +# Windows. +# +# The HTML Help Workshop contains a compiler that can convert all HTML output +# generated by doxygen into a single compiled HTML file (.chm). Compiled HTML +# files are now used as the Windows 98 help format, and will replace the old +# Windows help format (.hlp) on all Windows platforms in the future. Compressed +# HTML files also contain an index, a table of contents, and you can search for +# words in the documentation. The HTML workshop also contains a viewer for +# compressed HTML files. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +GENERATE_HTMLHELP = NO + +# The CHM_FILE tag can be used to specify the file name of the resulting .chm +# file. You can add a path in front of the file if the result should not be +# written to the html output directory. +# This tag requires that the tag GENERATE_HTMLHELP is set to YES. + +CHM_FILE = + +# The HHC_LOCATION tag can be used to specify the location (absolute path +# including file name) of the HTML help compiler (hhc.exe). If non-empty, +# doxygen will try to run the HTML help compiler on the generated index.hhp. +# The file has to be specified with full path. +# This tag requires that the tag GENERATE_HTMLHELP is set to YES. + +HHC_LOCATION = + +# The GENERATE_CHI flag controls if a separate .chi index file is generated +# (YES) or that it should be included in the master .chm file (NO). +# The default value is: NO. +# This tag requires that the tag GENERATE_HTMLHELP is set to YES. + +GENERATE_CHI = NO + +# The CHM_INDEX_ENCODING is used to encode HtmlHelp index (hhk), content (hhc) +# and project file content. +# This tag requires that the tag GENERATE_HTMLHELP is set to YES. + +CHM_INDEX_ENCODING = + +# The BINARY_TOC flag controls whether a binary table of contents is generated +# (YES) or a normal table of contents (NO) in the .chm file. Furthermore it +# enables the Previous and Next buttons. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTMLHELP is set to YES. + +BINARY_TOC = NO + +# The TOC_EXPAND flag can be set to YES to add extra items for group members to +# the table of contents of the HTML help documentation and to the tree view. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTMLHELP is set to YES. + +TOC_EXPAND = NO + +# If the GENERATE_QHP tag is set to YES and both QHP_NAMESPACE and +# QHP_VIRTUAL_FOLDER are set, an additional index file will be generated that +# can be used as input for Qt's qhelpgenerator to generate a Qt Compressed Help +# (.qch) of the generated HTML documentation. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +GENERATE_QHP = NO + +# If the QHG_LOCATION tag is specified, the QCH_FILE tag can be used to specify +# the file name of the resulting .qch file. The path specified is relative to +# the HTML output folder. +# This tag requires that the tag GENERATE_QHP is set to YES. + +QCH_FILE = + +# The QHP_NAMESPACE tag specifies the namespace to use when generating Qt Help +# Project output. For more information please see Qt Help Project / Namespace +# (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#namespace). +# The default value is: org.doxygen.Project. +# This tag requires that the tag GENERATE_QHP is set to YES. + +QHP_NAMESPACE = org.doxygen.Project + +# The QHP_VIRTUAL_FOLDER tag specifies the namespace to use when generating Qt +# Help Project output. For more information please see Qt Help Project / Virtual +# Folders (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#virtual- +# folders). +# The default value is: doc. +# This tag requires that the tag GENERATE_QHP is set to YES. + +QHP_VIRTUAL_FOLDER = doc + +# If the QHP_CUST_FILTER_NAME tag is set, it specifies the name of a custom +# filter to add. For more information please see Qt Help Project / Custom +# Filters (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#custom- +# filters). +# This tag requires that the tag GENERATE_QHP is set to YES. + +QHP_CUST_FILTER_NAME = + +# The QHP_CUST_FILTER_ATTRS tag specifies the list of the attributes of the +# custom filter to add. For more information please see Qt Help Project / Custom +# Filters (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#custom- +# filters). +# This tag requires that the tag GENERATE_QHP is set to YES. + +QHP_CUST_FILTER_ATTRS = + +# The QHP_SECT_FILTER_ATTRS tag specifies the list of the attributes this +# project's filter section matches. Qt Help Project / Filter Attributes (see: +# http://qt-project.org/doc/qt-4.8/qthelpproject.html#filter-attributes). +# This tag requires that the tag GENERATE_QHP is set to YES. + +QHP_SECT_FILTER_ATTRS = + +# The QHG_LOCATION tag can be used to specify the location of Qt's +# qhelpgenerator. If non-empty doxygen will try to run qhelpgenerator on the +# generated .qhp file. +# This tag requires that the tag GENERATE_QHP is set to YES. + +QHG_LOCATION = + +# If the GENERATE_ECLIPSEHELP tag is set to YES, additional index files will be +# generated, together with the HTML files, they form an Eclipse help plugin. To +# install this plugin and make it available under the help contents menu in +# Eclipse, the contents of the directory containing the HTML and XML files needs +# to be copied into the plugins directory of eclipse. The name of the directory +# within the plugins directory should be the same as the ECLIPSE_DOC_ID value. +# After copying Eclipse needs to be restarted before the help appears. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +GENERATE_ECLIPSEHELP = NO + +# A unique identifier for the Eclipse help plugin. When installing the plugin +# the directory name containing the HTML and XML files should also have this +# name. Each documentation set should have its own identifier. +# The default value is: org.doxygen.Project. +# This tag requires that the tag GENERATE_ECLIPSEHELP is set to YES. + +ECLIPSE_DOC_ID = org.doxygen.Project + +# If you want full control over the layout of the generated HTML pages it might +# be necessary to disable the index and replace it with your own. The +# DISABLE_INDEX tag can be used to turn on/off the condensed index (tabs) at top +# of each HTML page. A value of NO enables the index and the value YES disables +# it. Since the tabs in the index contain the same information as the navigation +# tree, you can set this option to YES if you also set GENERATE_TREEVIEW to YES. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +DISABLE_INDEX = NO + +# The GENERATE_TREEVIEW tag is used to specify whether a tree-like index +# structure should be generated to display hierarchical information. If the tag +# value is set to YES, a side panel will be generated containing a tree-like +# index structure (just like the one that is generated for HTML Help). For this +# to work a browser that supports JavaScript, DHTML, CSS and frames is required +# (i.e. any modern browser). Windows users are probably better off using the +# HTML help feature. Via custom style sheets (see HTML_EXTRA_STYLESHEET) one can +# further fine-tune the look of the index. As an example, the default style +# sheet generated by doxygen has an example that shows how to put an image at +# the root of the tree instead of the PROJECT_NAME. Since the tree basically has +# the same information as the tab index, you could consider setting +# DISABLE_INDEX to YES when enabling this option. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +GENERATE_TREEVIEW = NO + +# The ENUM_VALUES_PER_LINE tag can be used to set the number of enum values that +# doxygen will group on one line in the generated HTML documentation. +# +# Note that a value of 0 will completely suppress the enum values from appearing +# in the overview section. +# Minimum value: 0, maximum value: 20, default value: 4. +# This tag requires that the tag GENERATE_HTML is set to YES. + +ENUM_VALUES_PER_LINE = 4 + +# If the treeview is enabled (see GENERATE_TREEVIEW) then this tag can be used +# to set the initial width (in pixels) of the frame in which the tree is shown. +# Minimum value: 0, maximum value: 1500, default value: 250. +# This tag requires that the tag GENERATE_HTML is set to YES. + +TREEVIEW_WIDTH = 250 + +# If the EXT_LINKS_IN_WINDOW option is set to YES, doxygen will open links to +# external symbols imported via tag files in a separate window. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +EXT_LINKS_IN_WINDOW = NO + +# Use this tag to change the font size of LaTeX formulas included as images in +# the HTML documentation. When you change the font size after a successful +# doxygen run you need to manually remove any form_*.png images from the HTML +# output directory to force them to be regenerated. +# Minimum value: 8, maximum value: 50, default value: 10. +# This tag requires that the tag GENERATE_HTML is set to YES. + +FORMULA_FONTSIZE = 10 + +# Use the FORMULA_TRANPARENT tag to determine whether or not the images +# generated for formulas are transparent PNGs. Transparent PNGs are not +# supported properly for IE 6.0, but are supported on all modern browsers. +# +# Note that when changing this option you need to delete any form_*.png files in +# the HTML output directory before the changes have effect. +# The default value is: YES. +# This tag requires that the tag GENERATE_HTML is set to YES. + +FORMULA_TRANSPARENT = YES + +# Enable the USE_MATHJAX option to render LaTeX formulas using MathJax (see +# http://www.mathjax.org) which uses client side Javascript for the rendering +# instead of using pre-rendered bitmaps. Use this if you do not have LaTeX +# installed or if you want to formulas look prettier in the HTML output. When +# enabled you may also need to install MathJax separately and configure the path +# to it using the MATHJAX_RELPATH option. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +USE_MATHJAX = NO + +# When MathJax is enabled you can set the default output format to be used for +# the MathJax output. See the MathJax site (see: +# http://docs.mathjax.org/en/latest/output.html) for more details. +# Possible values are: HTML-CSS (which is slower, but has the best +# compatibility), NativeMML (i.e. MathML) and SVG. +# The default value is: HTML-CSS. +# This tag requires that the tag USE_MATHJAX is set to YES. + +MATHJAX_FORMAT = HTML-CSS + +# When MathJax is enabled you need to specify the location relative to the HTML +# output directory using the MATHJAX_RELPATH option. The destination directory +# should contain the MathJax.js script. For instance, if the mathjax directory +# is located at the same level as the HTML output directory, then +# MATHJAX_RELPATH should be ../mathjax. The default value points to the MathJax +# Content Delivery Network so you can quickly see the result without installing +# MathJax. However, it is strongly recommended to install a local copy of +# MathJax from http://www.mathjax.org before deployment. +# The default value is: http://cdn.mathjax.org/mathjax/latest. +# This tag requires that the tag USE_MATHJAX is set to YES. + +MATHJAX_RELPATH = http://cdn.mathjax.org/mathjax/latest + +# The MATHJAX_EXTENSIONS tag can be used to specify one or more MathJax +# extension names that should be enabled during MathJax rendering. For example +# MATHJAX_EXTENSIONS = TeX/AMSmath TeX/AMSsymbols +# This tag requires that the tag USE_MATHJAX is set to YES. + +MATHJAX_EXTENSIONS = + +# The MATHJAX_CODEFILE tag can be used to specify a file with javascript pieces +# of code that will be used on startup of the MathJax code. See the MathJax site +# (see: http://docs.mathjax.org/en/latest/output.html) for more details. For an +# example see the documentation. +# This tag requires that the tag USE_MATHJAX is set to YES. + +MATHJAX_CODEFILE = + +# When the SEARCHENGINE tag is enabled doxygen will generate a search box for +# the HTML output. The underlying search engine uses javascript and DHTML and +# should work on any modern browser. Note that when using HTML help +# (GENERATE_HTMLHELP), Qt help (GENERATE_QHP), or docsets (GENERATE_DOCSET) +# there is already a search function so this one should typically be disabled. +# For large projects the javascript based search engine can be slow, then +# enabling SERVER_BASED_SEARCH may provide a better solution. It is possible to +# search using the keyboard; to jump to the search box use + S +# (what the is depends on the OS and browser, but it is typically +# , /