$Id: low-level-x86.txt,v 1.2 2004/07/11 17:12:28 mikpe Exp $

PERFCTRS X86 LOW-LEVEL API

==========================

See low-level-api.txt for the common low-level API.

This document only describes x86-specific behaviour.

For detailed hardware control register layouts, see

the manufacturers' documentation.

Contents

========

- Supported processors

- Contents of <asm-i386/perfctr.h>

- Processor-specific Notes

- Implementation Notes

Supported processors

====================

- Intel P5, P5MMX, P6, P4.

- AMD K7, K8. (P6 clones, with some changes)

- Cyrix 6x86MX, MII, and III. (good P5 clones)

- Centaur WinChip C6, 2, and 3. (bad P5 clones)

- VIA C3. (bad P6 clone)

- Any generic x86 with a TSC.

Contents of <asm-i386/perfctr.h>

================================

"struct perfctr_sum_ctrs"

-------------------------

struct perfctr_sum_ctrs {

	unsigned long long tsc;

	unsigned long long pmc[18];

};

The pmc[] array has room for 18 counters.

"struct perfctr_cpu_control"

----------------------------

struct perfctr_cpu_control {

	unsigned int tsc_on;

	unsigned int nractrs;		/* # of a-mode counters */

	unsigned int nrictrs;		/* # of i-mode counters */

	unsigned int pmc_map[18];

	unsigned int evntsel[18];	/* one per counter, even on P5 */

	struct {

		unsigned int escr[18];

		unsigned int pebs_enable;	/* for replay tagging */

		unsigned int pebs_matrix_vert;	/* for replay tagging */

	} p4;

	int ireset[18];			/* < 0, for i-mode counters */

	unsigned int _reserved1;

	unsigned int _reserved2;

	unsigned int _reserved3;

	unsigned int _reserved4;

};

The per-counter arrays have room for 18 elements.

ireset[] values must be negative, since overflow occurs on

the negative-to-non-negative transition.

The p4 sub-struct contains P4-specific control data:

- escr[]: the control data to write to the ESCR register

  associatied with the counter

- pebs_enable: the control data to write to the PEBS_ENABLE MSR

- pebs_matrix_vert: the control data to write to the

  PEBS_MATRIX_VERT MSR

"struct perfctr_cpu_state"

--------------------------

struct perfctr_cpu_state {

	unsigned int cstatus;

	struct {	/* k1 is opaque in the user ABI */

		unsigned int id;

		int isuspend_cpu;

	} k1;

	/* The two tsc fields must be inlined. Placing them in a

	   sub-struct causes unwanted internal padding on x86-64. */

	unsigned int tsc_start;

	unsigned long long tsc_sum;

	struct {

		unsigned int map;

		unsigned int start;

		unsigned long long sum;

	} pmc[18];	/* the size is not part of the user ABI */

#ifdef __KERNEL__

	struct perfctr_cpu_control control;

	unsigned int p4_escr_map[18];

#endif

};

The k1 sub-struct is used by the low-level driver for

caching purposes. "id" identifies the control data, and

"isuspend_cpu" identifies the CPU on which the i-mode

counters were last suspended.

The pmc[] array has room for 18 elements.

p4_escr_map[] is computed from control by the low-level driver,

and provides the MSR number for the counter's associated ESCR.

User-space overflow signal handler items

----------------------------------------

#ifdef __KERNEL__

#define SI_PMC_OVF	(__SI_FAULT|'P')

#else

#define SI_PMC_OVF	('P')

#endif

#define si_pmc_ovf_mask	_sifields._pad[0]

Kernel-internal API

-------------------

In perfctr_cpu_update_control(), the is_global parameter controls

whether monitoring the other thread (T1) on HT P4s is permitted

or not. On other processors the parameter is ignored.

SMP kernels define CONFIG_PERFCTR_CPUS_FORBIDDEN_MASK and

"extern cpumask_t perfctr_cpus_forbidden_mask;".

On HT P4s, resource conflicts can occur because both threads

(T0 and T1) in a processor share the same perfctr registers.

To prevent conflicts, only thread 0 in each processor is allowed

to access the counters. perfctr_cpus_forbidden_mask contains the

smp_processor_id()s of each processor's thread 1, and it is the

responsibility of the high-level driver to ensure that it never

accesses the perfctr state from a forbidden thread.

Overflow interrupt handling requires local APIC support in the kernel.

Processor-specific Notes

========================

General

-------

pmc_map[] contains a counter number, as used by the RDPMC instruction.

It never contains an MSR number.

Counters are 32, 40, or 48 bits wide. The driver always only

reads the low 32 bits. This avoids performance issues, and

errata on some processors.

Writing to counters or their control registers tends to be

very expensive. This is why a-mode counters only use read

operations on the counter registers. Caching of control

register contents is done to avoid writing them. "Suspend CPU"

is recorded for i-mode counters to avoid writing the counter

registers when the counters are resumed (their control

registers must be written at both suspend and resume, however).

Some processors are unable to stop the counters (Centaur/VIA),

and some are unable to reinitialise them to arbitrary values (P6).

Storing the counters' total counts in the hardware counters

would break as soon as context-switches occur. This is another

reason why the accumulate-differences method for maintaining the

counter values is used.

Intel P5

--------

The hardware stores both counters' control data in a single

control register, the CESR MSR. The evntsel values are

limited to 16 bits each, and are combined by the low-level

driver to form the value for the CESR. Apart from that,

the evntsel values are direct images of the CESR.

Bits 0xFE00 in an evntsel value are reserved.

At least one evntsel CPL bit (0x00C0) must be set.

For Cyrix' P5 clones, evntsel bits 0xFA00  are reserved.

For Centaur's P5 clones, evntsel bits 0xFF00 are reserved.

It has no CPL bits to set. The TSC is broken and cannot be used.

Intel P6

--------

The evntsel values are mapped directly onto the counters'

EVNTSEL control registers.

The global enable bit (22) in EVNTSEL0 must be set. That bit is

reserved in EVNTSEL1.

Bits 21 and 19 (0x00280000) in each evntsel are reserved.

For an i-mode counter, bit 20 (0x00100000) of its evntsel must be

set. For a-mode counters, that bit must not be set.

Hardware quirk: Counters are 40 bits wide, but writing to a

counter only writes the low 32 bits: remaining bits are

sign-extended from bit 31.

AMD K7/K8

---------

Similar to Intel P6. The main difference is that each evntsel has

its own enable bit, which must be set.

VIA C3

------

Superficially similar to Intel P6, but only PERFCTR1/EVNTSEL1

are programmable. pmc_map[0] must be 1, if nractrs == 1.

Bits 0xFFFFFE00 in the evntsel are reserved. There are no auxiliary

control bits to set.

Generic

-------

Only permits TSC sampling, with tsc_on == 1 and nractrs == nrictrs == 0

in the control data.

Intel P4

--------

For each counter, its evntsel[] value is mapped onto its CCCR

control register, and its p4.escr[] value is mapped onto its

associated ESCR control register.

The ESCR register number is computed from the hardware counter

number (from pmc_map[]) and the ESCR SELECT field in the CCCR,

and is cached in p4_escr_map[].

pmc_map[] contains the value to pass to RDPMC when reading the

counter. It is strongly recommended to set bit 31 (fast rdpmc).

In each evntsel/CCCR value:

- the OVF, OVF_PMI_T1 and hardware-reserved bits (0xB80007FF)

  are reserved and must not be set

- bit 11 (EXTENDED_CASCADE) is only permitted on P4 models >= 2,

  and for counters 12 and 15-17

- bits 16 and 17 (ACTIVE_THREAD) must both be set on non-HT processors

- at least one of bits 12 (ENABLE), 30 (CASCADE), or 11 (EXTENDED_CASCADE)

  must be set

- bit 26 (OVF_PMI_T0) must be clear for a-mode counters, and set

  for i-mode counters; if bit 25 (FORCE_OVF) also is set, then

  the corresponding ireset[] value must be exactly -1

In each p4.escr[] value:

- bit 32 is reserved and must not be set

- the CPL_T1 field (bits 0 and 1) must be zero except on HT processors

  when global-mode counters are used

- IQ_ESCR0 and IQ_ESCR1 can only be used on P4 models <= 2

PEBS is not supported, but the replay tagging bits in PEBS_ENABLE

and PEBS_MATRIX_VERT may be used.

If p4.pebs_enable is zero, then p4.pebs_matrix_vert must also be zero.

If p4.pebs_enable is non-zero:

- only bits 24, 10, 9, 2, 1, and 0 may be set; note that in contrast

  to Intel's documentation, bit 25 (ENABLE_PEBS_MY_THR) is not needed

  and must not be set

- bit 24 (UOP_TAG) must be set

- at least one of bits 10, 9, 2, 1, or 0 must be set

- in p4.pebs_matrix_vert, all bits except 1 and 0 must be clear,

  and at least one of bits 1 and 0 must be set

Implementation Notes

====================

Caching

-------

Each 'struct perfctr_cpu_state' contains two cache-related fields:

- 'id': a unique identifier for the control data contents

- 'isuspend_cpu': the identity of the CPU on which a state containing

  interrupt-mode counters was last suspended

To this the driver adds a per-CPU cache, recording:

- the 'id' of the control data currently in that CPU

- the current contents of each control register

When perfctr_cpu_update_control() has validated the new control data,

it also updates the id field.

The driver's internal 'write_control' function, called from the

perfctr_cpu_resume() API function, first checks if the state's id

matches that of the CPU's cache, and if so, returns. Otherwise

it checks each control register in the state and updates those

that do not match the cache. Finally, it writes the state's id

to the cache. Tests on various x86 processor types have shown that

MSR writes are very expensive: the purpose of these cache checks

is to avoid MSR writes whenever possible.

Unlike accumulation-mode counters, interrupt-mode counters must be

physically stopped when suspended, primilarly to avoid overflow

interrupts in contexts not expecting them, and secondarily to avoid

increments to the counters themselves (see below).

When suspending interrupt-mode counters, the driver:

- records the CPU identity in the per-CPU cache

- stops each interrupt-mode counter by disabling its control register

- lets the cache and state id values remain the same

Later, when resuming interrupt-mode counters, the driver:

- if the state and cache id values match:

  * the cache id is cleared, to force a reload of the control

    registers stopped at suspend (see below)

  * if the state's "suspend" CPU identity matches the current CPU,

    the counter registers are still valid, and the procedure returns

- if the procedure did not return above, it then loops over each

  interrupt-mode counter:

  * the counter's control register is physically disabled, unless

    the cache indicates that it already is disabled; this is necessary

    to prevent premature events and overflow interrupts if the CPU's

    registers previously belonged to some other state

  * then the counter register itself is restored

After this interrupt-mode specific resume code is complete, the

driver continues by calling 'write_control' as described above.

The state and cache ids will not match, forcing write_control to

reload the disabled interrupt-mode control registers.

Call-site Backpatching

----------------------

The x86 family of processors is quite diverse in how their

performance counters work and are accessed. There are three

main designs (P5, P6, and P4) with several variations.

To handle this the processor type detection and initialisation

code sets up a number of function pointers to point to the

correct procedures for the actual CPU type.

Calls via function pointers are more expensive than direct calls,

so the driver actually performs direct calls to wrappers that

backpatch the original call sites to instead call the actual

CPU-specific functions in the future.

Unsynchronised code backpatching in SMP systems doesn't work

on Intel P6 processors due to an erratum, so the driver performs

a "finalise backpatching" step after the CPU-specific function

pointers have been set up. This step invokes the API procedures

on a temporary state object, set up to force every backpatchable

call site to be invoked and adjusted.

Several low-level API procedures are called in the context-switch

path by the per-process perfctrs kernel extension, which motivates

the efforts to reduce runtime overheads as much as possible.

Overflow Interrupts

-------------------

The x86 hardware enables overflow interrupts via the local

APIC's LVTPC entry, which is only present in P6/K7/K8/P4.

The low-level driver supports overflow interrupts as follows:

- It reserves a local APIC vector, 0xee, as LOCAL_PERFCTR_VECTOR.

- It adds a local APIC exception handler to entry.S, which

  invokes the driver's smp_perfctr_interrupt() procedure.

- It adds code to i8259.c to bind the LOCAL_PERFCTR_VECTOR

  interrupt gate to the exception handler in entry.S.

- During processor type detection, it records whether the

  processor supports the local APIC, and sets up function pointers

  for the suspend and resume operations on interrupt-mode counters.

- When the low-level driver is activated, it enables overflow

  interrupts by writing LOCAL_PERFCTR_VECTOR to each CPU's APIC_LVTPC.

- Overflow interrupts now end up in smp_perfctr_interrupt(), which

  ACKs the interrupt and invokes the interrupt handler installed

  by the high-level service/driver.

- When the low-level driver is deactivated, it disables overflow

  interrupts by masking APIC_LVTPC in each CPU. It then releases

  the local APIC back to the NMI watchdog.

At compile-time, the low-level driver indicates overflow interrupt

support by enabling CONFIG_PERFCTR_INTERRUPT_SUPPORT. If the feature

is also available at runtime, it sets the PERFCTR_FEATURE_PCINT flag

in the perfctr_info object.