:compat-mode: legacy

= Fencing =

////

We prefer [[ch-fencing]], but older versions of asciidoc don't deal well

with that construct for chapter headings

////

anchor:ch-fencing[Chapter 6, Fencing]

indexterm:[Fencing, Configuration]

indexterm:[STONITH, Configuration]

== What Is Fencing? ==

'Fencing' is the ability to make a node unable to run resources, even when that

node is unresponsive to cluster commands.

Fencing is also known as 'STONITH', an acronym for "Shoot The Other Node In The

Head", since the most common fencing method is cutting power to the node.

Another method is "fabric fencing", cutting the node's access to some

capability required to run resources (such as network access or a shared disk).

== Why Is Fencing Necessary? ==

Fencing protects your data from being corrupted by malfunctioning nodes or

unintentional concurrent access to shared resources.

Fencing protects against the "split brain" failure scenario, where cluster

nodes have lost the ability to reliably communicate with each other but are

still able to run resources. If the cluster just assumed that uncommunicative

nodes were down, then multiple instances of a resource could be started on

different nodes.

The effect of split brain depends on the resource type. For example, an IP

address brought up on two hosts on a network will cause packets to randomly be

sent to one or the other host, rendering the IP useless. For a database or

clustered file system, the effect could be much more severe, causing data

corruption or divergence.

Fencing also is used when a resource cannot otherwise be stopped. If a failed

resource fails to stop, it cannot be recovered elsewhere. Fencing the

resource's node is the only way to ensure the resource is recoverable.

Users may also configure the +on-fail+ property of any resource operation to

+fencing+, in which case the cluster will fence the resource's node if the

operation fails.

== Fence Devices ==

A 'fence device' (or 'fencing device') is a special type of resource that

provides the means to fence a node.

Examples of fencing devices include intelligent power switches and IPMI devices

that accept SNMP commands to cut power to a node, and iSCSI controllers that

allow SCSI reservations to be used to cut a node's access to a shared disk.

Since fencing devices will be used to recover from loss of networking

connectivity to other nodes, it is essential that they do not rely on the same

network as the cluster itself, otherwise that network becomes a single point of

failure.

Since loss of a node due to power outage is indistinguishable from loss of

network connectivity to that node, it is also essential that at least one fence

device for a node does not share power with that node. For example, an on-board

IPMI controller that shares power with its host should not be used as the sole

fencing device for that host.

Since fencing is used to isolate malfunctioning nodes, no fence device should

rely on its target functioning properly. This includes, for example, devices

that ssh into a node and issue a shutdown command (such devices might be

suitable for testing, but never for production).

== Fence Agents ==

A 'fence agent' (or 'fencing agent') is a +stonith+-class resource agent.

The fence agent standard provides commands (such as +off+ and +reboot+) that

the cluster can use to fence nodes. As with other resource agent classes,

this allows a layer of abstraction so that Pacemaker doesn't need any knowledge

about specific fencing technologies -- that knowledge is isolated in the agent.

== When a Fence Device Can Be Used ==

Fencing devices do not actually "run" like most services. Typically, they just

provide an interface for sending commands to an external device.

Additionally, fencing may be initiated by Pacemaker, by other cluster-aware software

such as DRBD or DLM, or manually by an administrator, at any point in the

cluster life cycle, including before any resources have been started.

To accommodate this, Pacemaker does not require the fence device resource to be

"started" in order to be used. Whether a fence device is started or not

determines whether a node runs any recurring monitor for the device, and gives

the node a slight preference for being chosen to execute fencing using that

device.

By default, any node can execute any fencing device. If a fence device is

disabled by setting its +target-role+ to Stopped, then no node can use that

device. If mandatory location constraints prevent a specific node from

"running" a fence device, then that node will never be chosen to execute

fencing using the device. A node may fence itself, but the cluster will choose

that only if no other nodes can do the fencing.

A common configuration scenario is to have one fence device per target node.

In such a case, users often configure anti-location constraints so that

the target node does not monitor its own device. The best practice is to make

the constraint optional (i.e. a finite negative score rather than +-INFINITY+),

so that the node can fence itself if no other nodes can.

== Limitations of Fencing Resources ==

Fencing resources have certain limitations that other resource classes don't:

* They may have only one set of meta-attributes and one set of instance

  attributes.

* If <<ch-rules,rules>> are used to determine fencing resource options, these

  may only be evaluated when first read, meaning that later changes to the

  rules will have no effect. Therefore, it is better to avoid confusion and not

  use rules at all with fencing resources.

These limitations could be revisited if there is significant user demand.

== Special Options for Fencing Resources ==

The table below lists special instance attributes that may be set for any

fencing resource ('not' meta-attributes, even though they are interpreted by

pacemaker rather than the fence agent). These are also listed in the man page

for +pacemaker-fenced+.

.Additional Properties of Fencing Resources

[width="95%",cols="8m,3,6,<12",options="header",align="center"]

|=========================================================

|Field

|Type

|Default

|Description

|stonith-timeout

|NA

|NA

a|Older versions used this to override the default period to wait for a STONITH (reboot, on, off) action to complete for this device.

 It has been replaced by the +pcmk_reboot_timeout+ and +pcmk_off_timeout+ properties.

 indexterm:[stonith-timeout,Fencing]

 indexterm:[Fencing,Property,stonith-timeout]

////

 (not yet implemented)

 priority

 integer

 0

 The priority of the STONITH resource. Devices are tried in order of highest priority to lowest.

 indexterm  priority,Fencing 

 indexterm  Fencing,Property,priority 

////

|provides

|string

|

|Any special capability provided by the fence device. Currently, only one such

 capability is meaningful: +unfencing+ (see <<s-unfencing>>).

 indexterm:[provides,Fencing]

 indexterm:[Fencing,Property,provides]

|pcmk_host_map

|string

|

|A mapping of host names to ports numbers for devices that do not support host names.

 Example: +node1:1;node2:2,3+ tells the cluster to use port 1 for

 *node1* and ports 2 and 3 for *node2*. If +pcmk_host_check+ is explicitly set

 to +static-list+, either this or +pcmk_host_list+ must be set.

 indexterm:[pcmk_host_map,Fencing]

 indexterm:[Fencing,Property,pcmk_host_map]

|pcmk_host_list

|string

|

|A list of machines controlled by this device. If +pcmk_host_check+ is

 explicitly set to +static-list+, either this or +pcmk_host_map+ must be set.

 indexterm:[pcmk_host_list,Fencing]

 indexterm:[Fencing,Property,pcmk_host_list]

|pcmk_host_check

|string

|A value appropriate to other configuration options and

 device capabilities (see note below)

a|How to determine which machines are controlled by the device.

 Allowed values:

* +dynamic-list:+ query the device via the "list" command

* +static-list:+ check the +pcmk_host_list+ or +pcmk_host_map+ attribute

* +status:+ query the device via the "status" command

* +none:+ assume every device can fence every machine

indexterm:[pcmk_host_check,Fencing]

indexterm:[Fencing,Property,pcmk_host_check]

|pcmk_delay_max

|time

|0s

|Enable a random delay of up to the time specified before executing fencing

actions. This is sometimes used in two-node clusters to ensure that the

nodes don't fence each other at the same time. The overall delay introduced

by pacemaker is derived from this random delay value adding a static delay so

that the sum is kept below the maximum delay.

indexterm:[pcmk_delay_max,Fencing]

indexterm:[Fencing,Property,pcmk_delay_max]

|pcmk_delay_base

|time

|0s

|Enable a static delay before executing fencing actions. This can be used

 e.g. in two-node clusters to ensure that the nodes don't fence each other,

 by having separate fencing resources with different values. The node that is

 fenced with the shorter delay will lose a fencing race. The overall delay

 introduced by pacemaker is derived from this value plus a random delay such

 that the sum is kept below the maximum delay.

indexterm:[pcmk_delay_base,Fencing]

indexterm:[Fencing,Property,pcmk_delay_base]

|pcmk_action_limit

|integer

|1

|The maximum number of actions that can be performed in parallel on this

 device, if the cluster option +concurrent-fencing+ is +true+. -1 is unlimited.

indexterm:[pcmk_action_limit,Fencing]

indexterm:[Fencing,Property,pcmk_action_limit]

|pcmk_host_argument

|string

|port

|'Advanced use only.' Which parameter should be supplied to the resource agent

to identify the node to be fenced. Some devices do not support the standard

+port+ parameter or may provide additional ones. Use this to specify an

alternate, device-specific parameter. A value of +none+ tells the

cluster not to supply any additional parameters.

 indexterm:[pcmk_host_argument,Fencing]

 indexterm:[Fencing,Property,pcmk_host_argument]

|pcmk_reboot_action

|string

|reboot

|'Advanced use only.' The command to send to the resource agent in order to

reboot a node. Some devices do not support the standard commands or may provide

additional ones. Use this to specify an alternate, device-specific command.

 indexterm:[pcmk_reboot_action,Fencing]

 indexterm:[Fencing,Property,pcmk_reboot_action]

|pcmk_reboot_timeout

|time

|60s

|'Advanced use only.' Specify an alternate timeout to use for `reboot` actions

instead of the value of +stonith-timeout+. Some devices need much more or less

time to complete than normal. Use this to specify an alternate, device-specific

timeout.

 indexterm:[pcmk_reboot_timeout,Fencing]

 indexterm:[Fencing,Property,pcmk_reboot_timeout]

 indexterm:[stonith-timeout,Fencing]

 indexterm:[Fencing,Property,stonith-timeout]

|pcmk_reboot_retries

|integer

|2

|'Advanced use only.' The maximum number of times to retry the `reboot` command

within the timeout period. Some devices do not support multiple connections, and

operations may fail if the device is busy with another task, so Pacemaker will

automatically retry the operation, if there is time remaining. Use this option

to alter the number of times Pacemaker retries before giving up.

 indexterm:[pcmk_reboot_retries,Fencing]

 indexterm:[Fencing,Property,pcmk_reboot_retries]

|pcmk_off_action

|string

|off

|'Advanced use only.' The command to send to the resource agent in order to

shut down a node. Some devices do not support the standard commands or may provide

additional ones. Use this to specify an alternate, device-specific command.

 indexterm:[pcmk_off_action,Fencing]

 indexterm:[Fencing,Property,pcmk_off_action]

|pcmk_off_timeout

|time

|60s

|'Advanced use only.' Specify an alternate timeout to use for `off` actions

instead of the value of +stonith-timeout+. Some devices need much more or less

time to complete than normal. Use this to specify an alternate, device-specific

timeout.

 indexterm:[pcmk_off_timeout,Fencing]

 indexterm:[Fencing,Property,pcmk_off_timeout]

 indexterm:[stonith-timeout,Fencing]

 indexterm:[Fencing,Property,stonith-timeout]

|pcmk_off_retries

|integer

|2

|'Advanced use only.' The maximum number of times to retry the `off` command

within the timeout period. Some devices do not support multiple connections, and

operations may fail if the device is busy with another task, so Pacemaker will

automatically retry the operation, if there is time remaining. Use this option

to alter the number of times Pacemaker retries before giving up.

 indexterm:[pcmk_off_retries,Fencing]

 indexterm:[Fencing,Property,pcmk_off_retries]

|pcmk_list_action

|string

|list

|'Advanced use only.' The command to send to the resource agent in order to

list nodes. Some devices do not support the standard commands or may provide

additional ones. Use this to specify an alternate, device-specific command.

 indexterm:[pcmk_list_action,Fencing]

 indexterm:[Fencing,Property,pcmk_list_action]

|pcmk_list_timeout

|time

|60s

|'Advanced use only.' Specify an alternate timeout to use for `list` actions

instead of the value of +stonith-timeout+. Some devices need much more or less

time to complete than normal. Use this to specify an alternate, device-specific

timeout.

 indexterm:[pcmk_list_timeout,Fencing]

 indexterm:[Fencing,Property,pcmk_list_timeout]

|pcmk_list_retries

|integer

|2

|'Advanced use only.' The maximum number of times to retry the `list` command

within the timeout period. Some devices do not support multiple connections, and

operations may fail if the device is busy with another task, so Pacemaker will

automatically retry the operation, if there is time remaining. Use this option

to alter the number of times Pacemaker retries before giving up.

 indexterm:[pcmk_list_retries,Fencing]

 indexterm:[Fencing,Property,pcmk_list_retries]

|pcmk_monitor_action

|string

|monitor

|'Advanced use only.' The command to send to the resource agent in order to

report extended status. Some devices do not support the standard commands or may provide

additional ones. Use this to specify an alternate, device-specific command.

 indexterm:[pcmk_monitor_action,Fencing]

 indexterm:[Fencing,Property,pcmk_monitor_action]

|pcmk_monitor_timeout

|time

|60s

|'Advanced use only.' Specify an alternate timeout to use for `monitor` actions

instead of the value of +stonith-timeout+. Some devices need much more or less

time to complete than normal. Use this to specify an alternate, device-specific

timeout.

 indexterm:[pcmk_monitor_timeout,Fencing]

 indexterm:[Fencing,Property,pcmk_monitor_timeout]

|pcmk_monitor_retries

|integer

|2

|'Advanced use only.' The maximum number of times to retry the `monitor` command

within the timeout period. Some devices do not support multiple connections, and

operations may fail if the device is busy with another task, so Pacemaker will

automatically retry the operation, if there is time remaining. Use this option

to alter the number of times Pacemaker retries before giving up.

 indexterm:[pcmk_monitor_retries,Fencing]

 indexterm:[Fencing,Property,pcmk_monitor_retries]

|pcmk_status_action

|string

|status

|'Advanced use only.' The command to send to the resource agent in order to

report status. Some devices do not support the standard commands or may provide

additional ones. Use this to specify an alternate, device-specific command.

 indexterm:[pcmk_status_action,Fencing]

 indexterm:[Fencing,Property,pcmk_status_action]

|pcmk_status_timeout

|time

|60s

|'Advanced use only.' Specify an alternate timeout to use for `status` actions

instead of the value of +stonith-timeout+. Some devices need much more or less

time to complete than normal. Use this to specify an alternate, device-specific

timeout.

 indexterm:[pcmk_status_timeout,Fencing]

 indexterm:[Fencing,Property,pcmk_status_timeout]

|pcmk_status_retries

|integer

|2

|'Advanced use only.' The maximum number of times to retry the `status` command

within the timeout period. Some devices do not support multiple connections, and

operations may fail if the device is busy with another task, so Pacemaker will

automatically retry the operation, if there is time remaining. Use this option

to alter the number of times Pacemaker retries before giving up.

 indexterm:[pcmk_status_retries,Fencing]

 indexterm:[Fencing,Property,pcmk_status_retries]

|=========================================================

[NOTE]

====

The default value for +pcmk_host_check+ is +static-list+ if either

+pcmk_host_list+ or +pcmk_host_map+ is configured. If neither of those are

configured, the default is +dynamic-list+ if the fence device supports the list

action, or +status+ if the fence device supports the status action but not the

list action. If none of those conditions apply, the default is +none+.

====

[[s-unfencing]]

== Unfencing ==

With fabric fencing (such as cutting network or shared disk access rather than

power), it is expected that the cluster will fence the node, and

then a system administrator must manually investigate what went wrong, correct

any issues found, then reboot (or restart the cluster services on) the node.

Once the node reboots and rejoins the cluster, some fabric fencing devices

require an explicit command to restore the node's access. This capability is

called 'unfencing' and is typically implemented as the fence agent's +on+

command.

If any cluster resource has +requires+ set to +unfencing+, then that resource

will not be probed or started on a node until that node has been unfenced.

== Fence Devices Dependent on Other Resources ==

In some cases, a fence device may require some other cluster resource (such as

an IP address) to be active in order to function properly.

This is obviously undesirable in general: fencing may be required when the

depended-on resource is not active, or fencing may be required because the node

running the depended-on resource is no longer responding.

However, this may be acceptable under certain conditions:

* The dependent fence device should not be able to target any node that is

  allowed to run the depended-on resource.

* The depended-on resource should not be disabled during production operation.

* The +concurrent-fencing+ cluster property should be set to +true+. Otherwise,

  if both the node running the depended-on resource and some node targeted by

  the dependent fence device need to be fenced, the fencing of the node

  running the depended-on resource might be ordered first, making the second

  fencing impossible and blocking further recovery. With concurrent fencing,

  the dependent fence device might fail at first due to the depended-on

  resource being unavailable, but it will be retried and eventually succeed

  once the resource is brought back up.

Even under those conditions, there is one unlikely problem scenario. The DC

always schedules fencing of itself after any other fencing needed, to avoid

unnecessary repeated DC elections. If the dependent fence device targets the

DC, and both the DC and a different node running the depended-on resource need

to be fenced, the DC fencing will always fail and block further recovery. Note,

however, that losing a DC node entirely causes some other node to become DC and

schedule the fencing, so this is only a risk when a stop or other operation

with +on-fail+ set to +fencing+ fails on the DC.

== Configuring Fencing ==

. Find the correct driver:

+

----

# stonith_admin --list-installed

----

. Find the required parameters associated with the device

  (replacing $AGENT_NAME with the name obtained from the previous step):

+

----

# stonith_admin --metadata --agent $AGENT_NAME

----

. Create a file called +stonith.xml+ containing a primitive resource

  with a class of +stonith+, a type equal to the agent name obtained earlier,

  and a parameter for each of the values returned in the previous step.

. If the device does not know how to fence nodes based on their uname,

  you may also need to set the special +pcmk_host_map+ parameter.  See

  `man pacemaker-fenced` for details.

. If the device does not support the `list` command, you may also need

  to set the special +pcmk_host_list+ and/or +pcmk_host_check+

  parameters.  See `man pacemaker-fenced` for details.

. If the device does not expect the victim to be specified with the

  `port` parameter, you may also need to set the special

  +pcmk_host_argument+ parameter. See `man pacemaker-fenced` for details.

. Upload it into the CIB using cibadmin:

+

----

# cibadmin -C -o resources --xml-file stonith.xml

----

. Set +stonith-enabled+ to true:

+

----

# crm_attribute -t crm_config -n stonith-enabled -v true

----

. Once the stonith resource is running, you can test it by executing the

  following (although you might want to stop the cluster on that machine

  first):

+

----

# stonith_admin --reboot nodename

----

=== Example Fencing Configuration ===

Assume we have a chassis containing four nodes and an IPMI device

active on 192.0.2.1. We would choose the `fence_ipmilan` driver,

and obtain the following list of parameters:

.Obtaining a list of Fence Agent Parameters

====

----

# stonith_admin --metadata -a fence_ipmilan

----

[source,XML]

----

<resource-agent name="fence_ipmilan" shortdesc="Fence agent for IPMI over LAN">

  <symlink name="fence_ilo3" shortdesc="Fence agent for HP iLO3"/>

  <symlink name="fence_ilo4" shortdesc="Fence agent for HP iLO4"/>

  <symlink name="fence_idrac" shortdesc="Fence agent for Dell iDRAC"/>

  <symlink name="fence_imm" shortdesc="Fence agent for IBM Integrated Management Module"/>

  <longdesc>

  </longdesc>

  <vendor-url>

  </vendor-url>

  <parameters>

    <parameter name="auth" unique="0" required="0">

      <getopt mixed="-A"/>

      <content type="string"/>

      <shortdesc lang="en">

      </shortdesc>

    </parameter>

    <parameter name="ipaddr" unique="0" required="1">

      <getopt mixed="-a"/>

      <content type="string"/>

      <shortdesc lang="en">

      </shortdesc>

    </parameter>

    <parameter name="passwd" unique="0" required="0">

      <getopt mixed="-p"/>

      <content type="string"/>

      <shortdesc lang="en">

      </shortdesc>

    </parameter>

    <parameter name="passwd_script" unique="0" required="0">

      <getopt mixed="-S"/>

      <content type="string"/>

      <shortdesc lang="en">

      </shortdesc>

    </parameter>

    <parameter name="lanplus" unique="0" required="0">

      <getopt mixed="-P"/>

      <content type="boolean"/>

      <shortdesc lang="en">

      </shortdesc>

    </parameter>

    <parameter name="login" unique="0" required="0">

      <getopt mixed="-l"/>

      <content type="string"/>

      <shortdesc lang="en">

      </shortdesc>

    </parameter>

    <parameter name="action" unique="0" required="0">

      <getopt mixed="-o"/>

      <content type="string" default="reboot"/>

      <shortdesc lang="en">

      </shortdesc>

    </parameter>

    <parameter name="timeout" unique="0" required="0">

      <getopt mixed="-t"/>

      <content type="string"/>

      <shortdesc lang="en">

      </shortdesc>

    </parameter>

    <parameter name="cipher" unique="0" required="0">

      <getopt mixed="-C"/>

      <content type="string"/>

      <shortdesc lang="en">

      </shortdesc>

    </parameter>

    <parameter name="method" unique="0" required="0">

      <getopt mixed="-M"/>

      <content type="string" default="onoff"/>

      <shortdesc lang="en">

      </shortdesc>

    </parameter>

    <parameter name="power_wait" unique="0" required="0">

      <getopt mixed="-T"/>

      <content type="string" default="2"/>

      <shortdesc lang="en">

      </shortdesc>

    </parameter>

    <parameter name="delay" unique="0" required="0">

      <getopt mixed="-f"/>

      <content type="string"/>

      <shortdesc lang="en">

      </shortdesc>

    </parameter>

    <parameter name="privlvl" unique="0" required="0">

      <getopt mixed="-L"/>

      <content type="string"/>

      <shortdesc lang="en">

      </shortdesc>

    </parameter>

    <parameter name="verbose" unique="0" required="0">

      <getopt mixed="-v"/>

      <content type="boolean"/>

      <shortdesc lang="en">

      </shortdesc>

    </parameter>

  </parameters>

  <actions>

    <action name="on"/>

    <action name="off"/>

    <action name="reboot"/>

    <action name="status"/>

    <action name="diag"/>

    <action name="list"/>

    <action name="monitor"/>

    <action name="metadata"/>

    <action name="stop" timeout="20s"/>

    <action name="start" timeout="20s"/>

  </actions>

</resource-agent>

----

====

Based on that, we would create a fencing resource fragment that might look

like this:

.An IPMI-based Fencing Resource

====

[source,XML]

----

<primitive id="Fencing" class="stonith" type="fence_ipmilan" >

  <instance_attributes id="Fencing-params" >

    <nvpair id="Fencing-passwd" name="passwd" value="testuser" />

    <nvpair id="Fencing-login" name="login" value="abc123" />

    <nvpair id="Fencing-ipaddr" name="ipaddr" value="192.0.2.1" />

    <nvpair id="Fencing-pcmk_host_list" name="pcmk_host_list" value="pcmk-1 pcmk-2" />

  </instance_attributes>

  <operations >

    <op id="Fencing-monitor-10m" interval="10m" name="monitor" timeout="300s" />

  </operations>

</primitive>

----

====

Finally, we need to enable fencing:

----

# crm_attribute -t crm_config -n stonith-enabled -v true

----

== Fencing Topologies ==

Pacemaker supports fencing nodes with multiple devices through a feature called

'fencing topologies'. Fencing topologies may be used to provide alternative

devices in case one fails, or to require multiple devices to all be executed

successfully in order to consider the node successfully fenced, or even a

combination of the two.

Create the individual devices as you normally would, then define one or more

+fencing-level+ entries in the +fencing-topology+ section of the configuration.

* Each fencing level is attempted in order of ascending +index+. Allowed

  values are 1 through 9.

* If a device fails, processing terminates for the current level.

  No further devices in that level are exercised, and the next level is attempted instead.

* If the operation succeeds for all the listed devices in a level, the level is deemed to have passed.

* The operation is finished when a level has passed (success), or all levels have been attempted (failed).

* If the operation failed, the next step is determined by the scheduler

  and/or the controller.

Some possible uses of topologies include:

* Try on-board IPMI, then an intelligent power switch if that fails

* Try fabric fencing of both disk and network, then fall back to power fencing

  if either fails

* Wait up to a certain time for a kernel dump to complete, then cut power to

  the node

.Properties of Fencing Levels

[width="95%",cols="1m,<3",options="header",align="center"]

|=========================================================

|Field

|Description

|id

|A unique name for the level

 indexterm:[id,fencing-level]

 indexterm:[Fencing,fencing-level,id]

|target

|The name of a single node to which this level applies

 indexterm:[target,fencing-level]

 indexterm:[Fencing,fencing-level,target]

|target-pattern

|An extended regular expression (as defined in

 http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09_04[POSIX])

 matching the names of nodes to which this level applies

 indexterm:[target-pattern,fencing-level]

 indexterm:[Fencing,fencing-level,target-pattern]

|target-attribute

|The name of a node attribute that is set (to +target-value+) for nodes to

 which this level applies

 indexterm:[target-attribute,fencing-level]

 indexterm:[Fencing,fencing-level,target-attribute]

|target-value

|The node attribute value (of +target-attribute+) that is set for nodes to

 which this level applies

 indexterm:[target-attribute,fencing-level]

 indexterm:[Fencing,fencing-level,target-attribute]

|index

|The order in which to attempt the levels.

 Levels are attempted in ascending order 'until one succeeds'.

 Valid values are 1 through 9.

 indexterm:[index,fencing-level]

 indexterm:[Fencing,fencing-level,index]

|devices

|A comma-separated list of devices that must all be tried for this level

 indexterm:[devices,fencing-level]

 indexterm:[Fencing,fencing-level,devices]

|=========================================================

.Fencing topology with different devices for different nodes

====

[source,XML]

----

 <cib crm_feature_set="3.0.6" validate-with="pacemaker-1.2" admin_epoch="1" epoch="0" num_updates="0">

  <configuration>

    ...

    <fencing-topology>

      <fencing-level id="f-p1.1" target="pcmk-1" index="1" devices="poison-pill"/>

      <fencing-level id="f-p1.2" target="pcmk-1" index="2" devices="power"/>

      <fencing-level id="f-p2.1" target="pcmk-2" index="1" devices="disk,network"/>

      <fencing-level id="f-p2.2" target="pcmk-2" index="2" devices="power"/>

    </fencing-topology>

    ...

  <configuration>

  <status/>

</cib>

----

====

=== Example Dual-Layer, Dual-Device Fencing Topologies ===

The following example illustrates an advanced use of +fencing-topology+ in a cluster with the following properties:

* 3 nodes (2 active prod-mysql nodes, 1 prod_mysql-rep in standby for quorum purposes)

* the active nodes have an IPMI-controlled power board reached at 192.0.2.1 and 192.0.2.2

* the active nodes also have two independent PSUs (Power Supply Units)

  connected to two independent PDUs (Power Distribution Units) reached at

  198.51.100.1 (port 10 and port 11) and 203.0.113.1 (port 10 and port 11)

* the first fencing method uses the `fence_ipmi` agent

* the second fencing method uses the `fence_apc_snmp` agent targetting 2 fencing devices (one per PSU, either port 10 or 11)

* fencing is only implemented for the active nodes and has location constraints

* fencing topology is set to try IPMI fencing first then default to a "sure-kill" dual PDU fencing

In a normal failure scenario, STONITH will first select +fence_ipmi+ to try to kill the faulty node.

Using a fencing topology, if that first method fails, STONITH will then move on to selecting +fence_apc_snmp+ twice:

* once for the first PDU 

* again for the second PDU 

The fence action is considered successful only if both PDUs report the required status. If any of them fails, STONITH loops back to the first fencing method, +fence_ipmi+, and so on until the node is fenced or fencing action is cancelled.

.First fencing method: single IPMI device

Each cluster node has it own dedicated IPMI channel that can be called for fencing using the following primitives:

[source,XML]

----

<primitive class="stonith" id="fence_prod-mysql1_ipmi" type="fence_ipmilan">

  <instance_attributes id="fence_prod-mysql1_ipmi-instance_attributes">

    <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-ipaddr" name="ipaddr" value="192.0.2.1"/>

    <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-action" name="action" value="off"/>

    <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-login" name="login" value="fencing"/>

    <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-passwd" name="passwd" value="finishme"/>

    <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-verbose" name="verbose" value="true"/>

    <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql1"/>

    <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-lanplus" name="lanplus" value="true"/>

  </instance_attributes>

</primitive>

<primitive class="stonith" id="fence_prod-mysql2_ipmi" type="fence_ipmilan">

  <instance_attributes id="fence_prod-mysql2_ipmi-instance_attributes">

    <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-ipaddr" name="ipaddr" value="192.0.2.2"/>

    <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-action" name="action" value="off"/>

    <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-login" name="login" value="fencing"/>

    <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-passwd" name="passwd" value="finishme"/>

    <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-verbose" name="verbose" value="true"/>

    <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql2"/>

    <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-lanplus" name="lanplus" value="true"/>

  </instance_attributes>

</primitive>

----

.Second fencing method: dual PDU devices

Each cluster node also has two distinct power channels controlled by two

distinct PDUs. That means a total of 4 fencing devices configured as follows:

- Node 1, PDU 1, PSU 1 @ port 10

- Node 1, PDU 2, PSU 2 @ port 10

- Node 2, PDU 1, PSU 1 @ port 11

- Node 2, PDU 2, PSU 2 @ port 11

The matching fencing agents are configured as follows:

[source,XML]

----

<primitive class="stonith" id="fence_prod-mysql1_apc1" type="fence_apc_snmp">

  <instance_attributes id="fence_prod-mysql1_apc1-instance_attributes">

    <nvpair id="fence_prod-mysql1_apc1-instance_attributes-ipaddr" name="ipaddr" value="198.51.100.1"/>

    <nvpair id="fence_prod-mysql1_apc1-instance_attributes-action" name="action" value="off"/>

    <nvpair id="fence_prod-mysql1_apc1-instance_attributes-port" name="port" value="10"/>

    <nvpair id="fence_prod-mysql1_apc1-instance_attributes-login" name="login" value="fencing"/>

    <nvpair id="fence_prod-mysql1_apc1-instance_attributes-passwd" name="passwd" value="fencing"/>

    <nvpair id="fence_prod-mysql1_apc1-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql1"/>

  </instance_attributes>

</primitive>

<primitive class="stonith" id="fence_prod-mysql1_apc2" type="fence_apc_snmp">

  <instance_attributes id="fence_prod-mysql1_apc2-instance_attributes">

    <nvpair id="fence_prod-mysql1_apc2-instance_attributes-ipaddr" name="ipaddr" value="203.0.113.1"/>

    <nvpair id="fence_prod-mysql1_apc2-instance_attributes-action" name="action" value="off"/>

    <nvpair id="fence_prod-mysql1_apc2-instance_attributes-port" name="port" value="10"/>

    <nvpair id="fence_prod-mysql1_apc2-instance_attributes-login" name="login" value="fencing"/>

    <nvpair id="fence_prod-mysql1_apc2-instance_attributes-passwd" name="passwd" value="fencing"/>

    <nvpair id="fence_prod-mysql1_apc2-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql1"/>

  </instance_attributes>

</primitive>

<primitive class="stonith" id="fence_prod-mysql2_apc1" type="fence_apc_snmp">

  <instance_attributes id="fence_prod-mysql2_apc1-instance_attributes">

    <nvpair id="fence_prod-mysql2_apc1-instance_attributes-ipaddr" name="ipaddr" value="198.51.100.1"/>

    <nvpair id="fence_prod-mysql2_apc1-instance_attributes-action" name="action" value="off"/>

    <nvpair id="fence_prod-mysql2_apc1-instance_attributes-port" name="port" value="11"/>

    <nvpair id="fence_prod-mysql2_apc1-instance_attributes-login" name="login" value="fencing"/>

    <nvpair id="fence_prod-mysql2_apc1-instance_attributes-passwd" name="passwd" value="fencing"/>

    <nvpair id="fence_prod-mysql2_apc1-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql2"/>

  </instance_attributes>

</primitive>

<primitive class="stonith" id="fence_prod-mysql2_apc2" type="fence_apc_snmp">

  <instance_attributes id="fence_prod-mysql2_apc2-instance_attributes">

    <nvpair id="fence_prod-mysql2_apc2-instance_attributes-ipaddr" name="ipaddr" value="203.0.113.1"/>

    <nvpair id="fence_prod-mysql2_apc2-instance_attributes-action" name="action" value="off"/>

    <nvpair id="fence_prod-mysql2_apc2-instance_attributes-port" name="port" value="11"/>

    <nvpair id="fence_prod-mysql2_apc2-instance_attributes-login" name="login" value="fencing"/>

    <nvpair id="fence_prod-mysql2_apc2-instance_attributes-passwd" name="passwd" value="fencing"/>

    <nvpair id="fence_prod-mysql2_apc2-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql2"/>

  </instance_attributes>

</primitive>

----

.Location Constraints 

To prevent STONITH from trying to run a fencing agent on the same node it is

supposed to fence, constraints are placed on all the fencing primitives:

[source,XML]

----

<constraints>

  <rsc_location id="l_fence_prod-mysql1_ipmi" node="prod-mysql1" rsc="fence_prod-mysql1_ipmi" score="-INFINITY"/>

  <rsc_location id="l_fence_prod-mysql2_ipmi" node="prod-mysql2" rsc="fence_prod-mysql2_ipmi" score="-INFINITY"/>

  <rsc_location id="l_fence_prod-mysql1_apc2" node="prod-mysql1" rsc="fence_prod-mysql1_apc2" score="-INFINITY"/>

  <rsc_location id="l_fence_prod-mysql1_apc1" node="prod-mysql1" rsc="fence_prod-mysql1_apc1" score="-INFINITY"/>

  <rsc_location id="l_fence_prod-mysql2_apc1" node="prod-mysql2" rsc="fence_prod-mysql2_apc1" score="-INFINITY"/>

  <rsc_location id="l_fence_prod-mysql2_apc2" node="prod-mysql2" rsc="fence_prod-mysql2_apc2" score="-INFINITY"/>

</constraints>