|
Packit |
13e616 |
|
|
Packit |
13e616 |
QoS Management in OpenSM
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
==============================================================================
|
|
Packit |
13e616 |
Table of contents
|
|
Packit |
13e616 |
==============================================================================
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
1. Overview
|
|
Packit |
13e616 |
2. Full QoS Policy File
|
|
Packit |
13e616 |
3. Simplified QoS Policy Definition
|
|
Packit |
13e616 |
4. Policy File Syntax Guidelines
|
|
Packit |
13e616 |
5. Examples of Full Policy File
|
|
Packit |
13e616 |
6. Simplified QoS Policy - Details and Examples
|
|
Packit |
13e616 |
7. SL2VL Mapping and VL Arbitration
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
==============================================================================
|
|
Packit |
13e616 |
1. Overview
|
|
Packit |
13e616 |
==============================================================================
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
When QoS in OpenSM is enabled (-Q or --qos), OpenSM looks for QoS Policy file.
|
|
Packit |
13e616 |
The default name of OpenSM QoS policy file is
|
|
Packit |
13e616 |
/usr/local/etc/opensm/qos-policy.conf. The default may be changed by using -Y
|
|
Packit |
13e616 |
or --qos_policy_file option with OpenSM.
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
During fabric initialization and at every heavy sweep OpenSM parses the QoS
|
|
Packit |
13e616 |
policy file, applies its settings to the discovered fabric elements, and
|
|
Packit |
13e616 |
enforces the provided policy on client requests. The overall flow for such
|
|
Packit |
13e616 |
requests is:
|
|
Packit |
13e616 |
- The request is matched against the defined matching rules such that the
|
|
Packit |
13e616 |
QoS Level definition is found.
|
|
Packit |
13e616 |
- Given the QoS Level, path(s) search is performed with the given
|
|
Packit |
13e616 |
restrictions imposed by that level.
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
There are two ways to define QoS policy:
|
|
Packit |
13e616 |
- Full policy, where the policy file syntax provides an administrator
|
|
Packit |
13e616 |
various ways to match PathRecord/MultiPathRecord (PR/MPR) request and
|
|
Packit |
13e616 |
enforce various QoS constraints on the requested PR/MPR
|
|
Packit |
13e616 |
- Simplified QoS policy definition, where an administrator would be able to
|
|
Packit |
13e616 |
match PR/MPR requests by various ULPs and applications running on top of
|
|
Packit |
13e616 |
these ULPs.
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
While the full policy syntax is very flexible, in many cases the simplified
|
|
Packit |
13e616 |
policy definition would be sufficient.
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
==============================================================================
|
|
Packit |
13e616 |
2. Full QoS Policy File
|
|
Packit |
13e616 |
==============================================================================
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
QoS policy file has the following sections:
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
I) Port Groups (denoted by port-groups).
|
|
Packit |
13e616 |
This section defines zero or more port groups that can be referred later by
|
|
Packit |
13e616 |
matching rules (see below). Port group lists ports by:
|
|
Packit |
13e616 |
- Port GUID
|
|
Packit |
13e616 |
- Port name, which is a combination of NodeDescription and IB port number
|
|
Packit |
13e616 |
- PKey, which means that all the ports in the subnet that belong to
|
|
Packit |
13e616 |
partition with a given PKey belong to this port group
|
|
Packit |
13e616 |
- Partition name, which means that all the ports in the subnet that belong
|
|
Packit |
13e616 |
to partition with a given name belong to this port group
|
|
Packit |
13e616 |
- Node type, where possible node types are: CA, SWITCH, ROUTER, ALL, and
|
|
Packit |
13e616 |
SELF (SM's port).
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
II) QoS Setup (denoted by qos-setup).
|
|
Packit |
13e616 |
This section describes how to set up SL2VL and VL Arbitration tables on
|
|
Packit |
13e616 |
various nodes in the fabric.
|
|
Packit |
13e616 |
However, this is not supported in OpenSM currently.
|
|
Packit |
13e616 |
SL2VL and VLArb tables should be configured in the OpenSM options file
|
|
Packit |
13e616 |
(default location - /usr/local/etc/opensm/opensm.conf).
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
III) QoS Levels (denoted by qos-levels).
|
|
Packit |
13e616 |
Each QoS Level defines Service Level (SL) and a few optional fields:
|
|
Packit |
13e616 |
- MTU limit
|
|
Packit |
13e616 |
- Rate limit
|
|
Packit |
13e616 |
- PKey
|
|
Packit |
13e616 |
- Packet lifetime
|
|
Packit |
13e616 |
When path(s) search is performed, it is done with regards to restriction that
|
|
Packit |
13e616 |
these QoS Level parameters impose.
|
|
Packit |
13e616 |
One QoS level that is mandatory to define is a DEFAULT QoS level. It is
|
|
Packit |
13e616 |
applied to a PR/MPR query that does not match any existing match rule.
|
|
Packit |
13e616 |
Similar to any other QoS Level, it can also be explicitly referred by any
|
|
Packit |
13e616 |
match rule.
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
IV) QoS Matching Rules (denoted by qos-match-rules).
|
|
Packit |
13e616 |
Each PathRecord/MultiPathRecord query that OpenSM receives is matched against
|
|
Packit |
13e616 |
the set of matching rules. Rules are scanned in order of appearance in the QoS
|
|
Packit |
13e616 |
policy file such as the first match takes precedence.
|
|
Packit |
13e616 |
Each rule has a name of QoS level that will be applied to the matching query.
|
|
Packit |
13e616 |
A default QoS level is applied to a query that did not match any rule.
|
|
Packit |
13e616 |
Queries can be matched by:
|
|
Packit |
13e616 |
- Source port group (whether a source port is a member of a specified group)
|
|
Packit |
13e616 |
- Destination port group (same as above, only for destination port)
|
|
Packit |
13e616 |
- PKey
|
|
Packit |
13e616 |
- QoS class
|
|
Packit |
13e616 |
- Service ID
|
|
Packit |
13e616 |
To match a certain matching rule, PR/MPR query has to match ALL the rule's
|
|
Packit |
13e616 |
criteria. However, not all the fields of the PR/MPR query have to appear in
|
|
Packit |
13e616 |
the matching rule.
|
|
Packit |
13e616 |
For instance, if the rule has a single criterion - Service ID, it will match
|
|
Packit |
13e616 |
any query that has this Service ID, disregarding rest of the query fields.
|
|
Packit |
13e616 |
However, if a certain query has only Service ID (which means that this is the
|
|
Packit |
13e616 |
only bit in the PR/MPR component mask that is on), it will not match any rule
|
|
Packit |
13e616 |
that has other matching criteria besides Service ID.
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
==============================================================================
|
|
Packit |
13e616 |
3. Simplified QoS Policy Definition
|
|
Packit |
13e616 |
==============================================================================
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
Simplified QoS policy definition comprises of a single section denoted by
|
|
Packit |
13e616 |
qos-ulps. Similar to the full QoS policy, it has a list of match rules and
|
|
Packit |
13e616 |
their QoS Level, but in this case a match rule has only one criterion - its
|
|
Packit |
13e616 |
goal is to match a certain ULP (or a certain application on top of this ULP)
|
|
Packit |
13e616 |
PR/MPR request, and QoS Level has only one constraint - Service Level (SL).
|
|
Packit |
13e616 |
The simplified policy section may appear in the policy file in combine with
|
|
Packit |
13e616 |
the full policy, or as a stand-alone policy definition.
|
|
Packit |
13e616 |
See more details and list of match rule criteria below.
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
==============================================================================
|
|
Packit |
13e616 |
4. Policy File Syntax Guidelines
|
|
Packit |
13e616 |
==============================================================================
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
- Empty lines are ignored.
|
|
Packit |
13e616 |
- Leading and trailing blanks, as well as empty lines, are ignored, so
|
|
Packit |
13e616 |
the indentation in the example is just for better readability.
|
|
Packit |
13e616 |
- Comments are started with the pound sign (#) and terminated by EOL.
|
|
Packit |
13e616 |
- Any keyword should be the first non-blank in the line, unless it's a
|
|
Packit |
13e616 |
comment.
|
|
Packit |
13e616 |
- Keywords that denote section/subsection start have matching closing
|
|
Packit |
13e616 |
keywords.
|
|
Packit |
13e616 |
- Having a QoS Level named "DEFAULT" is a must - it is applied to PR/MPR
|
|
Packit |
13e616 |
requests that didn't match any of the matching rules.
|
|
Packit |
13e616 |
- Any section/subsection of the policy file is optional.
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
==============================================================================
|
|
Packit |
13e616 |
5. Examples of Full Policy File
|
|
Packit |
13e616 |
==============================================================================
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
As mentioned earlier, any section of the policy file is optional, and
|
|
Packit |
13e616 |
the only mandatory part of the policy file is a default QoS Level.
|
|
Packit |
13e616 |
Here's an example of the shortest policy file:
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
qos-levels
|
|
Packit |
13e616 |
qos-level
|
|
Packit |
13e616 |
name: DEFAULT
|
|
Packit |
13e616 |
sl: 0
|
|
Packit |
13e616 |
end-qos-level
|
|
Packit |
13e616 |
end-qos-levels
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
Port groups section is missing because there are no match rules, which means
|
|
Packit |
13e616 |
that port groups are not referred anywhere, and there is no need defining
|
|
Packit |
13e616 |
them. And since this policy file doesn't have any matching rules, PR/MPR query
|
|
Packit |
13e616 |
won't match any rule, and OpenSM will enforce default QoS level.
|
|
Packit |
13e616 |
Essentially, the above example is equivalent to not having QoS policy file
|
|
Packit |
13e616 |
at all.
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
The following example shows all the possible options and keywords in the
|
|
Packit |
13e616 |
policy file and their syntax:
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
#
|
|
Packit |
13e616 |
# See the comments in the following example.
|
|
Packit |
13e616 |
# They explain different keywords and their meaning.
|
|
Packit |
13e616 |
#
|
|
Packit |
13e616 |
port-groups
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
port-group # using port GUIDs
|
|
Packit |
13e616 |
name: Storage
|
|
Packit |
13e616 |
# "use" is just a description that is used for logging
|
|
Packit |
13e616 |
# Other than that, it is just a comment
|
|
Packit |
13e616 |
use: SRP Targets
|
|
Packit |
13e616 |
port-guid: 0x10000000000001, 0x10000000000005-0x1000000000FFFA
|
|
Packit |
13e616 |
port-guid: 0x1000000000FFFF
|
|
Packit |
13e616 |
end-port-group
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
port-group
|
|
Packit |
13e616 |
name: Virtual Servers
|
|
Packit |
13e616 |
# The syntax of the port name is as follows:
|
|
Packit |
13e616 |
# "node_description/Pnum".
|
|
Packit |
13e616 |
# node_description is compared to the NodeDescription of the node,
|
|
Packit |
13e616 |
# and "Pnum" is a port number on that node.
|
|
Packit |
13e616 |
port-name: vs1 HCA-1/P1, vs2 HCA-1/P1
|
|
Packit |
13e616 |
end-port-group
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
# using partitions defined in the partition policy
|
|
Packit |
13e616 |
port-group
|
|
Packit |
13e616 |
name: Partitions
|
|
Packit |
13e616 |
partition: Part1
|
|
Packit |
13e616 |
pkey: 0x1234
|
|
Packit |
13e616 |
end-port-group
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
# using node types: CA, ROUTER, SWITCH, SELF (for node that runs SM)
|
|
Packit |
13e616 |
# or ALL (for all the nodes in the subnet)
|
|
Packit |
13e616 |
port-group
|
|
Packit |
13e616 |
name: CAs and SM
|
|
Packit |
13e616 |
node-type: CA, SELF
|
|
Packit |
13e616 |
end-port-group
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
end-port-groups
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
qos-setup
|
|
Packit |
13e616 |
# This section of the policy file describes how to set up SL2VL and VL
|
|
Packit |
13e616 |
# Arbitration tables on various nodes in the fabric.
|
|
Packit |
13e616 |
# However, this is not supported in OpenSM currently - the section is
|
|
Packit |
13e616 |
# parsed and ignored. SL2VL and VLArb tables should be configured in the
|
|
Packit |
13e616 |
# OpenSM options file (by default - /usr/local/etc/opensm/opensm.conf).
|
|
Packit |
13e616 |
end-qos-setup
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
qos-levels
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
# Having a QoS Level named "DEFAULT" is a must - it is applied to
|
|
Packit |
13e616 |
# PR/MPR requests that didn't match any of the matching rules.
|
|
Packit |
13e616 |
qos-level
|
|
Packit |
13e616 |
name: DEFAULT
|
|
Packit |
13e616 |
use: default QoS Level
|
|
Packit |
13e616 |
sl: 0
|
|
Packit |
13e616 |
end-qos-level
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
# the whole set: SL, MTU-Limit, Rate-Limit, PKey, Packet Lifetime
|
|
Packit |
13e616 |
qos-level
|
|
Packit |
13e616 |
name: WholeSet
|
|
Packit |
13e616 |
sl: 1
|
|
Packit |
13e616 |
mtu-limit: 4
|
|
Packit |
13e616 |
rate-limit: 5
|
|
Packit |
13e616 |
pkey: 0x1234
|
|
Packit |
13e616 |
packet-life: 8
|
|
Packit |
13e616 |
end-qos-level
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
end-qos-levels
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
# Match rules are scanned in order of their appearance in the policy file.
|
|
Packit |
13e616 |
# First matched rule takes precedence.
|
|
Packit |
13e616 |
qos-match-rules
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
# matching by single criteria: QoS class
|
|
Packit |
13e616 |
qos-match-rule
|
|
Packit |
13e616 |
use: by QoS class
|
|
Packit |
13e616 |
qos-class: 7-9,11
|
|
Packit |
13e616 |
# Name of qos-level to apply to the matching PR/MPR
|
|
Packit |
13e616 |
qos-level-name: WholeSet
|
|
Packit |
13e616 |
end-qos-match-rule
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
# show matching by destination group and service id
|
|
Packit |
13e616 |
qos-match-rule
|
|
Packit |
13e616 |
use: Storage targets
|
|
Packit |
13e616 |
destination: Storage
|
|
Packit |
13e616 |
service-id: 0x10000000000001, 0x10000000000008-0x10000000000FFF
|
|
Packit |
13e616 |
qos-level-name: WholeSet
|
|
Packit |
13e616 |
end-qos-match-rule
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
qos-match-rule
|
|
Packit |
13e616 |
source: Storage
|
|
Packit |
13e616 |
use: match by source group only
|
|
Packit |
13e616 |
qos-level-name: DEFAULT
|
|
Packit |
13e616 |
end-qos-match-rule
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
qos-match-rule
|
|
Packit |
13e616 |
use: match by all parameters
|
|
Packit |
13e616 |
qos-class: 7-9,11
|
|
Packit |
13e616 |
source: Virtual Servers
|
|
Packit |
13e616 |
destination: Storage
|
|
Packit |
13e616 |
service-id: 0x0000000000010000-0x000000000001FFFF
|
|
Packit |
13e616 |
pkey: 0x0F00-0x0FFF
|
|
Packit |
13e616 |
qos-level-name: WholeSet
|
|
Packit |
13e616 |
end-qos-match-rule
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
end-qos-match-rules
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
==============================================================================
|
|
Packit |
13e616 |
6. Simplified QoS Policy - Details and Examples
|
|
Packit |
13e616 |
==============================================================================
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
Simplified QoS policy match rules are tailored for matching ULPs (or some
|
|
Packit |
13e616 |
application on top of a ULP) PR/MPR requests. This section has a list of
|
|
Packit |
13e616 |
per-ULP (or per-application) match rules and the SL that should be enforced
|
|
Packit |
13e616 |
on the matched PR/MPR query.
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
Match rules include:
|
|
Packit |
13e616 |
- Default match rule that is applied to PR/MPR query that didn't match any
|
|
Packit |
13e616 |
of the other match rules
|
|
Packit |
13e616 |
- SDP
|
|
Packit |
13e616 |
- SDP application with a specific target TCP/IP port range
|
|
Packit |
13e616 |
- SRP with a specific target IB port GUID
|
|
Packit |
13e616 |
- RDS
|
|
Packit |
13e616 |
- iSER
|
|
Packit |
13e616 |
- iSER application with a specific target TCP/IP port range
|
|
Packit |
13e616 |
- IPoIB with a default PKey
|
|
Packit |
13e616 |
- IPoIB with a specific PKey
|
|
Packit |
13e616 |
- any ULP/application with a specific Service ID in the PR/MPR query
|
|
Packit |
13e616 |
- any ULP/application with a specific PKey in the PR/MPR query
|
|
Packit |
13e616 |
- any ULP/application with a specific target IB port GUID in the PR/MPR query
|
|
Packit |
13e616 |
- any ULP/application with a specific source IB port GUID in the PR/MPR query
|
|
Packit |
13e616 |
- any ULP/application with a specific source or target IB port GUID in the
|
|
Packit |
13e616 |
PR/MPR query
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
Since any section of the policy file is optional, as long as basic rules of
|
|
Packit |
13e616 |
the file are kept (such as no referring to nonexisting port group, having
|
|
Packit |
13e616 |
default QoS Level, etc), the simplified policy section (qos-ulps) can serve
|
|
Packit |
13e616 |
as a complete QoS policy file.
|
|
Packit |
13e616 |
The shortest policy file in this case would be as follows:
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
qos-ulps
|
|
Packit |
13e616 |
default : 0 #default SL
|
|
Packit |
13e616 |
end-qos-ulps
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
It is equivalent to the previous example of the shortest policy file, and it
|
|
Packit |
13e616 |
is also equivalent to not having policy file at all.
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
Below is an example of simplified QoS policy with all the possible keywords:
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
qos-ulps
|
|
Packit |
13e616 |
default : 0 # default SL
|
|
Packit |
13e616 |
sdp, port-num 30000 : 0 # SL for application running on top
|
|
Packit |
13e616 |
# of SDP when a destination
|
|
Packit |
13e616 |
# TCP/IPport is 30000
|
|
Packit |
13e616 |
sdp, port-num 10000-20000 : 0
|
|
Packit |
13e616 |
sdp : 1 # default SL for any other
|
|
Packit |
13e616 |
# application running on top of SDP
|
|
Packit |
13e616 |
rds : 2 # SL for RDS traffic
|
|
Packit |
13e616 |
iser, port-num 900 : 0 # SL for iSER with a specific target
|
|
Packit |
13e616 |
# port
|
|
Packit |
13e616 |
iser : 3 # default SL for iSER
|
|
Packit |
13e616 |
ipoib, pkey 0x0001 : 0 # SL for IPoIB on partition with
|
|
Packit |
13e616 |
# pkey 0x0001
|
|
Packit |
13e616 |
ipoib : 4 # default IPoIB partition,
|
|
Packit |
13e616 |
# pkey=0x7FFF
|
|
Packit |
13e616 |
any, service-id 0x6234 : 6 # match any PR/MPR query with a
|
|
Packit |
13e616 |
# specific Service ID
|
|
Packit |
13e616 |
any, pkey 0x0ABC : 6 # match any PR/MPR query with a
|
|
Packit |
13e616 |
# specific PKey
|
|
Packit |
13e616 |
srp, target-port-guid 0x1234 : 5 # SRP when SRP Target is located on
|
|
Packit |
13e616 |
# a specified IB port GUID
|
|
Packit |
13e616 |
any, target-port-guid 0x0ABC-0xFFFFF : 6 # match any PR/MPR query with
|
|
Packit |
13e616 |
# a specific target port GUID
|
|
Packit |
13e616 |
any, source-port-guid 0x5678 : 7 # match any PR/MPR query with
|
|
Packit |
13e616 |
# a specific source port
|
|
Packit |
13e616 |
# GUID
|
|
Packit |
13e616 |
any, source-target-port-guid 0x9abcd : 8 # match any PR/MPR query with
|
|
Packit |
13e616 |
# a specific source or target port
|
|
Packit |
13e616 |
# GUID
|
|
Packit |
13e616 |
end-qos-ulps
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
Similar to the full policy definition, matching of PR/MPR queries is done in
|
|
Packit |
13e616 |
order of appearance in the QoS policy file such as the first match takes
|
|
Packit |
13e616 |
precedence, except for the "default" rule, which is applied only if the query
|
|
Packit |
13e616 |
didn't match any other rule.
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
All other sections of the QoS policy file take precedence over the qos-ulps
|
|
Packit |
13e616 |
section. That is, if a policy file has both qos-match-rules and qos-ulps
|
|
Packit |
13e616 |
sections, then any query is matched first against the rules in the
|
|
Packit |
13e616 |
qos-match-rules section, and only if there was no match, the query is matched
|
|
Packit |
13e616 |
against the rules in qos-ulps section.
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
Note that some of these match rules may overlap, so in order to use the
|
|
Packit |
13e616 |
simplified QoS definition effectively, it is important to understand how each
|
|
Packit |
13e616 |
of the ULPs is matched:
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
6.1 IPoIB
|
|
Packit |
13e616 |
IPoIB query is matched by PKey. Default PKey for IPoIB partition is 0x7fff, so
|
|
Packit |
13e616 |
the following three match rules are equivalent:
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
ipoib : <SL>
|
|
Packit |
13e616 |
ipoib, pkey 0x7fff : <SL>
|
|
Packit |
13e616 |
any, pkey 0x7fff : <SL>
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
6.2 SDP
|
|
Packit |
13e616 |
SDP PR query is matched by Service ID. The Service-ID for SDP is
|
|
Packit |
13e616 |
0x000000000001PPPP, where PPPP are 4 hex digits holding the remote TCP/IP Port
|
|
Packit |
13e616 |
Number to connect to. The following two match rules are equivalent:
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
sdp : <SL>
|
|
Packit |
13e616 |
any, service-id 0x0000000000010000-0x000000000001ffff : <SL>
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
6.3 RDS
|
|
Packit |
13e616 |
Similar to SDP, RDS PR query is matched by Service ID. The Service ID for RDS
|
|
Packit |
13e616 |
is 0x000000000106PPPP, where PPPP are 4 hex digits holding the remote TCP/IP
|
|
Packit |
13e616 |
Port Number to connect to. Default port number for RDS is 0x48CA, which makes
|
|
Packit |
13e616 |
a default Service-ID 0x00000000010648CA. The following two match rules are
|
|
Packit |
13e616 |
equivalent:
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
rds : <SL>
|
|
Packit |
13e616 |
any, service-id 0x00000000010648CA : <SL>
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
6.4 iSER
|
|
Packit |
13e616 |
Similar to RDS, iSER query is matched by Service ID, where the the Service ID
|
|
Packit |
13e616 |
is also 0x000000000106PPPP. Default port number for iSER is 0x0CBC, which makes
|
|
Packit |
13e616 |
a default Service-ID 0x0000000001060CBC. The following two match rules are
|
|
Packit |
13e616 |
equivalent:
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
iser : <SL>
|
|
Packit |
13e616 |
any, service-id 0x0000000001060CBC : <SL>
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
6.5 SRP
|
|
Packit |
13e616 |
Service ID for SRP varies from storage vendor to vendor, thus SRP query is
|
|
Packit |
13e616 |
matched by the target IB port GUID. The following two match rules are
|
|
Packit |
13e616 |
equivalent:
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
srp, target-port-guid 0x1234 : <SL>
|
|
Packit |
13e616 |
any, target-port-guid 0x1234 : <SL>
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
Note that any of the above ULPs might contain target port GUID in the PR
|
|
Packit |
13e616 |
query, so in order for these queries not to be recognized by the QoS manager
|
|
Packit |
13e616 |
as SRP, the SRP match rule (or any match rule that refers to the target port
|
|
Packit |
13e616 |
guid only) should be placed at the end of the qos-ulps match rules.
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
6.6 MPI
|
|
Packit |
13e616 |
SL for MPI is manually configured by MPI admin. OpenSM is not forcing any SL
|
|
Packit |
13e616 |
on the MPI traffic, and that's why it is the only ULP that did not appear in
|
|
Packit |
13e616 |
the qos-ulps section.
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
==============================================================================
|
|
Packit |
13e616 |
7. SL2VL Mapping and VL Arbitration
|
|
Packit |
13e616 |
==============================================================================
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
OpenSM cached options file has a set of QoS related configuration parameters,
|
|
Packit |
13e616 |
that are used to configure SL2VL mapping and VL arbitration on IB ports.
|
|
Packit |
13e616 |
These parameters are:
|
|
Packit |
13e616 |
- Max VLs: the maximum number of VLs that will be on the subnet.
|
|
Packit |
13e616 |
- High limit: the limit of High Priority component of VL Arbitration
|
|
Packit |
13e616 |
table (IBA 7.6.9).
|
|
Packit |
13e616 |
- VLArb low table: Low priority VL Arbitration table (IBA 7.6.9) template.
|
|
Packit |
13e616 |
- VLArb high table: High priority VL Arbitration table (IBA 7.6.9) template.
|
|
Packit |
13e616 |
- SL2VL: SL2VL Mapping table (IBA 7.6.6) template. It is a list of VLs
|
|
Packit |
13e616 |
corresponding to SLs 0-15 (Note that VL15 used here means drop this SL).
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
There are separate QoS configuration parameters sets for various target types:
|
|
Packit |
13e616 |
CAs, routers, switch external ports, and switch's enhanced port 0. The names
|
|
Packit |
13e616 |
of such parameters are prefixed by "qos_<type>_" string. Here is a full list
|
|
Packit |
13e616 |
of the currently supported sets:
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
qos_ca_ - QoS configuration parameters set for CAs.
|
|
Packit |
13e616 |
qos_rtr_ - parameters set for routers.
|
|
Packit |
13e616 |
qos_sw0_ - parameters set for switches' port 0.
|
|
Packit |
13e616 |
qos_swe_ - parameters set for switches' external ports.
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
Here's the example of typical default values for CAs and switches' external
|
|
Packit |
13e616 |
ports (hard-coded in OpenSM initialization):
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
qos_ca_max_vls 15
|
|
Packit |
13e616 |
qos_ca_high_limit 0
|
|
Packit |
13e616 |
qos_ca_vlarb_high 0:4,1:0,2:0,3:0,4:0,5:0,6:0,7:0,8:0,9:0,10:0,11:0,12:0,13:0,14:0
|
|
Packit |
13e616 |
qos_ca_vlarb_low 0:0,1:4,2:4,3:4,4:4,5:4,6:4,7:4,8:4,9:4,10:4,11:4,12:4,13:4,14:4
|
|
Packit |
13e616 |
qos_ca_sl2vl 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,7
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
qos_swe_max_vls 15
|
|
Packit |
13e616 |
qos_swe_high_limit 0
|
|
Packit |
13e616 |
qos_swe_vlarb_high 0:4,1:0,2:0,3:0,4:0,5:0,6:0,7:0,8:0,9:0,10:0,11:0,12:0,13:0,14:0
|
|
Packit |
13e616 |
qos_swe_vlarb_low 0:0,1:4,2:4,3:4,4:4,5:4,6:4,7:4,8:4,9:4,10:4,11:4,12:4,13:4,14:4
|
|
Packit |
13e616 |
qos_swe_sl2vl 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,7
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
VL arbitration tables (both high and low) are lists of VL/Weight pairs.
|
|
Packit |
13e616 |
Each list entry contains a VL number (values from 0-14), and a weighting value
|
|
Packit |
13e616 |
(values 0-255), indicating the number of 64 byte units (credits) which may be
|
|
Packit |
13e616 |
transmitted from that VL when its turn in the arbitration occurs. A weight
|
|
Packit |
13e616 |
of 0 indicates that this entry should be skipped. If a list entry is
|
|
Packit |
13e616 |
programmed for VL15 or for a VL that is not supported or is not currently
|
|
Packit |
13e616 |
configured by the port, the port may either skip that entry or send from any
|
|
Packit |
13e616 |
supported VL for that entry.
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
Note, that the same VLs may be listed multiple times in the High or Low
|
|
Packit |
13e616 |
priority arbitration tables, and, further, it can be listed in both tables.
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
The limit of high-priority VLArb table (qos_<type>_high_limit) indicates the
|
|
Packit |
13e616 |
number of high-priority packets that can be transmitted without an opportunity
|
|
Packit |
13e616 |
to send a low-priority packet. Specifically, the number of bytes that can be
|
|
Packit |
13e616 |
sent is high_limit times 4K bytes.
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
A high_limit value of 255 indicates that the byte limit is unbounded.
|
|
Packit |
13e616 |
Note: if the 255 value is used, the low priority VLs may be starved.
|
|
Packit |
13e616 |
A value of 0 indicates that only a single packet from the high-priority table
|
|
Packit |
13e616 |
may be sent before an opportunity is given to the low-priority table.
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
Keep in mind that ports usually transmit packets of size equal to MTU.
|
|
Packit |
13e616 |
For instance, for 4KB MTU a single packet will require 64 credits, so in order
|
|
Packit |
13e616 |
to achieve effective VL arbitration for packets of 4KB MTU, the weighting
|
|
Packit |
13e616 |
values for each VL should be multiples of 64.
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
Below is an example of SL2VL and VL Arbitration configuration on subnet:
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
qos_ca_max_vls 15
|
|
Packit |
13e616 |
qos_ca_high_limit 6
|
|
Packit |
13e616 |
qos_ca_vlarb_high 0:4
|
|
Packit |
13e616 |
qos_ca_vlarb_low 0:0,1:64,2:128,3:192,4:0,5:64,6:64,7:64
|
|
Packit |
13e616 |
qos_ca_sl2vl 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,7
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
qos_swe_max_vls 15
|
|
Packit |
13e616 |
qos_swe_high_limit 6
|
|
Packit |
13e616 |
qos_swe_vlarb_high 0:4
|
|
Packit |
13e616 |
qos_swe_vlarb_low 0:0,1:64,2:128,3:192,4:0,5:64,6:64,7:64
|
|
Packit |
13e616 |
qos_swe_sl2vl 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,7
|
|
Packit |
13e616 |
|
|
Packit |
13e616 |
In this example, there are 8 VLs configured on subnet: VL0 to VL7. VL0 is
|
|
Packit |
13e616 |
defined as a high priority VL, and it is limited to 6 x 4KB = 24KB in a single
|
|
Packit |
13e616 |
transmission burst. Such configuration would suit a VL that needs low latency
|
|
Packit |
13e616 |
and uses a small MTU when transmitting packets. Rest of the VLs are defined as
|
|
Packit |
13e616 |
low priority VLs with different weights, while VL4 is effectively turned off.
|