OpenSM Performance manager HOWTO
================================

Introduction
============

OpenSM now includes a performance manager which collects port counters from
the subnet and stores them internally in OpenSM.

Some of the features of the performance manager are:

	1) Collect port data and error counters per v1.2.1 spec and store in
	   64 bit internal counts.
	2) Automatic reset of counters when they reach approximately 3/4 full.
	   (While not guaranteeing that counts will not be missed, this does
	   keep counts incrementing as best as possible given the current
	   spec limitations.)
	3) Basic warnings in the OpenSM log on "critical" errors like symbol
	   errors.
	4) Automatically detects "outside" resets of counters and adjusts to
	   continue collecting data.
	5) Can be run when OpenSM is in standby or inactive states in
	   addition to master state.

Known issues are:

	1) Data counters will be lost on high data rate links.  Sweeping the
	   fabric fast enough for even a DDR link is not practical.
	2) Default partition support only.


Setup and Usage
===============

Using the Performance Manager consists of 3 steps:

	1) compiling in support for the perfmgr (Optionally: the console
	   socket as well)
	2) enabling the perfmgr and console in opensm.conf
	3) retrieving data which has been collected.
	   3a) using console to "dump data"
	   3b) using a plugin module to store the data to your own
	       "database"

Step 1: Compile in support for the Performance Manager
------------------------------------------------------

At this time, it is really best to enable the console socket option as well.
OpenSM can be run in an "interactive" mode.  But with the console socket
option turned on one can also make a connection to a running OpenSM.  By
default, only "loopback" is enabled with the console with socket being a
compile time option.  Regardless, please be aware of your network security
configuration for as the commands presented in the console can affect the
operation of your subnet.


Step 2: Enable the perfmgr and console in opensm.conf
-----------------------------------------------------

Turning the Performance Manager on is pretty easy, set the following options in
the opensm.conf config file.  (Default location is
/usr/local/etc/opensm/opensm.conf)

	# Turn it all on
	perfmgr TRUE

	# redirection enable
	perfmgr_redir TRUE

	# sweep time in seconds
	perfmgr_sweep_time_s 180

	# Max outstanding queries
	perfmgr_max_outstanding_queries 500

	# Ignore CAs on sweep
	perfmgr_ignore_cas FALSE

	# Remove missing nodes from DB
	perfmgr_rm_nodes TRUE

	# Log error counters to opensm.log
	perfmgr_log_errors TRUE

	# Query PerfMgt Get(ClassPortInfo) for extended capabilities
	# Extended capabilities include 64 bit extended counters
	# and transmit wait support
	perfmgr_query_cpi TRUE

	# Log xmit_wait errors
	perfmgr_xmit_wait_log FALSE

	# If logging xmit_wait's; set threshold
	perfmgr_xmit_wait_threshold 65535

	# Dump file to dump the events to
	event_db_dump_file /var/log/opensm_port_counters.log

Also, enable the console socket and configure the port for it to listen to if
desired.

	# console [off|local|loopback|socket]
	console socket

	# Telnet port for console (default 10000)
	console_port 10000

	"local" is only useful if you run OpenSM in the foreground.


Step 3: Retrieve data which has been collected
----------------------------------------------

Step 3a: Using console dump function
------------------------------------

The console command "perfmgr dump_counters" will dump counters to the file
specified in the opensm.conf file.  In the example above
"/var/log/opensm_port_counters.log"

Example output is below:

<snip>
"SW1 wopr ISR9024D (MLX4 FW)" 0x8f10400411f56 port 1 (Since Mon May 12 13:27:14 2008)
     symbol_err_cnt       : 0
     link_err_recover     : 0
     link_downed          : 0
     rcv_err              : 0
     rcv_rem_phys_err     : 0
     rcv_switch_relay_err : 2
     xmit_discards        : 0
     xmit_constraint_err  : 0
     rcv_constraint_err   : 0
     link_integrity_err   : 0
     buf_overrun_err      : 0
     vl15_dropped         : 0
     xmit_data            : 470435
     rcv_data             : 405956
     xmit_pkts            : 8954
     rcv_pkts             : 6900
     unicast_xmit_pkts    : 0
     unicast_rcv_pkts     : 0
     multicast_xmit_pkts  : 0
     multicast_rcv_pkts   : 0
</snip>


Step 3b: Using a plugin module
------------------------------

If you want a more automated method of retrieving the data OpenSM provides a
plugin interface to extend OpenSM.  The header file is osm_event_plugin.h.
The functions you register with this interface will be called when data is
collected.  You can then use that data as appropriate.

An example plugin can be configured at compile time using the
"--enable-default-event-plugin" option on the configure line.  This plugin is
very simple.  It logs "events" received from the performance manager to a log
file.  I don't recommend using this directly but rather use it as a template to
create your own plugin.