Blame numad.8

Packit 6ad14e
.TH "numad" "8" "1.0.0" "Bill Gray" "Administration"
Packit 6ad14e
.SH "NAME"
Packit 6ad14e
.LP
Packit 6ad14e
numad \- A user\-level daemon that provides placement advice and process
Packit 6ad14e
management for efficient use of CPUs and memory on systems with NUMA topology.
Packit 6ad14e
.SH "SYNOPSIS"
Packit 6ad14e
.LP
Packit 6ad14e
numad [\fI\-dhvV\fP]
Packit 6ad14e
.br
Packit 6ad14e
.LP
Packit 6ad14e
numad  [\fI\-C 0|1\fP]
Packit 6ad14e
.br
Packit 6ad14e
.LP
Packit 6ad14e
numad  [\fI\-H THP_hugepage_scan_sleep_ms\fP]
Packit 6ad14e
.br
Packit 6ad14e
.LP
Packit 6ad14e
numad  [\fI\-i [min_interval:]max_interval\fP]
Packit 6ad14e
.br
Packit 6ad14e
.LP
Packit 6ad14e
numad  [\fI\-K 0|1\fP]
Packit 6ad14e
.br
Packit 6ad14e
.LP
Packit 6ad14e
numad  [\fI\-l log_level\fP]
Packit 6ad14e
.br
Packit 6ad14e
.LP
Packit 6ad14e
numad  [\fI\-m target_memory_locality\fP]
Packit 6ad14e
.br
Packit 6ad14e
.LP
Packit 6ad14e
numad  [\fI\-p PID\fP]
Packit 6ad14e
.br
Packit 6ad14e
.LP
Packit 6ad14e
numad  [\fI\-r PID\fP]
Packit 6ad14e
.br
Packit 6ad14e
.LP
Packit 6ad14e
numad  [\fI\-R reserved-CPU-list\fP]
Packit 6ad14e
.br
Packit 6ad14e
.LP
Packit 6ad14e
numad  [\fI\-S 0|1\fP]
Packit 6ad14e
.br
Packit 6ad14e
.LP
Packit 6ad14e
numad  [\fI\-t logical_CPU_percent\fP]
Packit 6ad14e
.br
Packit 6ad14e
.LP
Packit 6ad14e
numad  [\fI\-u target_utilization\fP]
Packit 6ad14e
.br
Packit 6ad14e
.LP
Packit 6ad14e
numad  [\fI\-w NCPUS[:MB]\fP]
Packit 6ad14e
.br
Packit 6ad14e
.LP
Packit 6ad14e
numad  [\fI\-x PID\fP]
Packit 6ad14e
.br
Packit 6ad14e
.SH "DESCRIPTION"
Packit 6ad14e
.LP
Packit 6ad14e
Numad is a system daemon that monitors NUMA topology and resource usage. It
Packit 6ad14e
will attempt to locate processes for efficient NUMA locality and affinity,
Packit 6ad14e
dynamically adjusting to changing system conditions.  Numad also provides
Packit 6ad14e
guidance to assist management applications with initial manual binding of CPU
Packit 6ad14e
and memory resources for their processes.  Note that numad is primarily
Packit 6ad14e
intended for server consolidation environments, where there might be multiple
Packit 6ad14e
applications or multiple virtual guests running on the same server system.
Packit 6ad14e
Numad is most likely to have a positive effect when processes can be localized
Packit 6ad14e
in a subset of the system's NUMA nodes.  If the entire system is dedicated to a
Packit 6ad14e
large in-memory database application, for example -- especially if memory
Packit 6ad14e
accesses will likely remain unpredictable -- numad will probably not improve
Packit 6ad14e
performance.
Packit 6ad14e
.SH "OPTIONS"
Packit 6ad14e
.LP
Packit 6ad14e
.TP
Packit 6ad14e
\fB\-C\fR <\fI0|1\fP>
Packit 6ad14e
This option controls whether or not numad treats inactive file cache as
Packit 6ad14e
available memory. By default, numad assumes it can count inactive file cache as
Packit 6ad14e
"free" memory when considering resources to match with processes.  Specify
Packit 6ad14e
\fI\-C 0\fP if numad should instead consider inactive file cache as a consumed
Packit 6ad14e
resource.
Packit 6ad14e
.TP
Packit 6ad14e
\fB\-d\fR
Packit 6ad14e
Debug output in log, sets the log level to LOG_DEBUG.  Same effect as \fI\-l 7\fP.
Packit 6ad14e
.TP
Packit 6ad14e
\fB\-h\fR
Packit 6ad14e
Display usage help information and then exit.
Packit 6ad14e
.TP
Packit 6ad14e
\fB\-H\fR  <\fITHP_scan_sleep_ms\fP>
Packit 6ad14e
Set the desired transparent hugepage scan interval in ms.  The
Packit 6ad14e
.na
Packit 6ad14e
/sys/kernel/mm/tranparent_hugepage/khugepaged/scan_sleep_millisecs
Packit 6ad14e
.ad
Packit 6ad14e
tunable is usually set to 10000ms by the operating system.  The default is
Packit 6ad14e
changed by numad to be 1000ms since it is helpful for the hugepage daemon to be
Packit 6ad14e
more aggressive when memory moves between nodes.  Specifying (\fI\-H 0\fP) will
Packit 6ad14e
cause numad to retain the system default value.  You can also make the hugepage
Packit 6ad14e
daemon more or less aggressive by specifying an alternate value with this
Packit 6ad14e
option.  For example, setting this value to 100ms (\fI\-H 100\fP) might improve
Packit 6ad14e
the performance of workloads which use many transparent hugepages.
Packit 6ad14e
.TP
Packit 6ad14e
\fB\-i\fR <\fI[min_interval:]max_interval\fP>
Packit 6ad14e
Sets the time interval that numad waits between system scans, in seconds to
Packit 6ad14e
<\fImax_interval\fP>. Default <\fImax_interval\fP> is 15 seconds, default
Packit 6ad14e
<\fImin_interval\fP> is 5 seconds.  Setting a <\fImax_interval\fP> of zero will
Packit 6ad14e
cause the daemon to exit.  (This is the normal mechanism to terminate the
Packit 6ad14e
daemon.)  A bigger <\fImax_interval\fP> will decrease numad overhead but also
Packit 6ad14e
decrease responsiveness to changing loads.  The default numad max_interval can
Packit 6ad14e
be changed in the numad.conf file.
Packit 6ad14e
.TP
Packit 6ad14e
\fB\-K\fR <\fI0|1\fP>
Packit 6ad14e
This option controls whether numad keeps interleaved memory spread across NUMA
Packit 6ad14e
nodes, or attempts to merge interleaved memory to local NUMA nodes.  The
Packit 6ad14e
default is to merge interleaved memory.  This is the appropriate setting to
Packit 6ad14e
localize processes in a subset of the system's NUMA nodes.  If you are running
Packit 6ad14e
a large, single-instance application that allocates interleaved memory because
Packit 6ad14e
the workload will have continuous unpredictable memory access patterns (e.g. a
Packit 6ad14e
large in-memory database), you might get better results by specifying \fI\-K
Packit 6ad14e
1\fP to instruct numad to keep interleaved memory distributed.
Packit 6ad14e
.TP
Packit 6ad14e
\fB\-l\fR <\fIlog_level\fP>
Packit 6ad14e
Sets the log level to <\fIlog_level\fP>.  Reasonable choices are 5, 6, or 7.
Packit 6ad14e
The default value is 5.  Note that CPU values are scaled by a factor of 100
Packit 6ad14e
internally and in the numad log files.  Unfortunately, you don't actually have
Packit 6ad14e
that many CPUs.
Packit 6ad14e
.TP
Packit 6ad14e
\fB\-m\fR  <\fItarget_memory_locality\fP>
Packit 6ad14e
Set the desired memory locality threshold to stop moving process memory.  Numad
Packit 6ad14e
might stop retrying to coalesce process memory when more than this percentage
Packit 6ad14e
of the process's memory is already localized in the target node(s).  The
Packit 6ad14e
default is 90%. Numad will frequently localize more than the localization
Packit 6ad14e
threshold percent, but it will not necessarily do so.  Decrease the threshold
Packit 6ad14e
to allow numad to leave more process memory distributed on various nodes.
Packit 6ad14e
Increase the threshold to instruct numad to try to localize more memory.
Packit 6ad14e
Acceptable values are between 50 and 100 percent.  Note that setting the target
Packit 6ad14e
memory locality to 100% might cause numad to continually retry to move memory
Packit 6ad14e
that the kernel will never succesfully move.
Packit 6ad14e
.TP
Packit 6ad14e
\fB\-p\fR <\fIPID\fP>
Packit 6ad14e
Add PID to explicit inclusion list of processes to consider for managing, if
Packit 6ad14e
the process also uses significant resources.  Multiple \fI\-p PID\fP options
Packit 6ad14e
can be specified at daemon start, but after daemon start, only one PID can be
Packit 6ad14e
added to the inclusion list per subsequent numad invocation.  Use with \-S to
Packit 6ad14e
precisely control the scope of processes numad can manage.  Note that the
Packit 6ad14e
specified process will not necessarily be actively managed unless it also meets
Packit 6ad14e
numad's significance threshold -- which is currently 300MB and half of a CPU.
Packit 6ad14e
.TP
Packit 6ad14e
\fB\-r\fR <\fIPID\fP>
Packit 6ad14e
Remove PID from both the explicit inclusion and the exclusion lists of
Packit 6ad14e
processes.  After daemon start, only one PID can be removed from the explicit
Packit 6ad14e
process lists per subsequent numad invocation.  Use with \-S and \-p and \-x to
Packit 6ad14e
precisely control the scope of processes numad can manage.
Packit 6ad14e
.TP
Packit 6ad14e
\fB\-R\fR <\fICPU_LIST\fP>
Packit 6ad14e
Specify a list of CPUs that numad should assume are reserved for non-numad use.
Packit 6ad14e
No processes will be bound to the specified CPUs by numad.  This option is
Packit 6ad14e
effective only when starting numad.  You cannot change reserved CPUs
Packit 6ad14e
dynamically while numad is already running.
Packit 6ad14e
.TP
Packit 6ad14e
\fB\-S\fR <\fI0|1\fP>
Packit 6ad14e
This option controls whether numad scans all system processes or only the
Packit 6ad14e
processes on the explicit inclusion PID list.  The default is to scan all
Packit 6ad14e
processes.  Use \fI\-S 0\fP to scan only the explicit inclusion PID list.  Use
Packit 6ad14e
\fI\-S 1\fP to again scan all system processes (excepting those on the explicit
Packit 6ad14e
exclusion list).  Starting numad as
Packit 6ad14e
.br
Packit 6ad14e
\fInumad \-S 0 \-p <PID-1> \-p <PID-2> \-p <PID-3>\fP
Packit 6ad14e
.br
Packit 6ad14e
will limit scanning, and thus also automatic NUMA management, to only those
Packit 6ad14e
three explicitly specified processes.
Packit 6ad14e
.TP
Packit 6ad14e
\fB\-t\fR  <\fIlogical_CPU_percent\fP>
Packit 6ad14e
Specify the resource value of logical CPUs.  Hardware threads typically share
Packit 6ad14e
most core resources, and so logical CPUs add only a fraction of CPU power for
Packit 6ad14e
many workloads.  By default numad considers logical CPUs to be only 20 percent
Packit 6ad14e
of a dedicated hardware core.
Packit 6ad14e
.TP
Packit 6ad14e
\fB\-u\fR  <\fItarget_utilization\fP>
Packit 6ad14e
Set the desired maximum consumption percentage of a node. Default is 85%.
Packit 6ad14e
Decrease the target value to maintain more available resource margin on each
Packit 6ad14e
node.  Increase the target value to more exhaustively consume node resources.
Packit 6ad14e
If you have sized your workloads to precisely fit inside a NUMA node,
Packit 6ad14e
specifying (\fI\-u 100\fP) might improve system performance by telling numad to
Packit 6ad14e
go ahead and consume all the resources in each node.  It is possible to specify
Packit 6ad14e
values up to 130 percent to oversubscribe CPUs in the nodes, but memory
Packit 6ad14e
utilization is always capped at 100%.  Use oversubscription values very
Packit 6ad14e
carefully.
Packit 6ad14e
.TP
Packit 6ad14e
\fB\-v\fR
Packit 6ad14e
Verbose output in log, sets the log level to LOG_INFO.  Same effect as \fI\-l 6\fP.
Packit 6ad14e
.TP
Packit 6ad14e
\fB\-V\fR
Packit 6ad14e
Display version information and exit.
Packit 6ad14e
.TP
Packit 6ad14e
\fB\-w\fR <\fINCPUS[:MB]\fP>
Packit 6ad14e
Queries numad for the best NUMA nodes to bind an entity that needs
Packit 6ad14e
<\fINCPUS\fP>.  The amount of memory (in MBs) is optional, but should normally
Packit 6ad14e
be specified as well <\fI:MB\fP> so numad can recommend NUMA nodes with
Packit 6ad14e
available CPU capacity and adequate free memory.  This query option can be used
Packit 6ad14e
regardless of whether numad is running as a daemon.  (An invocation using this
Packit 6ad14e
option when numad is not running as a daemon, will not cause the daemon to
Packit 6ad14e
start.) Output of this option is a string that contains a NUMA node list.  For
Packit 6ad14e
example: 2\-3,6.  The recommended node list could be saved in a shell variable
Packit 6ad14e
(e.g., NODES) and then used as the node list parameter in a
Packit 6ad14e
.br
Packit 6ad14e
\fInumactl \-m $NODES \-N $NODES ... \fP
Packit 6ad14e
.br
Packit 6ad14e
command.  See numactl(8).
Packit 6ad14e
.TP
Packit 6ad14e
\fB\-x\fR <\fIPID\fP>
Packit 6ad14e
Add PID to explicit exclusion list of processes to blacklist from managing.
Packit 6ad14e
Multiple \fI\-x PID\fP options can be specified at daemon start, but after
Packit 6ad14e
daemon start, only one PID can be added to the exclusion list per subsequent
Packit 6ad14e
numad invocation.  Use with \-S to precisely control the scope of processes
Packit 6ad14e
numad can manage.
Packit 6ad14e
.SH "FILES"
Packit 6ad14e
.LP
Packit 6ad14e
\fI/usr/bin/numad\fP
Packit 6ad14e
.br
Packit 6ad14e
\fI/etc/numad.conf\fP
Packit 6ad14e
.br
Packit 6ad14e
\fI/var/log/numad.log\fP
Packit 6ad14e
.br
Packit 6ad14e
\fI/var/run/numad.pid\fP
Packit 6ad14e
.SH "ENVIRONMENT VARIABLES"
Packit 6ad14e
.LP
Packit 6ad14e
.TP
Packit 6ad14e
None.
Packit 6ad14e
.SH "EXAMPLES"
Packit 6ad14e
.LP
Packit 6ad14e
Numad can be run as a system daemon and can be managed by the
Packit 6ad14e
standard init mechanisms of the host.
Packit 6ad14e
.LP
Packit 6ad14e
If interactive (manual) control is desired, you can start the daemon manually by typing:
Packit 6ad14e
.LP
Packit 6ad14e
/usr/bin/numad
Packit 6ad14e
.LP
Packit 6ad14e
Subsequent numad invocations while the daemon is running can be used to dynamically change most run-time options.
Packit 6ad14e
.LP
Packit 6ad14e
You can terminate numad from running by typing:
Packit 6ad14e
.LP
Packit 6ad14e
/usr/bin/numad -i0
Packit 6ad14e
.SH "AUTHORS"
Packit 6ad14e
.LP
Packit 6ad14e
Bill Gray <bgray@redhat.com>
Packit 6ad14e
.SH "SEE ALSO"
Packit 6ad14e
.LP
Packit 6ad14e
numactl(8)