.\"

.de CO

\dB\\$1\fP \fI\\$2\fP

..

.TH CRASH 8

.SH NAME

crash \- Analyze Linux crash dump data or a live system

.SH SYNOPSIS

.B crash

[\fIOPTION\fR]... \fINAMELIST MEMORY-IMAGE[@ADDRESS]    (dumpfile form)\fR

.br

.B crash

[\fIOPTION\fR]... \fI[NAMELIST]                         (live system form)\fR

.SH DESCRIPTION

.B Crash

is a tool for interactively analyzing the state of the Linux system

while it is running, or after a kernel crash has occurred and a 

core dump has been created by the 

.I netdump,

.I diskdump,

.I LKCD,

.I kdump,

.I xendump

or

.I kvmdump

facilities.  It is loosely based on the SVR4 UNIX crash 

command, but has been significantly enhanced

by completely merging it with the 

.B gdb(1)

debugger. The marriage of the two effectively combines the 

kernel-specific nature of the traditional UNIX crash utility with the 

source code level debugging capabilities of 

.B gdb(1). 

In the

.I dumpfile form,

both a NAMELIST and a MEMORY-IMAGE argument must be entered.

In the

.I live system form,

the NAMELIST argument must be entered if 

the kernel's

.I vmlinux 

file is not

located in a known location, such as the 

.I /usr/lib/debug/lib/modules/<kernel-version>

directory.  

The

.B crash

utility has also been extended to support the analysis

of dumpfiles generated by a crash of the Xen hypervisor.  In that

case, the NAMELIST argument must be that of the 

.I xen-syms

binary.

Live system analysis is not supported for the Xen hypervisor.

The 

.B crash

utility command set consists of common kernel core analysis tools 

such as kernel stack back traces of all processes, source code disassembly,

formatted kernel structure and variable displays, virtual memory data, 

dumps of linked-lists, etc., along with several commands that delve 

deeper into specific kernel subsystems.  Appropriate 

.B gdb

commands may also be entered, which in

turn are passed on to the 

.B gdb 

module for execution. 

If desired, commands may be placed in either a

.I $HOME/.crashrc

file and/or in a 

.I .crashrc 

file in the current directory.

During initialization, the commands in 

.I $HOME/.crashrc

are executed first, followed by those in the 

.I ./.crashrc

file.

The 

.B crash 

utility is designed to be independent of Linux version 

dependencies. When new kernel source code impacts the

correct functionality of 

.B crash

and its command set, the utility will 

be updated to recognize new kernel code changes, while

maintaining backwards compatibility with earlier releases.

.SH OPTIONS

.de BS

\fB\\$1\fP\ \fR\\$2\fP

..

.TP

.BI NAMELIST

This is a pathname to an uncompressed kernel image

(a 

.I vmlinux 

file), or a Xen hypervisor image (a 

.I xen-syms

file) 

which has been compiled with the "-g" option.

If using the

.I dumpfile form,

a 

.I vmlinux 

file may be compressed in either gzip or bzip2 formats.

.TP

.BI MEMORY-IMAGE[@ADDRESS]

A kernel core dump file created by the

.I netdump,

.I diskdump,

.I LKCD

.I kdump,

.I xendump

or

.I kvmdump

facilities.  

If a MEMORY-IMAGE argument is not entered, the session will be invoked on

the live system, which typically requires root privileges because of

the device file used to access system RAM.  By default,

.I /dev/crash

will be used if it exists.  If it does not exist, then

.I /dev/mem 

will be used; but if the kernel has been configured 

with 

.B CONFIG_STRICT_DEVMEM, 

then

.I /proc/kcore 

will be used.

It is permissible to explicitly enter 

.I /dev/crash, 

.I /dev/mem

or 

.I /proc/kcore.

An @ADDRESS value must be appended to the MEMORY-IMAGE if the dumpfile

is a raw RAM dumpfile that has no header information describing the file

contents.  Multiple MEMORY-IMAGE@ADDRESS ordered pairs may be entered,

with each dumpfile containing a contiguous block of RAM, where the ADDRESS

value is the physical start address of the block expressed in hexadecimal.

The physical address value(s) will be used to create a temporary ELF header

in /var/tmp, which will only exist during the crash session.  If a raw RAM

dumpile represents a live memory source, such as that specified by the QEMU

mem-path argument of a memory-backend-file object, then "live:" must be

prepended to the MEMORY-IMAGE name.

.TP

.BI mapfile

If the NAMELIST file is not the same kernel that is

running (live system form), or the kernel that was running 

when the system crashed (dumpfile form), then the

.I System.map

file of the original kernel should be entered on the command line.

.P

.BI -h \ [option]

.br

.BI \--help \ [option]

.RS

Without an 

.I option

argument, display a 

.B crash

usage help message.  If the 

.I option

argument is a

.B crash

command name, the help page for that command is displayed.  If it is

the string "input", a page describing the various

.B crash

command line input options is displayed.  If it is the string "output", a

page describing command line output options is displayed.  

If it is the string "all", then all of the possible help messages

are displayed.  After the help message is displayed, 

.B crash

exits.

.RE

.TP

.B \-s

Silently proceed directly to the "crash>" prompt without displaying

any version, GPL, or 

.B crash

initialization data during startup, and by default, runtime command

output is not passed to any scrolling command.

.TP

.BI \-i \ file

Execute the command(s) contained in

.I file

prior to displaying the "crash>" prompt for interactive

user input.

.TP

.BI \-d \ num

Set the internal debug level.

The higher the number, the more debugging data will be printed when

.B crash

initializes and runs.

.TP

.B \-S

Use 

.I /boot/System.map

as the

.I mapfile\fP.

.TP

.B \-e \fIvi | emacs\fR

Set the 

.B readline(3) 

command line editing mode to "vi" or "emacs".  The default

editing mode is "vi".

.TP

.B \-f

Force the usage of a compressed 

.I vmlinux 

file if its original name

does not start with "vmlinux".

.TP

.B \-k

Indicate that the NAMELIST file is an LKCD "Kerntypes" debuginfo file.

.TP

.BI -g \ [namelist]

Determine if a  

.I vmlinux 

or 

.I xen-syms 

namelist file contains debugging data.

.TP

.B \-t

Display the system-crash timestamp and exit. 

.TP

.B \-L

Attempt to lock all of its virtual address space into memory by calling mlockall(MCL_CURRENT|MCL_FUTURE) 

during initialization.  If the system call fails, an error message will be displayed,

but the session continues.

.TP

.BI \-c \ tty-device

Open the 

.I tty-device 

as the console used for debug messages.

.TP

.BI \-p \ page-size

If a processor's page size cannot be determined by the dumpfile,

and the processor default cannot be used, use 

.I page-size.

.TP

.BI \-o \ filename

Only used with the MEMORY-IMAGE@ADDRESS format for raw RAM dumpfiles, 

specifies a filename of a new ELF vmcore that will be created and used

as the dumpfile.  It will be saved to allow future use as a standalone

vmcore, replacing the original raw RAM dumpfile.

.P

.B -m \fIoption=value\fR

.br

.B --machdep \fIoption=value\fR

.RS

Pass an option and value pair to machine-dependent code.  These

architecture-specific option/pairs should only be

required in very rare circumstances:

.P

.nf

X86_64:

  phys_base=<physical-address>

  irq_eframe_link=<value>

  irq_stack_gap=<value>

  max_physmem_bits=<value>

  kernel_image_size=<value>

  vm=orig       (pre-2.6.11 virtual memory address ranges)

  vm=2.6.11     (2.6.11 and later virtual memory address ranges)

  vm=xen        (Xen kernel virtual memory address ranges)

  vm=xen-rhel4  (RHEL4 Xen kernel virtual address ranges)

  vm=5level     (5-level page tables)

  page_offset=<PAGE_OFFSET-value>

PPC64:

  vm=orig

  vm=2.6.14     (4-level page tables)

IA64:

  phys_start=<physical-address>

  init_stack_size=<size>

  vm=4l         (4-level page tables)

ARM:  

  phys_base=<physical-address>

ARM64:  

  phys_offset=<physical-address>

  kimage_voffset=<kimage_voffset-value>

  max_physmem_bits=<value>

X86:

  page_offset=<CONFIG_PAGE_OFFSET-value>

.fi

.RE

.TP

.B \-x

Automatically load extension modules from a particular directory. 

If a directory is specified in the 

.B CRASH_EXTENSIONS

shell environment

variable, then that directory will be used.  Otherwise 

.I /usr/lib64/crash/extensions

(64-bit architectures) or 

.I /usr/lib/crash/extensions

(32-bit architectures) will be used;

if they do not exist, then the 

.I ./extensions directory will be used.

.TP

.BI --active

Track only the active task on each cpu.

.TP

.BI --buildinfo

Display the crash binary's build date, the user ID of the builder,

the hostname of the machine where the build was done, the target 

architecture, the version number, and the compiler version.

.TP

.BI --memory_module \ modname

Use the

.I modname

as an alternative kernel module to the 

.I crash.ko

module that creates the

.I /dev/crash

device.

.TP

.BI --memory_device \ device

Use

.I device

as an alternative device to the 

.I /dev/crash, /dev/mem

or

.I /proc/kcore

devices.

.TP

.BI --log \ dumpfile

Dump the contents of the kernel log buffer.  A kernel namelist

argument is not necessary, but the dumpfile must contain the

VMCOREINFO data taken from the original /proc/vmcore ELF header.

.TP

.B --no_kallsyms

Do not use kallsyms-generated symbol information contained within 

kernel module object files.

.TP

.B --no_modules

Do not access or display any kernel module related information. 

.TP

.B --no_ikconf

Do not attempt to read configuration data that was built into kernels

configured with 

.B CONFIG_IKCONFIG.

.TP

.B --no_data_debug

Do not verify the validity of all structure member offsets and structure 

sizes that it uses.

.TP

.B --no_kmem_cache

Do not initialize the kernel's slab cache infrastructure, and commands that

use kmem_cache-related data will not work.

.TP

.B --no_elf_notes

Do not use the registers from the ELF NT_PRSTATUS notes saved in a compressed kdump header

for backtraces.

.TP

.B --kmem_cache_delay

Delay the initialization of the kernel's slab cache infrastructure until

it is required by a run-time command.

.TP

.B --readnow

Pass this flag to the embedded 

.B gdb

module, which will override its two-stage strategy that it uses for reading

symbol tables from the NAMELIST.

.TP

.B --smp

Specify that the system being analyzed is an SMP kernel.

.P

.B -v

.br

.B --version

.RS

Display the version of the

.B crash

utility, the version of the embedded

.B gdb

module, GPL information, and copyright notices.

.RE

.TP

.BI --cpus \ number

Specify the 

.I number 

of cpus in the SMP system being analyzed.

.TP

.BI --osrelease \ dumpfile

Display the OSRELEASE vmcoreinfo string from a kdump 

.I dumpfile 

header.

.TP

.BI --hyper

Force the session to be that of a Xen hypervisor.

.TP

.BI --p2m_mfn \ pfn

When a Xen Hypervisor or its dom0 kernel crashes, the dumpfile

is typically analyzed with either the Xen hypervisor or the dom0 kernel.

It is also possible to analyze any of the guest domU kernels if 

the pfn_to_mfn_list_list 

.I pfn 

value of the guest kernel is passed on the

command line along with its NAMELIST and the  

dumpfile.

.TP

.BI --xen_phys_start \ physical-address

Supply the base physical address of the Xen hypervisor's text and static data

for older xendump dumpfiles that did not pass that information in the dumpfile

header.

.TP

.B --zero_excluded

If the makedumpfile(8) facility has filtered a compressed kdump dumpfile to

exclude various types of non-essential pages, or has marked a compressed or

ELF kdump dumpfile as incomplete due to an ENOSPC or other error during its

creation, any attempt to read missing pages will fail.  With this flag, reads

from any of those pages will return zero-filled memory.

.TP

.B --no_panic

Do not attempt to find the task that was running when the kernel crashed.

Set the initial context to that of the "swapper" task on cpu 0.

.TP

.B --more

Use 

.I /bin/more 

as the command output scroller, overriding the default of 

.I /usr/bin/less

and any settings in either 

.I ./.crashrc 

or

.I $HOME/.crashrc. 

.TP

.B --less

Use 

.I /usr/bin/less 

as the command output scroller, overriding 

any settings in either 

.I ./.crashrc 

or 

.I $HOME/.crashrc. 

.TP

.B --hex

Set the default command output radix to 16, overriding the default radix of 10,

and any radix settings in either

.I ./.crashrc

or

.I $HOME/.crashrc.

.TP

.B --dec

Set the default command output radix to 10, overriding any 

radix settings in either

.I ./.crashrc

or 

.I $HOME/.crashrc.  This is the default radix setting.

.TP

.B --CRASHPAGER

Use the output paging command defined in the 

.B CRASHPAGER

shell environment

variable, overriding any settings in either 

.I ./.crashrc 

or 

.I $HOME/.crashrc.

.TP

.B --no_scroll

Do not pass run-time command output to any scrolling command. 

.TP

.B --no_strip

Do not strip cloned kernel text symbol names.

.TP

.B --no_crashrc

Do not execute the commands in either

.I $HOME/.crashrc

or 

.I ./.crashrc.

.TP

.BI --mod \ directory

When loading the debuginfo data of kernel modules with the 

.I mod -S

command, search for their object files in

.I directory

instead of in the standard location.

.TP

.BI --kaslr \ offset | auto

If an x86_64 kernel was configured with 

.B CONFIG_RANDOMIZE_BASE,

the offset value is equal to the difference between the symbol values 

compiled into the vmlinux file and their relocated KASLR values.  If set to

auto, the KASLR offset value will be automatically calculated.

.TP

.BI --reloc \ size

When analyzing live x86 kernels that were configured with a

.B CONFIG_PHYSICAL_START

value that is larger than its

.B CONFIG_PHYSICAL_ALIGN

value, then it will be necessary to enter

a relocation size equal to the difference between the two values.

.TP

.BI --hash \ count

Set the number of internal hash queue heads used for list gathering

and verification.  The default count is 32768.

.TP

.B --minimal

Bring up a session that is restricted to the 

.I log, dis, rd, sym, eval, set

and 

.I exit

commands.  This option may provide a way to

extract some minimal/quick information from a corrupted or truncated

dumpfile, or in situations where one of the several kernel subsystem

initialization routines would abort the 

.B crash

session.

.TP

.BI --kvmhost \ [32|64]

When examining an x86 KVM guest dumpfile, this option specifies

that the KVM host that created the dumpfile was an x86 (32-bit) 

or an x86_64 (64-bit) machine, overriding the automatically 

determined value.

.TP

.BI --kvmio \ <size>

override the automatically-calculated KVM guest I/O hole size.

.TP

.BI --offline \ [show|hide]

Show or hide command output that is related to offline cpus.  The

default setting is show.

.SH COMMANDS

Each 

.B crash

command generally falls into one of the following categories:

.TP

.I Symbolic display

Displays of kernel text/data, which take full advantage of the power of 

.B gdb

to format and display data structures symbolically.

.TP

.I System state

The majority of 

.B crash

commands consist of a set of "kernel-aware" 

commands, which delve into various kernel subsystems on a system-wide 

or per-task basis. 

.TP

.I Utility functions

A set of useful helper commands serving various purposes, some simple, 

others quite powerful. 

.TP

.I Session control

Commands that control the 

.B crash

session itself.

.PP

The following alphabetical list consists of a very simple overview of each 

.B crash

command.

However, since individual commands often have several options resulting in 

significantly different output, it is suggested that the full description

of each command be viewed by executing

.I crash\ -h\ \fI<command>\fP, 

or during a 

.B crash

session by simply entering

.B \fIhelp command\fP. 

.TP

.I *

"pointer to" is shorthand for either the

.I struct

or

.I union

commands.  It displays the contents of a kernel structure or union.

.TP

.I alias

creates a single-word alias for a command.

.TP

.I ascii

displays an ascii chart or translates a numeric value into its ascii components.

.TP

.I bt

displays a task's kernel-stack backtrace.  If it is given the

.I \-a

option, it displays the stack traces of the active tasks on all CPUs.

It is often used with the

.I foreach

command to display the backtraces of all tasks with one command.

.TP

.I btop

translates a byte value (physical offset) to its page number.

.TP

.I dev

displays data concerning the character and block device

assignments, I/O port usage, I/O memory usage, and PCI device data. 

.TP

.I dis

disassembles memory, either entire kernel functions, from a

location for a specified number of instructions, or from the start of a

function up to a specified memory location.

.TP

.I eval

evaluates an expression or numeric type and displays the result

in hexadecimal, decimal, octal and binary.

.TP

.I exit

causes

.B crash

to exit.

.TP

.I extend

dynamically loads or unloads 

.B crash

shared object extension modules.

.TP

.I files

displays information about open files in a context.

.TP

.I foreach

repeats a specified command for the specified (or all) tasks

in the system.

.TP

.I fuser

displays the tasks using the specified file or socket.

.TP

.I gdb

passes its argument to the embedded

.B gdb

module.  It is useful for executing

.B gdb 

commands that have the same name as

.B crash

commands.

.TP

.I help

alone displays the command menu; if followed by a command name, a full

description of a command, its options, and examples are displayed.

Its output is far more complete and useful than this man page.

.TP

.I ipcs

displays data about the System V IPC facilities.

.TP

.I irq

displays data concerning interrupt request numbers and

bottom-half interrupt handling. 

.TP

.I kmem

displays information about the use of kernel memory.

.TP

.I list

displays the contents of a linked list.

.TP

.I log

displays the kernel log_buf contents in chronological order.

.TP

.I mach

displays data specific to the machine type.

.TP

.I mod

displays information about the currently installed kernel modules,

or adds or deletes symbolic or debugging information about specified kernel

modules.

.TP

.I mount

displays information about the currently-mounted filesystems.

.TP

.I net

display various network related data.

.TP

.I p

passes its arguments to the

.B gdb

"print" command for evaluation and display.

.TP

.I ps

displays process status for specified, or all, processes

in the system.

.TP

.I pte

translates the hexadecimal contents of a PTE into its physical

page address and page bit settings.

.TP

.I ptob

translates a page frame number to its byte value.

.TP

.I ptov

translates a hexadecimal physical address into a kernel 

virtual address.

.TP

.I q

is an alias for the "exit" command.

.TP

.I rd

displays the contents of memory, with the output formatted

in several different manners.

.TP

.I repeat

repeats a command indefinitely, optionally delaying a given

number of seconds between each command execution.

.TP

.I runq

displays the tasks on the run queue.

.TP

.I search

searches a range of user or kernel memory space for given value.

.TP

.I set

either sets a new context, or gets the current context for

display.

.TP

.I sig

displays signal-handling data of one or more tasks.

.TP

.I struct

displays either a structure definition or the contents of a

kernel structure at a specified address.

.TP

.I swap

displays information about each configured swap device.

.TP

.I sym

translates a symbol to its virtual address, or a static 

kernel virtual address to its symbol -- or to a symbol-plus-offset value,

if appropriate.

.TP

.I sys

displays system-specific data.

.TP

.I task

displays the contents of a task_struct.

.TP

.I tree

displays the contents of a red-black tree or a radix tree.

.TP

.I timer

displays the timer queue entries, both old- and new-style,

in chronological order.

.TP

.I union

is similar to the

.I struct

command, except that it works on kernel unions.

.TP

.I vm

displays basic virtual memory information of a context.

.TP

.I vtop

translates a user or kernel virtual address to its physical

address.

.TP

.I waitq

walks the wait queue list displaying the tasks which 

are blocked on the specified wait queue.

.TP

.I whatis

displays the definition of structures, unions, typedefs or

text/data symbols.

.TP

.I wr

modifies the contents of memory on a live system. 

It can only be used if

.I /dev/mem

is the device file being used to access system RAM, and should obviously be used with great care.

.PP

When 

.B crash

is invoked with a Xen hypervisor binary as the NAMELIST, the

command set is slightly modified.  The

.I *, alias, ascii, bt, dis, eval, exit, extend,

.I gdb, help, list, log, p, pte, rd, repeat,

.I search, set, struct, sym, sys, union,

.I whatis, wr

and 

.I q

commands are the same as above.  The following commands

are specific to the Xen hypervisor:

.TP

.I domain

displays the contents of the domain structure for selected, or all, domains.

.TP

.I doms

displays domain status for selected, or all, domains.

.TP

.I dumpinfo

displays Xen dump information for selected, or all, cpus.

.TP

.I pcpus

displays physical cpu information for selected, or all, cpus.

.TP

.I vcpus

displays vcpu status for selected, or all, vcpus.

.SH FILES

.TP

.I .crashrc

Initialization commands.  The file can be located in the user's

.B HOME 

directory and/or the current directory.  Commands found in the

.I .crashrc

file in the 

.B HOME

directory are executed before those in the current directory's 

.I .crashrc

file.

.SH ENVIRONMENT

.TP

.B EDITOR

Command input is read using

.BR readline(3).

If

.B EDITOR

is set to

.I emacs

or

.I vi

then suitable keybindings are used.  If 

.B EDITOR

is not set, then

.I vi

is used.  This can be overridden by

.B set vi

or 

.B set emacs

commands located in a

.IR .crashrc 

file, or by entering

.B -e emacs

on the

.B crash

command line.

.TP

.B CRASHPAGER

If

.B CRASHPAGER

is set, its value is used as the name of the program to which command output will be sent. 

If not, then command output is sent to 

.B /usr/bin/less -E -X 

by default.

.TP

.B CRASH_MODULE_PATH

Specifies an alternative directory tree to search for kernel module

object files.

.TP

.B CRASH_EXTENSIONS

Specifies a directory containing extension modules that will be loaded

automatically if the 

.B -x

command line option is used.

.SH NOTES

.PP

If

.B crash

does not work, look for a newer version: kernel evolution frequently makes

.B crash

updates necessary.

.PP

The command

.B set scroll off

will cause output to be sent directly to

the terminal rather than through a paging program.  This is useful,

for example, if you are running

.B crash

in a window of

.BR emacs .

.SH AUTHOR

Dave Anderson <anderson@redhat.com> wrote

.B crash.

.TP

Jay Fenlason <fenlason@redhat.com> and Dave Anderson <anderson@redhat.com> wrote this man page.

.SH "SEE ALSO"

.PP

The

.I help

command within

.B crash

provides more complete and accurate documentation than this man page.

.PP

.I https://github.com/crash-utility

- the home page of the

.B crash

utility.

.PP

.BR netdump (8),

.BR gdb (1),

.BR makedumpfile(8)