White Paper: Crash Utility
Original author: Dave Anderson
Converted to reStructuredText by Rui Yang with AI assistance.
Note
Some of this documentation may be out of sync with the current code and needs updating. Patches in this area are welcome; please send them to the crash-utility mailing list.
Content
Abstract
The crash analysis utility is loosely based on the SVR4 UNIX crash
command, but has been significantly enhanced by completely merging it with the
GNU gdb debugger. The marriage of the two effectively combines the
kernel-specific nature of the traditional UNIX crash utility with the
source code level debugging capabilities of gdb. The utility can be used
to investigate:
Live Linux systems
Linux kernel core dumps created by the Kdump facility
Compressed Linux kernel core dumps created by the makedumpfile command (from kdump dumpfiles)
Linux kernel core dumps created from by the Red Hat Netdump facility
Linux kernel core dumps created from by the Red Hat Diskdump facility
Compressed Linux kernel core dumps created by the Red Hat Diskdump facility
Xen host Linux kernel core dumps created by the Kdump facility
Xen guest Linux kernel core dumps created by the original xendump facility
Xen guest Linux kernel core dumps created by the ELF-format xendump facility
Xen hypervisor core dumps created by the Kdump facility
KVM guest Linux kernel core dumps created by the
virsh dumpfacilitys390 Linux kernel core dumps created by the IBM standalone core dump facility.
s390x Linux kernel core dumps created by the IBM standalone core dump facility.
Linux kernel core dumps created by the LKCD (Linux Kernel Crash Dumps) Sourceforge project
Linux kernel core dumps created by the Mcore patch offered by Mission Critical Linux
The current set of commands consist of common kernel core analysis tools such
as kernel stack back traces of all processes, source code disassembly,
formatted kernel structure and variable displays, virtual memory data, dumps
of linked-lists, etc., along with several commands that delve deeper into
specific kernel subsystems. Relevant gdb commands may also be entered,
which in turn are passed on to the gdb module for execution.
The crash utility is designed to be independent of Linux version
dependencies. When new kernel source code impacts the correct functionality of
crash and its command set, the utility will be updated to recognize new
kernel code changes while maintaining backwards compatibility with earlier
releases. The most current version of the crash utility may be found here:
https://crash-utility.github.io/
Why Crash?
The Linux operating system originally lacked a built-in, traditional UNIX-like kernel crash dump facility. This was initially addressed by the Mission Critical Linux Mcore kernel patch and the LKCD (Linux Kernel Crash Dump) kernel patch from SGI in 1999, and later by the Red Hat Netdump facility in 2002, and the Red hat Diskdump facility in 2004. The upstream Linux community finally settled upon the adoption of the Kdump crash dump facility in 2006.
However, the creation of a kernel crash dump file is only half of the picture; a utility is required to be able to recognize the dumpfile format in order to read it, and to offer a useful set of commands to make sense of it.
Furthermore, to examine the contents of a live system's kernel internals from
user space, the only readily available option has been to use gdb on
/proc/kcore. While gdb is an incredibly powerful tool, it is designed
to debug user programs, and is not at all "kernel-aware". Consequently, using
gdb alone has limited usefulness when looking at kernel memory,
essentially constrained to the printing of kernel data structures if the
vmlinux file was built with the -g C flag, the disassembly of kernel
text, and raw data dumps. Furthermore, distributions such as Red Hat
Enterprise Linux have limited the access to /proc/kcore, making it
unusable as a kernel memory source.
As far as kernel crash dump files are concerned, the Red Hat Netdump and
uncompressed Diskdump facilities, and the Kdump facility create dump files
that are readable by gdb, but aside from giving it the capability of
displaying the panicking task's stack trace, it has the same constraints as
when reading /proc/kcore. However, gdb cannot read LKCD, Mcore, Xen or
s390/s390x dump files.
That being the state of things, the crash utility was developed as a
convenient means to cover all bases, i.e., all listed
dumpfile formats as well as live systems. Moreover, it is
also designed to be easily enhanced to suit the specific needs of the kernel
developers or analysts using it; the builtin command set can easily be
extended or enhanced, and external command modules may be written and
dynamically attached.
Prerequisites
The crash utility has the following prerequisites:
- kernel object file:
vmlinuxkernel object file, often referred to as the namelist in thisdocument, which must have been built with the-gC flag so that itwill contain the debug data required for symbolic debugging.In RHEL3 installations, the
vmlinuxfile associated with the running kernel is split into two files, a stripped version found in the/bootdirectory; which has have the operating system release string appended to it, for example,vmlinux-2.4.21-4.ELsmp. The stripped file in/bootcontains a link to its associated debuginfo file, which is located in the/usr/lib/debug/bootdirectory.In RHEL4, RHEL5 and RHEL6 installations, the
vmlinuxfile is part of the kernel debuginfo package, and is found in the relevant/usr/lib/debug/lib/modules/<release>directory.Ideally the kernel object file is the same kernel object file that is associated with the memory image. However, in circumstances where the
vmlinuxfile associated with the crash dump or live system was not built with the-gflag, there are work-arounds discussed later in the Invocation section.- memory image:
/dev/memor its replacement in RHEL4/RHEL5/RHEL6, the/dev/crashdriver. If no dump file argument is issued on thecrashcommand line, livesystem memory will be used by default. When examining a live system, rootprivileges are required.- platform processor types:
crashutility is actively developed and tested on the x86, x86_64,ia64, ppc64, arm, s390 and s390x processors. Legacy support for the Alpha and32-bit PowerPC platforms exists, but no longer actively maintained.- Linux kernel versions:
crashutility is backwards-compatible to at least Red Hat 6.0 (Linuxversion 2.2.5-15), up to Red Hat Enterprise Linux 5 (Linux version 2.6.18+).Due to the constantly shifting sands of the upstream kernel internals,immediate support for the latest kernel versions cannot be guaranteed.However, modifications are constantly being implemented to support changes inupstream kernel versions. The intent has always been to make the utilityindependent of Linux version dependencies, building in recognition of majorkernel code changes so as to adapt to new kernel versions, while maintainingbackwards compatibility.
Build Procedure
Starting with the RHEL3 release, the crash utility is automatically
installed during system installation if the Development Tools package set
is selected. However, for all other kernel versions, or if it was not selected
during system installation, the binary RPM can be installed, or if desired,
the sources re-built and installed.
If the crash utility is not pre-installed, and if all dependencies are met
on the target system, install the binary RPM like so:
# rpm -ivh crash-4.0-8.11.i386.rpm
Preparing... ########################################### [100%]
1:crash ########################################### [100%]
#
The crash executable will be installed in the /usr/bin directory.
Alternatively, the crash source code can be rebuilt. The crash
utility's source files come packaged in two typical formats, a
compressed tar image. or a source RPM file,
So, for example, crash version 4.0-8.11 can be built from either
crash-4.0-8.11.tar.gz or crash-4.0-8.11.src.rpm.
The latest "upstream" version of the crash utility, available in both file
formats, can be found here:
https://crash-utility.github.io/
In either case, the source file layout consists of a top-level directory
containing a set of crash-specific files, a compressed tar image
containing the full, unmodified, gdb source tree, and a small number of
modified gdb files required to merge the two entities. The build procedure
does the following:
the unmodified
gdbsources are extracted into a subdirectory of the top-levelcrashsource directory, and overlayed by the small set of modifiedgdbfiles.the files in the
gdbsource tree are built first, creating thelibbfd.a,libreadline.a,libopcodes.a,libiberty.aandlibgdb.alibraries.the
crashsources files in the top-level directory are then compiled into acrashlib.alibrary.the objects are all linked into the
crashexecutable, located in the top-level directory.
Depending upon the speed of the host system, the complete build may take
several minutes, primarily due to the time consumed by the build of the
gdb portion.
Building from the tar image
To build from the compressed tar image, simply uncompress/extract the source
files, cd into the resultant source directory, and enter make:
# tar xvzmf crash-4.0-8.11.tar.gz
crash-4.0-8.11/
crash-4.0-8.11/main.c
crash-4.0-8.11/tools.c
crash-4.0-8.11/global_data.c
crash-4.0-8.11/memory.c
crash-4.0-8.11/filesys.c
crash-4.0-8.11/help.c
crash-4.0-8.11/task.c
crash-4.0-8.11/kernel.c
crash-4.0-8.11/test.c
crash-4.0-8.11/gdb_interface.c
crash-4.0-8.11/configure.c
crash-4.0-8.11/net.c
crash-4.0-8.11/dev.c
crash-4.0-8.11/alpha.c
crash-4.0-8.11/x86.c
crash-4.0-8.11/ppc.c
crash-4.0-8.11/ia64.c
crash-4.0-8.11/s390.c
crash-4.0-8.11/s390x.c
crash-4.0-8.11/s390dbf.c
crash-4.0-8.11/ppc64.c
crash-4.0-8.11/x86_64.c
crash-4.0-8.11/extensions.c
crash-4.0-8.11/remote.c
crash-4.0-8.11/va_server.c
crash-4.0-8.11/va_server_v1.c
crash-4.0-8.11/symbols.c
crash-4.0-8.11/cmdline.c
crash-4.0-8.11/lkcd_common.c
crash-4.0-8.11/lkcd_v1.c
crash-4.0-8.11/lkcd_v2_v3.c
crash-4.0-8.11/lkcd_v5.c
crash-4.0-8.11/lkcd_v7.c
crash-4.0-8.11/lkcd_v8.c
crash-4.0-8.11/lkcd_fix_mem.c
crash-4.0-8.11/s390_dump.c
crash-4.0-8.11/lkcd_x86_trace.c
crash-4.0-8.11/netdump.c
crash-4.0-8.11/diskdump.c
crash-4.0-8.11/xendump.c
crash-4.0-8.11/unwind.c
crash-4.0-8.11/unwind_decoder.c
crash-4.0-8.11/unwind_x86_32_64.c
crash-4.0-8.11/xen_hyper.c
crash-4.0-8.11/xen_hyper_command.c
crash-4.0-8.11/xen_hyper_global_data.c
crash-4.0-8.11/xen_hyper_dump_tables.c
crash-4.0-8.11/defs.h
crash-4.0-8.11/xen_hyper_defs.h
crash-4.0-8.11/va_server.h
crash-4.0-8.11/vas_crash.h
crash-4.0-8.11/netdump.h
crash-4.0-8.11/diskdump.h
crash-4.0-8.11/xendump.h
crash-4.0-8.11/unwind.h
crash-4.0-8.11/unwind_i.h
crash-4.0-8.11/rse.h
crash-4.0-8.11/unwind_x86.h
crash-4.0-8.11/unwind_x86_64.h
crash-4.0-8.11/lkcd_vmdump_v1.h
crash-4.0-8.11/lkcd_vmdump_v2_v3.h
crash-4.0-8.11/lkcd_dump_v5.h
crash-4.0-8.11/lkcd_dump_v7.h
crash-4.0-8.11/lkcd_dump_v8.h
crash-4.0-8.11/lkcd_x86_trace.h
crash-4.0-8.11/lkcd_fix_mem.h
crash-4.0-8.11/ibm_common.h
crash-4.0-8.11/Makefile
crash-4.0-8.11/gdb-6.1/
crash-4.0-8.11/gdb-6.1/gdb/
crash-4.0-8.11/gdb-6.1/gdb/Makefile.in
crash-4.0-8.11/gdb-6.1/gdb/main.c
crash-4.0-8.11/gdb-6.1/gdb/symtab.c
crash-4.0-8.11/gdb-6.1/gdb/target.c
crash-4.0-8.11/gdb-6.1/gdb/symfile.c
crash-4.0-8.11/gdb-6.1/gdb/elfread.c
crash-4.0-8.11/gdb-6.1/gdb/ui-file.c
crash-4.0-8.11/gdb-6.1/gdb/utils.c
crash-4.0-8.11/gdb-6.1/gdb/dwarf2read.c
crash-4.0-8.11/gdb-6.1/gdb/ppc-linux-tdep.c
crash-4.0-8.11/gdb-6.1/Makefile.in
crash-4.0-8.11/gdb-6.1/include/
crash-4.0-8.11/gdb-6.1/include/obstack.h
crash-4.0-8.11/gdb-6.1.patch
crash-4.0-8.11/COPYING
crash-4.0-8.11/.rh_rpm_package
crash-4.0-8.11/crash.8
crash-4.0-8.11/extensions/
crash-4.0-8.11/extensions/Makefile
crash-4.0-8.11/extensions/echo.c
crash-4.0-8.11/extensions/dminfo.c
crash-4.0-8.11/extensions/libsial/
crash-4.0-8.11/extensions/libsial/Makefile
crash-4.0-8.11/extensions/libsial/mkbaseop.c
crash-4.0-8.11/extensions/libsial/README
crash-4.0-8.11/extensions/libsial/README.sial
crash-4.0-8.11/extensions/libsial/sial_alloc.c
crash-4.0-8.11/extensions/libsial/sial_api.c
crash-4.0-8.11/extensions/libsial/sial_api.h
crash-4.0-8.11/extensions/libsial/sial_builtin.c
crash-4.0-8.11/extensions/libsial/sial_case.c
crash-4.0-8.11/extensions/libsial/sial_define.c
crash-4.0-8.11/extensions/libsial/sial_func.c
crash-4.0-8.11/extensions/libsial/sial.h
crash-4.0-8.11/extensions/libsial/sial_input.c
crash-4.0-8.11/extensions/libsial/sial.l
crash-4.0-8.11/extensions/libsial/sial-lsed
crash-4.0-8.11/extensions/libsial/sial_member.c
crash-4.0-8.11/extensions/libsial/sial_node.c
crash-4.0-8.11/extensions/libsial/sial_num.c
crash-4.0-8.11/extensions/libsial/sial_op.c
crash-4.0-8.11/extensions/libsial/sialpp.l
crash-4.0-8.11/extensions/libsial/sialpp-lsed
crash-4.0-8.11/extensions/libsial/sialpp.y
crash-4.0-8.11/extensions/libsial/sial_print.c
crash-4.0-8.11/extensions/libsial/sial_stat.c
crash-4.0-8.11/extensions/libsial/sial_str.c
crash-4.0-8.11/extensions/libsial/sial_type.c
crash-4.0-8.11/extensions/libsial/sial_util.c
crash-4.0-8.11/extensions/libsial/sial_var.c
crash-4.0-8.11/extensions/libsial/sial.y
crash-4.0-8.11/extensions/sial.c
crash-4.0-8.11/extensions/sial.mk
crash-4.0-8.11/gdb-6.1.tar.gz
crash-4.0-8.11/README
# cd crash-4.0-8.11
# make
TARGET: X86
CRASH: 4.0-8.11
GDB: 6.1
gdb-6.1/gdb/CONTRIBUTE
gdb-6.1/gdb/COPYING
gdb-6.1/gdb/ChangeLog
gdb-6.1/gdb/ChangeLog-1990
gdb-6.1/gdb/ChangeLog-1991
gdb-6.1/gdb/ChangeLog-1992
gdb-6.1/gdb/ChangeLog-1993
gdb-6.1/gdb/ChangeLog-1994
gdb-6.1/gdb/ChangeLog-1995
gdb-6.1/gdb/ChangeLog-1996
gdb-6.1/gdb/ChangeLog-1997
gdb-6.1/gdb/ChangeLog-1998
gdb-6.1/gdb/ChangeLog-1999
gdb-6.1/gdb/ChangeLog-2000
gdb-6.1/gdb/ChangeLog-2001
gdb-6.1/gdb/ChangeLog-2002
gdb-6.1/gdb/ChangeLog-2003
gdb-6.1/gdb/ChangeLog-3.x
gdb-6.1/gdb/MAINTAINERS
gdb-6.1/gdb/NEWS
gdb-6.1/gdb/PROBLEMS
gdb-6.1/gdb/README
gdb-6.1/gdb/TODO
gdb-6.1/gdb/abug-rom.c
gdb-6.1/gdb/acconfig.h
gdb-6.1/gdb/acinclude.m4
gdb-6.1/gdb/aclocal.m4
gdb-6.1/gdb/ada-exp.y
gdb-6.1/gdb/ada-lang.c
gdb-6.1/gdb/ada-lang.h
(complete output not shown)
ar -rs crashlib.a main.o tools.o global_data.o memory.o filesys.o help.o task.o
build_data.o kernel.o test.o gdb_interface.o net.o dev.o alpha.o x86.o ppc.o ia6
4.o s390.o s390x.o s390dbf.o ppc64.o x86_64.o extensions.o remote.o va_server.o
va_server_v1.o symbols.o cmdline.o lkcd_common.o lkcd_v1.o lkcd_v2_v3.o lkcd_v5.
o lkcd_v7.o lkcd_v8.o lkcd_fix_mem.o s390_dump.o netdump.o diskdump.o xendump.o
lkcd_x86_trace.o unwind_v1.o unwind_v2.o unwind_v3.o unwind_x86_32_64.o xen_hype
r.o xen_hyper_command.o xen_hyper_global_data.o xen_hyper_dump_tables.o
ar: creating crashlib.a
gcc -g -O2 \
-o `cat mergeobj` libgdb.a \
../bfd/libbfd.a ../readline/libreadline.a ../opcodes/libopcod
es.a ../libiberty/libiberty.a -lm -lncurses ../libiberty/libiberty.a -ldl
-rdynamic `cat mergelibs`
#
The resultant crash executable will be located in the top-level source
directory. Install it in /usr/bin by entering:
# make install
/usr/bin/install crash /usr/bin
#
Building from the source RPM
To build from the source RPM, install the crash-4.0-8.11.src.rpm, cd
to the appropriate SPECS directory, and build the package:
# rpm -Uvh crash-4.0-8.11.src.rpm
1:crash ########################################### [100%]
# cd /usr/src/redhat/SPECS
# rpmbuild -ba crash.spec
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.19005
+ umask 022
+ cd /usr/src/redhat/BUILD
+ LANG=C
+ export LANG
+ unset DISPLAY
+ cd /usr/src/redhat/BUILD
+ rm -rf crash-4.0-8.11
+ /bin/gzip -dc /usr/src/redhat/SOURCES/crash-4.0-8.11.tar.gz
+ tar -xvvf -
drwxr-xr-x root/root 0 2002-09-12 16:51:40 crash-4.0-8.11/
-rwxrwxr-x root/root 31916 2002-09-12 16:51:40 crash-4.0-8.11/main.c
-rwxrwxr-x root/root 103454 2002-09-12 16:51:40 crash-4.0-8.11/tools.c
-rwxrwxr-x root/root 5802 2002-09-12 16:51:40 crash-4.0-8.11/global_data.c
-rwxrwxr-x root/root 225343 2002-09-12 16:51:40 crash-4.0-8.11/memory.c
-rwxrwxr-x root/root 75492 2002-09-12 16:51:40 crash-4.0-8.11/filesys.c
-rwxrwxr-x root/root 211519 2002-09-12 16:51:40 crash-4.0-8.11/help.c
-rwxrwxr-x root/root 110604 2002-09-12 16:51:40 crash-4.0-8.11/task.c
-rwxrwxr-x root/root 101805 2002-09-12 16:51:40 crash-4.0-8.11/kernel.c
-rwxrwxr-x root/root 2198 2002-09-12 16:51:40 crash-4.0-8.11/test.c
-rwxrwxr-x root/root 18949 2002-09-12 16:51:40 crash-4.0-8.11/gdb_interface.c
-rwxrwxr-x root/root 20239 2002-09-12 16:51:40 crash-4.0-8.11/configure.c
-rwxrwxr-x root/root 29931 2002-09-12 16:51:40 crash-4.0-8.11/net.c
-rwxrwxr-x root/root 99654 2002-09-12 16:51:40 crash-4.0-8.11/dev.c
-rwxrwxr-x root/root 76146 2002-09-12 16:51:40 crash-4.0-8.11/alpha.c
-rwxrwxr-x root/root 74638 2002-09-12 16:51:40 crash-4.0-8.11/x86.c
-rwxrwxr-x root/root 42109 2002-09-12 16:51:40 crash-4.0-8.11/ppc.c
-rwxrwxr-x root/root 76357 2002-09-12 16:51:40 crash-4.0-8.11/ia64.c
(complete output not shown)
Requires: libc.so.6 libc.so.6(GLIBC_2.0) libc.so.6(GLIBC_2.1) libc.so.6(GLIBC_2.
2) libc.so.6(GLIBC_2.2.3) libc.so.6(GLIBC_2.3) libdl.so.2 libdl.so.2(GLIBC_2.0)
libdl.so.2(GLIBC_2.1) libm.so.6 libm.so.6(GLIBC_2.0) libncurses.so.5 libz.so.1
Processing files: crash-debuginfo-4.0-8.11
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(PayloadFilesHave
Prefix) <= 4.0-1
Checking for unpackaged file(s): /usr/lib/rpm/check-files /var/tmp/crash-root
Wrote: /usr/src/redhat/SRPMS/crash-4.0-8.11.src.rpm
Wrote: /usr/src/redhat/RPMS/i386/crash-4.0-8.11.i386.rpm
Wrote: /usr/src/redhat/RPMS/i386/crash-debuginfo-4.0-8.11.i386.rpm
#
Lastly, install the binary RPM, which copies the crash executable to the
/usr/bin directory:
# rpm -ivh /usr/src/redhat/RPMS/i386/crash-4.0-8.11.i386.rpm
Preparing... ########################################### [100%]
1:crash
#
Invocation
When crash is run on a dumpfile, at least two arguments are always
required:
The kernel object filename, often referred to as the kernel namelist. When initially built from the kernel sources, its name is
vmlinux. In RHEL3 installations, it is copied to the/bootdirectory, where the operating system release number is appended, as invmlinux-2.4.21-4.ELsmp. In RHEL4, RHEL5 and RHEL6 installations, thevmlinuxfile is part of the kernel debuginfo package, and is located in the relevant/usr/lib/debug/lib/modules/<release>directory.The dumpfile name, typically named
vmcore.
For example, if both files are located in the current directory:
# crash vmlinux vmcore
When crash is run on a live system, /dev/mem is used as the memory
image. In RHEL4, RHEL5 and RHEL6, the /dev/mem driver is restricted on x86
and x86_64 systems, and is replaced by the /dev/crash driver. In any case,
only the kernel object filename is required:
# crash vmlinux
Furthermore, when crash is run on a live system, the vmlinux argument
is not required when the kernel object file is located in any of the
following locations:
/boot/usr/lib/debug/lib/modules/<release>/(root directory)any subdirectory of
/usr/src/usr/src/redhat/BUILD/kernel-x.x.x/linux-<release>
When the vmlinux file is not entered on the command line, a search will be
made in all of the directories above until a kernel object file is found that
contains a version string that matches the running kernel, as indicated by
/proc/version. If a matching kernel is found, then crash may be
invoked on a live system simply by entering:
# crash
In the examples above, it is presumed that the vmlinux kernel has been
built with the -g C flag, which traditionally has not been done by
default. To address this requirement, starting with Red Hat Enterprise Linux 3
(RHEL3), all RHEL kernels are now built with -g C flag. The manner of
accessing the debug data for RHEL3,
RHEL4, RHEL5 and
RHEL6 kernels is described in the following sections.
Unfortunately, since RHEL2.1 kernels are not built with -g, the kernel
must be rebuilt; directions for rebuilding
RHEL2.1
kernels can be found
here.
RHEL3 Kernels
In RHEL3, the vmlinux kernel debug information is stripped and stored in a
separate debuginfo file. The stripped vmlinux file in /boot has an
embedded link to its associated debuginfo file in /usr/lib/debug/boot, so
that the crash utility (and the built-in gdb module) knows where to
find it:
RHEL3 kernel namelist in |
RHEL3 kernel debuginfo file in |
vmlinux-<release>.ELvmlinux-<release>.ELsmpvmlinux-<release>.ELhugemem |
vmlinux-<release>.EL.debugvmlinux-<release>.ELsmp.debugvmlinux-<release>.ELhugemem.debug |
The debuginfo files for a specific kernel <release> come from a separate RPM
that must be installed for the crash utility to work. For example, the
i686 RPM for the examples above would be named kernel-debuginfo-<release>.i686.rpm, and would install the debuginfo file for all three
of the kernel flavors.
For example, to run crash on a live system, the associated debuginfo
package must be installed:
# uname -r
2.4.21-4.ELsmp
# rpm -ivh kernel-debuginfo-2.4.21-4.EL.i686.rpm
Preparing... ########################################### [100%]
1:kernel-debuginfo ########################################### [100%]
# ls /usr/lib/debug/boot
vmlinux-2.4.21-4.EL.debug
vmlinux-2.4.21-4.ELhugemem.debug
vmlinux-2.4.21-4.ELsmp.debug
#
Accordingly, if the running kernel's vmlinux file is in one the search
locations above, and its associated debuginfo file is located in the
/usr/lib/debug/boot directory or in the current directory from which
crash is invoked, no arguments are required to run on a live system:
# crash
However, if the linked debuginfo file is not in either of those locations, it
can be added to the crash command line along with the vmlinux
filename. So, for example, if the debuginfo file was located in /tmp:
# crash /boot/vmlinux-2.4.21-4.ELsmp /tmp/vmlinux-2.4.21-4.ELsmp.debug
For analyzing dumpfiles however, the vmlinux file name must be on the
command line along with the dumpfile name, as in the following examples:
# crash /boot/vmlinux-2.4.21-4.ELsmp vmcore
or if the debuginfo file is not in the standard location:
# crash /boot/vmlinux-2.4.21-4.ELsmp /tmp/vmlinux-2.4.21-4.ELsmp.debug vmcore
RHEL4 Kernels
The procedure has been made much simpler for RHEL4 kernels. The kernel is
built with the -g flag, and the resultant vmlinux file is stored in
the associated debuginfo package. After installing the debuginfo package, the
vmlinux file for each kernel flavor of a given RHEL4 release will be
installed in the directory named:
/usr/lib/debug/lib/modules/<release><flavor>/vmlinux
where for i686 kernels, <flavor> can be either hugemem, smp, or
nothing (for uniprocessor kernels). For example:
# uname -r
2.6.9-6.39.ELsmp
# rpm -ivh kernel-debuginfo-2.6.9-6.39.EL.i686.rpm
Preparing... ########################################### [100%]
1:kernel-debuginfo ########################################### [100%]
#
# find /usr/lib/debug/lib/modules/2.6.9-6.39.EL* -name vmlinux
/usr/lib/debug/lib/modules/2.6.9-6.39.ELhugemem/vmlinux
/usr/lib/debug/lib/modules/2.6.9-6.39.ELsmp/vmlinux
/usr/lib/debug/lib/modules/2.6.9-6.39.EL/vmlinux
#
Once the debuginfo package is installed, crash can be invoked on the live
system with no arguments, because the vmlinux file will be found
automatically:
# crash
To run crash on a dumpfile, however, the appropriate vmlinux file and
the dumpfile name must both be on the command line, as in:
# crash /usr/lib/debug/lib/modules/2.6.9-6.39.ELsmp/vmlinux vmcore
RHEL5 Kernels
RHEL5 kernels are also built with the -g flag, and the resultant
vmlinux file is stored in an associated debuginfo package. Unlike RHEL4,
the kernel debuginfo packages are split into one package per flavor plus a
"common" package user by all kernel flavors. After installing the debuginfo
package, the vmlinux file for each kernel flavor of a given RHEL5 release
will be installed in the directory named:
/usr/lib/debug/lib/modules/<release><flavor>/vmlinux
where for i686 kernels, there are 4 possible <flavor>s, either PAE,
xen, debug, or nothing (for non-PAE SMP kernels). For example:
# rpm -ivh kernel-debuginfo-2.6.18-66.el5.i686.rpm \
kernel-PAE-debuginfo-2.6.18-66.el5.i686.rpm \
kernel-xen-debuginfo-2.6.18-66.el5.i686.rpm \
kernel-debuginfo-common-2.6.18-66.el5.i686.rpm
Preparing... ########################################### [100%]
1:kernel-debuginfo-common########################################### [ 25%]
2:kernel-debuginfo ########################################### [ 50%]
3:kernel-PAE-debuginfo ########################################### [ 75%]
4:kernel-xen-debuginfo ########################################### [100%]
# find /usr/lib/debug/lib/modules/2.6.18-66.el5* -name vmlinux
/usr/lib/debug/lib/modules/2.6.18-66.el5/vmlinux
/usr/lib/debug/lib/modules/2.6.18-66.el5PAE/vmlinux
/usr/lib/debug/lib/modules/2.6.18-66.el5xen/vmlinux
#
Once the debuginfo package is installed, crash can be invoked on the live
system with no arguments, because the vmlinux file will be found
automatically:
# crash
To run crash on a dumpfile, however, the appropriate vmlinux file and
the dumpfile name must both be on the command line, as in:
# crash /usr/lib/debug/lib/modules/2.6.18-66.el5/vmlinux vmcore
RHEL6 Kernels
RHEL6 kernels are also built with the -g flag, and the resultant
vmlinux file is stored in an associated debuginfo package. Like RHEL5, the
kernel debuginfo packages are split into one package per flavor plus a
"common" package user by all kernel flavors. After installing the debuginfo
package, the vmlinux file for each kernel flavor of a given RHEL6 release
will be installed in the directory named:
/usr/lib/debug/lib/modules/<release><flavor>/vmlinux
where for x86_64 kernels, there are only 2 possible <flavor>s, either the
standard kernel or the debug kernel. For example:
# rpm -ivh kernel-debuginfo-common-2.6.32-70.el6.x86_64.rpm \
kernel-debuginfo-2.6.32-70.el6.x86_64.rpm \
kernel-debug-debuginfo-2.6.32-70.el6.x86_64.rpm
Preparing... ########################################### [100%]
1:kernel-debuginfo-common########################################### [ 25%]
2:kernel-debuginfo ########################################### [ 50%]
3:kernel-debug-debuginfo ########################################### [100%]
# find /usr/lib/debug/lib/modules/2.6.32-70.el6* -name vmlinux
/usr/lib/debug/lib/modules/2.6.32-70.el6/vmlinux
/usr/lib/debug/lib/modules/2.6.32-70.el6debug/vmlinux
#
Once the debuginfo package is installed, crash can be invoked on the live
system with no arguments, because the vmlinux file will be found
automatically:
# crash
To run crash on a dumpfile, however, the appropriate vmlinux file and
the dumpfile name must both be on the command line, as in:
# crash /usr/lib/debug/lib/modules/2.6.32-70.el6/vmlinux vmcore
RHEL2.1 Kernels (or kernels built without -g flag)
If the running kernel was not built with the -g C flag, then it is
necessary to rebuild a kernel of the same configuration with the -g C
flag. The essential change done by this kernel rebuild task is a modification
of top-level Makefile of the kernel source tree, such that the CFLAGS
definition contains the -g flag. For example, this is the line that must
be changed:
CFLAGS := $(CPPFLAGS) -Wall -Wstrict-prototypes -Wno-trigraphs -O2 \
-fno-strict-aliasing -fno-common
by adding the -g flag:
CFLAGS := $(CPPFLAGS) -Wall -Wstrict-prototypes -Wno-trigraphs -O2 \
-fno-strict-aliasing -fno-common -g
For example, since RHEL2.1 kernels are not built with -g, a kernel
rebuild is required. For a detailed example of how to rebuild a RHEL2.1 kernel
with the -g flag, please refer to these
directions.
Upon rebuilding the kernel, a new vmlinux file will be created that
contains the debug data required by crash. However, the symbol values will
not match those of the running or dumped kernel. To deal with this inequity,
the actual symbol values can be gathered from either the original non-debug
vmlinux file or from its associated System.map file. That being the
case, two arguments must be supplied to crash to fully describe the
running/dumped kernel, the newly-created vmlinux file compiled with
-g, as well as a source of the real symbol values. So, for example, if the
vmlinux file built with -g were renamed to vmlinux.dbg, the
invocation line would look like this on a live system:
# crash vmlinux vmlinux.dbg
(or)
# crash /boot/System.map vmlinux.dbg
(or)
# crash -S vmlinux.debug
The -S argument above is simply an alternative to entering the default
/boot/System.map string.
Similarly, when looking at a dumpfile, two arguments are required to describe
the dumped kernel, along with the vmcore image:
# crash vmlinux vmlinux.dbg vmcore
(or)
# crash /boot/System.map vmlinux.dbg vmcore
(or)
# crash -S vmlinux.dbg vmcore
Again, for a detailed example of how to rebuild a RHEL2.1 kernel with the
-g flag, refer to these
directions.
Invocation output
The arguments may be entered in any order. If the file arguments are not in
the current directory, absolute pathnames must be used. When in doubt, simply
enter crash -h to get an explanation of the command line arguments:
# crash -h
Usage:
crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist] [dumpfile]
[namelist]
The [namelist] argument is a pathname to an uncompressed kernel image
(a vmlinux file) that has been compiled with the "-g" switch, or
that has an accessible, associated, debuginfo file. If the [dumpfile]
argument is entered, then the [namelist] argument must be entered
If the [namelist] argument is not entered when running on a live
system, a search will be made in several typical directories for
for a kernel namelist file that matches the live system.
[dumpfile]
The [dumpfile] argument is a pathname to a kernel memory core dump
file. If the [dumpfile] argument is not entered, the session will be
invoked on the live system using /dev/mem, which usually requires root
privileges.
[mapfile]
If the live system kernel, or the kernel from which the [dumpfile]
was derived, was not compiled with the -g switch, then the additional
[mapfile] argument is required. The [mapfile] argument may consist
of either the associated System.map file, or the non-debug kernel
namelist. However, if the [mapfile] argument is used, then the
[namelist] argument must be a kernel namelist of a similar kernel
version that was built with the -g switch.
[-S]
Use "/boot/System.map" as the [mapfile].
Examples when running on a live system:
$ crash
$ crash /usr/tmp/vmlinux
$ crash /boot/System.map vmlinux.dbg
$ crash -S vmlinux.dbg
$ crash vmlinux vmlinux.dbg
Examples when running on a dumpfile:
$ crash vmlinux vmcore
$ crash /boot/System.map vmlinux.dbg vmcore
$ crash -S vmlinux.dbg vmcore
$ crash vmlinux vmlinux.dbg vmcore
[-h [opt]]
The -h option alone displays this message. If the [opt] argument is
a crash command name, the help page for that command is displayed. If
the string "input" is entered, a page describing the various crash
command line input options is displayed. If the string "output" is
entered, a page describing command line output options is displayed.
[-v]
Display the versions of crash and gdb making up this executable.
[-s]
Do not display any version, GPL, or crash initialization data; proceed
directly to the "crash>" prompt.
[-i file]
Execute the crash command(s) in [file] prior to accepting any user
input from the "crash>" prompt.
[-d num]
Set crash debug level [num]. The higher the number, the more debug data
will be printed during crash runtime.
Given that all invocation arguments are in order, here is an example of a
successful invocation on a dumpfile, running a kernel that was built with
-g, along with a vmcore dump file was created by the Red Hat Netdump
facility:
# crash vmlinux-2.4.20-2.1.15.entsmp vmcore
crash 4.0-8.11
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
KERNEL: vmlinux-2.4.20-2.1.15.entsmp
DUMPFILE: vmcore
CPUS: 1
DATE: Wed Mar 12 10:12:56 2003
UPTIME: 00:38:25
LOAD AVERAGE: 1.16, 0.74, 0.30
TASKS: 60
NODENAME: dhcp64-220.boston.redhat.com
RELEASE: 2.4.20-2.1.15.entsmp
VERSION: #1 SMP Tue Mar 11 16:12:22 EST 2003
MACHINE: i686 (501 Mhz)
MEMORY: 128 MB
PANIC: "Oops: 0002" (check log for details)
PID: 0
COMMAND: "swapper"
TASK: c038e000
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash>
This next example shows the output when the panicking kernel was not built
with -g. In this case, a similar kernel type was built with -g, and
the resultant kernel object file was renamed as vmlinux.dbg. Note that
there will be a message concerning the patching of gdb data; this
indicates that the non-matching symbol values from the vmlinux.dbg are
being over-written by the correct symbol values found in the original
vmlinux file:
# crash vmlinux vmlinux.dbg vmcore
crash 4.0-8.11
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
please wait... (patching 16053 gdb minimal_symbol values)
DEBUG KERNEL: vmlinux.dbg
DUMPFILE: vmcore
CPUS: 1
DATE: Wed Mar 27 11:02:31 2002
UPTIME: 00:07:24
LOAD AVERAGE: 0.43, 0.42, 0.19
TASKS: 68
NODENAME: anderson.boston.redhat.com
RELEASE: 2.4.9-26beta.48enterprise
VERSION: #1 SMP Thu Mar 21 12:33:05 EST 2002
MACHINE: i686 (501 Mhz)
MEMORY: 128 MB
PANIC: "Oops: 0002" (check log for details)
PID: 1696
COMMAND: "insmod"
TASK: c74de000
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash>
Invocation on a live system looks essentially the same, except that the
DUMPFILE will be indicated as /dev/mem. In the following example, no
arguments were entered, because the running RHEL3 kernel was found in the
/boot directory, and its associated debuginfo file in the
/usr/lib/debug/boot directory. The debuginfo file is listed next to the
DEBUGINFO tag:
# crash
crash 4.0-8.11
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
KERNEL: /boot/vmlinux-2.4.21-4.ELhugemem
DEBUGINFO: /usr/lib/debug/boot/vmlinux-2.4.21-4.ELhugemem.debug
DUMPFILE: /dev/mem
CPUS: 2
DATE: Thu Aug 21 11:24:38 2003
UPTIME: 1 days, 23:14:11
LOAD AVERAGE: 0.14, 0.10, 0.08
TASKS: 106
NODENAME: crash.boston.redhat.com
RELEASE: 2.4.21-4.ELhugemem
VERSION: #1 SMP Wed Aug 13 21:33:31 EDT 2003
MACHINE: i686 (1993 Mhz)
MEMORY: 511.5 MB
PID: 4757
COMMAND: "crash"
TASK: 19b82000
CPU: 1
STATE: TASK_RUNNING (ACTIVE)
crash>
Invocation Errors
Invocation errors will cause the crash session to abort upon
initialization. Typically they occur as the result of one of the following
reasons:
The
vmlinuxfile contains no debug data (i.e., was built without the-gflag), and no additional debug kernel object file name was entered on the command line. The error message will be of the form:crash: /boot/vmlinux-2.4.18-14: no debugging data available
The
vmlinuxfile does not match the dumpfile. The error message will be of the form:crash: vmlinux and tmp/vmcore do not match!
The
vmlinuxfile could not be found on a live system. The error message will be of the form:crash: cannot find booted kernel -- please enter namelist argument
The associated debuginfo file cannot be found. The error message will be of the form:
crash: /boot/vmlinux-2.4.21-4.ELsmp: no debugging data available crash: vmlinux-2.4.21-4.ELsmp.debug: debuginfo file not found
The
crashutility binary does not match thevmlinuxand/orvmcorearguments. The error message will be of the form:WARNING: machine type mismatch: crash utility: X86 vmlinux: X86_64 crash: vmlinux: not a supported file format
Command Input
Upon a successful session invocation on a dump file or a live kernel, the
crash> prompt will appear. Interactive crash commands are gathered
using the GNU readline library, taking advantage of its command line
history mechanism, and its vi or emacs command line editing modes.
Commands may also be issued to crash from a file.
Command Line History
The command line history consists of a numbered list of previously-run
commands. The full list of commands may be viewed by entering h at any
time. For example:
crash> h
[1] bt -a
[2] ps
[3] foreach bt
[4] set
[5] dis -rl c0221141
crash>
Commands in the history list may be re-run in the following manners
To re-run the last command executed, simply enter
ror!!and then ENTER.Enter
rfollowed by the appropriate history list number, and then ENTER.Enter
rfollowed by a uniquely-identifying set of characters from the beginning of the previously-entered command string, and then ENTER.Recycle back through the command history list using the up-arrow and down-arrow keys until the desired command is re-displayed, and then ENTER.
Recycle back through the command history list using the key-strokes appropriate for the command line editing mode being used (
vioremacs) until the desired command is re-displayed, and then ENTER.
Command Line Editing
The command line editing mode may be set to either vi (the default) or
emacs. The mode may set in the following manners, listed in increasing
order of precedence:
Do none of the following, in which case the default
viediting mode will be used.Set the
EDITORenvironment variable to eithervioremacs.Create an entry in a
.crashrcfile in the user's$HOMEdirectory. The entry must be a line of one of the following forms:set vi set emacs
Create an entry in a
.crashrcfile be located in the current directory, in the form shown in (3) above.Use the
-ecommand line option, as in:# crash -e [vi | emacs] ...
Given either editing mode, any previously entered command line can be brought back by entering the mode-specific key-stroke(s), the command line edited using the proper mode, and then run by hitting ENTER.
Command Line Input from a File
An input file consisting of a list of commands may be fed to crash in the
following manners:
Upon invocation, as in:
# crash -i inputfile
Upon invocation, by entering the commands in a
.crashrcfile, which can be either in the user's$HOMEdirectory or in the current directory.On the command line during a
crashsession, as in:crash> < inputfile
In all of the three cases above, after the list of commands in the file have
completed, the crash> prompt will appear and commands may then be entered
interactively (unless one of the file commands happens to be the exit
command).
Numerical Arguments
Numerical arguments are typically presumed to be decimal unless the argument
contains an a, b, c, d, e or f. In those cases, the
preceding 0x is not required. For hexadecimal numbers that do not
contain one of those 6 characters, the preceding 0x is required. So, for
example, a value of 1 gigabyte would have to be expressed as 0x40000000,
whereas 3 gigabytes could be expressed as c0000000.
It should be noted that several commands will only accept hexadecimal numerical arguments. For example, the rd ("read") command only accepts hexadecimal addresses. Therefore a read from user address of 0x40017000 could be entered as:
crash> rd 40017000 40
40017000: 20000824 00000010 00000048 00000063 $.. ....H...c...
40017010: 00000082 000000ba 000000bb 000000cd ................
40017020: 000000ce 000000cf 000000d7 000000db ................
40017030: 000000dc 000000dd 000000de 000000e2 ................
40017040: 000000ed 00000167 6c676e45 20687369 ....g...English
40017050: 61636f6c 6620656c 7420726f 55206568 locale for the U
40017060: 46004153 20656572 74666f53 65726177 SA.Free Software
40017070: 756f4620 7461646e 2c6e6f69 636e4920 Foundation, Inc
40017080: 3935002e 6d655420 20656c70 63616c50 ..59 Temple Plac
40017090: 202d2065 74697553 33332065 42202c30 e - Suite 330, B
Command Output
crash commands can often be verbose, and it's helpful to control the
output, as well as to be able to scroll backwards to view previous command
output. So, by default, command output that would overflow the user's display
screen is piped to /usr/bin/less, along with a prompt line that informs
the user how to scroll forward, backward, or to quit the command. For example,
here is an example of what a
ps command might look
like:
crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
0 0 0 c030a000 RU 0.0 0 0 [swapper]
1 0 0 cff98000 IN 0.2 1412 468 init
2 1 0 c1446000 IN 0.0 0 0 [keventd]
3 1 0 cfffa000 IN 0.0 0 0 [kapm-idled]
4 0 0 cfff8000 IN 0.0 0 0 [ksoftirqd_CPU0]
5 0 0 cffee000 IN 0.0 0 0 [kswapd]
6 0 0 cffec000 IN 0.0 0 0 [kreclaimd]
7 0 0 c1826000 IN 0.0 0 0 [bdflush]
8 0 0 c1824000 IN 0.0 0 0 [kupdated]
9 1 0 cff90000 IN 0.0 0 0 [mdrecoveryd]
13 1 0 cf07a000 IN 0.0 0 0 [kjournald]
89 1 0 ce804000 IN 0.0 0 0 [khubd]
184 1 0 ce4d4000 IN 0.0 0 0 [kjournald]
572 1 0 cd938000 IN 0.0 440 48 dhcpcd
637 1 0 ce4a4000 IN 0.2 1476 612 syslogd
642 1 0 cd92c000 IN 0.2 2092 432 klogd
663 1 0 ce2bc000 IN 0.2 1564 612 portmap
691 1 0 cd84a000 IN 0.3 1652 668 rpc.statd
803 1 0 cd756000 IN 0.2 1400 452 apmd
828 1 0 cd6c2000 IN 0.3 18024 684 ypbind
830 828 0 cd76e000 IN 0.3 18024 684 ypbind
831 830 0 cd71c000 IN 0.3 18024 684 ypbind
-- MORE -- forward: <SPACE>, <ENTER> or j backward: b or k quit: q
This default output scrolling behavior can be turned off by entering the
following line in a .crashrc file located in either the $HOME or
current directories:
set scroll off
During runtime, the following commands (or their respective builtin aliases) can be used to turn the scrolling behavior off, and back on, again:
crash> set scroll off
scroll: off
crash> set scroll on
scroll: on
crash> alias
ORIGIN ALIAS COMMAND
builtin man help
builtin ? help
builtin quit q
builtin sf set scroll off
builtin sn set scroll on
builtin hex set radix 16
builtin dec set radix 10
builtin g gdb
builtin px p -x
builtin pd p -d
builtin for foreach
builtin size *
builtin dmesg log
builtin last ps -l
crash> sf
scroll: off
crash> sn
scroll: on
crash>
Alternatively, command output may be redirected to a pipe or to a file using standard shell redirection syntax. For examples:
crash> task | grep uid
uid = 3369,
euid = 3369,
suid = 3369,
fsuid = 3369,
crash> foreach bt > bt.all
crash> ps >> process.data
crash> kmem -i | grep SLAB > slab.pages
crash>
When a command's output is redirected to a pipe or file, the default
/usr/bin/less behavior is turned off for that particular command.
Numerical Output
The default numerical output radix for non-pointer values is decimal, which is
most often noticed when using the builtin gdb capability of printing
formatted data structures. During runtime, the following commands (or their
respective builtin aliases) can be used to toggle the output radix from
decimal to hexadecimal, and back again:
crash> set radix 16
output radix: 16 (hex)
crash> set scroll 10
output radix: 10 (decimal)
crash> alias
ORIGIN ALIAS COMMAND
builtin man help
builtin ? help
builtin quit q
builtin sf set scroll off
builtin sn set scroll on
builtin hex set radix 16
builtin dec set radix 10
builtin g gdb
builtin px p -x
builtin pd p -d
builtin for foreach
builtin size *
builtin dmesg log
crash> hex
output radix: 16 (hex)
crash> dec
output radix: 10 (decimal)
crash>
Alternatively, the px or pd aliases coerce the "print" command
p, to override the
current output radix. For example, here the changing value of jiffies on a
live system is printed using the current default radix, then in hexadecimal,
and lastly in decimal:
crash> p jiffies
jiffies = $4 = 69821055
crash> px jiffies
jiffies = $5 = 0x42963aa
crash> pd jiffies
jiffies = $6 = 69821656
crash>
Crash Context
Upon a successful invocation of a crash session, one of the existing Linux
tasks is selected as the current context. It is important to be aware of
the current context because several crash commands are
"context-sensitive", meaning that the command is executed from the view-point
of the current context. Therefore, the output of context-sensitive commands
can vary depending upon which context is current.
Upon invocation of a crash session, the selection of the current context
is based upon the following criteria:
On dumpfiles:
The task that was running when
die()was called.The task that was running when
panic()was called.The task that was running when an ALT-SYSRQ-c keyboard interrupt was received.
The task that was running when the character "c" was echoed to
/proc/sysrq-trigger.
On a live system:
the
crashtask itself.
The current context selection is shown in the session invocation data. For
example, here is a session begun on a dumpfile that was created when an
insmod task's attempt to install a module resulted in an "oops" violation:
# crash tmp/vm*
crash 4.0-8.11
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
KERNEL: tmp/vmlinux
DEBUG KERNEL: tmp/vmlinux.dbg
DUMPFILE: tmp/vmcore
CPUS: 1
DATE: Wed Mar 27 11:02:31 2002
UPTIME: 00:07:24
LOAD AVERAGE: 0.43, 0.42, 0.19
TASKS: 68
NODENAME: anderson.boston.redhat.com
RELEASE: 2.4.9-26beta.48enterprise
VERSION: #1 SMP Thu Mar 21 12:33:05 EST 2002
MACHINE: i686 (501 Mhz)
MEMORY: 128 MB
PANIC: "Oops: 0002" (check log for details)
PID: 1696
COMMAND: "insmod"
TASK: c74de000
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash>
During runtime, the current context can always be displayed by entering the set command with no arguments:
crash> set
PID: 1696
COMMAND: "insmod"
TASK: c74de000
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash>
Changing the Crash Context
The current context can be changed to a new task via the
set command. Either
of two "handles" may be used to identify a task, the PID number, or the kernel
address of the task's task_struct. For example:
crash> set 1
PID: 1
COMMAND: "init"
TASK: c7f98000
CPU: 0
STATE: TASK_RUNNING
crash> set c0a52000
PID: 1503
COMMAND: "cat"
TASK: c0a52000
CPU: 0
STATE: TASK_INTERRUPTIBLE
crash>
Alternatively, the current context can be set to the task running on a given
CPU number, or back to the panicking task. Using the same dumpfile session
shown above, in which there is only one CPU, the original context may be
restored using the -c CPU-number or the -p ("panic task") options:
crash> set -c 0
PID: 1696
COMMAND: "insmod"
TASK: c74de000
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash> set -p
PID: 1696
COMMAND: "insmod"
TASK: c74de000
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash>
Context-Sensitive Commands
It is important to be aware that several crash commands are
context-sensitive. For example, the
files command
displays data about the open files of a task. If it is issued with no
arguments, it displays the open files data of the current context. In this
example, the current context happens to be PID 642, the klogd daemon:
crash> files
PID: 642 TASK: cd92c000 CPU: 0 COMMAND: "klogd"
ROOT: / CWD: /
FD FILE DENTRY INODE TYPE PATH
0 ce06c800 ce29ec60 cd8df900 REG /proc/kmsg
1 ce06cf20 ce29ebe0 cd8df740 SOCK socket:/[858]
2 ce06c5c0 ce2423a0 ce462c80 REG /boot/System.map-2.4.9-e.3enterprise
However, if the files command is issued with either of the two task handles as an argument, then it will display the open files data of the specified task. In this example, PID 12731 is specified:
crash> files 12731
PID: 12731 TASK: c8150000 CPU: 0 COMMAND: "vi"
ROOT: / CWD: /tmp
FD FILE DENTRY INODE TYPE PATH
0 c988cd80 ced919a0 c87fc3c0 CHR /dev/pts/11
1 c988cd80 ced919a0 c87fc3c0 CHR /dev/pts/11
2 c988cd80 ced919a0 c87fc3c0 CHR /dev/pts/11
4 c2927ae0 c6cad8a0 cd6d5040 REG /tmp/.crontab.12730.swp
5 c2927a80 c6cad9a0 c5764ac0 REG /tmp/crontab.12730
This type of context-sensitive behaviour is also exhibited by the vm, bt, sig, set, net and task commands. Unless a PID or task address is specified as an argument, the output will reflect data concerning the current context.
Other commands may simply default to the current context. For example, the rd command can read memory from an address that is specified as a user-space address. Since the rd command does not accept a PID or task address as an argument, it would be necessary to be aware that the user-space access will come from the address space of the current context.
Builtin Help
Readily available help information is built into the crash utility. During
a session, entering the
help command with
no argument shows the following menu:
crash> help
crash version: 4.0-8.11 gdb version: 6.1
For help on any command above, enter "help <command>".
For help on input options, enter "help input".
For help on output options, enter "help output".
crash>
Each command has its own man-like help page, which can be viewed by
clicking on the command name above. Each help page details the syntax of the
command and its available options, a description of the command in general, a
description of each option, and a set of examples. During a crash session,
a command's help page can be displayed by entering
help followed by
the command name. So, for example, to get help on how to use the
set command:
crash> help set
NAME
set - set a process context or internal crash variable
SYNOPSIS
set [pid | taskp | [-c cpu] | -p] | [crash_variable [setting]] | -v
DESCRIPTION
This command either sets a new context, or gets the current context for
display. The context can be set by the use of:
pid a process PID.
taskp a hexadecimal task_struct pointer.
-c cpu sets the context to the active task on a cpu (dumpfiles only).
-p sets the context to the panic task, or back to the crash task on
a live system.
-v display the current state of internal crash variables.
If no argument is entered, the current context is displayed. The context
consists of the PID, the task pointer, the CPU, and task state.
This command may also be used to set internal crash variables. If no value
argument is entered, the current value of the crash variable is shown. These
are the crash variables, acceptable arguments, and purpose:
scroll on | off controls output scrolling.
scroll less /usr/bin/less as the output scrolling program.
scroll more /bin/more as the output scrolling program.
scroll CRASHPAGER use CRASHPAGER environment variable as the
output scrolling program.
radix 10 | 16 sets output radix to 10 or 16.
refresh on | off controls internal task list refresh.
print_max number set maximum number of array elements to print.
console device-name sets debug console device.
debug number sets crash debug level.
core on | off if on, drops core when the next error message
is displayed.
hash on | off controls internal list verification.
silent on | off turns off initialization messages; turns off
crash prompt during input file execution.
(scrolling is turned off if silent is on)
edit vi | emacs set line editing mode (from .crashrc file only).
namelist filename name of kernel (from .crashrc file only).
dumpfile filename name of core dumpfile (from .crashrc file only).
zero_excluded on | off controls whether excluded pages from a dumpfile
should return zero-filled memory.
Internal variables may be set in four manners:
1. entering the set command in $HOME/.crashrc.
2. entering the set command in .crashrc in the current directory.
3. executing an input file containing the set command.
4. during runtime with this command.
During initialization, $HOME/.crashrc is read first, followed by the
.crashrc file in the current directory. Set commands in the .crashrc file
in the current directory override those in $HOME/.crashrc. Set commands
entered with this command or by runtime input file override those
defined in either .crashrc file. Multiple set command arguments or argument
pairs may be entered in one command line.
EXAMPLES
Set the current context to task c2fe8000:
crash> set c2fe8000
PID: 15917
COMMAND: "bash"
TASK: c2fe8000
CPU: 0
STATE: TASK_INTERRUPTIBLE
Set the context back to the panicking task:
crash> set -p
PID: 698
COMMAND: "gen12"
TASK: f9d78000
CPU: 2
STATE: TASK_RUNNING (PANIC)
Turn off output scrolling:
crash> set scroll off
scroll: off (/usr/bin/less)
Show the current state of crash internal variables:
crash> set -v
scroll: on (/usr/bin/less)
radix: 10 (decimal)
refresh: on
print_max: 256
console: /dev/pts/2
debug: 0
core: off
hash: on
silent: off
edit: vi
namelist: vmlinux
dumpfile: vmcore
zero_excluded: off
Show the current context:
crash> set
PID: 1525
COMMAND: "bash"
TASK: c1ede000
CPU: 0
STATE: TASK_INTERRUPTIBLE
If for some reason a crash session cannot be invoked, but help information
for a particular crash command is desired, the same help page can be
displayed from a shell command line using the -h option to crash:
# crash -h ascii
NAME
ascii - translate a hexadecimal string to ASCII
SYNOPSIS
ascii value ...
DESCRIPTION
Translates 32-bit or 64-bit hexadecimal values to ASCII. If no argument
is entered, an ASCII chart is displayed.
EXAMPLES
Translate the hexadecimal value of 0x62696c2f7273752f to ASCII:
crash> ascii 62696c2f7273752f
62696c2f7273752f: /usr/lib
Display an ASCII chart:
crash> ascii
0 1 2 3 4 5 6 7
+-------------------------------
0 | NUL DLE SP 0 @ P ' p
1 | SOH DC1 ! 1 A Q a q
2 | STX DC2 " 2 B R b r
3 | ETX DC3 # 3 C S c s
4 | EOT DC4 $ 4 D T d t
5 | ENQ NAK % 5 E U e u
6 | ACK SYN & 6 F V f v
7 | BEL ETB ` 7 G W g w
8 | BS CAN ( 8 H X h x
9 | HT EM ) 9 I Y i y
A | LF SUB * : J Z j z
B | VT ESC + ; K [ k {
C | FF FS , < L \ l |
D | CR GS _ = M ] m }
E | SO RS . > N ^ n ~
F | SI US / ? O - o DEL
#
Lastly, help concerning command input and output can be displayed by entering
help input or
help output
during runtime, or crash -h input or crash -h output from a shell
command line.
The Command Set
Each crash command generally falls into one of the following categories:
The remainder of this section breaks the command set into categories, and
gives a short description of each command in that category. However, for
complete details and examples, recall that the crash utility has a
self-contained help page for each command; to view the full help page, click
on the command name next to its description below.
Symbolic Display of Kernel Text or Data
The following commands typically take full advantage of the power of gdb
to display kernel data structures symbolically.
Command |
Description |
|---|---|
Displays a formatted kernel data structure type located at a given address, or
at an address referred to by a symbol; if no address is specified, the
structure definition is displayed. The output can be narrowed down to a
singular member of the structure, or to display the offset of every member
from the beginning of the structure. A count may be appended to display an
array of structures. Its usage is so common that two short-cuts exist such
that the user need not enter the "
|
|
Same as |
|
"Pointer-to" command which can be used in lieu of entering |
|
Displays the contents of a kernel variable; the arguments are passed on to
|
|
Displays all available symbol table information concerning a data type or a data symbol. |
|
Translates a kernel symbol name to its kernel virtual address and section, or a kernel virtual address to its symbol name and section. It can also be used to dump the complete list of kernel symbols, or to query the symbol list for all symbols containing a given sub-string. |
|
Disassembles the text of complete kernel function, or from a specified address for a given number of instructions, or from the beginning of a function up to a specified address. |
System State
The majority of crash commands come from the following set of
"kernel-aware" commands, which delve into various kernel subsystems on a
system-wide or per-task basis. The task-specific commands are
context-sensitive, meaning that they act upon the current context unless a PID
or task address is specified as an argument.
Command |
Description |
|---|---|
Arguably the most useful |
|
Displays data concerning the character and block device assignments, I/O port usage, I/O memory usage, and PCI device data. |
|
This context-sensitive command displays the task's current root directory and working directories, and then for each open file descriptor, shows:
Another option acts upon a specified
It can be called from the foreach wrapper command. |
|
Displays a list of tasks that reference a specified filename or |
|
Display data concerning interrupt request numbers and bottom-half handling. |
|
This command has numerous options that delve into the state of several kernel memory subsystems:
Also, given an address, this command searches the symbol table, the slab
subsystem, the free list, the |
|
Dumps the kernel message buffer chronologically, accounting for any wrap-around. |
|
Displays machine and/or processor specific data. |
|
Displays the list of currently-loaded kernel modules. More importantly, it loads the debug data from the module object files if they are available, allowing symbolic debugging capability of kernel modules. |
|
For each mounted filesystem, or for just a specified filesystem, displays:
Options exist to dump a list of a specified filesystem's open files or dirty
inodes. Filesystems may be specified by |
|
Displays various network-related data:
It can be called from the foreach wrapper command. |
|
Useful process status command, in typical Linux
Also has an option to show a task's parental hierarchy back to the |
|
This command translates the contents of a PTE into its physical page address and page bit settings, or if it references a swap location, the swap device and offset. |
|
Displays list of tasks on the run queue. |
|
A context-sensitive command which displays a task's signal information, including:
Other options list the signal number/names combination for a processor type,
and translate the contents of a |
|
For each configured swap device, this command displays the same data that is
shown by the Linux command |
|
Re-displays the same system-related data that is seen during
Other options display information concerning the system call table, and one
allows the |
|
This context-sensitive command displays a task's complete |
|
Displays the timer queue entries in chronological order, listing the target
function names, the current value of |
|
This powerful, context-sensitive command displays a wealth of information concerning a task's virtual memory data, including:
Other options translate the flags of a |
|
This context-sensitive command translates a user or kernel virtual address to its physical address. Also displayed are:
It can be called from the foreach wrapper command. |
|
Lists the tasks linked on a specified kernel wait queue. |
Utility Functions
The following commands are a set of useful helper commands serving various purposes, some simple, others quite powerful.
Command |
Description |
|---|---|
Translates a numerical value into its ASCII components; with no arguments, displays an ASCII chart. |
|
Translates a byte value (physical address) to its page number. |
|
A simple calculator, evaluates an expression and displays the result in hexadecimal, decimal, octal and binary, and optionally showing the bit numbers set in the result. |
|
Dumps the entries of a linked list of structures. It can handle lists of
structures that are singly-linked with simple "next" pointers, or those with
embedded |
|
translates a page frame number to its byte value (physical address). |
|
Translates a physical address into a kernel virtual address by adding the
appropriate |
|
Searches a range of user or kernel memory space for given value, with an optional "don't care" bit-mask argument. |
|
Displays a specified amount of user virtual, kernel virtual, or physical memory in several formats, such as 8, 16, 32 or 64 bit values, hexadecimal or decimal, symbolically, and with ASCII translations. When reading user virtual addresses, the command is context-sensitive. |
|
Modifies the contents of memory on a live system. Write permission on
|
Session Control Commands
The following commands typcally aid in the efficient running of a crash
session.
Command |
Description |
|---|---|
Creates a single-word alias for a command string. Several aliases are built
into |
|
Shuts down the |
|
Extend the |
|
Quite often it is helpful, or even necessary, to run the same
The identifiers above may be mixed if it makes sense, such as using a combination of PIDs, task addresses, and command names. The context-sensitive commands that can be issued to the selected tasks are: A header containing the PID, task address, CPU and command name will be pre-pended before the command output for each selected task. |
|
This command passes its arguments directly to |
|
This wrapper command repeats a |
|
This primary purpose for this command is to set the |
|
Shuts down the |
Crash Usage: A Case Study
The steps taken to debug a kernel crash dump are not etched in stone, and the
crash commands used to debug a kernel issue vary according to the problem
exhibited. The section contains of a case study that shows
how the capabilities of the crash utility were used to to debug a specific
kernel problem. However, before doing so, it should be noted that the
following commands are typically the most commonly-used:
Display the backtrace of the current context, or as specified with arguments. This command is typically the first command entered after starting a dumpfile session. Since the initial context is the panic context, it will show the function trace leading up to the kernel panic.
bt -awill show the trace of the active task on each CPU, since there may be an interrelationship between the panicking task on one CPU and the running task(s) on the other CPU(s). Whenbtis given as the argument to foreach. displays the backtraces of all tasks.Print the contents of a data structure at a specified address. This command is so common that it is typically unnecessary to enter the
structcommand name on the command line; if the first command line argument is not acrashorgdbcommand, but it is the name of a known data structure, then all the command line arguments are passed to thestructcommand. So for example, the following two commands yield the same result:crash> struct vm_area_struct d3cb2600crash> vm_area_struct d3cb2600Set a new task context by PID, task address, or cpu. Since several
crashcommands are context-sensitive, it's helpful to be able to change the context to avoid having to pass the PID or task address to those context-sensitive commands in order to access the data of a task that is not the current context.Prints the contents of a kernel variable; since it's a gateway to the
gdbmodule, it can also be used to print complex C language expressions.Read memory, which may be either kernel virtual, user virtual, or physical, and display it several different formats and sizes.
Lists basic task information for each process; it can also display parent and child hierarchies.
Dump the kernel
log_buf, which often contains clues leading up to a subsequent kernel crash.Execute a
crashcommand on all tasks, or those specified, in the system; can be used with bt, vm, task, files, net, set, sig and vtop.Dump the open file descriptor data of a task; most usefully, the
file,dentryandinodestructure addresses for each open file descriptor.Dump the virtual memory map of a task, including the vital information concerning each
vm_area_structmaking up a task's address space. It can also dump the physical address of each page in the address space, or if not mapped, its location in a file or on the swap device.
A Case Study: "kernel BUG at pipe.c:120!"
Upon bringing up a crash session, a great deal of information can be
gained just by the invocation data. Here is what what displayed in this
particular case:
...
KERNEL: vmlinux-2.4.9-e.10.13enterprise-g
DUMPFILE: vmcore-incomplete
CPUS: 2
DATE: Mon Feb 17 08:20:56 2003
UPTIME: 4 days, 20:04:41
LOAD AVERAGE: 0.95, 1.04, 1.25
TASKS: 110
NODENAME: testbox.redhat.com
RELEASE: 2.4.9-e.10.13enterprise
VERSION: #1 SMP Mon Feb 3 12:59:26 EST 2003
MACHINE: i686 (2788 Mhz)
MEMORY: 6 GB
PANIC: "kernel BUG at pipe.c:120!"
PID: 20571
COMMAND: "imp"
TASK: d1566000
CPU: 1
STATE: TASK_RUNNING (PANIC)
crash>
In this case the PANIC string "kernel BUG at pipe.c:120!" points to
the exact kernel source code line at which the panic occurred.
Then, getting a backtrace of panicking task is typically the first order of the day:
crash> bt
PID: 20571 TASK: d1566000 CPU: 1 COMMAND: "imp"
#0 [d1567e44] die at c010785c
#1 [d1567e54] do_invalid_op at c0107b2c
#2 [d1567f0c] error_code (via invalid_op) at c01073de
EAX: 0000001d EBX: ed87b2e0 ECX: c02f6064 EDX: 00005fa1 EBP: 00001000
DS: 0018 ESI: f640e740 ES: 0018 EDI: 00001000
CS: 0010 EIP: c0150b6d ERR: ffffffff EFLAGS: 00010292
#3 [d1567f48] pipe_read at c0150b6d
#4 [d1567f6c] sys_read at c01468d4
#5 [d1567fc0] system_call at c01072dc
EAX: 00000003 EBX: 0000000a ECX: 40b4e05c EDX: 00002000
DS: 002b ESI: 00002000 ES: 002b EDI: 40b4e05c
SS: 002b ESP: bffe9e88 EBP: bffe9eb8
CS: 0023 EIP: 40aaa1d4 ERR: 00000003 EFLAGS: 00000286
The backtrace shows that the call to die() was generated by an
invalid_op exception. The exception was caused by the BUG() call in
the pipe_read() function:
if (count && PIPE_WAITING_WRITERS(*inode) &&
!(filp->f_flags & O_NONBLOCK)) {
/*
* We know that we are going to sleep: signal
* writers synchronously that there is more
* room.
*/
wake_up_interruptible_sync(PIPE_WAIT(*inode));
if (!PIPE_EMPTY(*inode))
BUG();
goto do_more_read;
}
In the code segment above, the pipe_read() code has previously down'd
the semaphore of the inode associated with the pipe, giving it exclusive
access. It had read all data in the pipe, but still needed more to satisfy the
count requested. Finding that there was a writer with more data -- and who
was waiting on the semaphore -- it woke up the writer. However, after doing
the wakeup, it did a sanity-check on the pipe contents, and found that it was
no longer empty -- which is theoretically impossible since it was still
holding the semaphore. It appeared that the writer process wrote to the pipe
while the reader process still had exclusive access -- somehow overriding the
semaphore.
Since the semaphore mechanism was seemingly not working, it was first
necessary to look at the actual semaphore structure associated with the
pipe's inode. This first required looking at the first argument to the
pipe_read() function; the
whatis command
shows that it is a struct file pointer:
crash> whatis pipe_read
ssize_t pipe_read(struct file *, char *, size_t, loff_t *);
crash>
Using the bt -f
option, each frame in the backtrace is expanded to show all stack data in the
frame. Looking at the expansion of the sys_read() frame, we can see that
the last thing pushed on the stack before calling pipe_read() was the
file pointer address of edf3f740:
...
#3 [d1567f48] pipe_read at c0150b6d
[RA: c01468d6 SP: d1567f4c FP: d1567f6c SIZE: 36]
d1567f4c: c026701c 00000078 fffffff2 00001000
d1567f5c: 00000000 edf3f740 ffffffea 00002000
d1567f6c: c01468d6
#4 [d1567f6c] sys_read at c01468d4
[RA: c01072e3 SP: d1567f70 FP: d1567fc0 SIZE: 84]
d1567f70: edf3f740 40b4f05c 00002000 edf3f760
d1567f80: c03683d0 fffffffb 00000001 c0120d3b
d1567f90: 00000046 00000046 0000000b c0350960
d1567fa0: 0000000b f639eb00 c0108e0e 00000020
d1567fb0: d1566000 00002000 40b4e05c bffe9eb8
d1567fc0: c01072e3
...
The task at hand is finding the inode containing the suspect semaphore from
the file structure address. The file structure's f_dentry member
points to its dentry structure, whose d_inode member in turn points to
the pipe's inode. The
struct command
can be used to dump the complete contents of a data structure at a given
address; by tagging the .member onto the structure name, we can print just
the member desired. By following the structure chain, the inode address can be
determined like so:
crash> struct file.f_dentry edf3f740
f_dentry = 0xdb0ec440,
crash> struct dentry.d_inode db0ec440
d_inode = 0xf640e740,
crash> struct inode.i_sem f640e740
i_sem = {
count = {
counter = 2
},
sleepers = 0,
wait = {
lock = {
lock = 1
},
task_list = {
next = 0xf640e7ac,
prev = 0xf640e7ac
}
}
},
crash>
The dump of the semaphore structure above showed the problem: the counter
value of 2 is illegal. It should never be greater than 1; in this case
a value of 2 allows two successful down operations, i.e., giving two
tasks access to the pipe at the same time.
(As an aside, determining the inode address above could also be accomplished
by using the context-sensitive
files command,
which dumps the associated file, dentry and inode structure
addresses for each open file descriptor of a task. The dumped file descriptor
list would contain one with a reference to the file structure at
edf3f740, and would also show the associated inode address of
f640e740.)
Before getting a dumpfile, this same panic had occurred several times. It was
erroneously presumed that the problem was in the pipe-handling code, but it
was eventually determined not to be the case. By instrumenting a kernel with
debug code, the starting counter value of a pipe was found to be 3.
Compounding that problem was the fact that the inode slab cache is one of a
few special cases that presume that the freed inode's contents are left in a
legitimate state so that they do not have to be completely reinitialized with
each subsequent reallocation. So when the pipe's inode was created, it
received an inode with a bogus counter value.
Confirming the existence of bogus inode structures in the slab cache was a multi-stepped procedure. Using the command kmem command to access the inode slab cache, we can get the addresses of all free and currently-allocated inodes. Since there are typically several thousand inodes, the output is extremely verbose, but here is the beginning of it:
crash> kmem -S inode_cache
CACHE NAME OBJSIZE ALLOCATED TOTAL SLABS SSIZE
c7666564 inode_cache 448 11563 12339 1371 4k
SLAB MEMORY TOTAL ALLOCATED FREE
d1d82000 d1d82040 9 9 0
FREE / [ALLOCATED]
[d1d82040]
[d1d82200]
[d1d823c0]
[d1d82580]
[d1d82740]
[d1d82900]
[d1d82ac0]
[d1d82c80]
[d1d82e40]
SLAB MEMORY TOTAL ALLOCATED FREE
f4e52000 f4e52040 9 7 2
FREE / [ALLOCATED]
f4e52040 (cpu 1 cache)
f4e52200 (cpu 1 cache)
[f4e523c0]
[f4e52580]
[f4e52740]
[f4e52900]
[f4e52ac0]
[f4e52c80]
[f4e52e40]
...
In the truncated output above, all of the inode address in the slab cache
are dumped; the ones currently in use are surrounded by brackets, the free
ones are not. So, for example, the inodes at addresses f4e52040 and
f4e52200 are free; the others are not. The full output was
piped to a script that pulled out just the free inode
addresses (i.e., output lines starting with three spaces), and
redirected them into a file. The file was modified to
be a crash input file by making each
extracted inode address to be the arguments of the
struct command,
using its short-cut method that allows the dropping of the
struct command name; therefore the input file contained hundreds of
crash commands of the form:
inode.i_sem f4e52040
inode.i_sem f4e52200
inode.i_sem f5cdc040
inode.i_sem f5cdc200
inode.i_sem f5cdc3c0
inode.i_sem f5cdc580
...
Note that the
struct command
would be used by default above, as documented in its help page; if the first
command line argument is not a crash or gdb command, but it is the
name of a known data structure, it passes the arguments to the
struct command.
Using the capability of feeding an input file, in this case consisting of hundreds of short-cut struct commands like those above, the output was again quite verbose, consisting of structure member dumps of the form:
crash> < input.file
crash> inode.i_sem f4e52040
i_sem = {
count = {
counter = 1
},
sleepers = 0,
wait = {
lock = {
lock = 1
},
task_list = {
next = 0xf4e520ac,
prev = 0xf4e520ac
}
}
},
crash> inode.i_sem f4e52200
i_sem = {
count = {
counter = 1
},
sleepers = 0,
wait = {
lock = {
lock = 1
},
task_list = {
next = 0xf4e5226c,
prev = 0xf4e5226c
}
}
},
...
However, it was a simple matter of piping the output
to grep, and looking for counter values not equal to 1:
crash> < input.file | grep counter | grep -v "= 1"
counter = 3
counter = 3
counter = 3
counter = 3
crash>
This turned out to be the smoking gun. Another round of debugging with an
instrumented kernel that trapped attempts to free an inode with a semaphore
counter of 3 caught the perpetrator in the act.
Command Extensions
Frequently users wish to add an additional option to an existing crash
command, or add a brand new command, in order to address a kernel issue they
are debugging or developing. For those reasons, the crash utility was
designed with extensibility in mind. There are two ways to add new
functionality:
adding new code and compiling it into the
crashexecutable,creating a shared object library that can be dynamically loaded by using the extend command.
This section consists of a quick guide that describes how to get started using both methods.
Adding new code and compiling it into the crash executable
The current set of crash commands can be seen by entering the
help command with
no arguments:
crash> help
* files mod runq union
alias foreach mount search vm
ascii fuser net set vtop
bt gdb p sig waitq
btop help ps struct whatis
dev irq pte swap wr
dis kmem ptob sym q
eval list ptov sys
exit log rd task
extend mach repeat timer
crash version: 4.0-8.11 gdb version: 6.1
For help on any command above, enter "help ".
For help on input options, enter "help input".
For help on output options, enter "help output".
crash>
For each command in the menu above, there is an entry in a data structure in
the file global_data.c, which is located in the top-level directory of the
crash source code tree:
/*
* To add a new command, declare it in defs.h and enter it in this table.
*/
struct command_table_entry base_command_table[] = {
{"*", cmd_pointer, help_pointer, 0},
{"alias", cmd_alias, help_alias, 0},
{"ascii", cmd_ascii, help_ascii, 0},
{"bt", cmd_bt, help_bt, REFRESH_TASK_TABLE},
{"btop", cmd_btop, help_btop, 0},
{"dev", cmd_dev, help_dev, 0},
{"dis", cmd_dis, help_dis, 0},
{"eval", cmd_eval, help_eval, 0},
{"exit", cmd_quit, help_exit, 0},
{"extend", cmd_extend, help_extend, 0},
{"files", cmd_files, help_files, REFRESH_TASK_TABLE},
{"foreach", cmd_foreach, help_foreach, REFRESH_TASK_TABLE},
{"fuser", cmd_fuser, help_fuser, REFRESH_TASK_TABLE},
{"gdb", cmd_gdb, help_gdb, REFRESH_TASK_TABLE},
{"help", cmd_help, help_help, 0},
{"irq", cmd_irq, help_irq, 0},
{"kmem", cmd_kmem, help_kmem, 0},
{"list", cmd_list, help__list, REFRESH_TASK_TABLE},
{"log", cmd_log, help_log, 0},
{"mach", cmd_mach, help_mach, 0},
{"mod", cmd_mod, help_mod, 0},
{"mount", cmd_mount, help_mount, 0},
{"net", cmd_net, help_net, REFRESH_TASK_TABLE},
{"p", cmd_p, help_p, 0},
{"ps", cmd_ps, help_ps, REFRESH_TASK_TABLE},
{"pte", cmd_pte, help_pte, 0},
{"ptob", cmd_ptob, help_ptob, 0},
{"ptov", cmd_ptov, help_ptov, 0},
{"q", cmd_quit, help_quit, 0},
{"rd", cmd_rd, help_rd, 0},
{"repeat", cmd_repeat, help_repeat, 0},
{"runq", cmd_runq, help_runq, REFRESH_TASK_TABLE},
{"search", cmd_search, help_search, 0},
{"set", cmd_set, help_set, REFRESH_TASK_TABLE},
{"sig", cmd_sig, help_sig, REFRESH_TASK_TABLE},
{"struct", cmd_struct, help_struct, 0},
{"swap", cmd_swap, help_swap, 0},
{"sym", cmd_sym, help_sym, 0},
{"sys", cmd_sys, help_sys, REFRESH_TASK_TABLE},
{"task", cmd_task, help_task, REFRESH_TASK_TABLE},
{"test", cmd_test, NULL, HIDDEN_COMMAND},
{"timer", cmd_timer, help_timer, 0},
{"union", cmd_union, help_union, 0},
{"vm", cmd_vm, help_vm, REFRESH_TASK_TABLE},
{"vtop", cmd_vtop, help_vtop, REFRESH_TASK_TABLE},
{"waitq", cmd_waitq, help_waitq, REFRESH_TASK_TABLE},
{"whatis", cmd_whatis, help_whatis, 0},
{"wr", cmd_wr, help_wr, 0},
{(char *)NULL}
};
Each entry consists of the following simple data structure:
struct command_table_entry { /* one for each command in menu */
char *name;
cmd_func_t func;
char **help_data;
ulong flags;
};
The structure members consist of:
nameThe character string that appears in the help menu output. It must be unique among other
crashcommands, and preferably should not clash with anygdbcommand.
funcA pointer to the function that performs the command. To add an option to an existing command, find the file that contains this function, and modify it as desired.
help_dataA pointer to an array of character strings that make up the help data for that command. Although optional, it's certainly useful to create help data for any new commands or options.
flagsREFRESH_TASK_TABLE, the set of running tasks on a live system will beupdated just prior to executing the command.IfHIDDEN_COMMAND, the command will not be shown in the help menu. Theonly command with this flag is thetestcommand, whose functioncmd_test()exists solely as an builtin aid for quickly developing new ortemporary commands.
For a newly-written command to appear in the help menu, it simply requires a
reference to it in the structure above. To test a new command without adding
it to the menu, use the hidden "test" command, found in test.c:
void
cmd_test(void)
{
int c;
while ((c = getopt(argcnt, args, "")) != EOF) {
switch(c)
{
default:
argerrs++;
break;
}
}
if (argerrs)
cmd_usage(pc->curcmd, SYNOPSIS);
while (args[optind]) {
;
optind++;
}
}
The test command contains the basic template used by crash commands to
accept dash-arguments, which are fielded by the getopt() routine, while
all other command line arguments are fielded in the while loop. To add an
option to the test command (or any other existing command), simply fit it
into the appropriate argument-gathering mechanism. Or, for that matter, if no
arguments are required, put the functionality at the end of the command's
function. Here is a trivial example of a change to the cmd_test()
function, with the modifications highlighted:
void
cmd_test(void)
{
int c;
while ((c = getopt(argcnt, args, "xa:")) != EOF) {
switch(c)
{
case 'x':
fprintf(fp, "arg: -x\n");
break;
case 'a':
fprintf(fp, "arg: -a %s\n", optarg);
break;
default:
argerrs++;
break;
}
}
if (argerrs)
cmd_usage(pc->curcmd, SYNOPSIS);
while (args[optind]) {
fprintf(fp, "arg: %s\n", args[optind]);
;
optind++;
}
fprintf(fp, "do test work here...\n");
}
Then re-compile crash by entering make:
# make
TARGET: X86
CRASH: 4.0-8.11
GDB: 6.1
cc -c -g -DX86 -D_FILE_OFFSET_BITS=64 build_data.c
cc -c -g -DX86 -D_FILE_OFFSET_BITS=64 test.c
ar -rs crashlib.a main.o tools.o global_data.o memory.o filesys.o help.o task.o
build_data.o kernel.o test.o gdb_interface.o net.o dev.o alpha.o x86.o ppc.o ia6
4.o s390.o s390x.o s390dbf.o ppc64.o x86_64.o extensions.o remote.o va_server.o
va_server_v1.o symbols.o cmdline.o lkcd_common.o lkcd_v1.o lkcd_v2_v3.o lkcd_v5.
o lkcd_v7.o lkcd_v8.o lkcd_fix_mem.o s390_dump.o netdump.o diskdump.o xendump.o
lkcd_x86_trace.o unwind_v1.o unwind_v2.o unwind_v3.o unwind_x86_32_64.o xen_hype
r.o xen_hyper_command.o xen_hyper_global_data.o xen_hyper_dump_tables.o
gcc -g -O2 \
-o `cat mergeobj` libgdb.a \
../bfd/libbfd.a ../readline/libreadline.a ../opcodes/libopcod
es.a ../libiberty/libiberty.a -lm -lncurses ../libiberty/libiberty.a -ldl
-rdynamic `cat mergelibs`
#
Then run the test command with the tree possible argument types:
crash> test -x -a dasharg this that the other
arg: -x
arg: -a dasharg
arg: this
arg: that
arg: the
arg: other
do test work here...
crash>
The easiest way to implement a new command's functionality is to use an existing command as a template. There is a broad range of utility routines that handle argument strings, read memory, access data structures and their members, display data, and so on, that obviate the need to reinvent the wheel to accomplish a task. Look at what other similar commands do, and copy their mechanisms.