Kernel Crash Dump Guide - Red Hat Customer Portal

Transcript

Red Hat Enterprise Linux 7 Kernel Crash Dump Guide Kernel Crash Dump Configuration and Analysis Mark Flitter Jaromír Hradílek Petr Bokoč Red Hat Enterprise Linux 7 Kernel Crash Dump Guide Kernel Crash Dump Configuration and Analysis Mark Flitter Red Hat Custo mer Co ntent Services [email protected] m Jaro mír Hradílek Red Hat Custo mer Co ntent Services Petr Bo ko č Red Hat Custo mer Co ntent Services Legal Notice Co pyright © 20 16 Red Hat, Inc. This do cument is licensed by Red Hat under the Creative Co mmo ns Attributio n-ShareAlike 3.0 Unpo rted License. If yo u distribute this do cument, o r a mo dified versio n o f it, yo u must pro vide attributio n to Red Hat, Inc. and pro vide a link to the o riginal. If the do cument is mo dified, all Red Hat trademarks must be remo ved. Red Hat, as the licenso r o f this do cument, waives the right to enfo rce, and agrees no t to assert, Sectio n 4 d o f CC-BY-SA to the fullest extent permitted by applicable law. Red Hat, Red Hat Enterprise Linux, the Shado wman lo go , JBo ss, OpenShift, Fedo ra, the Infinity lo go , and RHCE are trademarks o f Red Hat, Inc., registered in the United States and o ther co untries. Linux ® is the registered trademark o f Linus To rvalds in the United States and o ther co untries. Java ® is a registered trademark o f Oracle and/o r its affiliates. XFS ® is a trademark o f Silico n Graphics Internatio nal Co rp. o r its subsidiaries in the United States and/o r o ther co untries. MySQL ® is a registered trademark o f MySQL AB in the United States, the Euro pean Unio n and o ther co untries. No de.js ® is an o fficial trademark o f Jo yent. Red Hat So ftware Co llectio ns is no t fo rmally related to o r endo rsed by the o fficial Jo yent No de.js o pen so urce o r co mmercial pro ject. The OpenStack ® Wo rd Mark and OpenStack lo go are either registered trademarks/service marks o r trademarks/service marks o f the OpenStack Fo undatio n, in the United States and o ther co untries and are used with the OpenStack Fo undatio n's permissio n. We are no t affiliated with, endo rsed o r spo nso red by the OpenStack Fo undatio n, o r the OpenStack co mmunity. All o ther trademarks are the pro perty o f their respective o wners. Abstract The Kernel Crash Dump Guide do cuments ho w to co nfigure, test, and use the kdump crash co llectio n service o n Red Hat Enterprise Linux 7, and pro vides a brief o verview o f ho w to analyze the resulting co re dump using the crash debugging utility. It is o riented to wards system administrato rs with a basic understanding o f the Red Hat Enterprise Linux system. T able of Cont ent s T able of Contents . .hapt ⁠C . . . .er . .1. .. Int . . .roduct . . . . . .ion . . .t.o. kdump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2. . . . . . . . . . ⁠1.1. Ab o ut kd ump and kexec 2 ⁠1.2. Memo ry Req uirements 2 . .hapt ⁠C . . . .er . .2. .. Inst . . . .alling . . . . .and . . . .Configuring . . . . . . . . . . .kdump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3. . . . . . . . . . ⁠2 .1. Ins talling kd ump 3 ⁠2 .2. Co nfig uring kd ump o n the Co mmand Line 4 ⁠2 .3. Co nfig uring kd ump in the G rap hic al Us er Interfac e 8 ⁠2 .4. Tes ting the kd ump Co nfig uratio n 14 ⁠2 .5. Ad d itio nal Res o urc es 14 . .hapt ⁠C . . . .er . .3. . .Analyz . . . . . .ing . . . a. .Core . . . . Dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1. 6. . . . . . . . . . ⁠3 .1. Ins talling the c ras h Utility 16 ⁠3 .2. Running the c ras h Utility 16 ⁠3 .3. Dis p laying the Mes s ag e Buffer 17 ⁠3 .4. Dis p laying a Bac ktrac e 18 ⁠3 .5. Dis p laying a Pro c es s Status 19 ⁠3 .6 . Dis p laying Virtual Memo ry Info rmatio n 19 ⁠3 .7. Dis p laying O p en Files ⁠3 .8 . Exiting the Utility 20 20 . .ppendix ⁠A . . . . . . . A. . . Frequent . . . . . . . . ly . . Asked . . . . . .Q . .uest . . . .ions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2. 1. . . . . . . . . . . .ppendix ⁠A . . . . . . . B. . . .Support . . . . . . .ed . . .kdump . . . . . .Configurat . . . . . . . . . ions . . . . .and . . . .T.arget . . . . .s. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2. 3. . . . . . . . . . ⁠B .1. Memo ry Req uirements fo r kd ump ⁠B .2. Minimum Thres ho ld fo r Auto matic Memo ry Res ervatio n ⁠B .3. Sup p o rted kd ump Targ ets 23 23 24 ⁠B .4. Sup p o rted kd ump Filtering Levels ⁠B .5. Sup p o rted Default Ac tio ns 24 25 ⁠B .6 . Es timating Kd ump Siz e 25 . .ppendix ⁠A . . . . . . . C. . . Revision . . . . . . . . .Hist . . . ory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2. 7. . . . . . . . . . 1 Kernel Crash Dump G uide Chapter 1. Introduction to kdump 1.1. About kdump and kexec K d u mp is a kernel crash dumping mechanism that allows you to save the contents of the system's memory for later analysis. It relies on kexec, which can be used to boot a Linux kernel from the context of another kernel, bypass BIOS, and preserve the contents of the first kernel's memory that would otherwise be lost. In case of a system crash, kdump uses kexec to boot into a second kernel (a capture kernel). This second kernel resides in a reserved part of the system memory that is inaccessible to the first kernel. The second kernel then captures the contents of the crashed kernel's memory (a crash dump) and saves it. 1.2. Memory Requirement s In order for kdump to be able to capture a kernel crash dump and save it for further analysis, a part of the system memory has to be permanently reserved for the capture kernel. When reserved, this part of the system memory is not available to main kernel. The memory requirements vary based on certain system parameters. One of the major factors is the system's hardware architecture. To find out the exact name of the machine architecture (such as x86 _6 4 ) and print it to standard output, type the following command at a shell prompt: uname -m Another factor which influences the amount of memory to be reserved is the total amount of installed system memory. For example, on the x86_64 architecture, the amount of reserved memory will be 160 MB + 2 bits for every 4 KB of RAM. On a system with 1 TB of total physical memory installed, this means 224 MB (160 MB + 64 MB). For a complete list of memory requirements for kdump based on the system architecture and the amount of physical memory, see Section B.1, “ Memory Requirements for kdump” . On many systems, kdump can estimate the amount of required memory and reserve it automatically. This behavior is enabled by default, but only works on systems that have more than a certain amount of total available memory, which varies based on the system architecture. See Section B.2, “ Minimum Threshold for Automatic Memory Reservation” for a list of minimum requirements for automatic memory reservation based on the system architecture. If the system has less than the minimum amount of memory required for the automatic allocation to work or if your use case requires a different value, you can configure the amount of reserved memory manually. For information on how to do so on the command line, see Section 2.2.1, “ Configuring the Memory Usage” . For information on how to configure the amount of reserved memory in the graphical user interface, see Section 2.3.1, “ Configuring the Memory Usage” . Important It is highly recommended to test the configuration after setting up the kdump service, even when using the automatic memory reservation. For instructions on how to test your configuration, see Section 2.4, “ Testing the kdump Configuration” . 2 ⁠Chapt er 2 . Inst alling and Configuring kdump Chapter 2. Installing and Configuring kdump 2.1. Inst alling kdump In many cases, the kd ump service is installed and activated by default on new Red Hat Enterprise Linux 7 installations. The An aco n d a installer provides a screen for kdump configuration when performing an interactive installation using the graphical or text interface. The installer screen is titled Kd ump and is available from the main Instal l ati o n Summary screen, and only allows limited configuration - you can only select whether kdump will be enabled and how much memory will be reserved. Information about memory requirements for kdump is available in Section B.1, “ Memory Requirements for kdump” . The Kdump configuration screen in the installer is documented in the Red Hat Enterprise Linux 7 Installation Guide. Note In previous releases of Red Hat Enterprise Linux, kdump configuration was available in the First b o o t utility which was automatically executed after the installation finished and the system rebooted for the first time. Starting with Red Hat Enterprise Linux 7.1, kdump configuration has been moved into the installer. Some installation options, such as custom Kickstart installations, may not install or enable kdump by default. If this is the case on your system, and you want to install kdump additionally, execute the following command as ro o t at a shell prompt: # yum i nstal l kexec-to o l s This will install kdump and all other necessary packages, assuming your system has an active subscription or a custom repository containing the kexec-tools package for your system's architecture. Note If you do not know whether kdump is installed on your system, you can check using rpm: $ rpm -q kexec-to o l s Additionally, a graphical configuration tool is available, but not installed by default if you use the command described above. To install this utility, which is described in Section 2.3, “ Configuring kdump in the Graphical User Interface” , use the following command as ro o t: # yum i nstal l system-co nfi g -kd ump For more information on how to install new packages in Red Hat Enterprise Linux 7 using the Yu m package manager, see the Red Hat Enterprise Linux 7 System Administrator's Guide. 3 Kernel Crash Dump G uide Important A limitation in the current implementation of the Intel IO MMU driver can occasionally prevent the kd ump service from capturing the core dump image. To use kdump on Intel architectures reliably, it is advised that the IOMMU support is disabled. 2.2. Configuring kdump on t he Command Line 2.2.1. Configuring t he Memory Usage Memory reserved for the kdump kernel is always reserved during system boot, which means that the amount of memory is specified in the system's boot loader configuration. This section will explain how to change the amount of reserved memory on AMD 64 and Intel 64 systems and IBM Power Systems servers using the G R U B 2 boot loader, and on IBM System z using z ip l. Pro ced u re 2.1. C h an g in g Memo ry O p t io n s in G R U B 2 1. Open the /etc/d efaul t/g rub configuration file as ro o t using a plain text editor such as vim or G ed it . 2. In this file, locate the line beginning with G R UB_C MD LINE_LINUX. The line will look similar to the following: GRUB_CMDLINE_LINUX="rd.lvm.lv=rhel/swap crashkernel = auto rd.lvm.lv=rhel/root rhgb quiet" Note the highlighted crashkernel = option; this is where the reserved memory is configured. 3. Change the value of the crashkernel = option to the amount of memory you want to reserve. For example, to reserve 128 MB of memory, use the following: crashkernel=128M Note There are multiple ways to configure the memory reserved - for example, you can define an offset or multiple memory amounts based on how much RAM is available in the system at startup. This is described further in this section. Then, save the file and exit the editor. 4. Finally, regenerate the G R U B 2 configuration using the edited d efaul t file. If your system uses BIOS firmware, execute the following command: # g rub2-mkco nfi g -o /bo o t/g rub2/g rub. cfg On a system with UEFI firmware, execute the following instead: # g rub2-mkco nfi g -o /bo o t/efi /EFI/red hat/g rub. cfg 4 ⁠Chapt er 2 . Inst alling and Configuring kdump After finishing the procedure above, the boot loader is reconfigured and the amount of memory you have specified in its configuration file will be reserved after the next reboot. Pro ced u re 2.2. C h an g in g Memo ry O p t io n s in z ip l 1. Open the /etc/zi pl . co nf configuration file as ro o t using a plain text editor such as vim or G ed it . 2. In this file, locate the parameters= section, and edit the crashkernel = parameter (or add it if not present). For example, to reserve 128 MB of memory, use the following: crashkernel=128M Note There are multiple ways to configure the memory reserved - for example, you can define an offset or multiple memory amounts based on how much RAM is available in the system at startup. This is described further in this section. Then, save the file and exit the editor. 3. Finally, regenerate the z ip l configuration: # zi pl Note Executing only the zi pl command with no additional options will use default values. See the zi pl (8) man page for information about available options. After finishing the procedure above, the boot loader is reconfigured and the amount of memory you have specified in its configuration file will be reserved after the next reboot. The crashkernel = option can be defined in multiple ways. The auto value enables automatic configuration of reserved memory based on the total amount of memory in the system, following the guidelines described in Section B.1, “ Memory Requirements for kdump” . Replace the auto value with a specific amount of memory to change this behavior. For example, to reserve 128 MB of memory, use the following: crashkernel=128M You can also set the amount of reserved memory to be variable, depending on the total amount of installed memory. The syntax is for variable memory reservation is crashkernel = : ,: . For example: crashkernel=512M-2G:64M,2G-:128M The above example will reserve 64 MB of memory if the total amount of system memory is 512 MB or higher and lower than 2 GB. If the total amount of memory is more than 2 GB, 128 MB will be reserved for kdump instead. 5 Kernel Crash Dump G uide On some systems, it might be necessary to reserve memory with a certain fixed offset. If the offset is set, the reserved memory will begin there. To offset the reserved memory, use the following syntax: crashkernel=128M@ 16M The example above means that kdump will reserve 128 MB of memory starting at 16 MB (physical address 0x01000000). If the offset parameter is set to 0 or omitted entirely, kdump will offset the reserved memory automatically. This syntax can also be used when setting a variable memory reservation as described above; in this case, the offset is always specified last (for example, crashkernel = 512M-2G : 6 4 M,2G -: 128M@ 16 M). 2.2.2. Configuring t he kdump T ype When a kernel crash is captured, the core dump can be either stored as a file in a local file system, written directly to a device, or sent over a network using the NFS (Network File System) or SSH (Secure Shell) protocol. Only one of these options can be set at the moment, and the default option is to store the vmco re file in the /var/crash/ directory of the local file system. To change this, as ro o t, open the /etc/kd ump. co nf configuration file in a text editor and edit the options as described below. To change the local directory in which the core dump is to be saved, remove the hash sign (“ #” ) from the beginning of the #path /var/crash line, and replace the value with a desired directory path. path /usr/local/cores Important In Red Hat Enterprise Linux 7, the directory defined as the kdump target using the path directive must exist when the kd ump systemd service is started - otherwise the service will fail. This behavior is different from earlier releases of Red Hat Enterprise Linux, where the directory was being created automatically if it did not exist when starting the service. Optionally, if you wish to write the file to a different partition, follow the same procedure with the one of the lines beginning with #ext4 . Here, you can use either a device name (the #ext4 /d ev/vg /l v_kd ump line), a file system label (the #ext4 LABEL= /bo o t line) or a UUID (the #ext4 UUID = 0 3138356 -5e6 1-4 ab3-b58e-2750 7ac4 19 37 line). Change the file system type as well as the device name, label or UUID to the desired values. For example: ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937 Important Specifying storage devices using a LABEL= or UUID = is recommended. D isk device names such as /d ev/sd a3 are not guaranteed to be consistent across reboot. See the Red Hat Enterprise Linux 7 Storage Administration Guide for information about persistent disk device naming. 6 ⁠Chapt er 2 . Inst alling and Configuring kdump Important When dumping to D ASD on s390x hardware, it is essential that the dump devices are correctly specified in /etc/d asd . co nf before proceeding. To write the dump directly to a device, remove the hash sign (“ #” ) from the beginning of the #raw /d ev/vg /l v_kd ump line, and replace the value with a desired device name. For example: raw /dev/sdb1 To store the dump to a remote machine using the NFS protocol, remove the hash sign (“ #” ) from the beginning of the #nfs my. server. co m: /expo rt/tmp line, and replace the value with a valid hostname and directory path. For example: nfs penguin.example.com:/export/cores To store the dump to a remote machine using the SSH protocol, remove the hash sign (“ #” ) from the beginning of the #ssh user@ my. server. co m line, and replace the value with a valid username and hostname. To include your SSH key in the configuration as well, remove the hash sign (“ #” ) from the beginning of the #sshkey /ro o t/. ssh/kd ump_i d _rsa line and change the value to the location of a key valid on the server you are trying to dump to. For example: ssh john@ penguin.example.com sshkey /root/.ssh/mykey For information on how to configure an SSH server and set up a key-based authentication, see the Red Hat Enterprise Linux 7 System Administrator's Guide. For a complete list of currently supported and unsupported targets sorted by type, see Table B.3, “ Supported kdump Targets” . 2.2.3. Configuring t he Core Collect or To reduce the size of the vmco re dump file, kd ump allows you to specify an external application (a core collector) to compress the data, and optionally leave out all irrelevant information. Currently, the only fully supported core collector is maked umpfi l e. To enable the core collector, as ro o t, open the /etc/kd ump. co nf configuration file in a text editor, remove the hash sign (“ #” ) from the beginning of the #co re_co l l ecto r maked umpfi l e -l -messag e-l evel 1 -d 31 line, and edit the command line options as described below. To enable the dump file compression, add the -c parameter. For example: core_collector makedumpfile -c To remove certain pages from the dump, add the -d value parameter, where value is a sum of values of pages you want to omit as described in Table B.4, “ Supported Filtering Levels” . For example, to remove both zero and free pages, use the following: core_collector makedumpfile -d 17 -c See the maked umpfi l e(8) man page for a complete list of available options. 7 Kernel Crash Dump G uide 2.2.4 . Configuring t he Default Act ion By default, when kd ump fails to create a core dump at the target location specified in Section 2.2.2, “ Configuring the kdump Type” , the root file system is mounted and kd ump attempts to save the core locally. To change this behavior, as ro o t, open the /etc/kd ump. co nf configuration file in a text editor, remove the hash sign (“ #” ) from the beginning of the #d efaul t shel l line, and replace the value with a desired action as described in Table B.5, “ Supported D efault Actions” . For example: default reboot 2.2.5. Enabling t he Service To start the kd ump daemon at boot time, type the following at a shell prompt as ro o t: systemctl enabl e kd ump. servi ce This will enable the service for mul ti -user. targ et. Similarly, typing systemctl sto p kd ump will disable it. To start the service in the current session, use the following command as ro o t: systemctl start kd ump. servi ce Important In Red Hat Enterprise Linux 7, the directory defined as the kdump target must exist when the kd ump systemd service is started - otherwise the service will fail. This behavior is different from earlier releases of Red Hat Enterprise Linux, where the directory was being created automatically if it did not exist when starting the service. For more information on systemd and configuring services in general, see the Red Hat Enterprise Linux 7 System Administrator's Guide. 2.3. Configuring kdump in t he Graphical User Int erface To start the K ern el D u mp C o n f ig u rat io n utility, select Act ivit ies → O t h er → K ern el crash d u mp s from the panel, or type system-co nfi g -kd ump at a shell prompt. You will be presented with a window as shown in Figure 2.1, “ Basic Settings” . The utility allows you to configure kd ump as well as to enable or disable starting the service at boot time. When you are done, click Appl y to save the changes. Unless you are already authenticated, you will be prompted to enter the superuser password. The utility will also remind you that you must reboot the system in order to apply any changes you have made to the configuration. 8 ⁠Chapt er 2 . Inst alling and Configuring kdump Important On IBM System z or PowerPC systems with SELi nux running in Enforcing mode, the kdumpgui_run_bootloader Boolean must be enabled before launching the Kernel D ump Configuration utility. This Boolean allows system-config-kdump to run the boot loader in the bootloader_t SELinux domain. To permanently enable the Boolean, run the following command as root; # setsebool -P kdumpgui_run_bootloader 1 Important When dumping to D ASD on s390x hardware, it is essential that the dump devices are correctly specified in /etc/d asd . co nf before proceeding. 2.3.1. Configuring t he Memory Usage The Basi c Setti ng s tab enables you to configure the amount of memory that is reserved for the kd ump kernel. To do so, select the Manual setti ng s radio button, and click the up and down arrow buttons next to the New kd ump Memo ry field to increase or decrease the amount of memory to be reserved. Notice that the Usabl e Memo ry field changes accordingly showing you the remaining memory that will be available to the system. See Section 1.2, “ Memory Requirements” for more information on kdump's memory requirements. 9 Kernel Crash Dump G uide Fig u re 2.1. B asic Set t in g s 2.3.2. Configuring t he kdump T ype The T arg et Setti ng s tab allows you to specify the target location for the vmco re dump. The dump can be either stored as a file in a local file system, written directly to a device, or sent over a network using the NFS (Network File System) or SSH (Secure Shell) protocol. 10 ⁠Chapt er 2 . Inst alling and Configuring kdump Fig u re 2.2. T arg et Set t in g s To save the dump to the local file system, select the Lo cal fi l esystem radio button. Optionally, you can customize the settings by choosing a different partition from the P arti ti o n drop-down list and a target directory using the P ath field. Important In Red Hat Enterprise Linux 7, the directory defined as the kdump target must exist when the kd ump systemd service is started - otherwise the service will fail. This behavior is different from earlier releases of Red Hat Enterprise Linux, where the directory was being created automatically if it did not exist when starting the service. To write the dump directly to a device, select the R aw d evi ce radio button, and choose the desired target device from the drop-down list next to it. To send the dump to a remote machine over a network connection, select the Netwo rk radio button. To use the NFS protocol, select the NFS radio button, and fill the Server name and P ath to d i recto ry fields. To use the SSH protocol, select the SSH radio button, and fill the Server name, P ath to d i recto ry, and User name fields with the remote server address, target directory, and a valid user name respectively. 11 Kernel Crash Dump G uide For information on how to configure an SSH server and set up a key-based authentication, see the Red Hat Enterprise Linux 7 System Administrator's Guide. For a complete list of currently supported targets, see Table B.3, “ Supported kdump Targets” . 2.3.3. Configuring t he Core Collect or The Fi l teri ng Setti ng s tab enables you to select the filtering level for the vmco re dump. Fig u re 2.3. Filt erin g Set t in g s To exclude the zero pag e, cache pag e, cache pri vate, user d ata, or free pag e from the dump, select the checkbox next to the appropriate label. 2.3.4 . Configuring t he Default Act ion To choose what action to perform when kd ump fails to create a core dump, select an appropriate option from the D efaul t acti o n drop-down list. Available options are d u mp t o ro o t f s an d reb o o t (the default action which attempts to save the core locally and then reboots the system), reb o o t (to reboot the system), sh ell (to present a user with an interactive shell prompt), h alt (to halt the system), and p o wero f f (to power the system off). 12 ⁠Chapt er 2 . Inst alling and Configuring kdump Fig u re 2.4 . Filt erin g Set t in g s To customize the options that are passed to the maked umpfi l e core collector, edit the C o re co l l ecto r text field; see Section 2.2.3, “ Configuring the Core Collector” for more information. 2.3.5. Enabling t he Service To start the kd ump service at boot time, click the Enabl e button on the toolbar and then click the Appl y button. This will enable and activate the service for mul ti -user. targ et. Clicking the D i sabl e button and confirming by clicking the Appl y button will disable the service immediately. Important In Red Hat Enterprise Linux 7, the directory defined as the kdump target must exist when the kd ump systemd service is started - otherwise the service will fail. This behavior is different from earlier releases of Red Hat Enterprise Linux, where the directory was being created automatically if it did not exist when starting the service. For more information on systemd targets and configuring services in general, see the Red Hat Enterprise Linux 7 System Administrator's Guide. 13 Kernel Crash Dump G uide 2.4 . T est ing t he kdump Configurat ion Warning The commands below will cause the kernel to crash. Use caution when following these steps, and by no means use them on a production system. To test the configuration, reboot the system with kd ump enabled, and make sure that the service is running: ~]# systemctl i s-acti ve kd ump active Then type the following commands at a shell prompt: echo 1 > /pro c/sys/kernel /sysrq echo c > /pro c/sysrq -tri g g er This will force the Linux kernel to crash, and the address-YYYY-MM-DD-HH:MM:SS/vmco re file will be copied to the location you have selected in the configuration (that is, to /var/crash/ by default). Note In addition to confirming the validity of the configuration, this action can also be used to record how long a crash dump will take to complete if it is performed under a representative test load. 2.5. Addit ional Resources 2.5.1. Inst alled Document at ion kd u mp .co n f (5) — a manual page for the /etc/kd ump. co nf configuration file containing the full documentation of available options. z ip l.co n f (5) — a manual page for the /etc/zi pl . co nf configuration file. z ip l(8) — a manual page for the zi pl boot loader utility for IBM System z. maked u mp f ile(8) — a manual page for the maked umpfi l e core collector. kexec(8) — a manual page for kexec. crash (8) — a manual page for the crash utility. /usr/share/d o c/kexec-to o l s-version/kexec-kd ump-ho wto . txt — an overview of the kd ump and kexec installation and usage. 2.5.2. Online Document at ion h t t p s://access.red h at .co m/sit e/so lu t io n s/6 038 14 ⁠Chapt er 2 . Inst alling and Configuring kdump The Red Hat Knowledgebase article about the kexec and kd ump configuration. h t t p s://access.red h at .co m/sit e/so lu t io n s/223773 The Red Hat Knowledgebase article about supported kd ump targets. h t t p ://p eo p le.red h at .co m/an d erso n / The crash utility homepage. h t t p s://www.g n u .o rg /so f t ware/g ru b / The G R U B 2 boot loader homepage and documentation. 15 Kernel Crash Dump G uide Chapter 3. Analyzing a Core Dump To determine the cause of the system crash, you can use the crash utility, which provides an interactive prompt very similar to the GNU D ebugger (GD B). This utility allows you to interactively analyze a running Linux system as well as a core dump created by netd ump, d i skd ump, xend ump, or kd ump. 3.1. Inst alling t he crash Ut ilit y To install the crash analyzing tool, execute the following command from a shell prompt as ro o t: yum i nstal l crash In addition to crash , it is also necessary to install the kernel-debuginfo package that corresponds to your running kernel, which provides the data necessary for dump analysis. To install kernel-debuginfo we use the d ebug i nfo -i nstal l command as ro o t: d ebug i nfo -i nstal l kernel For more information on how to install new packages in Red Hat Enterprise Linux using the Yu m package manager, see the Red Hat Enterprise Linux 7 System Administrator's Guide. 3.2. Running t he crash Ut ilit y To start the utility, type the command in the following form at a shell prompt: crash /var/crash//vmco re /usr/l i b/d ebug /l i b/mo d ul es//vml i nux Note that the version should be the same that was captured by kd ump. To find out which kernel you are currently running, use the uname -r command. Examp le 3.1. R u n n in g t h e crash u t ilit y ~]# crash /usr/l i b/d ebug /l i b/mo d ul es/2. 6 . 32-6 9 . el 6 . i 6 86 /vml i nux \ /var/crash/127. 0 . 0 . 1-20 10 -0 8-25-0 8: 4 5: 0 2/vmco re crash 5.0.0-23.el6 Copyright (C) 2002-2010 Red Hat, Inc. Copyright (C) 2004, 2005, 2006 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for 16 ⁠Chapt er 3. Analyz ing a Core Dump details. GNU gdb (GDB) 7.0 Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-pc-linux-gnu"... KERNEL: DUMPFILE: DUMP] CPUS: DATE: UPTIME: LOAD AVERAGE: TASKS: NODENAME: RELEASE: VERSION: MACHINE: MEMORY: PANIC: PID: COMMAND: TASK: CPU: STATE: /usr/lib/debug/lib/modules/2.6.32-69.el6.i686/vmlinux /var/crash/127.0.0.1-2010-08-25-08:45:02/vmcore [PARTIAL 4 Wed Aug 25 08:44:47 2010 00:09:02 0.00, 0.01, 0.00 140 hp-dl320g5-02.lab.bos.redhat.com 2.6.32-69.el6.i686 #1 SMP Tue Aug 24 10:31:45 EDT 2010 i686 (2394 Mhz) 8 GB "Oops: 0002 [#1] SMP " (check log for details) 5591 "bash" f196d560 [THREAD_INFO: ef4da000] 2 TASK_RUNNING (PANIC) crash> 3.3. Displaying t he Message Buffer To display the kernel message buffer, type the l o g command at the interactive prompt. Examp le 3.2. D isp layin g t h e kern el messag e b u f f er crash> l o g ... several lines omitted ... EIP: 0060:[] EFLAGS: 00010096 CPU: 2 EIP is at sysrq_handle_crash+0xf/0x20 EAX: 00000063 EBX: 00000063 ECX: c09e1c8c EDX: 00000000 ESI: c0a09ca0 EDI: 00000286 EBP: 00000000 ESP: ef4dbf24 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 Process bash (pid: 5591, ti=ef4da000 task=f196d560 task.ti=ef4da000) Stack: c068146b c0960891 c0968653 00000003 00000000 00000002 efade5c0 c06814d0 <0> fffffffb c068150f b7776000 f2600c40 c0569ec4 ef4dbf9c 00000002 b7776000 17 Kernel Crash Dump G uide <0> efade5c0 00000002 b7776000 c0569e60 c051de50 ef4dbf9c f196d560 ef4dbfb4 Call Trace: [] ? __handle_sysrq+0xfb/0x160 [] ? write_sysrq_trigger+0x0/0x50 [] ? write_sysrq_trigger+0x3f/0x50 [] ? proc_reg_write+0x64/0xa0 [] ? proc_reg_write+0x0/0xa0 [] ? vfs_write+0xa0/0x190 [] ? sys_write+0x41/0x70 [] ? syscall_call+0x7/0xb Code: a0 c0 01 0f b6 41 03 19 d2 f7 d2 83 e2 03 83 e0 cf c1 e2 04 09 d0 88 41 03 f3 c3 90 c7 05 c8 1b 9e c0 01 00 00 00 0f ae f8 89 f6 05 00 00 00 00 01 c3 89 f6 8d bc 27 00 00 00 00 8d 50 d0 83 EIP: [] sysrq_handle_crash+0xf/0x20 SS:ESP 0068:ef4dbf24 CR2: 0000000000000000 Type hel p l o g for more information on the command usage. Note The kernel message buffer includes the most essential information about the system crash and, as such, it is always dumped first in to the vmco re-d mesg . txt file. This is useful when an attempt to get the full vmco re file failed, for example because of lack of space on the target location. By default, vmco re-d mesg . txt is located in the /var/crash/ directory. 3.4 . Displaying a Backt race To display the kernel stack trace, type the bt command at the interactive prompt. You can use bt to display the backtrace of a single process. Examp le 3.3. D isp layin g t h e kern el st ack t race crash> bt PID: 5591 TASK: f196d560 CPU: 2 COMMAND: "bash" #0 [ef4dbdcc] crash_kexec at c0494922 #1 [ef4dbe20] oops_end at c080e402 #2 [ef4dbe34] no_context at c043089d #3 [ef4dbe58] bad_area at c0430b26 #4 [ef4dbe6c] do_page_fault at c080fb9b #5 [ef4dbee4] error_code (via page_fault) at c080d809 EAX: 00000063 EBX: 00000063 ECX: c09e1c8c EDX: 00000000 EBP: 00000000 DS: 007b ESI: c0a09ca0 ES: 007b EDI: 00000286 GS: 00e0 CS: 0060 EIP: c068124f ERR: ffffffff EFLAGS: 00010096 #6 [ef4dbf18] sysrq_handle_crash at c068124f #7 [ef4dbf24] __handle_sysrq at c0681469 #8 [ef4dbf48] write_sysrq_trigger at c068150a #9 [ef4dbf54] proc_reg_write at c0569ec2 #10 [ef4dbf74] vfs_write at c051de4e 18 ⁠Chapt er 3. Analyz ing a Core Dump #11 [ef4dbf94] sys_write at c051e8cc #12 [ef4dbfb0] system_call at c0409ad5 EAX: ffffffda EBX: 00000001 ECX: b7776000 EDX: 00000002 DS: 007b ESI: 00000002 ES: 007b EDI: b7776000 SS: 007b ESP: bfcb2088 EBP: bfcb20b4 GS: 0033 CS: 0073 EIP: 00edc416 ERR: 00000004 EFLAGS: 00000246 Type hel p bt for more information on the command usage. 3.5. Displaying a Process St at us To display status of processes in the system, type the ps command at the interactive prompt. You can use ps to display the status of a single process. Examp le 3.4 . D isp layin g t h e st at u s o f p ro cesses in t h e syst em crash> ps PID PPID CPU TASK > 0 0 0 c09dc560 > 0 0 1 f7072030 0 0 2 f70a3a90 > 0 0 3 f70ac560 1 0 1 f705ba90 ... several lines omitted ... 5566 1 1 f2592560 5567 1 2 ef427560 5587 5132 0 f196d030 > 5591 5587 2 f196d560 ST RU RU RU RU IN %MEM 0.0 0.0 0.0 0.0 0.0 IN IN IN RU 0.0 0.0 0.0 0.0 VSZ 0 0 0 0 2828 12876 12876 11064 5084 RSS COMM 0 [swapper] 0 [swapper] 0 [swapper] 0 [swapper] 1424 init 784 784 3184 1648 auditd auditd sshd bash Type hel p ps for more information on the command usage. 3.6. Displaying Virt ual Memory Informat ion To display basic virtual memory information, type the vm command at the interactive prompt. You can use vm to display information on a single process. Examp le 3.5. D isp layin g virt u al memo ry in f o rmat io n o f t h e cu rren t co n t ext crash> vm PID: 5591 TASK: f196d560 CPU: 2 MM PGD RSS TOTAL_VM f19b5900 ef9c6000 1648k 5084k VMA START END FLAGS f1bb0310 242000 260000 8000875 f26af0b8 260000 261000 8100871 efbc275c 261000 262000 8100873 efbc2a18 268000 3ed000 8000075 efbc23d8 3ed000 3ee000 8000070 efbc2888 3ee000 3f0000 8100071 efbc2cd4 3f0000 3f1000 8100073 COMMAND: "bash" FILE /lib/ld-2.12.so /lib/ld-2.12.so /lib/ld-2.12.so /lib/libc-2.12.so /lib/libc-2.12.so /lib/libc-2.12.so /lib/libc-2.12.so 19 Kernel Crash Dump G uide efbc243c 3f1000 3f4000 100073 efbc28ec 3f6000 3f9000 8000075 /lib/libdl-2.12.so efbc2568 3f9000 3fa000 8100071 /lib/libdl-2.12.so efbc2f2c 3fa000 3fb000 8100073 /lib/libdl-2.12.so f26af888 7e6000 7fc000 8000075 /lib/libtinfo.so.5.7 f26aff2c 7fc000 7ff000 8100073 /lib/libtinfo.so.5.7 efbc211c d83000 d8f000 8000075 /lib/libnss_files-2.12.so efbc2504 d8f000 d90000 8100071 /lib/libnss_files-2.12.so efbc2950 d90000 d91000 8100073 /lib/libnss_files-2.12.so f26afe00 edc000 edd000 4040075 f1bb0a18 8047000 8118000 8001875 /bin/bash f1bb01e4 8118000 811d000 8101873 /bin/bash f1bb0c70 811d000 8122000 100073 f26afae0 9fd9000 9ffa000 100073 ... several lines omitted ... Type hel p vm for more information on the command usage. 3.7. Displaying Open Files To display information about open files, type the fi l es command at the interactive prompt. You can use fi l es to display files opened by only one selected process. Examp le 3.6 . D isp layin g in f o rmat io n ab o u t o p en f iles o f t h e cu rren t co n t ext crash> fi l es PID: 5591 TASK: f196d560 CPU: 2 ROOT: / CWD: /root FD FILE DENTRY INODE 0 f734f640 eedc2c6c eecd6048 1 efade5c0 eee14090 f00431d4 2 f734f640 eedc2c6c eecd6048 10 f734f640 eedc2c6c eecd6048 255 f734f640 eedc2c6c eecd6048 COMMAND: "bash" TYPE PATH CHR /pts/0 REG /proc/sysrq-trigger CHR /pts/0 CHR /pts/0 CHR /pts/0 Type hel p fi l es for more information on the command usage. 3.8. Exit ing t he Ut ilit y To exit the interactive prompt and terminate crash , type exi t or q . Examp le 3.7. Exit in g t h e crash u t ilit y crash> exi t ~]# 20 ⁠Appendix A. Frequent ly Asked Q uest ions Appendix A. Frequently Asked Questions Q: Wh at Met h o d s o f D u mp in g are Availab le f o r Virt u al Mach in es? A: In most cases, the kd ump mechanism will be sufficient for obtaining a memory dump from a machine after a crash or panic. This can be set up in the same manner as installations to bare metal. However, in some cases, it may be necessary to work directly with the hypervisor to obtain a crash dump. There are two mechanisms available with l i bvi rt to achieve this; pvpani c and vi rsh d ump. Both of these methods are described in the Virtualization D eployment and Administration Guide. The pvpani c mechanism can be found at Virtualization D eployment and Administration Guide - Setting a Panic D evice. The vi rsh d ump command is discussed in Virtualization D eployment and Administration Guide - Creating a D ump File of a D omain's Core. Q: H o w d o I u p lo ad a larg e d u mp f ile t o R ed H at Su p p o rt Services? A: In some cases, it might be necessary to send a kernel crash dump file to Red Hat Global Support Services for analysis. However, the dump file can be very large, even after being filtered. Since files larger than 250 MB cannot be uploaded directly through the Red Hat Customer Portal when opening a new support case, an FTP server is provided by Red Hat for uploading large files. The FTP server's address is d ro pbo x. red hat. co m and the files are to be uploaded in the /i nco mi ng / directory. Your FTP client needs to be set into passive mode; if your firewall does not allow this mode, you may use the o ri g i n-d ro pbo x. red hat. co m server using active mode. Make sure that the uploaded files are compressed using a program such as g z ip and properly and descriptively named. Using your support case number in the file name is recommended. After successfuly uploading all necessary files, provide the engineer in charge of your support case with the exact file name and its SHA1 or MD 5 checksum. For more specific instructions and additional information, see https://access.redhat.com/site/solutions/2112. Q: H o w mu ch t ime is n eed ed f o r a crash d u mp ? A: It is often necessary, for the purposes of disaster recovery planning, to know how long a dump will take to complete. However, the length of time it takes to complete a crash dump is highly dependent on the amount of memory being copied to disk and the speed of the interfaces between RAM and storage. 21 Kernel Crash Dump G uide For any test of timings, the system must be operating under a representative load, otherwise the page exclusion choices can present a false view of kdump behaviour with a fully loaded production system. This discrepancy will be observed more particularly when working with very large quantities of RAM. Storage interfaces should also be considered in your planning when assessing time to dump. Because of network constrants, a connection dumping over ssh for example, may take longer to complete than a locally attached SATA disk. 22 ⁠Appendix B. Support ed kdump Configurat ions and T arget s Appendix B. Supported kdump Configurations and Targets B.1. Memory Requirement s for kdump In order for kdump to be able to capture a kernel crash dump and save it for further analysis, a part of the system memory has to be permanently reserved for the capture kernel. The table below contains a list of minimum memory requirements for kdump based on the system's architecture and total available physical memory. For information on how to change memory settings on the command line, see Section 2.2.1, “ Configuring the Memory Usage” . For instructions on how to set up the amount of reserved memory in the graphical user interface, see Section 2.3.1, “ Configuring the Memory Usage” . T ab le B .1. Min imu m Amo u n t o f R eserved Memo ry R eq u ired f o r kd u mp Arch it ect u re Availab le Memo ry Min imu m R eserved Memo ry AMD 64 and Intel 64 (x86 _6 4 ) 2 GB and more IBM POWER (ppc6 4 ) 2 GB to 4 GB 4 GB to 32 GB 32 GB to 64 GB 64 GB to 128 GB 128 GB and more 2 GB and more 160 MB + 2 bits for every 4 KB of RAM. For a system with 1 TB of memory, 224 MB is the minimum (160 + 64 MB). 256 MB of RAM. 512 MB of RAM. 1 GB of RAM. 2 GB or RAM. 4 GB of RAM. 160 MB + 2 bits for every 4 KB of RAM. For a system with 1 TB of memory, 224 MB is the minimum (160 + 64 MB). IBM System z (s39 0 x) B.2. Minimum T hreshold for Aut omat ic Memory Reservat ion On some systems, it is possible to allocate memory for kdump automatically, either by using the crashkernel = auto parameter in the bootloader's configuration file, or by enabling this option in the graphical configuration utility. For this automatic reservation to work, however, a certain amount of total memory needs to be available in the system. This amount differs based on the system's architecture. The table below lists the thresholds for automatic memory allocation. If the system has less memory than specified in the table, memory will have to be reserved manually. For information on how to change these settings on the command line, see Section 2.2.1, “ Configuring the Memory Usage” . For instructions on how to change the amount of reserved memory in the graphical user interface, see Section 2.3.1, “ Configuring the Memory Usage” . T ab le B .2. Min imu m Amo u n t o f Memo ry R eq u ired f o r Au t o mat ic Memo ry R eservat io n Arch it ect u re R eq u ired Memo ry AMD 64 and Intel 64 (x86 _6 4 ) IBM POWER (ppc6 4 ) IBM System z (s39 0 x) 2 GB 2 GB 4 GB 23 Kernel Crash Dump G uide B.3. Support ed kdump T arget s When a kernel crash is captured, the core dump can be either written directly to a device, stored as a file on a local file system, or sent over a network. The table below contains a complete list of dump targets that are currently supported or explicitly unsupported by kdump. For information on how to configure the target type on the command line, see Section 2.2.2, “ Configuring the kdump Type” . For information on how to do so in the graphical user interface, see Section 2.3.2, “ Configuring the kdump Type” . T ab le B .3. Su p p o rt ed kd u mp T arg et s T yp e Su p p o rt ed T arg et s U n su p p o rt ed T arg et s Raw device All locally attached raw disks and partitions. ext2, ext3, ext4 , btrfs and xfs file systems on directly attached disk drives, hardware RAID logical drives, LVM devices, and md rai d arrays. Remote directories accessed using the NFS or SSH protocol over IP v4 . — Local file system Remote directory Any local file system not explicitly listed as supported in this table, including the auto type (automatic file system detection). Remote directories on the ro o tfs file system accessed using the NFS protocol. Remote directories accessed using the Remote directories accessed using the i SC SI protocol over both hardware i SC SI protocol on be2i scsi and software initiators. hardware. Multipath-based storages. — — Remote directories accessed over IP v6 . Remote directories accessed using the SMB/C IFS protocol. Remote directories accessed using the FC o E (Fibre Channel over Ethernet) protocol. Remote directories accessed using wireless network interfaces. B.4 . Support ed kdump Filt ering Levels To reduce the size of the dump file, kdump uses the maked umpfi l e core collector to compress the data and optionally leave out irrelevant information. The table below contains a complete list of filtering levels that are currently supported by the maked umpfi l e utility. For instructions on how to configure the core collector on the command line, see Section 2.2.3, “ Configuring the Core Collector” . For information on how to do so in the graphical user interface, see Section 2.3.3, “ Configuring the Core Collector” . T ab le B .4 . Su p p o rt ed Filt erin g Levels O p t io n D escrip t io n 1 2 4 Z ero pages Cache pages Cache private 24 ⁠Appendix B. Support ed kdump Configurat ions and T arget s O p t io n D escrip t io n 8 16 User pages Free pages Note The maked umpfi l e command supports removal of transparent huge pages and hugetlbfs pages on Red Hat Enterprise Linux 7.3 and later. Both these types of hugepages are considered User Pages and will be removed using the -8 level. B.5. Support ed Default Act ions By default, when kdump fails to create a core dump, it mounts the root file system and attempts to save the core locally. You can, however, configure kdump to perform a different operation in case it fails to save the core dump to the primary target. The table below lists all default actions that are currently supported by kdump. For detailed information on how to set up the default action on the command line, see Section 2.2.4, “ Configuring the D efault Action” . For information on how to do so in the graphical user interface, see Section 2.3.4, “ Configuring the D efault Action” . T ab le B .5. Su p p o rt ed D ef au lt Act io n s O p t io n D escrip t io n d ump_to _ro o tf s Attempt to save the core dump to the root file system. This option is especially useful in combination with a network target: if the network target is unreachable, this option configures kdump to save the core dump locally. The system is rebooted afterwards. Reboot the system, losing the core dump in the process. Halt the system, losing the core dump in the process. Power off the system, losing the core dump in the process. Run a shell session from within the initramfs, allowing the user to record the core dump manually. rebo o t hal t po wero ff shel l B.6. Est imat ing Kdump Siz e When planning and building your kd ump environment it is necessary to know how much space is required for the dump file before one is produced. The maked umpfi l e command can help with this. The --mem-usag e option provides a useful report about excludable pages, that can be used to determine which dump level you want to assign. This command should be run when the system is under representative load, otherwise maked umpfi l e will return a smaller value than is expected in your production environment. [root@ hostname ~]# makedumpfile --mem-usage /proc/kcore TYPE PAGES EXCLUDABLE DESCRIPTION ---------------------------------------------------------------------ZERO 501635 yes Pages filled with zero 25 Kernel Crash Dump G uide CACHE CACHE_PRIVATE private USER pages FREE KERN_DATA data 51657 5442 yes yes Cache pages Cache pages + 16301 yes User process 77738211 1333192 yes no Free pages Dumpable kernel Important The maked umpfi l e command reports in p ag es. This means that you must calculate the size of memory in use against the kernel page size, which in the Red Hat Enterprise Linux kernel, is 4 kilobytes (4096 bytes). 26 ⁠Appendix C. Revision Hist ory Appendix C. Revision History R evisio n 1.3- 2 Fri N o v 4 2016 Version for 7.3 GA publication. Mark Flit t er R evisio n 1.2- 9 T h u 18 Au g 2016 Mark Flit t er Updates for 7.3 Beta, addition of Z Series specific notes and estimating the size of vmcores. R evisio n 1.2- 0 Fri 06 Mar 2015 Pet r B o ko č Update fixing several issues such as wrong information for memory configuration and outdated screenshots. R evisio n 1.1- 3 Wed 18 Feb 2015 Pet r B o ko č Red Hat Enterprise Linux 7.1 GA release of the Kernel Crash D ump Guide. R evisio n 1.1- 0 Fri 05 D ec 2014 Pet r B o ko č Red Hat Enterprise Linux 7.1 Beta release of the Kernel Crash D ump Guide. R evisio n 1.0- 0 Mo n 02 Ju n 2014 Jaro mír H rad ílek Red Hat Enterprise Linux 7.0 GA release of the Kernel Crash D ump Guide. R evisio n 0.0- 8 Initial creation of the book. T h u Jan 17 2013 Jaro mír H rad ílek 27