# Kernel Panic Procedures

This article is a work in progress or otherwise under review
and does not represent current policy.

## Contents

1. Synopsis
2. Preliminary Notes
2.1. Obtaining a Kernel Dump
2.1.1. Example Crash: Force Panic from WSCons via KVM: Dell DRAC4
2.1.2. What Now
2.2. Processing the CORE Dump

## Synopsis

Although a few official NetBSD.org documents exist on the topics of using
the advanced kernel debugging using KGDB (Kernelized GNU Debugger (GDB)),
there are few documents which formalize a <b>"Kernel Panic/Crash Reporting Procedure"</b>
using a combination of DDB (the minimalist in-kernel debugger) in
combination with GDB after the crash.

http://www.netbsd.org/docs/kernel/#ddb


## Preliminary Notes

If the problem is easily re-created, try to obtain a kernel backtrace

The DDB is the minimalist kernel Debugger added by "options DDB" to the kernel

Obtain a backtrace at the "db{0}>" prompt using the "bt" command

Search the Mailing List Archives and Query the NetBSD.org PR database at
http://www.netbsd.org/support/query-pr.html

File a PR at http://www.netbsd.org/support/send-pr.html

Post the problem for the discussion on the appropriate mailing list.


## Obtaining a Kernel Dump

A kernel dump is possible to obtain from many kernel panics.
When at the DB prompt, simply execute:

	db{0}> sync

The dump of memory will be written to the swap partition.

At boot time, if rc.conf(5) has value "savecore=yes",
the swap file coredump will be saved to "/var/crash".

	savecore=yes
	savecore_flags="-N /netbsd -z"
	savecore_dir="/var/crash"

A gzip(1)-compressed file will be available for analysis with gdb(1).

Your swap partition must be at least the size of your physical RAM

Your "/var/crash" partition must have sufficient space to hold the same file.


## Example Crash: Force Panic from WSCons via KVM: Dell DRAC4

You can force a kernel panic from the console on amd64/i386 using the special key sequence: Control+Alt+Esc.

* Forcing the Kernel Panic from Console

TODO: <img alt="Forcing the Kernel Panic from Console" src="/images/thumb/6/6f/cf_NetBSD_KernPanic_Force.png/300px-cf_NetBSD_KernPanic_Force.png" width="300" height="249" border="0" class="thumbimage" />


You can then instruct the KDB to run a preliminary backtrace to get a general idea of what went wrong:

TODO: <img alt="Initial backtrace" src="/images/thumb/6/62/cf_NetBSD_KernPanic_InitialBacktrace.png/300px-cf_NetBSD_KernPanic_InitialBacktrace.png" width="300" height="249" border="0" class="thumbimage" />

You can then force a sync of the file system and and dump of the kernel memory into the swap partition:

TODO: <img alt="Forcing the RAM dump to Swap" src="/images/thumb/6/68/cf_NetBSD_KernPanic_ForceDump.png/300px-cf_NetBSD_KernPanic_ForceDump.png" width="300" height="225" border="0" class="thumbimage" />

At reboot, you will see the "/etc/rc.d/savecore" script archive and gzip the dump:

TODO: <img alt="Save Core Running" src="/images/thumb/9/94/cf_NetBSD_KernPanic_SaveCore.png/300px-cf_NetBSD_KernPanic_SaveCore.png" width="300" height="249" border="0" class="thumbimage" />

You can then load the core dump into standard <b>gdb(8)</b>


## What Now

You can submit the feedback as a PR to the NetBSD GNATS system.

If you wish to continue to debug, you may find this article useful:
<a href="/NetBSD_kernel_developer_cheat-sheet" title="NetBSD kernel developer cheat-sheet">NetBSD kernel developer cheat-sheet</a>


## Processing the CORE Dump

Heubert Feyrer has a great guide to analyzing kernel panic core dumps at: http://www.feyrer.de/NetBSD/bx/blosxom.cgi/index.front?-tags=gdb

Additionally, the following command below can be used to create a relatively useful backtrace:

<pre>
bash-4.0# cd /var/crash
bash-4.0# gunzip -d *gz
bash-4.0# gdb  --symbols=/netbsd.gdb --quiet --eval-command="file /netbsd.gdb" \ 
               --eval-command="target kvm netbsd.1.core" --eval-command "bt" \ 
               --eval-command "list" --eval-command "info all-registers" 2>&1
Load new symbol table from "/netbsd.gdb"? (y or n) y
Reading symbols from /netbsd.gdb...done.
#0  0xc047c9f8 in cpu_reboot (howto=256, bootstr=0x0) at /usr/src/sys/arch/i386/i386/machdep.c:927
927                     dumpsys();
#0  0xc047c9f8 in cpu_reboot (howto=256, bootstr=0x0) at /usr/src/sys/arch/i386/i386/machdep.c:927
#1  0xc01c3f2a in db_sync_cmd (addr=-1065223264, have_addr=false, count=-1071881791, modif=0xcc883c04 "[BINARY]") at /usr/src/sys/ddb/db_command.c:1304
#2  0xc01c45fa in db_command (last_cmdp=0xc07dfe3c) at /usr/src/sys/ddb/db_command.c:926
#3  0xc01c4856 in db_command_loop () at /usr/src/sys/ddb/db_command.c:583
#4  0xc01c7320 in db_trap (type=1, code=0) at /usr/src/sys/ddb/db_trap.c:101
#5  0xc0478855 in kdb_trap (type=1, code=0, regs=0xcc883e3c) at /usr/src/sys/arch/i386/i386/db_interface.c:229
#6  0xc047efe2 in trap (frame=0xcc883e3c) at /usr/src/sys/arch/i386/i386/trap.c:350
#7  0xc010cb80 in calltrap ()
#8  0xc047717c in breakpoint ()
#9  0xc02e3676 in wskbd_translate (id=0xc0833ae0, type=2, value=&lt;value optimized out&gt;) at /usr/src/sys/dev/wscons/wskbd.c:1586
#10 0xc02e386e in wskbd_input (dev=0xcc888800, type=2, value=1) at /usr/src/sys/dev/wscons/wskbd.c:682
#11 0xc054c27a in pckbd_input (vsc=0xcc0cc6a8, data=1) at /usr/src/sys/dev/pckbport/pckbd.c:584
#12 0xc02ba80d in pckbcintr (vsc=0xcc0d6ebc) at /usr/src/sys/dev/ic/pckbc.c:607
#13 0xc0465798 in intr_biglock_wrapper (vp=0xc2e853c0) at /usr/src/sys/arch/x86/x86/intr.c:617
#14 0xc01036d9 in Xintr_ioapic_edge3 ()
#15 0xc0477234 in x86_mwait ()
Previous frame inner to this frame (corrupt stack?)
922             /* Disable interrupts. */
923             splhigh();
924     
925             /* Do a dump if requested. */
926             if ((howto &amp; (RB_DUMP | RB_HALT)) == RB_DUMP)
927                     dumpsys();
928     
929     haltsys:
930             doshutdownhooks();
931     
eax            0x0      0
ecx            0x0      0
edx            0x0      0
ebx            0x100    256
esp            0xcc883bb8       0xcc883bb8
ebp            0xcc883bc0       0xcc883bc0
esi            0xc07dfe3c       -1065484740
edi            0x0      0
eip            0xc047c9f8       0xc047c9f8 &lt;cpu_reboot+368&gt;
eflags         0x0      [ ]
cs             0x0      0
ss             0x0      0
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0
st0            0        (raw 0x00000000000000000000)
st1            0        (raw 0x00000000000000000000)
st2            0        (raw 0x00000000000000000000)
st3            0        (raw 0x00000000000000000000)
st4            0        (raw 0x00000000000000000000)
st5            0        (raw 0x00000000000000000000)
st6            0        (raw 0x00000000000000000000)
st7            0        (raw 0x00000000000000000000)
fctrl          0x0      0
fstat          0x0      0
ftag           0x0      0
fiseg          0x0      0
fioff          0x0      0
foseg          0x0      0
fooff          0x0      0
fop            0x0      0
xmm0           
xmm1           
xmm2           
xmm3           
xmm4           
xmm5           
xmm6           
xmm7           
mm0            
mm1            
mm2            
mm3            
mm4            
mm5            
mm6            
mm7            
</pre>

NOTE: Aleksej Saushev <asau@inbox.ru> and Brian Seklecki <seklecki@noc.cfi.pgh.pa.us> will work on this content concurrently.