wikisrc/panic.mdwn - view

File: [NetBSD Developer Wiki] / wikisrc / Attic / panic.mdwn
Revision 1.3: download - view: text, annotated - select for diffs
Thu Dec 2 21:47:25 2010 UTC (13 years ago) by asau
Branches: MAIN
CVS tags: HEAD

Don't advertise non-native shell.

# Kernel Panic Procedures This article is a work in progress or otherwise under review and does not represent current policy. ## Contents 1. Synopsis 1. Preliminary Notes 1. Obtaining a Kernel Dump 1. Example Crash: Force Panic from WSCons via KVM: Dell DRAC4 1. What Now 1. Processing the CORE Dump ## Synopsis Although a few official NetBSD.org documents exist on the topics of using the advanced kernel debugging using KGDB (Kernelized GNU Debugger (GDB)), there are few documents which formalize a <b>"Kernel Panic/Crash Reporting Procedure"</b> using a combination of DDB (the minimalist in-kernel debugger) in combination with GDB after the crash. <http://www.netbsd.org/docs/kernel/#ddb> ## Preliminary Notes If the problem is easily re-created, try to obtain a kernel backtrace The DDB is the minimalist kernel Debugger added by "options DDB" to the kernel Obtain a backtrace at the "db{0}>" prompt using the "bt" command Search the Mailing List Archives and Query the NetBSD.org PR database at <http://www.netbsd.org/support/query-pr.html> File a PR at <http://www.netbsd.org/support/send-pr.html> Post the problem for the discussion on the appropriate mailing list. ## Obtaining a Kernel Dump A kernel dump is possible to obtain from many kernel panics. When at the DB prompt, simply execute: db{0}> sync The dump of memory will be written to the swap partition. At boot time, if rc.conf(5) has value "savecore=yes", the swap file coredump will be saved to "/var/crash". savecore=yes savecore_flags="-N /netbsd -z" savecore_dir="/var/crash" A gzip(1)-compressed file will be available for analysis with gdb(1). Your swap partition must be at least the size of your physical RAM Your "/var/crash" partition must have sufficient space to hold the same file. ## Example Crash: Force Panic from WSCons via KVM: Dell DRAC4 You can force a kernel panic from the console on amd64/i386 using the special key sequence: Control+Alt+Esc. * Forcing the Kernel Panic from Console TODO: <img alt="Forcing the Kernel Panic from Console" src="/images/thumb/6/6f/cf_NetBSD_KernPanic_Force.png/300px-cf_NetBSD_KernPanic_Force.png" width="300" height="249" border="0" class="thumbimage" /> You can then instruct the KDB to run a preliminary backtrace to get a general idea of what went wrong: TODO: <img alt="Initial backtrace" src="/images/thumb/6/62/cf_NetBSD_KernPanic_InitialBacktrace.png/300px-cf_NetBSD_KernPanic_InitialBacktrace.png" width="300" height="249" border="0" class="thumbimage" /> You can then force a sync of the file system and and dump of the kernel memory into the swap partition: TODO: <img alt="Forcing the RAM dump to Swap" src="/images/thumb/6/68/cf_NetBSD_KernPanic_ForceDump.png/300px-cf_NetBSD_KernPanic_ForceDump.png" width="300" height="225" border="0" class="thumbimage" /> At reboot, you will see the "/etc/rc.d/savecore" script archive and gzip the dump: TODO: <img alt="Save Core Running" src="/images/thumb/9/94/cf_NetBSD_KernPanic_SaveCore.png/300px-cf_NetBSD_KernPanic_SaveCore.png" width="300" height="249" border="0" class="thumbimage" /> You can then load the core dump into standard <b>gdb(8)</b> ## What Now You can submit the feedback as a PR to the NetBSD GNATS system. If you wish to continue to debug, you may find this article useful: (TODO!) <a href="/NetBSD_kernel_developer_cheat-sheet" title="NetBSD kernel developer cheat-sheet">NetBSD kernel developer cheat-sheet</a> ## Processing the CORE Dump Heubert Feyrer has a great guide to analyzing kernel panic core dumps at <http://www.feyrer.de/NetBSD/bx/blosxom.cgi/index.front?-tags=gdb> Additionally, the following command below can be used to create a relatively useful backtrace: <pre> localhost# cd /var/crash localhost# gunzip -d *gz localhost# gdb --symbols=/netbsd.gdb --quiet --eval-command="file /netbsd.gdb" \ --eval-command="target kvm netbsd.1.core" --eval-command "bt" \ --eval-command "list" --eval-command "info all-registers" 2>&1 Load new symbol table from "/netbsd.gdb"? (y or n) y Reading symbols from /netbsd.gdb...done. #0 0xc047c9f8 in cpu_reboot (howto=256, bootstr=0x0) at /usr/src/sys/arch/i386/i386/machdep.c:927 927 dumpsys(); #0 0xc047c9f8 in cpu_reboot (howto=256, bootstr=0x0) at /usr/src/sys/arch/i386/i386/machdep.c:927 #1 0xc01c3f2a in db_sync_cmd (addr=-1065223264, have_addr=false, count=-1071881791, modif=0xcc883c04 "[BINARY]") at /usr/src/sys/ddb/db_command.c:1304 #2 0xc01c45fa in db_command (last_cmdp=0xc07dfe3c) at /usr/src/sys/ddb/db_command.c:926 #3 0xc01c4856 in db_command_loop () at /usr/src/sys/ddb/db_command.c:583 #4 0xc01c7320 in db_trap (type=1, code=0) at /usr/src/sys/ddb/db_trap.c:101 #5 0xc0478855 in kdb_trap (type=1, code=0, regs=0xcc883e3c) at /usr/src/sys/arch/i386/i386/db_interface.c:229 #6 0xc047efe2 in trap (frame=0xcc883e3c) at /usr/src/sys/arch/i386/i386/trap.c:350 #7 0xc010cb80 in calltrap () #8 0xc047717c in breakpoint () #9 0xc02e3676 in wskbd_translate (id=0xc0833ae0, type=2, value=<value optimized out>) at /usr/src/sys/dev/wscons/wskbd.c:1586 #10 0xc02e386e in wskbd_input (dev=0xcc888800, type=2, value=1) at /usr/src/sys/dev/wscons/wskbd.c:682 #11 0xc054c27a in pckbd_input (vsc=0xcc0cc6a8, data=1) at /usr/src/sys/dev/pckbport/pckbd.c:584 #12 0xc02ba80d in pckbcintr (vsc=0xcc0d6ebc) at /usr/src/sys/dev/ic/pckbc.c:607 #13 0xc0465798 in intr_biglock_wrapper (vp=0xc2e853c0) at /usr/src/sys/arch/x86/x86/intr.c:617 #14 0xc01036d9 in Xintr_ioapic_edge3 () #15 0xc0477234 in x86_mwait () Previous frame inner to this frame (corrupt stack?) 922 /* Disable interrupts. */ 923 splhigh(); 924 925 /* Do a dump if requested. */ 926 if ((howto & (RB_DUMP | RB_HALT)) == RB_DUMP) 927 dumpsys(); 928 929 haltsys: 930 doshutdownhooks(); 931 eax 0x0 0 ecx 0x0 0 edx 0x0 0 ebx 0x100 256 esp 0xcc883bb8 0xcc883bb8 ebp 0xcc883bc0 0xcc883bc0 esi 0xc07dfe3c -1065484740 edi 0x0 0 eip 0xc047c9f8 0xc047c9f8 <cpu_reboot+368> eflags 0x0 [ ] cs 0x0 0 ss 0x0 0 ds 0x0 0 es 0x0 0 fs 0x0 0 gs 0x0 0 st0 0 (raw 0x00000000000000000000) st1 0 (raw 0x00000000000000000000) st2 0 (raw 0x00000000000000000000) st3 0 (raw 0x00000000000000000000) st4 0 (raw 0x00000000000000000000) st5 0 (raw 0x00000000000000000000) st6 0 (raw 0x00000000000000000000) st7 0 (raw 0x00000000000000000000) fctrl 0x0 0 fstat 0x0 0 ftag 0x0 0 fiseg 0x0 0 fioff 0x0 0 foseg 0x0 0 fooff 0x0 0 fop 0x0 0 xmm0 xmm1 xmm2 xmm3 xmm4 xmm5 xmm6 xmm7 mm0 mm1 mm2 mm3 mm4 mm5 mm6 mm7 </pre> NOTE: Aleksej Saushev <asau@inbox.ru> and Brian Seklecki <seklecki@noc.cfi.pgh.pa.us> will work on this content concurrently.