Annotation of wikisrc/panic.mdwn, revision 1.1
1.1 ! asau 1: # Kernel Panic Procedures
! 2:
! 3: This article is a work in progress or otherwise under review
! 4: and does not represent current policy.
! 5:
! 6: ## Contents
! 7:
! 8: 1. Synopsis
! 9: 2. Preliminary Notes
! 10: 2.1. Obtaining a Kernel Dump
! 11: 2.1.1. Example Crash: Force Panic from WSCons via KVM: Dell DRAC4
! 12: 2.1.2. What Now
! 13: 2.2. Processing the CORE Dump
! 14:
! 15: ## Synopsis
! 16:
! 17: Although a few official NetBSD.org documents exist on the topics of using
! 18: the advanced kernel debugging using KGDB (Kernelized GNU Debugger (GDB)),
! 19: there are few documents which formalize a <b>"Kernel Panic/Crash Reporting Procedure"</b>
! 20: using a combination of DDB (the minimalist in-kernel debugger) in
! 21: combination with GDB after the crash.
! 22:
! 23: http://www.netbsd.org/docs/kernel/#ddb
! 24:
! 25:
! 26: ## Preliminary Notes
! 27:
! 28: If the problem is easily re-created, try to obtain a kernel backtrace
! 29:
! 30: The DDB is the minimalist kernel Debugger added by "options DDB" to the kernel
! 31:
! 32: Obtain a backtrace at the "db{0}>" prompt using the "bt" command
! 33:
! 34: Search the Mailing List Archives and Query the NetBSD.org PR database at
! 35: http://www.netbsd.org/support/query-pr.html
! 36:
! 37: File a PR at http://www.netbsd.org/support/send-pr.html
! 38:
! 39: Post the problem for the discussion on the appropriate mailing list.
! 40:
! 41:
! 42: ## Obtaining a Kernel Dump
! 43:
! 44: A kernel dump is possible to obtain from many kernel panics.
! 45: When at the DB prompt, simply execute:
! 46:
! 47: db{0}> sync
! 48:
! 49: The dump of memory will be written to the swap partition.
! 50:
! 51: At boot time, if rc.conf(5) has value "savecore=yes",
! 52: the swap file coredump will be saved to "/var/crash".
! 53:
! 54: savecore=yes
! 55: savecore_flags="-N /netbsd -z"
! 56: savecore_dir="/var/crash"
! 57:
! 58: A gzip(1)-compressed file will be available for analysis with gdb(1).
! 59:
! 60: Your swap partition must be at least the size of your physical RAM
! 61:
! 62: Your "/var/crash" partition must have sufficient space to hold the same file.
! 63:
! 64:
! 65: ## Example Crash: Force Panic from WSCons via KVM: Dell DRAC4
! 66:
! 67: You can force a kernel panic from the console on amd64/i386 using the special key sequence: Control+Alt+Esc.
! 68:
! 69: * Forcing the Kernel Panic from Console
! 70:
! 71: TODO: <img alt="Forcing the Kernel Panic from Console" src="/images/thumb/6/6f/cf_NetBSD_KernPanic_Force.png/300px-cf_NetBSD_KernPanic_Force.png" width="300" height="249" border="0" class="thumbimage" />
! 72:
! 73:
! 74: You can then instruct the KDB to run a preliminary backtrace to get a general idea of what went wrong:
! 75:
! 76: TODO: <img alt="Initial backtrace" src="/images/thumb/6/62/cf_NetBSD_KernPanic_InitialBacktrace.png/300px-cf_NetBSD_KernPanic_InitialBacktrace.png" width="300" height="249" border="0" class="thumbimage" />
! 77:
! 78: You can then force a sync of the file system and and dump of the kernel memory into the swap partition:
! 79:
! 80: TODO: <img alt="Forcing the RAM dump to Swap" src="/images/thumb/6/68/cf_NetBSD_KernPanic_ForceDump.png/300px-cf_NetBSD_KernPanic_ForceDump.png" width="300" height="225" border="0" class="thumbimage" />
! 81:
! 82: At reboot, you will see the "/etc/rc.d/savecore" script archive and gzip the dump:
! 83:
! 84: TODO: <img alt="Save Core Running" src="/images/thumb/9/94/cf_NetBSD_KernPanic_SaveCore.png/300px-cf_NetBSD_KernPanic_SaveCore.png" width="300" height="249" border="0" class="thumbimage" />
! 85:
! 86: You can then load the core dump into standard <b>gdb(8)</b>
! 87:
! 88:
! 89: ## What Now
! 90:
! 91: You can submit the feedback as a PR to the NetBSD GNATS system.
! 92:
! 93: If you wish to continue to debug, you may find this article useful:
! 94: <a href="/NetBSD_kernel_developer_cheat-sheet" title="NetBSD kernel developer cheat-sheet">NetBSD kernel developer cheat-sheet</a>
! 95:
! 96:
! 97: ## Processing the CORE Dump
! 98:
! 99: Heubert Feyrer has a great guide to analyzing kernel panic core dumps at: http://www.feyrer.de/NetBSD/bx/blosxom.cgi/index.front?-tags=gdb
! 100:
! 101: Additionally, the following command below can be used to create a relatively useful backtrace:
! 102:
! 103: <pre>
! 104: bash-4.0# cd /var/crash
! 105: bash-4.0# gunzip -d *gz
! 106: bash-4.0# gdb --symbols=/netbsd.gdb --quiet --eval-command="file /netbsd.gdb" \
! 107: --eval-command="target kvm netbsd.1.core" --eval-command "bt" \
! 108: --eval-command "list" --eval-command "info all-registers" 2>&1
! 109: Load new symbol table from "/netbsd.gdb"? (y or n) y
! 110: Reading symbols from /netbsd.gdb...done.
! 111: #0 0xc047c9f8 in cpu_reboot (howto=256, bootstr=0x0) at /usr/src/sys/arch/i386/i386/machdep.c:927
! 112: 927 dumpsys();
! 113: #0 0xc047c9f8 in cpu_reboot (howto=256, bootstr=0x0) at /usr/src/sys/arch/i386/i386/machdep.c:927
! 114: #1 0xc01c3f2a in db_sync_cmd (addr=-1065223264, have_addr=false, count=-1071881791, modif=0xcc883c04 "[BINARY]") at /usr/src/sys/ddb/db_command.c:1304
! 115: #2 0xc01c45fa in db_command (last_cmdp=0xc07dfe3c) at /usr/src/sys/ddb/db_command.c:926
! 116: #3 0xc01c4856 in db_command_loop () at /usr/src/sys/ddb/db_command.c:583
! 117: #4 0xc01c7320 in db_trap (type=1, code=0) at /usr/src/sys/ddb/db_trap.c:101
! 118: #5 0xc0478855 in kdb_trap (type=1, code=0, regs=0xcc883e3c) at /usr/src/sys/arch/i386/i386/db_interface.c:229
! 119: #6 0xc047efe2 in trap (frame=0xcc883e3c) at /usr/src/sys/arch/i386/i386/trap.c:350
! 120: #7 0xc010cb80 in calltrap ()
! 121: #8 0xc047717c in breakpoint ()
! 122: #9 0xc02e3676 in wskbd_translate (id=0xc0833ae0, type=2, value=<value optimized out>) at /usr/src/sys/dev/wscons/wskbd.c:1586
! 123: #10 0xc02e386e in wskbd_input (dev=0xcc888800, type=2, value=1) at /usr/src/sys/dev/wscons/wskbd.c:682
! 124: #11 0xc054c27a in pckbd_input (vsc=0xcc0cc6a8, data=1) at /usr/src/sys/dev/pckbport/pckbd.c:584
! 125: #12 0xc02ba80d in pckbcintr (vsc=0xcc0d6ebc) at /usr/src/sys/dev/ic/pckbc.c:607
! 126: #13 0xc0465798 in intr_biglock_wrapper (vp=0xc2e853c0) at /usr/src/sys/arch/x86/x86/intr.c:617
! 127: #14 0xc01036d9 in Xintr_ioapic_edge3 ()
! 128: #15 0xc0477234 in x86_mwait ()
! 129: Previous frame inner to this frame (corrupt stack?)
! 130: 922 /* Disable interrupts. */
! 131: 923 splhigh();
! 132: 924
! 133: 925 /* Do a dump if requested. */
! 134: 926 if ((howto & (RB_DUMP | RB_HALT)) == RB_DUMP)
! 135: 927 dumpsys();
! 136: 928
! 137: 929 haltsys:
! 138: 930 doshutdownhooks();
! 139: 931
! 140: eax 0x0 0
! 141: ecx 0x0 0
! 142: edx 0x0 0
! 143: ebx 0x100 256
! 144: esp 0xcc883bb8 0xcc883bb8
! 145: ebp 0xcc883bc0 0xcc883bc0
! 146: esi 0xc07dfe3c -1065484740
! 147: edi 0x0 0
! 148: eip 0xc047c9f8 0xc047c9f8 <cpu_reboot+368>
! 149: eflags 0x0 [ ]
! 150: cs 0x0 0
! 151: ss 0x0 0
! 152: ds 0x0 0
! 153: es 0x0 0
! 154: fs 0x0 0
! 155: gs 0x0 0
! 156: st0 0 (raw 0x00000000000000000000)
! 157: st1 0 (raw 0x00000000000000000000)
! 158: st2 0 (raw 0x00000000000000000000)
! 159: st3 0 (raw 0x00000000000000000000)
! 160: st4 0 (raw 0x00000000000000000000)
! 161: st5 0 (raw 0x00000000000000000000)
! 162: st6 0 (raw 0x00000000000000000000)
! 163: st7 0 (raw 0x00000000000000000000)
! 164: fctrl 0x0 0
! 165: fstat 0x0 0
! 166: ftag 0x0 0
! 167: fiseg 0x0 0
! 168: fioff 0x0 0
! 169: foseg 0x0 0
! 170: fooff 0x0 0
! 171: fop 0x0 0
! 172: xmm0
! 173: xmm1
! 174: xmm2
! 175: xmm3
! 176: xmm4
! 177: xmm5
! 178: xmm6
! 179: xmm7
! 180: mm0
! 181: mm1
! 182: mm2
! 183: mm3
! 184: mm4
! 185: mm5
! 186: mm6
! 187: mm7
! 188: </pre>
! 189:
! 190: NOTE: Aleksej Saushev <asau@inbox.ru> and Brian Seklecki <seklecki@noc.cfi.pgh.pa.us> will work on this content concurrently.
CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb
![[BACK]](/icons/back.gif){kind=icon}
Return to panic.mdwn CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [NetBSD Developer Wiki] / wikisrc