Annotation of wikisrc/panic.mdwn, revision 1.5

1.1       asau        1: # Kernel Panic Procedures
                      2: 
                      3: This article is a work in progress or otherwise under review
                      4: and does not represent current policy.
                      5: 
                      6: ## Contents
                      7: 
                      8: 1. Synopsis
1.2       asau        9: 1. Preliminary Notes
                     10: 1. Obtaining a Kernel Dump
                     11: 1. Example Crash: Force Panic from WSCons via KVM: Dell DRAC4
                     12: 1. What Now
                     13: 1. Processing the CORE Dump
1.1       asau       14: 
                     15: ## Synopsis
                     16: 
                     17: Although a few official NetBSD.org documents exist on the topics of using
                     18: the advanced kernel debugging using KGDB (Kernelized GNU Debugger (GDB)),
                     19: there are few documents which formalize a <b>"Kernel Panic/Crash Reporting Procedure"</b>
                     20: using a combination of DDB (the minimalist in-kernel debugger) in
                     21: combination with GDB after the crash.
                     22: 
1.2       asau       23: <http://www.netbsd.org/docs/kernel/#ddb>
1.1       asau       24: 
                     25: 
                     26: ## Preliminary Notes
                     27: 
                     28: If the problem is easily re-created, try to obtain a kernel backtrace
                     29: 
                     30: The DDB is the minimalist kernel Debugger added by "options DDB" to the kernel
                     31: 
                     32: Obtain a backtrace at the "db{0}>" prompt using the "bt" command
                     33: 
                     34: Search the Mailing List Archives and Query the NetBSD.org PR database at
1.2       asau       35: <http://www.netbsd.org/support/query-pr.html>
1.1       asau       36: 
1.2       asau       37: File a PR at <http://www.netbsd.org/support/send-pr.html>
1.1       asau       38: 
                     39: Post the problem for the discussion on the appropriate mailing list.
                     40: 
                     41: 
                     42: ## Obtaining a Kernel Dump
                     43: 
                     44: A kernel dump is possible to obtain from many kernel panics.
                     45: When at the DB prompt, simply execute:
                     46: 
                     47:        db{0}> sync
                     48: 
                     49: The dump of memory will be written to the swap partition.
                     50: 
1.5     ! asau       51: At boot time the swap file coredump will be saved to "/var/crash".
        !            52: The following settings control this behaviour
        !            53: (they are set by default in /etc/defaults/rc.conf),
        !            54: you don't usually need to change those:
1.1       asau       55: 
                     56:        savecore=yes
                     57:        savecore_flags="-N /netbsd -z"
                     58:        savecore_dir="/var/crash"
                     59: 
                     60: A gzip(1)-compressed file will be available for analysis with gdb(1).
                     61: 
                     62: Your swap partition must be at least the size of your physical RAM
                     63: 
                     64: Your "/var/crash" partition must have sufficient space to hold the same file.
                     65: 
                     66: 
                     67: ## Example Crash: Force Panic from WSCons via KVM: Dell DRAC4
                     68: 
                     69: You can force a kernel panic from the console on amd64/i386 using the special key sequence: Control+Alt+Esc.
                     70: 
                     71: * Forcing the Kernel Panic from Console
                     72: 
                     73: TODO: <img alt="Forcing the Kernel Panic from Console" src="/images/thumb/6/6f/cf_NetBSD_KernPanic_Force.png/300px-cf_NetBSD_KernPanic_Force.png" width="300" height="249" border="0" class="thumbimage" />
                     74: 
                     75: 
                     76: You can then instruct the KDB to run a preliminary backtrace to get a general idea of what went wrong:
                     77: 
                     78: TODO: <img alt="Initial backtrace" src="/images/thumb/6/62/cf_NetBSD_KernPanic_InitialBacktrace.png/300px-cf_NetBSD_KernPanic_InitialBacktrace.png" width="300" height="249" border="0" class="thumbimage" />
                     79: 
                     80: You can then force a sync of the file system and and dump of the kernel memory into the swap partition:
                     81: 
                     82: TODO: <img alt="Forcing the RAM dump to Swap" src="/images/thumb/6/68/cf_NetBSD_KernPanic_ForceDump.png/300px-cf_NetBSD_KernPanic_ForceDump.png" width="300" height="225" border="0" class="thumbimage" />
                     83: 
                     84: At reboot, you will see the "/etc/rc.d/savecore" script archive and gzip the dump:
                     85: 
                     86: TODO: <img alt="Save Core Running" src="/images/thumb/9/94/cf_NetBSD_KernPanic_SaveCore.png/300px-cf_NetBSD_KernPanic_SaveCore.png" width="300" height="249" border="0" class="thumbimage" />
                     87: 
                     88: You can then load the core dump into standard <b>gdb(8)</b>
                     89: 
                     90: 
                     91: ## What Now
                     92: 
                     93: You can submit the feedback as a PR to the NetBSD GNATS system.
                     94: 
                     95: If you wish to continue to debug, you may find this article useful:
1.2       asau       96: (TODO!) <a href="/NetBSD_kernel_developer_cheat-sheet" title="NetBSD kernel developer cheat-sheet">NetBSD kernel developer cheat-sheet</a>
1.1       asau       97: 
                     98: 
                     99: ## Processing the CORE Dump
                    100: 
1.4       asau      101: Hubert Feyrer has a great guide to analyzing kernel panic core dumps at <http://www.feyrer.de/NetBSD/bx/blosxom.cgi/index.front?-tags=gdb>
1.1       asau      102: 
                    103: Additionally, the following command below can be used to create a relatively useful backtrace:
                    104: 
                    105: <pre>
1.3       asau      106: localhost# cd /var/crash
                    107: localhost# gunzip -d *gz
                    108: localhost# gdb  --symbols=/netbsd.gdb --quiet --eval-command="file /netbsd.gdb" \ 
                    109:                 --eval-command="target kvm netbsd.1.core" --eval-command "bt" \ 
                    110:                 --eval-command "list" --eval-command "info all-registers" 2>&1
1.1       asau      111: Load new symbol table from "/netbsd.gdb"? (y or n) y
                    112: Reading symbols from /netbsd.gdb...done.
                    113: #0  0xc047c9f8 in cpu_reboot (howto=256, bootstr=0x0) at /usr/src/sys/arch/i386/i386/machdep.c:927
                    114: 927                     dumpsys();
                    115: #0  0xc047c9f8 in cpu_reboot (howto=256, bootstr=0x0) at /usr/src/sys/arch/i386/i386/machdep.c:927
                    116: #1  0xc01c3f2a in db_sync_cmd (addr=-1065223264, have_addr=false, count=-1071881791, modif=0xcc883c04 "[BINARY]") at /usr/src/sys/ddb/db_command.c:1304
                    117: #2  0xc01c45fa in db_command (last_cmdp=0xc07dfe3c) at /usr/src/sys/ddb/db_command.c:926
                    118: #3  0xc01c4856 in db_command_loop () at /usr/src/sys/ddb/db_command.c:583
                    119: #4  0xc01c7320 in db_trap (type=1, code=0) at /usr/src/sys/ddb/db_trap.c:101
                    120: #5  0xc0478855 in kdb_trap (type=1, code=0, regs=0xcc883e3c) at /usr/src/sys/arch/i386/i386/db_interface.c:229
                    121: #6  0xc047efe2 in trap (frame=0xcc883e3c) at /usr/src/sys/arch/i386/i386/trap.c:350
                    122: #7  0xc010cb80 in calltrap ()
                    123: #8  0xc047717c in breakpoint ()
                    124: #9  0xc02e3676 in wskbd_translate (id=0xc0833ae0, type=2, value=&lt;value optimized out&gt;) at /usr/src/sys/dev/wscons/wskbd.c:1586
                    125: #10 0xc02e386e in wskbd_input (dev=0xcc888800, type=2, value=1) at /usr/src/sys/dev/wscons/wskbd.c:682
                    126: #11 0xc054c27a in pckbd_input (vsc=0xcc0cc6a8, data=1) at /usr/src/sys/dev/pckbport/pckbd.c:584
                    127: #12 0xc02ba80d in pckbcintr (vsc=0xcc0d6ebc) at /usr/src/sys/dev/ic/pckbc.c:607
                    128: #13 0xc0465798 in intr_biglock_wrapper (vp=0xc2e853c0) at /usr/src/sys/arch/x86/x86/intr.c:617
                    129: #14 0xc01036d9 in Xintr_ioapic_edge3 ()
                    130: #15 0xc0477234 in x86_mwait ()
                    131: Previous frame inner to this frame (corrupt stack?)
                    132: 922             /* Disable interrupts. */
                    133: 923             splhigh();
                    134: 924     
                    135: 925             /* Do a dump if requested. */
                    136: 926             if ((howto &amp; (RB_DUMP | RB_HALT)) == RB_DUMP)
                    137: 927                     dumpsys();
                    138: 928     
                    139: 929     haltsys:
                    140: 930             doshutdownhooks();
                    141: 931     
                    142: eax            0x0      0
                    143: ecx            0x0      0
                    144: edx            0x0      0
                    145: ebx            0x100    256
                    146: esp            0xcc883bb8       0xcc883bb8
                    147: ebp            0xcc883bc0       0xcc883bc0
                    148: esi            0xc07dfe3c       -1065484740
                    149: edi            0x0      0
                    150: eip            0xc047c9f8       0xc047c9f8 &lt;cpu_reboot+368&gt;
                    151: eflags         0x0      [ ]
                    152: cs             0x0      0
                    153: ss             0x0      0
                    154: ds             0x0      0
                    155: es             0x0      0
                    156: fs             0x0      0
                    157: gs             0x0      0
                    158: st0            0        (raw 0x00000000000000000000)
                    159: st1            0        (raw 0x00000000000000000000)
                    160: st2            0        (raw 0x00000000000000000000)
                    161: st3            0        (raw 0x00000000000000000000)
                    162: st4            0        (raw 0x00000000000000000000)
                    163: st5            0        (raw 0x00000000000000000000)
                    164: st6            0        (raw 0x00000000000000000000)
                    165: st7            0        (raw 0x00000000000000000000)
                    166: fctrl          0x0      0
                    167: fstat          0x0      0
                    168: ftag           0x0      0
                    169: fiseg          0x0      0
                    170: fioff          0x0      0
                    171: foseg          0x0      0
                    172: fooff          0x0      0
                    173: fop            0x0      0
                    174: xmm0           
                    175: xmm1           
                    176: xmm2           
                    177: xmm3           
                    178: xmm4           
                    179: xmm5           
                    180: xmm6           
                    181: xmm7           
                    182: mm0            
                    183: mm1            
                    184: mm2            
                    185: mm3            
                    186: mm4            
                    187: mm5            
                    188: mm6            
                    189: mm7            
                    190: </pre>
                    191: 
                    192: NOTE: Aleksej Saushev <asau@inbox.ru> and Brian Seklecki <seklecki@noc.cfi.pgh.pa.us> will work on this content concurrently.

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb