File:  [NetBSD Developer Wiki] / wikisrc / Attic / panic.mdwn
Revision 1.4: download - view: text, annotated - select for diffs
Sat Mar 3 14:56:48 2012 UTC (10 years, 7 months ago) by asau
Branches: MAIN
CVS tags: HEAD
Fix spelling.
From Moritz Wilhelmy via #netbsd

    1: # Kernel Panic Procedures
    2: 
    3: This article is a work in progress or otherwise under review
    4: and does not represent current policy.
    5: 
    6: ## Contents
    7: 
    8: 1. Synopsis
    9: 1. Preliminary Notes
   10: 1. Obtaining a Kernel Dump
   11: 1. Example Crash: Force Panic from WSCons via KVM: Dell DRAC4
   12: 1. What Now
   13: 1. Processing the CORE Dump
   14: 
   15: ## Synopsis
   16: 
   17: Although a few official NetBSD.org documents exist on the topics of using
   18: the advanced kernel debugging using KGDB (Kernelized GNU Debugger (GDB)),
   19: there are few documents which formalize a <b>"Kernel Panic/Crash Reporting Procedure"</b>
   20: using a combination of DDB (the minimalist in-kernel debugger) in
   21: combination with GDB after the crash.
   22: 
   23: <http://www.netbsd.org/docs/kernel/#ddb>
   24: 
   25: 
   26: ## Preliminary Notes
   27: 
   28: If the problem is easily re-created, try to obtain a kernel backtrace
   29: 
   30: The DDB is the minimalist kernel Debugger added by "options DDB" to the kernel
   31: 
   32: Obtain a backtrace at the "db{0}>" prompt using the "bt" command
   33: 
   34: Search the Mailing List Archives and Query the NetBSD.org PR database at
   35: <http://www.netbsd.org/support/query-pr.html>
   36: 
   37: File a PR at <http://www.netbsd.org/support/send-pr.html>
   38: 
   39: Post the problem for the discussion on the appropriate mailing list.
   40: 
   41: 
   42: ## Obtaining a Kernel Dump
   43: 
   44: A kernel dump is possible to obtain from many kernel panics.
   45: When at the DB prompt, simply execute:
   46: 
   47: 	db{0}> sync
   48: 
   49: The dump of memory will be written to the swap partition.
   50: 
   51: At boot time, if rc.conf(5) has value "savecore=yes",
   52: the swap file coredump will be saved to "/var/crash".
   53: 
   54: 	savecore=yes
   55: 	savecore_flags="-N /netbsd -z"
   56: 	savecore_dir="/var/crash"
   57: 
   58: A gzip(1)-compressed file will be available for analysis with gdb(1).
   59: 
   60: Your swap partition must be at least the size of your physical RAM
   61: 
   62: Your "/var/crash" partition must have sufficient space to hold the same file.
   63: 
   64: 
   65: ## Example Crash: Force Panic from WSCons via KVM: Dell DRAC4
   66: 
   67: You can force a kernel panic from the console on amd64/i386 using the special key sequence: Control+Alt+Esc.
   68: 
   69: * Forcing the Kernel Panic from Console
   70: 
   71: TODO: <img alt="Forcing the Kernel Panic from Console" src="/images/thumb/6/6f/cf_NetBSD_KernPanic_Force.png/300px-cf_NetBSD_KernPanic_Force.png" width="300" height="249" border="0" class="thumbimage" />
   72: 
   73: 
   74: You can then instruct the KDB to run a preliminary backtrace to get a general idea of what went wrong:
   75: 
   76: TODO: <img alt="Initial backtrace" src="/images/thumb/6/62/cf_NetBSD_KernPanic_InitialBacktrace.png/300px-cf_NetBSD_KernPanic_InitialBacktrace.png" width="300" height="249" border="0" class="thumbimage" />
   77: 
   78: You can then force a sync of the file system and and dump of the kernel memory into the swap partition:
   79: 
   80: TODO: <img alt="Forcing the RAM dump to Swap" src="/images/thumb/6/68/cf_NetBSD_KernPanic_ForceDump.png/300px-cf_NetBSD_KernPanic_ForceDump.png" width="300" height="225" border="0" class="thumbimage" />
   81: 
   82: At reboot, you will see the "/etc/rc.d/savecore" script archive and gzip the dump:
   83: 
   84: TODO: <img alt="Save Core Running" src="/images/thumb/9/94/cf_NetBSD_KernPanic_SaveCore.png/300px-cf_NetBSD_KernPanic_SaveCore.png" width="300" height="249" border="0" class="thumbimage" />
   85: 
   86: You can then load the core dump into standard <b>gdb(8)</b>
   87: 
   88: 
   89: ## What Now
   90: 
   91: You can submit the feedback as a PR to the NetBSD GNATS system.
   92: 
   93: If you wish to continue to debug, you may find this article useful:
   94: (TODO!) <a href="/NetBSD_kernel_developer_cheat-sheet" title="NetBSD kernel developer cheat-sheet">NetBSD kernel developer cheat-sheet</a>
   95: 
   96: 
   97: ## Processing the CORE Dump
   98: 
   99: Hubert Feyrer has a great guide to analyzing kernel panic core dumps at <http://www.feyrer.de/NetBSD/bx/blosxom.cgi/index.front?-tags=gdb>
  100: 
  101: Additionally, the following command below can be used to create a relatively useful backtrace:
  102: 
  103: <pre>
  104: localhost# cd /var/crash
  105: localhost# gunzip -d *gz
  106: localhost# gdb  --symbols=/netbsd.gdb --quiet --eval-command="file /netbsd.gdb" \ 
  107:                 --eval-command="target kvm netbsd.1.core" --eval-command "bt" \ 
  108:                 --eval-command "list" --eval-command "info all-registers" 2>&1
  109: Load new symbol table from "/netbsd.gdb"? (y or n) y
  110: Reading symbols from /netbsd.gdb...done.
  111: #0  0xc047c9f8 in cpu_reboot (howto=256, bootstr=0x0) at /usr/src/sys/arch/i386/i386/machdep.c:927
  112: 927                     dumpsys();
  113: #0  0xc047c9f8 in cpu_reboot (howto=256, bootstr=0x0) at /usr/src/sys/arch/i386/i386/machdep.c:927
  114: #1  0xc01c3f2a in db_sync_cmd (addr=-1065223264, have_addr=false, count=-1071881791, modif=0xcc883c04 "[BINARY]") at /usr/src/sys/ddb/db_command.c:1304
  115: #2  0xc01c45fa in db_command (last_cmdp=0xc07dfe3c) at /usr/src/sys/ddb/db_command.c:926
  116: #3  0xc01c4856 in db_command_loop () at /usr/src/sys/ddb/db_command.c:583
  117: #4  0xc01c7320 in db_trap (type=1, code=0) at /usr/src/sys/ddb/db_trap.c:101
  118: #5  0xc0478855 in kdb_trap (type=1, code=0, regs=0xcc883e3c) at /usr/src/sys/arch/i386/i386/db_interface.c:229
  119: #6  0xc047efe2 in trap (frame=0xcc883e3c) at /usr/src/sys/arch/i386/i386/trap.c:350
  120: #7  0xc010cb80 in calltrap ()
  121: #8  0xc047717c in breakpoint ()
  122: #9  0xc02e3676 in wskbd_translate (id=0xc0833ae0, type=2, value=&lt;value optimized out&gt;) at /usr/src/sys/dev/wscons/wskbd.c:1586
  123: #10 0xc02e386e in wskbd_input (dev=0xcc888800, type=2, value=1) at /usr/src/sys/dev/wscons/wskbd.c:682
  124: #11 0xc054c27a in pckbd_input (vsc=0xcc0cc6a8, data=1) at /usr/src/sys/dev/pckbport/pckbd.c:584
  125: #12 0xc02ba80d in pckbcintr (vsc=0xcc0d6ebc) at /usr/src/sys/dev/ic/pckbc.c:607
  126: #13 0xc0465798 in intr_biglock_wrapper (vp=0xc2e853c0) at /usr/src/sys/arch/x86/x86/intr.c:617
  127: #14 0xc01036d9 in Xintr_ioapic_edge3 ()
  128: #15 0xc0477234 in x86_mwait ()
  129: Previous frame inner to this frame (corrupt stack?)
  130: 922             /* Disable interrupts. */
  131: 923             splhigh();
  132: 924     
  133: 925             /* Do a dump if requested. */
  134: 926             if ((howto &amp; (RB_DUMP | RB_HALT)) == RB_DUMP)
  135: 927                     dumpsys();
  136: 928     
  137: 929     haltsys:
  138: 930             doshutdownhooks();
  139: 931     
  140: eax            0x0      0
  141: ecx            0x0      0
  142: edx            0x0      0
  143: ebx            0x100    256
  144: esp            0xcc883bb8       0xcc883bb8
  145: ebp            0xcc883bc0       0xcc883bc0
  146: esi            0xc07dfe3c       -1065484740
  147: edi            0x0      0
  148: eip            0xc047c9f8       0xc047c9f8 &lt;cpu_reboot+368&gt;
  149: eflags         0x0      [ ]
  150: cs             0x0      0
  151: ss             0x0      0
  152: ds             0x0      0
  153: es             0x0      0
  154: fs             0x0      0
  155: gs             0x0      0
  156: st0            0        (raw 0x00000000000000000000)
  157: st1            0        (raw 0x00000000000000000000)
  158: st2            0        (raw 0x00000000000000000000)
  159: st3            0        (raw 0x00000000000000000000)
  160: st4            0        (raw 0x00000000000000000000)
  161: st5            0        (raw 0x00000000000000000000)
  162: st6            0        (raw 0x00000000000000000000)
  163: st7            0        (raw 0x00000000000000000000)
  164: fctrl          0x0      0
  165: fstat          0x0      0
  166: ftag           0x0      0
  167: fiseg          0x0      0
  168: fioff          0x0      0
  169: foseg          0x0      0
  170: fooff          0x0      0
  171: fop            0x0      0
  172: xmm0           
  173: xmm1           
  174: xmm2           
  175: xmm3           
  176: xmm4           
  177: xmm5           
  178: xmm6           
  179: xmm7           
  180: mm0            
  181: mm1            
  182: mm2            
  183: mm3            
  184: mm4            
  185: mm5            
  186: mm6            
  187: mm7            
  188: </pre>
  189: 
  190: NOTE: Aleksej Saushev <asau@inbox.ru> and Brian Seklecki <seklecki@noc.cfi.pgh.pa.us> will work on this content concurrently.

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb