File:  [NetBSD Developer Wiki] / wikisrc / tutorials / panic.mdwn
Revision 1.7: download - view: text, annotated - select for diffs
Sat May 4 16:11:34 2019 UTC (18 months, 3 weeks ago) by sevan
Branches: MAIN
CVS tags: HEAD
s/category/section when it comes to manuals

    1: # Kernel Panic Procedures
    2: 
    3: This article is a work in progress or otherwise under review
    4: and does not represent current policy.
    5: 
    6: ## Contents
    7: 
    8: 1. Synopsis
    9: 1. Preliminary Notes
   10: 1. Obtaining a Kernel Dump
   11: 1. Finding which line caused the crash
   12: 1. Backtrace through trap() in GDB
   13: 1. Example Crash: Force Panic from WSCons via KVM: Dell DRAC4
   14: 1. What Now
   15: 1. Processing the core dump
   16: 
   17: ## Synopsis
   18: 
   19: Although a few official NetBSD.org documents exist on the topics of using
   20: the advanced kernel debugging using KGDB (Kernelized GNU Debugger (GDB)),
   21: there are few documents which formalize a <b>"Kernel Panic/Crash Reporting Procedure"</b>
   22: using a combination of DDB (the minimalist in-kernel debugger) in
   23: combination with GDB after the crash.
   24: 
   25: <http://www.netbsd.org/docs/kernel/#ddb>
   26: 
   27: 
   28: ## Preliminary Notes
   29: 
   30: If the problem is easily re-created, try to obtain a kernel backtrace
   31: 
   32: The DDB is the minimalist kernel Debugger added by `options DDB` to the kernel
   33: 
   34: Obtain a backtrace at the `db{0}>` prompt using the `bt` command
   35: 
   36: Search the [Mailing List Archives](http://mail-index.netbsd.org) and [Query the NetBSD.org PR database](http://www.netbsd.org/support/query-pr.html) for reports of similar issues.
   37: 
   38: Post the problem for the discussion on the appropriate mailing list.
   39: 
   40: 
   41: ## Obtaining a Kernel Dump
   42: 
   43: A kernel dump is possible to obtain from many kernel panics.
   44: When at the DB prompt, simply execute:
   45: 
   46: 	db{0}> sync
   47: 
   48: The dump of memory will be written to the swap partition.
   49: 
   50: At boot time the swap file coredump will be saved to `/var/crash`.
   51: The default settings to control this behaviour is in `/etc/defaults/rc.conf`
   52: and can be overriden in `/etc/rc.conf`
   53: 
   54: 	savecore=yes
   55: 	savecore_flags="-N /netbsd -z"
   56: 	savecore_dir="/var/crash"
   57: 
   58: A [[!template id=man name="gzip" section="1"]] compressed file will be available for analysis with [[!template id=man name="gdb" section="1"]] or [[!template id=man name="crash" section="8"]]. To load the core dump into `gdb`, after uncompressing it with [[!template id=man name="gunzip" section="1"]], use `target kvm /path/to/netbsd.core`.
   59: 
   60: Your swap partition must be at least the size of your physical RAM
   61: 
   62: Your `/var/crash` partition must have sufficient space to hold the same file.
   63: 
   64: 
   65: ## Finding which line caused the crash
   66: 
   67: With a back trace, it's possible to translate the an address to a line in source code.
   68: 
   69: 	Stopped in pid 496.1 (gdb) at netbsd:breakpoint+0x5: leave
   70: 
   71: To find the address of breakpoint function in the running kernel, use [[!template id=man name="nm" section="1"]].
   72: 
   73: 	nm /netbsd | grep breakpoint
   74: 	ffffffff8021df70 T breakpoint
   75: 	ffffffff8079d944 T db_breakpoint_cmd
   76: 	ffffffff81644b38 d db_breakpoint_list
   77: 	ffffffff81644b30 d db_breakpoints_inserted
   78: 	ffffffff8079d892 T db_clear_breakpoints
   79: 	ffffffff8079d7d0 t db_find_breakpoint
   80: 	ffffffff8079d824 T db_find_breakpoint_here
   81: 	ffffffff81644b40 d db_free_breakpoints
   82: 	ffffffff81644b48 d db_next_free_breakpoint
   83: 	ffffffff8079d835 T db_set_breakpoints
   84: 
   85: Then add `0x5` to the address (`0x5` is obtained from the panic message above, not a fixed value for all) and use [[!template id=man name="addr2line" section="1"]]
   86: 
   87: 	addr2line -e /netbsd ffffffff8021df75
   88: 
   89: ## Backtrace through trap() in GDB
   90: 
   91: In [[!template id=man name="gdb" section="1"]] import the stack script and run the `stack` command.
   92: 
   93: 	(gdb) source /usr/src/sys/arch/i386/gdbscripts/stack
   94: 
   95: See [port-i386/10313](http://gnats.netbsd.org/10313) for more info.
   96: 
   97: ## Example Crash: Force Panic from WSCons via KVM: Dell DRAC4
   98: 
   99: You can invoke the kernel debugger from the console on amd64/i386 using the special key sequence: Control+Alt+Esc. See the "Entering the debugger" section of [[!template id=man name="ddb" section="9"]] for the key sequence on other platforms.
  100: 
  101: Once in the debugger, you can instruct the KDB to run a preliminary backtrace to get a general idea of what went wrong using the `bt` command.
  102: 
  103: You can then force a sync of the file system and and dump of the kernel memory into the swap partition using the `sync` command.
  104: 
  105: On the subsequent boot, the `/etc/rc.d/savecore` script will perform the necessary tasks to archive and [[!template id=man name="gzip" section="1"]] the dump.
  106: 
  107: You can then load the core dump into [[!template id=man name="gdb" section="1"]] or [[!template id=man name="crash" section="8"]]
  108: 
  109: 
  110: ## What Now
  111: 
  112: You can submit the feedback as [a PR to the NetBSD GNATS system](http://www.netbsd.org/support/send-pr.html).
  113: 
  114: ## Processing the core dump
  115: 
  116: Hubert Feyrer has a [great guide to analyzing kernel panic core dumps](http://www.feyrer.de/NetBSD/bx/blosxom.cgi/index.front?-tags=gdb)
  117: 
  118: Additionally, the following command below can be used to create a relatively useful backtrace:
  119: 
  120: <pre>
  121: localhost# cd /var/crash
  122: localhost# gunzip -d *gz
  123: localhost# gdb  --symbols=/netbsd.gdb --quiet --eval-command="file /netbsd.gdb" \ 
  124:                 --eval-command="target kvm netbsd.1.core" --eval-command "bt" \ 
  125:                 --eval-command "list" --eval-command "info all-registers" 2>&1
  126: Load new symbol table from "/netbsd.gdb"? (y or n) y
  127: Reading symbols from /netbsd.gdb...done.
  128: #0  0xc047c9f8 in cpu_reboot (howto=256, bootstr=0x0) at /usr/src/sys/arch/i386/i386/machdep.c:927
  129: 927                     dumpsys();
  130: #0  0xc047c9f8 in cpu_reboot (howto=256, bootstr=0x0) at /usr/src/sys/arch/i386/i386/machdep.c:927
  131: #1  0xc01c3f2a in db_sync_cmd (addr=-1065223264, have_addr=false, count=-1071881791, modif=0xcc883c04 "[BINARY]") at /usr/src/sys/ddb/db_command.c:1304
  132: #2  0xc01c45fa in db_command (last_cmdp=0xc07dfe3c) at /usr/src/sys/ddb/db_command.c:926
  133: #3  0xc01c4856 in db_command_loop () at /usr/src/sys/ddb/db_command.c:583
  134: #4  0xc01c7320 in db_trap (type=1, code=0) at /usr/src/sys/ddb/db_trap.c:101
  135: #5  0xc0478855 in kdb_trap (type=1, code=0, regs=0xcc883e3c) at /usr/src/sys/arch/i386/i386/db_interface.c:229
  136: #6  0xc047efe2 in trap (frame=0xcc883e3c) at /usr/src/sys/arch/i386/i386/trap.c:350
  137: #7  0xc010cb80 in calltrap ()
  138: #8  0xc047717c in breakpoint ()
  139: #9  0xc02e3676 in wskbd_translate (id=0xc0833ae0, type=2, value=&lt;value optimized out&gt;) at /usr/src/sys/dev/wscons/wskbd.c:1586
  140: #10 0xc02e386e in wskbd_input (dev=0xcc888800, type=2, value=1) at /usr/src/sys/dev/wscons/wskbd.c:682
  141: #11 0xc054c27a in pckbd_input (vsc=0xcc0cc6a8, data=1) at /usr/src/sys/dev/pckbport/pckbd.c:584
  142: #12 0xc02ba80d in pckbcintr (vsc=0xcc0d6ebc) at /usr/src/sys/dev/ic/pckbc.c:607
  143: #13 0xc0465798 in intr_biglock_wrapper (vp=0xc2e853c0) at /usr/src/sys/arch/x86/x86/intr.c:617
  144: #14 0xc01036d9 in Xintr_ioapic_edge3 ()
  145: #15 0xc0477234 in x86_mwait ()
  146: Previous frame inner to this frame (corrupt stack?)
  147: 922             /* Disable interrupts. */
  148: 923             splhigh();
  149: 924     
  150: 925             /* Do a dump if requested. */
  151: 926             if ((howto &amp; (RB_DUMP | RB_HALT)) == RB_DUMP)
  152: 927                     dumpsys();
  153: 928     
  154: 929     haltsys:
  155: 930             doshutdownhooks();
  156: 931     
  157: eax            0x0      0
  158: ecx            0x0      0
  159: edx            0x0      0
  160: ebx            0x100    256
  161: esp            0xcc883bb8       0xcc883bb8
  162: ebp            0xcc883bc0       0xcc883bc0
  163: esi            0xc07dfe3c       -1065484740
  164: edi            0x0      0
  165: eip            0xc047c9f8       0xc047c9f8 &lt;cpu_reboot+368&gt;
  166: eflags         0x0      [ ]
  167: cs             0x0      0
  168: ss             0x0      0
  169: ds             0x0      0
  170: es             0x0      0
  171: fs             0x0      0
  172: gs             0x0      0
  173: st0            0        (raw 0x00000000000000000000)
  174: st1            0        (raw 0x00000000000000000000)
  175: st2            0        (raw 0x00000000000000000000)
  176: st3            0        (raw 0x00000000000000000000)
  177: st4            0        (raw 0x00000000000000000000)
  178: st5            0        (raw 0x00000000000000000000)
  179: st6            0        (raw 0x00000000000000000000)
  180: st7            0        (raw 0x00000000000000000000)
  181: fctrl          0x0      0
  182: fstat          0x0      0
  183: ftag           0x0      0
  184: fiseg          0x0      0
  185: fioff          0x0      0
  186: foseg          0x0      0
  187: fooff          0x0      0
  188: fop            0x0      0
  189: xmm0           
  190: xmm1           
  191: xmm2           
  192: xmm3           
  193: xmm4           
  194: xmm5           
  195: xmm6           
  196: xmm7           
  197: mm0            
  198: mm1            
  199: mm2            
  200: mm3            
  201: mm4            
  202: mm5            
  203: mm6            
  204: mm7            
  205: </pre>

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb