Annotation of wikisrc/panic.mdwn, revision 1.5
1.1 asau 1: # Kernel Panic Procedures
2:
3: This article is a work in progress or otherwise under review
4: and does not represent current policy.
5:
6: ## Contents
7:
8: 1. Synopsis
1.2 asau 9: 1. Preliminary Notes
10: 1. Obtaining a Kernel Dump
11: 1. Example Crash: Force Panic from WSCons via KVM: Dell DRAC4
12: 1. What Now
13: 1. Processing the CORE Dump
1.1 asau 14:
15: ## Synopsis
16:
17: Although a few official NetBSD.org documents exist on the topics of using
18: the advanced kernel debugging using KGDB (Kernelized GNU Debugger (GDB)),
19: there are few documents which formalize a <b>"Kernel Panic/Crash Reporting Procedure"</b>
20: using a combination of DDB (the minimalist in-kernel debugger) in
21: combination with GDB after the crash.
22:
1.2 asau 23: <http://www.netbsd.org/docs/kernel/#ddb>
1.1 asau 24:
25:
26: ## Preliminary Notes
27:
28: If the problem is easily re-created, try to obtain a kernel backtrace
29:
30: The DDB is the minimalist kernel Debugger added by "options DDB" to the kernel
31:
32: Obtain a backtrace at the "db{0}>" prompt using the "bt" command
33:
34: Search the Mailing List Archives and Query the NetBSD.org PR database at
1.2 asau 35: <http://www.netbsd.org/support/query-pr.html>
1.1 asau 36:
1.2 asau 37: File a PR at <http://www.netbsd.org/support/send-pr.html>
1.1 asau 38:
39: Post the problem for the discussion on the appropriate mailing list.
40:
41:
42: ## Obtaining a Kernel Dump
43:
44: A kernel dump is possible to obtain from many kernel panics.
45: When at the DB prompt, simply execute:
46:
47: db{0}> sync
48:
49: The dump of memory will be written to the swap partition.
50:
1.5 ! asau 51: At boot time the swap file coredump will be saved to "/var/crash".
! 52: The following settings control this behaviour
! 53: (they are set by default in /etc/defaults/rc.conf),
! 54: you don't usually need to change those:
1.1 asau 55:
56: savecore=yes
57: savecore_flags="-N /netbsd -z"
58: savecore_dir="/var/crash"
59:
60: A gzip(1)-compressed file will be available for analysis with gdb(1).
61:
62: Your swap partition must be at least the size of your physical RAM
63:
64: Your "/var/crash" partition must have sufficient space to hold the same file.
65:
66:
67: ## Example Crash: Force Panic from WSCons via KVM: Dell DRAC4
68:
69: You can force a kernel panic from the console on amd64/i386 using the special key sequence: Control+Alt+Esc.
70:
71: * Forcing the Kernel Panic from Console
72:
73: TODO: <img alt="Forcing the Kernel Panic from Console" src="/images/thumb/6/6f/cf_NetBSD_KernPanic_Force.png/300px-cf_NetBSD_KernPanic_Force.png" width="300" height="249" border="0" class="thumbimage" />
74:
75:
76: You can then instruct the KDB to run a preliminary backtrace to get a general idea of what went wrong:
77:
78: TODO: <img alt="Initial backtrace" src="/images/thumb/6/62/cf_NetBSD_KernPanic_InitialBacktrace.png/300px-cf_NetBSD_KernPanic_InitialBacktrace.png" width="300" height="249" border="0" class="thumbimage" />
79:
80: You can then force a sync of the file system and and dump of the kernel memory into the swap partition:
81:
82: TODO: <img alt="Forcing the RAM dump to Swap" src="/images/thumb/6/68/cf_NetBSD_KernPanic_ForceDump.png/300px-cf_NetBSD_KernPanic_ForceDump.png" width="300" height="225" border="0" class="thumbimage" />
83:
84: At reboot, you will see the "/etc/rc.d/savecore" script archive and gzip the dump:
85:
86: TODO: <img alt="Save Core Running" src="/images/thumb/9/94/cf_NetBSD_KernPanic_SaveCore.png/300px-cf_NetBSD_KernPanic_SaveCore.png" width="300" height="249" border="0" class="thumbimage" />
87:
88: You can then load the core dump into standard <b>gdb(8)</b>
89:
90:
91: ## What Now
92:
93: You can submit the feedback as a PR to the NetBSD GNATS system.
94:
95: If you wish to continue to debug, you may find this article useful:
1.2 asau 96: (TODO!) <a href="/NetBSD_kernel_developer_cheat-sheet" title="NetBSD kernel developer cheat-sheet">NetBSD kernel developer cheat-sheet</a>
1.1 asau 97:
98:
99: ## Processing the CORE Dump
100:
1.4 asau 101: Hubert Feyrer has a great guide to analyzing kernel panic core dumps at <http://www.feyrer.de/NetBSD/bx/blosxom.cgi/index.front?-tags=gdb>
1.1 asau 102:
103: Additionally, the following command below can be used to create a relatively useful backtrace:
104:
105: <pre>
1.3 asau 106: localhost# cd /var/crash
107: localhost# gunzip -d *gz
108: localhost# gdb --symbols=/netbsd.gdb --quiet --eval-command="file /netbsd.gdb" \
109: --eval-command="target kvm netbsd.1.core" --eval-command "bt" \
110: --eval-command "list" --eval-command "info all-registers" 2>&1
1.1 asau 111: Load new symbol table from "/netbsd.gdb"? (y or n) y
112: Reading symbols from /netbsd.gdb...done.
113: #0 0xc047c9f8 in cpu_reboot (howto=256, bootstr=0x0) at /usr/src/sys/arch/i386/i386/machdep.c:927
114: 927 dumpsys();
115: #0 0xc047c9f8 in cpu_reboot (howto=256, bootstr=0x0) at /usr/src/sys/arch/i386/i386/machdep.c:927
116: #1 0xc01c3f2a in db_sync_cmd (addr=-1065223264, have_addr=false, count=-1071881791, modif=0xcc883c04 "[BINARY]") at /usr/src/sys/ddb/db_command.c:1304
117: #2 0xc01c45fa in db_command (last_cmdp=0xc07dfe3c) at /usr/src/sys/ddb/db_command.c:926
118: #3 0xc01c4856 in db_command_loop () at /usr/src/sys/ddb/db_command.c:583
119: #4 0xc01c7320 in db_trap (type=1, code=0) at /usr/src/sys/ddb/db_trap.c:101
120: #5 0xc0478855 in kdb_trap (type=1, code=0, regs=0xcc883e3c) at /usr/src/sys/arch/i386/i386/db_interface.c:229
121: #6 0xc047efe2 in trap (frame=0xcc883e3c) at /usr/src/sys/arch/i386/i386/trap.c:350
122: #7 0xc010cb80 in calltrap ()
123: #8 0xc047717c in breakpoint ()
124: #9 0xc02e3676 in wskbd_translate (id=0xc0833ae0, type=2, value=<value optimized out>) at /usr/src/sys/dev/wscons/wskbd.c:1586
125: #10 0xc02e386e in wskbd_input (dev=0xcc888800, type=2, value=1) at /usr/src/sys/dev/wscons/wskbd.c:682
126: #11 0xc054c27a in pckbd_input (vsc=0xcc0cc6a8, data=1) at /usr/src/sys/dev/pckbport/pckbd.c:584
127: #12 0xc02ba80d in pckbcintr (vsc=0xcc0d6ebc) at /usr/src/sys/dev/ic/pckbc.c:607
128: #13 0xc0465798 in intr_biglock_wrapper (vp=0xc2e853c0) at /usr/src/sys/arch/x86/x86/intr.c:617
129: #14 0xc01036d9 in Xintr_ioapic_edge3 ()
130: #15 0xc0477234 in x86_mwait ()
131: Previous frame inner to this frame (corrupt stack?)
132: 922 /* Disable interrupts. */
133: 923 splhigh();
134: 924
135: 925 /* Do a dump if requested. */
136: 926 if ((howto & (RB_DUMP | RB_HALT)) == RB_DUMP)
137: 927 dumpsys();
138: 928
139: 929 haltsys:
140: 930 doshutdownhooks();
141: 931
142: eax 0x0 0
143: ecx 0x0 0
144: edx 0x0 0
145: ebx 0x100 256
146: esp 0xcc883bb8 0xcc883bb8
147: ebp 0xcc883bc0 0xcc883bc0
148: esi 0xc07dfe3c -1065484740
149: edi 0x0 0
150: eip 0xc047c9f8 0xc047c9f8 <cpu_reboot+368>
151: eflags 0x0 [ ]
152: cs 0x0 0
153: ss 0x0 0
154: ds 0x0 0
155: es 0x0 0
156: fs 0x0 0
157: gs 0x0 0
158: st0 0 (raw 0x00000000000000000000)
159: st1 0 (raw 0x00000000000000000000)
160: st2 0 (raw 0x00000000000000000000)
161: st3 0 (raw 0x00000000000000000000)
162: st4 0 (raw 0x00000000000000000000)
163: st5 0 (raw 0x00000000000000000000)
164: st6 0 (raw 0x00000000000000000000)
165: st7 0 (raw 0x00000000000000000000)
166: fctrl 0x0 0
167: fstat 0x0 0
168: ftag 0x0 0
169: fiseg 0x0 0
170: fioff 0x0 0
171: foseg 0x0 0
172: fooff 0x0 0
173: fop 0x0 0
174: xmm0
175: xmm1
176: xmm2
177: xmm3
178: xmm4
179: xmm5
180: xmm6
181: xmm7
182: mm0
183: mm1
184: mm2
185: mm3
186: mm4
187: mm5
188: mm6
189: mm7
190: </pre>
191:
192: NOTE: Aleksej Saushev <asau@inbox.ru> and Brian Seklecki <seklecki@noc.cfi.pgh.pa.us> will work on this content concurrently.
CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb
![[BACK]](/icons/back.gif){kind=icon}
Return to panic.mdwn CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [NetBSD Developer Wiki] / wikisrc