This HOWTO explains how to set up a test environment for symbolic debugging of the NetBSD kernel using a pair of QEMU virtual machines.
You need a computer running an OS capable of cross-building NetBSD (the "host system"). This can be NetBSD itself, Linux, or some other Unix-like OS. These instructions have been tested with NetBSD/amd64 6.1.4 and Debian 7 hosts. There should be at least 20 gigabytes of available disk space.
If your host system is running NetBSD, install the following packages from pkgsrc:
- emulators/qemu >= 2.0.0nb4
If your host system uses a package system other than pkgsrc, use that to install cvs, make, gcc, qemu, the Python pexpect library, and genisoimage or mkisofs. Also download and install the most recent anita package from http://www.gson.org/netbsd/anita/download/.
Building the target system
Check out the NetBSD-current sources from CVS and build a full NetBSD-current/i386 release with debug symbols using the build.sh script. The i386 port is the preferred test platform because the two other ports supported by anita are affected by known bugs: amd64 by PR 49276, and sparc by qemu bug 1335444.
If you do the build in a directory other than /usr/src, use the -fdebug-prefix-map option to ensure that the source file names embedded in the debug symbols point to /usr/src, which is where the sources will be installed on the target system. For example:
$ CVSROOT=anoncvs@anoncvs.NetBSD.org:/cvsroot cvs checkout -A -P src $ cd src $ ./build.sh -j 4 -V MKDEBUG=YES -V COPTS="-g -fdebug-prefix-map=$(pwd)=/usr/src" -O ../obj -m i386 -U release sourcesets
For best performance, change the number after "-j" to the number of CPU cores you have, or slightly more.
Installing the target system
Install the system in a virtual machine, including the debug symbols and source code:
$ cd .. $ anita --workdir work --disk-size 4G --memory-size 256M \ --sets kern-GENERIC,modules,base,etc,comp,debug,games,man,misc,tests,text,syssrc,src,sharesrc,gnusrc \ install $(pwd)/obj/releasedir/i386/
Booting the VMs
Next, start two qemu virtual machines, one to run the kernel being debugged (the "target VM") and another to run gdb (the "gdb VM").
The two VMs could be run on separate physical machines, but in this example, they are run on the same physical machine and share the same hard disk image. This sharing is made possible by the "-snapshot" option to qemu, which ensures that the disk image is not written to by either VM.
First start the target VM, enabling qemu's built-in GDB target stub on TCP port 1234:
$ qemu-system-i386 -nographic -snapshot -hda work/wd0.img -gdb tcp::1234
If you don't want everyone on the Internet to be able to debug your target, make sure incoming connections on port 1234 are blocked in your firewall.
In a second terminal window, start the gdb VM:
$ qemu-system-i386 -nographic -snapshot -hda work/wd0.img
Log in to the gdb VM as root and set up the network:
login: root # dhcpcd
Start gdb on the gdb VM and connect to the target:
# gdb /netbsd (gdb) target remote my.host.name:1234
where my.host.name is the domain name or IP address of the host system.
Now you should be able to get a stack trace and start debugging with full debug symbols and access to the source code:
(gdb) where (gdb) list
If the stack trace prints very slowly (like 30 seconds per stack frame), it's likely because you are using a version of qemu where the user-mode networking code fails to disable the Nagle algorithm. This is fixed in the qemu in pkgsrc, but you may run into it if your qemu is not installed via pkgsrc.
Here is a couple of useful qemu commands to know:
Ctrl-a b will send a break which will make the NetBSD VM enter the ddb kernel debugger.
Ctrl-a c will switch to the qemu monitor where you can enter commands like "quit" to exit qemu, or do things like saving/restoring the VM to/from a file.