I've spent a month on fixes and debugging issues around the tracing facilities in the kernel.

Distribution cleanup

As planned in the previous month, I've performed cleanup in the distribution:

  • Removal of unneeded PT_SET_SIGMASK and PT_GET_SIGMASK, followed by pullup to NetBSD-8.
  • regnsub(3) and regasub(3) API change has been abandoned as it causes backward compatibility bootstrap breakage from older distributions. Altering it could require additional changes in libtre, but it has been abandoned too. At the end I've finally removed just the USE_LIBTRE build option, replacing the libc's regex(3) implementation with an alternative library from libtre/agrep, because it no longer builds as a regex-replacement and the upstream development stalled few years ago.
  • Backport of _UC_MACHINE_FP() to NetBSD-8 has been finished, this means that we can use the same code in NetBSD-8 and NetBSD-current in the sources of sanitizers.

Improvements in the ATF ptrace(2) tests

I've performed the following operations in the ATF ptrace(2) and related tests, in the following commit order:

  • Correct all ATF failures in t_ptrace_x86_wait.h (debug registers)
  • ATF: Correct a race bug in attach2 (t_ptrace_wait*)
  • ATF: Reenable attach2 in t_ptrace_wait*
  • Add a new function in ATF t_ptrace_wait*: await_zombie_raw()
  • ATF t_ptrace_wait*: Disable debug messages in msg.h
  • ATF: Add new test race1 in t_ptrace_wait*
  • Add new ATF tests: kernel/t_zombie

Zombie detection race

I've detected that the operation polling for a single process status - using sysctl(3) - can report the same process twice, for the first time as an alive process and for the second time as a zombie. This used to break the ATF ptrace(2) tests where we have a polling function, detecting the state change of a dying process.

I've prepared an applied a fix for this case and the bug detected in ATF ptrace(2) tests is now gone. I've included additional regression tests as noted below, to catch the race in a dedicated t_zombie test-suite.

There is also a controversy here, as it's not specified by POSIX what happens when we are polling an unrelated process, that is not a child of its parent. This operation can fail and we the previous approach we could report the same process twice, while with the newer one reporting twice is much more unlikely on the cost of missing the process at times.

My preference is that a process between the state transition of alive->dying->dead and dead->zombie, can disappear for a while. I find this metaphor more natural rather than observing two entities that one is dying and the other is a zombie. There is no perfect solution to this process watch and just following the narrow cases requested with POSIX seems to be the proper solution and it de facto assumes such races.

Bohrbug in X86 Debug Registers

The release engineering machines were observing occasional failures with the X86 Debug Registers in the ptrace(2) ATF test-suite. This was appearing once a while, approximately quarterly, on both amd64 and i386 machines.

I've missed them previously in my original work a year ago, as this happened to be caused by the fact that this bug is not reproducible on newer (quicker? more cores?) Intel CPUs. This failure was reproduced only in software-emulation (slow!) qemu and on Intel Core 2 Duo.

An attempt to execute the same X86 Debug Register test on Intel i7 for 100M times does not make this crash to pop up even a single time.

I've spent the majority of the past month on researching this bug and it happened to be a bohrbug, this means that adding almost any debug code anywhere makes it disappear completely or make reproducible siginificantly less often.

It's not worth the space to describe the process of understanding this bug more closely, step by step - it's worth to mention the current state of the understanding of mine. We can observe a repeated syscall for the _lwp_kill(2) operation (executed with raise(SIGSTOP)), with the same trap frame (or very similar). I still don't know how is it possible.

I've finally added support to the NetBSD kernel - in my local copy of the sources - for Bochs-style debug console/protocol (ISA port 0xe9). This allows me to log internal qemu debug and NetBSD kernel messages into a single buffer and output to a single file. I need this property as matching two logging buffers isn't trivial, especially since just the logging of interrupt frames in qemu can quickly generate hundreds of megabytes of text. Assuming that I've reproduced the bohrbug after 20k executions of the same test and there is a lot of noise from unrelated kernel/hardware traps, it's a useful property. I'm planning to cleanup this code and submit to the NetBSD sources.

For completeness I had to patch the qemu isa-debugcon (Bochs-style debug console) source code to log into the same buffer as the internal qemu debug messages... sometimes bugs require non-conventional approaches to research them. Ideally I would like to have a rewind/record feature in qemu together with the gdb-server emulation, but we are still not there.

ptrace(2) status and plans

Now there is the X86 Debug Register race. Also, the following tests still marked as expected failures:

  • eventmask3
  • resume1
  • signal3, signal5, signal6, signal7, signal8, signal9
  • suspend1, suspend2
  • vfork1

This shows that the remaining major problems are in:

  • vfork(2)
  • signals
  • threads

Once I will squash the X86 Debug Register race, I plan to work on these bugs in this oder: vfork handling bugs, signals and threads.

I have in mind addition of a lot of a lot of tests verifying correct signal processing in traced programs.

In this iteration of ptrace(2) hardening, I plan to skip Machine-Dependent extensions (like AVXv2 registers) and extending core(5) files with additional features (for example we might want to know exact ABI or FPU hardware layout).

Plan for the next milestone

Keep working on the bug reproducible with X86 Debug Registers and switch to the remaining ptrace(2) failures.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:


Posted late Sunday night, April 2nd, 2018 Tags:

The NetBSD Project is pleased to announce NetBSD 8.0 RC 1, the first release candidate for the upcoming NetBSD 8.0 release.

25 years and a few days after the first official NetBSD release (NetBSD 0.8 on April 19, 1993) we are now quickly approaching the first final release from the netbsd-8 branch that has been in the work for more most of a year now.

The official RC1 announcement list these major changes compared to older releases:

  • USB stack rework, USB3 support added
  • In-kernel audio mixer
  • Reproducible builds
  • PaX MPROTECT (W^X) memory protection enforced by default on some architectures with fine-grained memory protection and suitable ELF formats: i386, amd64, evbarm, landisk, pmax
  • PaX ASLR enabled by default on:
    i386, amd64, evbarm, landisk, pmax, sparc64
  • MKPIE (position independent executables) by default for userland on: i386, amd64, arm, m68k, mips, sh3, sparc64
  • added can(4), a socket layer for CAN busses
  • added ipsecif(4) for route-based VPNs
  • made part of the network stack MP-safe
  • NET_MPSAFE kernel option is required to try
  • WAPBL stability and performance improvements

Specific to i386 and amd64 CPUs:
  • Meltdown mitigation: SVS (separate virtual address spaces)
  • Spectre mitigation (support in gcc, used by default for kernels)
  • SMAP support
  • (U)EFI bootloader

Various new drivers:
  • nvme(4) for modern solid state disks
  • iwm(4), a driver for Intel Wireless devices (AC7260, AC7265, AC3160...)
  • ixg(4): X540, X550 and newer device support.
  • ixv(4): Intel 10G Ethernet virtual function driver.
  • bta2dpd - new Bluetooth Advanced Audio Distribution Profile daemon

Many evbarm kernels now use FDT (flat device tree) information (loadable at boot time from an external file) for device configuration, the number of kernels has decreased but the numer of boards has vastly increased.

Lots of updates to 3rd party software included:
  • GCC 5.5 with support for Address Sanitizer and Undefined Behavior Sanitizer
  • GDB 7.12
  • GNU binutils 2.27
  • Clang/LLVM 3.8.1
  • OpenSSH 7.6
  • OpenSSL 1.0.2k
  • mdocml 1.14.1
  • acpica 20170303
  • ntp 4.2.8p11-o
  • dhcpcd 7.0.3
  • Lua 5.3.4

The NetBSD developers and the release engineering team have spent a lot of effort to make sure NetBSD 8.0 will be a superb release, but we have not yet fixed most of the accompanying documentation. So the included release notes and install documents will be updated before the final release, and also the above list of major items may lack important things.

Get NetBSD 8.0 RC1 from our CDN (provided by fastly) or one of the ftp mirrors.

Complete source and binaries for NetBSD are available for download at many sites around the world. A list of download sites providing FTP, AnonCVS, and other services may be found at http://www.NetBSD.org/mirrors/.

Please test RC1, so we can make the final release the best one ever so far. We are looking forward to your feedback. Please send-pr any bugs or mail us at releng at NetBSD.org for more general comments.

Posted at noon on Wednesday, April 25th, 2018 Tags: