Apr 2017
S M T W T F S
           
23 24
           

Archives

This page is a blog mirror of sorts. It pulls in articles from blog's feed and publishes them here (with a feed, too).

A feature set for debugging NetBSD applications (without threads) has been merged with upstream LLDB! The number of passing tests this month has been increased from 267/1235 to 622/1247. This is +133% within one month and approximately 50% of successfully passed tests in total! As usual regular housekeeping of ptrace(2) interfaces has been done on the NetBSD side.

During this month I've finished the needed Native Process Plugin with breakpoints fully supported. In order to achieve this I had to address bugs, add missing features and diligently debug the debugger sniffing on the GDB Remote Protocol line. Since NetBSD-8 is approaching, I have performed the also needed housekeeping on the base system distribution side.

What has been done in NetBSD

I've managed to achieve the following goals:

Clean up in ptrace(2) ATF tests

We have created some maintanance burden for the current ptrace(2) regression tests. The main issues with them is code duplication and the splitting between generic (Machine Independent) and port-specific (Machine Dependent) test files. I've eliminated some of the ballast and merged tests into the appropriate directory tests/lib/libc/sys/. The old location (tests/kernel) was a violation of the tests/README recommendation:

When adding new tests, please try to follow the following conventions.

1. For library routines, including system calls, the directory structure of
   the tests should follow the directory structure of the real source tree.
   For instance, interfaces available via the C library should follow:

        src/lib/libc/gen -> src/tests/lib/libc/gen
        src/lib/libc/sys -> src/tests/lib/libc/sys
        ...

The ptrace(2) interface inhabits src/lib/libc/sys so there is no reason not to move it to its proper home.

PTRACE_FORK on !x86 ports

Along with the motivation from Martin Husemann we have investigated the issue with PTRACE_FORK ATF regression tests. It was discovered that these tests aren't functional on evbarm, alpha, shark, sparc and sparc64 and likely on other non-x86 ports. We have discovered that there is a missing SIGTRAP emitted from the child, during the fork(2) handshake. The proper order of operations is as follows:

  • parent emits SIGTRAP with si_code=TRAP_CHLD and pe_set_event=pid of forkee
  • child emits SIGTRAP with si_code=TRAP_CHLD and pe_set_event=pid of forker

Only the x86 ports were emitting the second SIGTRAP signal.

The culprit reason has been investigated and narrowed down to the child_return() function in src/sys/arch/x86/x86/syscall.c:

void
child_return(void *arg)
{
    struct lwp *l = arg;
    struct trapframe *tf = l->l_md.md_regs;
    struct proc *p = l->l_proc;
    if (p->p_slflag & PSL_TRACED) {
        ksiginfo_t ksi;
        mutex_enter(proc_lock);
        KSI_INIT_EMPTY(&ksi);
        ksi.ksi_signo = SIGTRAP;
        ksi.ksi_lid = l->l_lid;
        kpsignal(p, &ksi, NULL);
        mutex_exit(proc_lock);
    }
    X86_TF_RAX(tf) = 0;
    X86_TF_RFLAGS(tf) &= ~PSL_C;
    userret(l);
    ktrsysret(SYS_fork, 0, 0);
}

This child_return() function was the only one among all the existing ones for other platforms to contain the needed code for SIGTRAP. The appropriate solution was installed by Martin, as we taught our featured signal routing subsystem to handle early signals from the fork(2) calls.

PT_SYSCALL and PT_SYSCALLEMU

Christos Zoulas addressed the misbehavior with tracing syscall entry and syscall exit code. We can again get an event inside a debugger that a debuggee attempts to trigger a syscall and later return from it. This means that a debugger can access the register layout before executing the appropriate kernel code and read it again after executing the syscall. This allows to monitor exact sysentry arguments and return the values afterwards. Another option is to fake the trap frame with new values, it's sometimes useful for debugging.

With the addition of PT_SYSCALLEMU we can implement a virtual kernel syscall monitor. It means that we can fake syscalls within a debugger. In order to achieve this feature, we need to use the PT_SYSCALL operation, catch SIGTRAP with si_code=TRAP_SCE (syscall entry), call PT_SYSCALLEMU and perform an emulated userspace syscall that would have been done by the kernel, followed by calling another PT_SYSCALL with si_code=TRAP_SCX.

This interface makes it possible to introduce the following into NetBSD: truss(1) from FreeBSD and strace(1) from Linux. There used to be a port of at least strace(1) in the past, but it's time to refresh this code. Another immediate consumer is of course in DTrace/libproc... as there are facilities within this library to trace the system call entry and exit in order to catch fork(2) events. Why to catch fork(2)? It can be useful to detach software breakpoints in order to detach them before cloning address space of forker for forkee; and after the operation reapply them again.

What has been done in LLDB

A lot of work has been done with the goal to get breakpoints functional. This target penetrated bugs in the existing local patches and unveiled missing features required to be added. My initial test was tracing a dummy hello-world application in C. I have sniffed the GDB Remote Protocol packets and compared them between Linux and NetBSD. This helped to streamline both versions and bring the NetBSD support to the required Linux level.

As a bonus the initial code for OpenBSD support was also added into the LLDB tree. At the moment OpenBSD only supports opening core(5) files with a single-thread. The same capability was also added for NetBSD.

By the end of March all local patches for LLDB were merged upstream! This resulted in NetBSD being among the first operating systems to use a Native Process Plugin framework with the debugserver capability, alongside Linux & Android. The current FreeBSD support in LLDB is dated and lagging behind and limited to local debugging. A majority of the work to bring FreeBSD on par could well be a case of s/NetBSD/FreeBSD/.

Among the features in the upstreamed NetBSD Process Plugin:

  • handling software breakpoints,
  • correctly attaching to a tracee,
  • supporting NetBSD specific ptrace(2),
  • monitoring process termination,
  • monitoring SIGTRAP events,
  • monitoring SIGSTOP events,
  • monitoring other signals events,
  • resuming the whole process,
  • getting memory region info perms,
  • reading memory from tracee,
  • writing memory to tracee,
  • reading ELF AUXV,
  • x86_64 GPR reading and writing,
  • detecting debuginfo of the basesystem programs located in /usr/libdata/debug
  • adding single step support,
  • adding execve(2) trap support,
  • placeholder for Floating Point Registers code,
  • initial code for the NetBSD specific core(5) files,
  • enabling ELF Aux Vector reading on the lldb client side,
  • enabling QPassSignals feature for NetBSD on the lldb client side,
  • enabling ProcessPOSIXLog on NetBSD,
  • minor tweaking.

Demo

It's getting rather difficult to present all the features of the NetBSD Process Plugin without making the example overly long. This is why I will restrict it to a very basic debugging session hosted on NetBSD.

$ cat crashme.c 
#include 

int
main(int argc, char **argv)
{
        int i = argv;

        while (i-- != 0)
                printf("argv[%d]=%s\n", i, argv[i]);

        return 0;
}
$ gcc -w -g -o crashme crashme.c # -w disable all warnings                                                                                          
$ ./crashme
Memory fault (core dumped) 
chieftec$ lldb ./crashme
(lldb) target create "./crashme"
Current executable set to './crashme' (x86_64).
(lldb) r
Process 612 launched: './crashme' (x86_64)
Process 612 stopped
* thread #1, stop reason = signal SIGSEGV: address access protected (fault address: 0x7f7ffec88518)
    frame #0: 0x000000000040089c crashme`main(argc=1, argv=0x00007f7fffdd6420) at crashme.c:9
   6            int i = argv;
   7    
   8            while (i-- != 0)
-> 9                    printf("argv[%d]=%s\n", i, argv[i]);
   10   
   11           return 0;
   12   }
(lldb) frame var
(int) argc = 1
(char **) argv = 0x00007f7fffdd6420
(int) i = -2268129
(lldb) # i looks wrong, checking argv...
(lldb) p *argv
(char *) $0 = 0x00007f7fffdd6940 "./crashme"
(lldb) # set a brakpoint and restart
(lldb) b main
Breakpoint 1: where = crashme`main + 15 at crashme.c:6, address = 0x000000000040087f
(lldb) r
There is a running process, kill it and restart?: [Y/n] y
Process 612 exited with status = 6 (0x00000006) got unexpected response to k packet: Sff
Process 80 launched: './crashme' (x86_64)
Process 80 stopped
* thread #1, stop reason = breakpoint 1.1
    frame #0: 0x000000000040087f crashme`main(argc=1, argv=0x00007f7fff3d17c8) at crashme.c:6
   3    int
   4    main(int argc, char **argv)
   5    {
-> 6            int i = argv;
   7    
   8            while (i-- != 0)
   9                    printf("argv[%d]=%s\n", i, argv[i]);
(lldb) frame var
(int) argc = 1
(char **) argv = 0x00007f7fff3d17c8
(int) i = 0
(lldb) n
Process 80 stopped
* thread #1, stop reason = step over
    frame #0: 0x0000000000400886 crashme`main(argc=1, argv=0x00007f7fff3d17c8) at crashme.c:8
   5    {
   6            int i = argv;
   7    
-> 8            while (i-- != 0)
   9                    printf("argv[%d]=%s\n", i, argv[i]);
   10   
   11           return 0;
(lldb) p i
(int) $1 = -12773432
(lldb) # gotcha i = argv, instead of argc!
(lldb) bt
* thread #1, stop reason = step over
  * frame #0: 0x0000000000400886 crashme`main(argc=1, argv=0x00007f7fff3d17c8) at crashme.c:8
    frame #1: 0x000000000040078b crashme`___start + 229
(lldb) disassemble --frame --mixed

   4    main(int argc, char **argv)
** 5    {

crashme`main:
    0x400870 <+0>:  pushq  %rbp
    0x400871 <+1>:  movq   %rsp, %rbp
    0x400874 <+4>:  subq   $0x20, %rsp
    0x400878 <+8>:  movl   %edi, -0x14(%rbp)
    0x40087b <+11>: movq   %rsi, -0x20(%rbp)

** 6            int i = argv;
   7    

    0x40087f <+15>: movq   -0x20(%rbp), %rax
    0x400883 <+19>: movl   %eax, -0x4(%rbp)

-> 8            while (i-- != 0)

->  0x400886 <+22>: jmp    0x4008b3                  ; <+67> at crashme.c:8

** 9                    printf("argv[%d]=%s\n", i, argv[i]);
   10   

    0x400888 <+24>: movl   -0x4(%rbp), %eax
    0x40088b <+27>: cltq   
    0x40088d <+29>: leaq   (,%rax,8), %rdx
    0x400895 <+37>: movq   -0x20(%rbp), %rax
    0x400899 <+41>: addq   %rdx, %rax
    0x40089c <+44>: movq   (%rax), %rdx
    0x40089f <+47>: movl   -0x4(%rbp), %eax
    0x4008a2 <+50>: movl   %eax, %esi
    0x4008a4 <+52>: movl   $0x400968, %edi           ; imm = 0x400968 
    0x4008a9 <+57>: movl   $0x0, %eax
    0x4008ae <+62>: callq  0x400630                  ; symbol stub for: printf

** 8            while (i-- != 0)

    0x4008b3 <+67>: movl   -0x4(%rbp), %eax
    0x4008b6 <+70>: leal   -0x1(%rax), %edx
    0x4008b9 <+73>: movl   %edx, -0x4(%rbp)
    0x4008bc <+76>: testl  %eax, %eax
    0x4008be <+78>: jne    0x400888                  ; <+24> at crashme.c:9

** 11           return 0;

    0x4008c0 <+80>: movl   $0x0, %eax

** 12   }

    0x4008c5 <+85>: leave  
    0x4008c6 <+86>: retq   
(lldb) version
lldb version 5.0.0 (http://llvm.org/svn/llvm-project/lldb/trunk revision 299109)
(lldb) platform status 
  Platform: host
    Triple: x86_64-unknown-netbsd7.99
OS Version: 7.99.67 (0799006700)
    Kernel: NetBSD 7.99.67 (GENERIC) #1: Mon Apr  3 08:09:29 CEST 2017  root@chieftec:/public/netbsd-root/sys/arch/amd64/compile/GENERIC
  Hostname: 127.0.0.1
WorkingDir: /public/lldb_devel
    Kernel: NetBSD
   Release: 7.99.67
   Version: NetBSD 7.99.67 (GENERIC) #1: Mon Apr  3 08:09:29 CEST 2017  root@chieftec:/public/netbsd-root/sys/arch/amd64/compile/GENERIC
(lldb) # thank you!
(lldb) q
Quitting LLDB will kill one or more processes. Do you really want to proceed: [Y/n] y

Plan for the next milestone

I've listed the following goals for the next milestone.

  • watchpoints support,
  • floating point registers support,
  • enhance core(5) and make it work for multiple threads
  • introduce PT_SETSTEP and PT_CLEARSTEP in ptrace(2)
  • support threads in the NetBSD Process Plugin
  • research F_GETPATH in fcntl(2)

Beyond the next milestone is x86 32-bit support.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:

http://netbsd.org/donations/#how-to-donate

Posted late Tuesday afternoon, April 4th, 2017 Tags: blog

The NetBSD Project is pleased to announce NetBSD 7.1, the first feature update of the NetBSD 7 release branch. It represents a selected subset of fixes deemed important for security or stability reasons, as well as new features and enhancements.

Some highlights of NetBSD 7.1 are:

  • Support for Raspberry Pi Zero.
  • Initial DRM/KMS support for NVIDIA graphics cards via nouveau (Disabled by default. Uncomment nouveau and nouveaufb in your kernel config to test).
  • The addition of vioscsi, a driver for the Google Compute Engine disk.
  • Linux compatibility improvements, allowing, e.g., the use of Adobe Flash Player 24.
  • wm(4):
    • C2000 KX and 2.5G support.
    • Wake On Lan support.
    • 82575 and newer SERDES based systems now work.
  • ODROID-C1 Ethernet now works.
  • Numerous bug fixes and stability improvements.

For more details, please see the release notes.

Complete source and binaries for NetBSD are available for download at many sites around the world. A list of download sites providing FTP, AnonCVS, SUP, and other services may be found at http://www.NetBSD.org/mirrors/.

Posted early Wednesday morning, March 15th, 2017 Tags: blog
During this month I've finished the needed work in the base distribution in order to host fully featured LLDB. Currently the ptrace(2) interfaces in NetBSD are, in terms of features, closely related to FreeBSD and Linux. There are only few bugs left with filed Problem Reports and alerting regression tests, however they do not interfere with the needed functions to move the port of the debugger forward.

This time I will shortly iterate over the finished tasks as I would like to present slides on ptrace(2) in NetBSD.

As usual hundreds of ATF tests were introduced, few Problem Reports filed and two patches upstreamed to LLDB.

What has been done in NetBSD

I've managed to achieve the following goals:

  1. Marked exect(3) obsolete in libc
  2. Removed libpthread_dbg(3) from the base distribution
  3. Added new ptrace(2) operations PT_SET_SIGMASK and PT_GET_SIGMASK
  4. Switch ptrace(2) PT_WATCHPOINT API to PT_GETDBRGS and PT_SETDBREGS
  5. Validation of ptrace(2) PT_SYSCALL

The exect(3) interface is no longer functional and it's the proper time to obsolete it. The libpthread_dbg(3) library is no longer needed, it became unnecessary along with the M:N thread model removal.

I originally added the PT_*ET_SIGMASK interface in order to help the criu port for NetBSD. This software is used to checkpoint programs on Linux. Debuggers can also have checkpointing support, for example GDB/Linux has this ability.

The debug registers are finally in the proper form on NetBSD. The previous API had defects as it was designed to be safe, but keeping it safe on the kernel side was impractical. I've finally decided to adapt it to the existing FreeBSD semantics and reuse PT_*ETDBREGS operations. I've verified this new API on real hardware amd64 (Intel i7) and i386 (Intel Pentium IV) with newly written 390 tests. It's worth noting that this API renders GDB to support hardware watchpoints on NetBSD almost out-of-the-box! However it's not free of bugs, after catching a watchpoint, GDB enters on a trap in a dynamic linker (?).

(gdb) c
Continuing.

Watchpoint 2: traceme

Old value = 0
New value = 16
main (argc=1, argv=0x7f7fff79fe30) at test.c:8
8               printf("traceme=%d\n", traceme);
(gdb) c
Continuing.

Watchpoint 2 deleted because the program has left the block in
which its expression is valid.
0x00007f7e3c000782 in _rtld_bind_start () from /usr/libexec/ld.elf_so
(gdb) c
Continuing.

Program received signal SIGTRAP, Trace/breakpoint trap.
0x00007f7e3c0007b5 in _rtld_bind_start () from /usr/libexec/ld.elf_so
(gdb) c
Continuing.
traceme=16
traceme=17
[Inferior 1 (process 28797) exited normally]

Sadly the PT_SYSCALL needs more work. The current implementation is currently broken and cannot stop the process on syscall entry.

What has been done in LLDB

As usual, I've pushed some ready patches to upstream LLDB. The first one covers Debug Register accessors and the second one adds proper thread identity detection on NetBSD.

Porting ptrace(2) software to NetBSD

I have prepared this presentation to illustrate the new code in the context of Linux and other BSDs.

To browse the slides one must use arrow buttons or the mouse scroll wheel. Press the h key for help.

Porting ptrace(2) software to NetBSD.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:

http://netbsd.org/donations/#how-to-donate

Posted early Wednesday morning, March 1st, 2017 Tags: blog

The second release candidate of NetBSD 7.1 is now available for download at:

http://cdn.NetBSD.org/pub/NetBSD/NetBSD-7.1_RC2/

Those of you who prefer to build from source can continue to follow the netbsd-7 branch or use the netbsd-7-1-RC2 tag.

Most changes made since 7.1_RC1 have been security fixes. See src/doc/CHANGES-7.1 for the full list.

Please help us out by testing 7.1_RC2. We love any and all feedback. Report problems through the usual channels (submit a PR or write to the appropriate list). More general feedback is welcome at releng@NetBSD.org.

Posted terribly early Friday morning, February 24th, 2017 Tags: blog

Introduction

I have been working on and off for almost a year trying to get reproducible builds (the same source tree always builds an identical cdrom) on NetBSD. I did not think at the time it would take as long or be so difficult, so I did not keep a log of all the changes I needed to make. I was also not the only one working on this. Other NetBSD developers have been making improvements for the past 6 years.

I would like to acknowledge the NetBSD build system (aka build.sh) which is a fully portable cross-build system. This build system has given us a head-start in the reproducible builds work.

I would also like to acknowledge the work done by the Debian folks who have provided a platform to run, test and analyze reproducible builds. Special mention to the diffoscope tool that gives an excellent overview of what's different between binary files, by finding out what they are (and if they are containers what they contain) and then running the appropriate formatter and diff program to show what's different for each file.

Finally other developers who have started, motivated and did a lot of work getting us here like Joerg Sonnenberger and Thomas Klausner for their work on reproducible builds, and Todd Vierling and Luke Mewburn for their work on build.sh.

Sources of difference

Here's is what we found that we needed to fix, how we chose to fix it and why, and where are we now.

There are many reasons why two separate builds from the same sources can be different. Here's an (incomplete) list:

  1. timestamps

    Many things like to keep track of timestamps, specially archive formats (tar(1), ar(1)), filesystems etc. The way to handle each is different, but the approach is to make them either produce files with a 0 timestamp (where it does not matter like ar), or with a specific timestamp when using 0 does not make sense (it is not useful to the user).
  2. dates/times/authors etc. embedded in source files

    Some programs like to report the date/time they were built, the author, the system they were built on etc. This can be done either by programmatically finding and creating source files containing that information during build time, or by using standard macros such as __DATE__, __TIME__ etc. Usually putting a constant time or eliding the information (such as we do with kernels and bootblocks) solves the problem.
  3. timezone sensitive code

    Certain filesystem formats (iso 9660 etc.) don't store raw timestamps but formatted times; to achieve this they convert from a timestamp to localtime, so they are affected by the timezone.
  4. directory order/build order

    The build order is not constant especially in the presence of parallel builds; neither is directory scan order. If those are used to create output files, the output files will need to be sorted so they become consistent.
  5. non-sanitized data stored into files

    Writing data structures into raw files can lead to problems. Running the same program in different operating systems or using ASLR makes those issues more obvious.
  6. symbolic links/paths

    Having paths embedded into binaries (specially for debugging information) can lead to binary differences. Propagation of the logical path can prove problematic.
  7. general tool inconsistencies

    gcc(1) profiling uses a PROFILE_HOOK macro on RISC targets that utilizes the "current function" number to produce labels. Processing order of functions is not guaranteed. gpt(8) creation involves uuid generation; these are generally random. block allocation on msdos filesystems had a random component. makefs(8) uses timezones with timestamps (iso9660), randomness for block selection (msdos), stores stray pointers in superblock (ffs).
  8. toolchain

    Every program that is used to generate other output needs to have consistent results. In NetBSD this is done with build.sh, which builds a set of tools from known sources before it can use those tools to build the rest of the system). There is a large number of tools. There are also internal issues with the tools that make their output non reproducible, such as nondeterministic symbol creation or capturing parts of the environment in debugging information.
  9. build information / tunables / environment

    There are many environment settings, or build variable settings that can affect the build. This needs to be kept constant across builds so we've changed the list of variables that are reported in Makefile.params:
    .if ${MKREPRO:Uno} != "yes"
    RELEASEVARS+=   BSDOBJDIR BSDSRCDIR BUILDID BUILDINFO BUILDSEED \
                    DESTDIR KERNARCHDIR KERNCONFDIR KERNOBJDIR KERNSRCDIR MAKE \
                    MAKEFLAGS NBUILDJOBS NETBSDSRCDIR OBJMACHINE OBJMACHINE_ARCH \
                    RELEASEDIR RELEASEMACHINEDIR TOOLDIR USR_OBJMACHINE X11SRCDIR
    .endif
    
  10. making sure that the source tree has no local changes

Variables controlling reproducible builds

Reproducible builds are controlled on NetBSD with two variables: MKREPRO (which can be set to yes or no) and MKREPRO_TIMESTAMP which is used to set the timestamp of the builds artifacts. This is usually set to the number of seconds from the epoch. The build.sh -P flag handles reproducible builds automatically: sets the MKREPRO variable to yes, and then finds the latest source file timestamp in the tree and sets MKREPRO_TIMESTAMP to that.

Handling timestamps

The first thing that we needed to understand was how to deal with timestamps. Some of the timestamps are not very useful (for example inside random ar archives) so we choose to 0 them out. Others though become annoying if they are all 0. What does it mean when you mount install media and all the dates on the files are Jan 1, 1970?

We decided that a better timestamp would be the timestamp of the most recently modified file in the source tree. Unfortunately this was not easy to find on NetBSD, because we are still using CVS as the source control system, and CVS does not have a good way to provide that. For that we wrote a tool called cvslatest, that scans the CVS metadata files (CVS/Entries) and finds the latest commit. This works well for freshly checked out trees (since CVS uses the source timestamp when checking out), but not with updated trees (because CVS uses the current time when updating files, so that make(1) thinks they've been modified). To fix that, we've added a new flag to the cvs(1) "update" command -t, that uses the source checkout time.

The build system needs now to evaluate the tree for the latest file running cvslatest(1) and find the latest timestamp in seconds from the Epoch which is set in the MKREPRO_TIMESTAMP variable. This is the same as SOURCE_DATE_EPOCH. Various Makefiles are using this variable and MKRERPO to determine how to produce consistent build artifacts.

For example many commands (tar(1), makefs(8), gpt(8), ...) have been modified to take a --timestamp or -T command line switch to generate output files that use the given timestamp, instead of the current time.

Other software (am-utils, acpica, bootblocks, kernel) used __DATE__ or __TIME__, or captured the user, machine, etc. from the environment and had to be changed to a constant time, user, machine, etc.

roff(7) documents used the td macro to generate the date of formatting in the document have been changed to conditionally use the macro based on register R, for example as in intro.me and then the Makefile was changed to set that register for MKREPRO.

Handling Order

We don't control the build order of things and we also don't control the directory order which can be filesystem dependent. The collation order also is environment specific, and sorting needs to be stable (we have not encountered that problem yet). Two different programs caused us problems here:

  • file(1) with the generation of the compiled magic file using directory order (fixed by changing file(1)).
  • install-info(1), texinfo(5) files that have no specific order. For that we developed another tool called sortinfo(1) that sorts those files as a post-process step.

Fortunately the filesystem builders and tar programs usually work with input directories that appear to have a consistent order so far, so we did not have to fix things there.

Permissions

NetBSD already keeps permissions for most things consistent in different ways:

  • the build system uses install(8) and specifies ownership and mode.
  • the mtree(8) program creates build artifacts using consistent ownership and permissions.

Nevertheless, the various architecture-specific distribution media installers used cp(1)/mkdir(1) and needed to be corrected.

Toolchain

Most of the issues found had to do with capturing the environment in debugging information. The two biggest issues were: DW_AT_Producer and DW_AT_comp_dir:

DW_AT_producer    : (indirect string, offset: 0x80): GNU C99 5.4.0 \
    -fno-canonical-system-headers -mtune=nocona \
    -march=x86-64 -g -O2 -std=gnu99 -fPIE -fstack-protector \
    -fdebug-prefix-map=$NETBSDSRCDIR=/usr/src \
    -fdebug-prefix-map=$X11SRCDIR=/usr/xsrc \
    -fdebug-regex-map=/usr/src/(.*)/obj.*=/usr/obj/\1 \
    -fdebug-regex-map=/usr/src/(.*)/obj.*/(.*)=/usr/obj/\1/\2 \
    --param ssp-buffer-size=1

Here you see two changes we made for reproducible builds:

  • We chose to allow variable names (and have gcc(1) expand them) for the source of the prefix map because the source tree location can vary. Others have chosen to skip -fdebug-prefix-map from the variables to be listed.
  • We added -fdebug-regex-map so that we could handle the NetBSD specific objdir build functionality. Object directories can have many flavors in NetBSD so it was difficult to use -fdebug-prefix-map to capture that.

DW_AT_comp_dir presented a different challenge. We got non-reproducibility when building on paths where either the source or the object directories contained symbolic links. Although gcc(1) does the right thing handling logical paths (respects $PWD), we found that there were problems both in the NetBSD sh(1) (fixed here) and in the NetBSD make(1) (fixed here). Unfortunately we can't depend on the shell to obey the logical path so we decided to go with:

    ${MAKE} -C other/dir
instead of:
    cd other/dir && ${MAKE}

This works because make(1) is a tool (part of the toolchain we provide) whereas sh(1) is not.

Another weird issue popped up on sparc64 where a single file in the whole source tree does not build reproducibly. This file is asn1_krb5_asn1.c which is generated in here. The problem is that when profiling on RISC machines gcc uses the PROFILE_HOOK macro which in turn uses the "function number" to generate labels. This number is assigned to each function in a source file as it is being compiled. Unfortunately this number is not deterministic because of optimization (a bug?), but fortunately turning optimization off fixes the problem.

Status and future work

As of 2017-02-20 we have fully reproducible builds on amd64 and sparc64. We are planning to work on the following areas:

  • Vary more parameters on the system build (filesystem types, build OS's)
  • Verify that cross building is reproducible
  • Verify that unprivileged builds work
  • Test on all the platforms
Posted late Monday afternoon, February 20th, 2017 Tags: blog
The LLVM project is a quickly moving target, this also applies to the LLVM debugger -- LLDB. It's actively used in several first-class operating systems, while - thanks to my spare time dedication - NetBSD joined the LLDB club in 2014, only lately the native support has been substantially improved and the feature set is quickly approaching the support level of Linux and FreeBSD. During this work 12 patches were committed to upstream, 12 patches were submitted to review, 11 new ATF were tests added, 2 NetBSD bugs filed and several dozens of commits were introduced in pkgsrc-wip, reducing the local patch set to mostly Native Process Plugin for NetBSD.

What has been done in NetBSD

1. Triagged issues of ptrace(2) in the DTrace/NetBSD support

Chuck Silvers works on improving DTrace in NetBSD and he has detected an issue when tracer signals are being ignored in libproc. The libproc library is a compatibility layer for DTrace simulating /proc capabilities on the SunOS family of systems.

I've verified that the current behavior of signal routing is incorrect. The NetBSD kernel correctly masks signals emitted by a tracee, not routing them to its tracer. On the other hand the masking rules in the inferior process blacklists signals generated by the kernel, which is incorrect and turns a debugger into a deaf listener. This is the case for libproc as signals were masked and software breakpoints triggering INT3 on i386/amd64 CPUs and SIGTRAP with TRAP_BRKP si_code wasn't passed to the tracer.

This isn't limited to turning a debugger into a deaf listener, but also a regular execution of software breakpoints requires: rewinding the program counter register by a single instruction, removing trap instruction and restoring the original instruction. When an instruction isn't restored and further code execution is pretty randomly affected, it resulted in execution anomalies and breaking of tracee.

A workaround for this is to disable signal masking in tracee.

Another drawback inspired by the DTrace code is to enhance PT_SYSCALL handling by introducing a way to distinguish syscall entry and syscall exit events. I'm planning to add dedicated si_codes for these scenarios. While there, there are users requesting PT_STEP and PT_SYSCALL tracing at the same time in an efficient way without involving heuristcs.

I've filed the mentioned bug:

I've added new ATF tests:

  • Verify that masking single unrelated signal does not stop tracer from catching other signals
  • Verify that masking SIGTRAP in tracee stops tracer from catching this raised signal
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching software breakpoints
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching single step trap
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching exec() breakpoint
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching PTRACE_FORK breakpoint
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching PTRACE_VFORK breakpoint
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching PTRACE_VFORK_DONE breakpoint
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching PTRACE_LWP_CREATE breakpoint
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching PTRACE_LWP_EXIT breakpoint

2. ELF Auxiliary Vectors

The ELF file format permits to transfer additional information for a process with a dedicated container of properties, it's named ELF Auxilary Vector. Every system has its dedicated way to read this information in a debugger from a tracee. The NetBSD approach is to transfer this vector with a ptrace(2) API PIOD_READ_AUXV. Our interface shares the API with OpenBSD. I filed a bug that our interface returns vector size of 8496 bytes, while OpenBSD has constant 64 bytes. It was diagnosed and fixed by Christos Zoluas that we were incorrectly counting bits and bytes and this enlarged the data streamlined. The bug was harmless and had no known side-effects besides large chunk of zeroed data.

There is also a prepared local patch extending NetBSD platform support to read information for this vector, it's primarily required for correct handling of PIE binaries. At the moment there is no interface similar to "info auxv" to the one from GDB. Unfortunately at the current stage, this code is still unused by NetBSD. I will return to it once the Native Process Plugin is enhanced.

I've filed the mentioned bug:

I've added new ATF test:

  • Verify PT_READ_AUXV called for tracee.

What has been done in LLDB

1. Resolving executable's name with sysctl(7)

In the past the way to retrieve a specified process' executable path name was using Linux-compatibile feature in procfs (/proc). The canonical solution on Linux is to resolve path of /proc/$PID/exe. Christos Zoulas added in DTrace port enhancements a solution similar to FreeBSD to retrieve this property with sysctl(7).

This new approach removes dependency on /proc mounted and Linux compatibility functionality.

Support for this has been submitted to LLDB and merged upstream:

2. Real-Time Signals

The key feature of the POSIX standard with Asynchronous I/O is to support Real-Time Signals. One of their use-cases is in .NET debugging facilities. Support for this set of signals was developed during Google Summer of Code 2016 by Charles Cui and reviewed and committed by Christos Zoulas.

I've extended the LLDB capabilities for NetBSD to recognize these signals in the NetBSDSignals class.

Support for this has been submitted to LLDB and merged upstream:

3. Conflict removal with system-wide six.py

The transition from Python 2.x to 3.x is still ongoing and will take a while. The current deadline support for the 2.x generation has been extended to 2020. One of the ways to keep both generations supported in the same source-code is to use the six.py library (py2 x py3 = 6.py). It abstracts commonly used constructs to support both language families.

The issue for packaging LLDB in NetBSD was to install this tiny library unconditionally to a system-wide location. There were several solutions to this approach:

  • drop Python 2.x support,
  • install six.py into subdirectory,
  • make an installation of six.py conditional.

The first solution would turn discussion into flamewar, the second one happened to be too difficult to be properly implemented as the changes were invasive and Python is used in several places of the code-base (tests, bindings...). The final solution was to introduce a new CMake option LLDB_USE_SYSTEM_SIX - disabled by default to retain the current behavior.

To properly implement LLDB_USE_SYSTEM_SIX, I had to dig into installation scripts combined in CMake and Python files. It wasn't helping that Python scripts were reinventing getopt(3) functionality.. and I had to alter it in order to introduce a new option --useSystemSix.

Support for this has been submitted to LLDB and merged upstream:

4. Do not pass non-POD type variables through variadic function

There was a long standing local patch in pkgsrc, added by Tobias Nygren and detected with Clang.

According to the C++11 standard 5.2.2/7:

Passing a potentially-evaluated argument of class type having a non-trivial copy constructor, a non-trivial move constructor, or a non-trivial destructor, with no corresponding parameter, is conditionally-supported with implementation-defined semantics.

A short example to trigger similar warning was presented by Joerg Sonnenberg:

#include <string>
#include <cstdarg>

void f(std::string msg, ...) {
  va_list ap;
  va_start(ap, msg);
}

This code compiled against libc++ gives:

test.cc:6:3: error: cannot pass object of non-POD type 'std::string' (aka 'basic_string<char, char_traits<char>,

allocator<char> >') through variadic function; call will abort at runtime [-Wnon-pod-varargs]

Support for this has been submitted to LLDB and merged upstream:

5. Add NetBSD support in Host::GetCurrentThreadID

Linux has a very specific thread model, where process is mostly equivalent to native thread and POSIX thread - it's completely different on other mainstream general-purpose systems. That said fallback support to translate pthread_t on NetBSD to retrieve the native integer identifier was incorrect. The proper NetBSD function to retrieve light-weigth process identification is to call _lwp_self(2).

Support for this has been submitted to LLDB and merged upstream:

6. Synchronize PlatformNetBSD with Linux

The old PlatformNetBSD code was based on the FreeBSD version. While the FreeBSD current one is still similar to the one from a year ago, it's inappropriate to handle a remote process plugin approach. This forced me to base refreshed code on Linux.

After realizing that PlatformPlugin on POSIX platforms suffers from code duplication, Pavel Labath helped out to eliminate common functions shared by other systems. This resulted in a shorter patch synchronizing PlatformNetBSD with Linux, this step opened room for FreeBSD to catch up.

Support for this has been submitted to LLDB and merged upstream:

7. Transform ProcessLauncherLinux to ProcessLauncherPosixFork

It is UNIX specific that signal handlers are global per application. This introduces issues with wait(2)-like functions called in tracers, as these functions tend to conflict with real-life libraries, notably GUI toolkits (where SIGCHLD events are handled).

The current best approach to this limitation is to spawn a forkee and establish a remote connection over the GDB protocol with a debugger frontend. ProcessLauncherLinux was prepared with this design in mind and I have added support for NetBSD. Once FreeBSD will catch up, they might reuse the same code.

Support for this has been submitted to LLDB and merged upstream:

8. Document that LaunchProcessPosixSpawn is used on NetBSD

Host::GetPosixspawnFlags was built for most POSIX platforms - however only Apple, Linux, FreeBSD and other-GLIBC ones (I assume Debian/kFreeBSD to be GLIBC-like) were documented. I've included NetBSD to this list..

Support for this has been submitted to LLDB and merged upstream:

  • Document that LaunchProcessPosixSpawn is used on NetBSD committed r293770

9. Switch std::call_once to llvm::call_once

There is a long-standing bug in libstdc++ on several platforms that std::call_once is broken for cryptic reasons. This motivated me to follow the approach from LLVM and replace it with homegrown fallback implementation llvm::call_once.

This change wasn't that simple at first sight as the original LLVM version used different semantics that disallowed straight definition of non-static once_flag. Thanks to cooperation with upstream the proper solution was coined and LLDB now works without known regressions on libstdc++ out-of-the-box.

Support for this has been submitted to LLVM, LLDB and merged upstream:

10. Other enhancements

I a had plan to push more code in this milestone besides the mentioned above tasks. Unfortunately not everything was testable at this stage.

Among the rescheduled projects:

  • In the NetBSD platform code conflict removal in GetThreadName / SetThreadName between pthread_t and lwpid_t. It looks like another bite from the Linux thread model. Proper solution to this requires pushing forward the Process Plugin for NetBSD.
  • Host::LaunchProcessPosixSpawn proper setting ::posix_spawnattr_setsigdefault on NetBSD - currently untestable.
  • Fix false positives - premature before adding more functions in NetBSD Native Process Plugin.

On the other hand I've fixed a build issue of one test on NetBSD:

Plan for the next milestone

I've listed the following goals for the next milestone.

  • mark exect(3) obsolete in libc
  • remove libpthread_dbg(3) from the base distribution
  • add new API in ptrace(2) PT_SET_SIGMASK and PT_GET_SIGMASK
  • add new API in ptrace(2) to resume and suspend a specific thread
  • finish switch of the PT_WATCHPOINT API in ptrace(2) to PT_GETDBREGS & PT_SETDBREGS
  • validate i386, amd64 and Xen proper support of new interfaces
  • upstream to LLDB accessors for debug registers on NetBSD/amd64
  • validate PT_SYSCALL and add a functionality to detect and distinguish syscall-entry syscall-exit events
  • validate accessors for general purpose and floating point registers

Post mortem

FreeBSD is catching up after NetBSD changes, e.g. with the following commit:

This move allows to introduce further reduction of code-duplication. There still is a lot of room for improvement. Another benefit for other software distributions, is that they can now appropriately resolve the six.py conflict without local patches.

These examples clearly show that streamlining NetBSD code results in improved support for other systems and creates a cleaner environment for introducing new platforms.

A pure NetBSD-oriented gain is improvement of system interfaces in terms of quality and functionality, especially since DTrace/NetBSD is a quick adopter of new interfaces.. and indirectly a sandbox to sort out bugs in ptrace(2).

The tasks in the next milestone will turn NetBSD's ptrace(2) to be on par with Linux and FreeBSD, this time with marginal differences.

To render it more clearly NetBSD will have more interfaces in read/write mode than FreeBSD has (and be closer to Linux here), on the other hand not so many properites will be available in a thread specific field under the PT_LWPINFO operation that caused suspension of the process.

Another difference is that FreeBSD allows to trace only one type of syscall events: on entry or on exit. At the moment this is not needed in existing software, although it's on the longterm wishlist in the GDB project for Linux.

It turned out that, I was overly optimistic about the feature set in ptrace(2), while the basic ones from the first milestone were enough to implement basic support in LLDB.. it would require me adding major work in heuristics as modern tracers no longer want to perform guessing what might happened in the code and what was the source of signal interruption.

This was the final motivation to streamline the interfaces for monitoring capabilities and now I'm adding remaining interfaces as they are also needed, if not readily in LLDB, there is DTrace and other software that is waiting for them now. Somehow I suspect that I will need them in LLDB sooner than expected.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue to fund projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:

http://netbsd.org/donations/#how-to-donate

Posted in the wee hours of Monday night, February 14th, 2017 Tags: blog

Background

I am using a sparc64 Sun Blade 2500 (silver) as a desktop machine - for my pretty light "desktop" needs. Besides the usual developer tools (editors, compilers, subversion, hg, git) and admin stuff (all text based) I need mpg123 and mserv for music queues, Gimp for image manipulation and of course Firefox.

Recently I updated all my installed pkgs to pkgsrc-current and as usual the new Firefox version failed to build.

Fortunately the issues were minor, as they all had been handled upstream for Firefox 52 already, all I needed to do was back-porting a few fixes. This made the pkg build, but after a few minutes of test browsing, it crashed. Not surprisingly this was reproducible, any web site trying to play audio triggered it. A bit surprising though: the same happened on an amd64 machine I tried next. After a bit digging the bug was easy to fix, and upstream already took the fix and committed it to the libcubeb repository.

Success!

So I am now happily editing this post using Firefox 51 on the Blade 2500.

I saw one crash in two days of browsing, but unfortunately could not (yet) reproduce it (I have gdb attached now). There will be future pkg updates certainly.

Future Obstacles

You may have read elsewhere that Firefox will start to require a working Rust compiler to build.

This is a bit unfortunate, as Rust (while academically interesting) is right now not a very good implementation language if you care about portability. The only available compiler requires a working LLVM back end, which we are still debugging. Our auto-builds produce sparc sets with LLVM, but the result is not fully working (due to what we believe being code gen bugs in LLVM). It seems we need to fix this soon (which would be good anyway, independent of the Rust issue). Besides the back end, only very recently traces of sparc64 support popped up in Rust. However, we still have a few firefox versions time to get it all going. I am optimistic.

Another upcoming change is that Cairo (currently used as 2D graphics back end, at least on sparc64) will be phased out and Skia will be the only supported software rendering target. Unfortunately Skia does (as of now) not support any big endian machine at all. I am looking for help getting Skia to work on big endian hardware in general, and sparc64 in particular.

Alternatives

Just in case, I tested a few other browsers and (so far) they all failed:
  • NetSurf
    Nice, small, has a few "tweaks" and does not yet support JavaScript good enough for many sites
  • Midori
    They call it "lightweight" but it is based on WebKit, which alone is a few times more heavy than all of Firefox. It crashes immediately at startup on sparc64 (I am investigating, but with low priority - actually I had to replace the hard disk in my machine to make enough room for the debug object files for WebKit - it takes ~20GB)

So, while it is a bit of a struggle to keep a modern browser working on my favorite odd-ball architecture, it seems we will get at least to the Firefox 52 ESR release, and that should give us enough time to get Rust working and hopefully continue with Firefox.

Posted Wednesday afternoon, February 8th, 2017 Tags: blog
Operating systems can be called monitors as they handle system calls from userland processes. A similar task is performed by debuggers as they implement monitors for traced applications and interpret various events that occurred in tracees and are messaged usually with signals to their tracers. During this month I have started a new Process Plugin within LLDB to incept NativeProcessNetBSD - copied from NativeProcessLinux - implementing basic functionality and handling all the needed events in the MonitorCallback() function. To achieve these tasks, I had to add a bunch of new ptrace(2) interfaces in the kernel to cover all that is required by LLDB monitors. The current Process Plugin for NetBSD is capable to start a process, catch all the needed events correctly and if applicable resume or step the process.

What has been done in NetBSD

1. Verified the full matrix of combinations of wait(2) and ptrace(2) in the following test-cases

  • verify basic PT_GET_SIGINFO call for SIGTRAP from tracee
  • verify basic PT_GET_SIGINFO and PT_SET_SIGINFO calls without modification of SIGINT from tracee
  • verify basic PT_GET_SIGINFO and PT_SET_SIGINFO calls with setting signal to new value
  • detect SIGTRAP TRAP_EXEC from tracee
  • verify single PT_STEP call with signal information check
  • verify that fork(2) is intercepted by ptrace(2) with EVENT_MASK set to PTRACE_FORK and reports correct signal information
  • verify that vfork(2) is intercepted by ptrace(2) with EVENT_MASK set to PTRACE_VFORK_DONE
  • verify that vfork(2) is intercepted by ptrace(2) with EVENT_MASK set to PTRACE_FORK | PTRACE_VFORK_DONE
  • verify that PTRACE_VFORK in EVENT_MASK is preserved
  • verify that PTRACE_VFORK_DONE in EVENT_MASK is preserved
  • verify that PTRACE_LWP_CREATE in EVENT_MASK is preserved
  • verify that PTRACE_LWP_EXIT in EVENT_MASK is preserved
  • verify that thread creation is intercepted by ptrace(2) with EVENT_MASK set to PTRACE_LWP_CREATE
  • verify that thread termination is intercepted by ptrace(2) with EVENT_MASK set to PTRACE_LWP_EXIT

2. GNU libstdc++ std::call_once bug investigation

The LLVM toolchain internally uses several std::call_once calls, within LLVM and LLDB codebase. The GNU libstdc++ library implements C++11 std::call_once with wrapping pthread_once(3) and a function pointer passed as a TLS (Thread Local Storage) variable. Currently this setup isn't functional with shared linking on NetBSD and apparently on few other systems and hardware platforms. This issue is still unresolved and its status is traced in PR 51139. As of now, there is a walkaround originally prepared for NetBSD, and adapted for others - namely llvm::call_once.

3. Improving documentation and other minor system parts

  • rename the SIGPOLL signal to SIGIO in the documentation of siginfo(2)
  • replace SIGPOLL references with SIGIO in comments of <sys/siginfo.h>
  • document SI_NOINFO in siginfo(2)
  • document SI_LWP in siginfo(2)
  • document SI_QUEUE in siginfo(2)
  • document SI_MESGQ in siginfo(2)
  • document TRAP_EXEC in siginfo(2)
  • document TRAP_CHLD in siginfo(2)
  • document TRAP_LWP in siginfo(2)
  • reference siginfo(2) for a SIGCHLD signal in ptrace(2)
  • remove unused macro for ATF_TP_ADD_TC_HAVE_DBREGS in src/tests/kernel/t_ptrace_wait.h
  • cleanup commented out code after revert of racy vfork(2) commit from year 2012
  • remove stub code to redefine PTRACE_FORK in ATF tests
  • ... and several microimprovements in the codebase

4. Documentation of ptrace(2) and explanation how debuggers work

  • document addr and data argument usage for PT_GET_PROCESS_STATE
  • document PT_SET_SIGINFO and PT_GET_SIGINFO
  • explain execve(2) handling and behavior, SIGTRAP & TRAP_EXEC
  • reference PaX MPROTECT restrictions for debuggers
  • explain software breakpoints handling and behavior, SIGTRAP & TRAP_BKPT
  • explain single step behavior, SIGTRAP & TRAP_TRACE
  • explain that PT_TRACE_ME does not send a SIGSTOP signal
  • list predefined MI symbols for help debuggers in port specific headers
  • document the current behavior of TRAP_CHLD
  • add more notes on PTRACE_FORK events
  • document PTRACE_VFORK and PTRACE_VFORK_DONE
  • document PTRACE_LWP_CREATE and PTRACE_LWP_EXIT
  • import HISTORY notes from FreeBSD

5. Introduction of new siginfo(2) codes for SIGTRAP

  • TRAP_EXEC - catch exec() events transforming the calling process into a new process
  • TRAP_CHLD - process child trap (fork, vfork and vfork done events)
  • TRAP_LWP - process thread traps (birth, termination)
  • TRAP_HWWPT - process hardware assisted watchpoints

6. New ptrace(2) interfaces

There were added new interfaces to the native ptrace(2) NetBSD API.

Interface to introspect and fake signal information

The PT_GET_SIGINFO interface is designed to read signal information sent to tracee. A program can also fake a signal to be passed to a debugged process, with included destination thread or the whole process. A debugger requires detailed signal information in order to distinguish the exact trap reason easily and precisely, for example whether a program stopped on a software defined breakpoint or a single step trap.

This interface is modeled after the Linux specific calls: PTRACE_GETSIGINFO and PTRACE_SETSIGINFO, but adapted for the NetBSD use-case. FreeBSD currently has no way to fake signal information.

I have modeled this interface to be most efficient in terms of determination what exact thread received the signal. Thanks to it, a process does not need to examine each thread separately and performs only a single ptrace(2) call. This makes a significant difference in a massively threaded software.

The signal information accessors introduce a new structure ptrace_siginfo:

/*
 * Signal Information structure
 */
typedef struct ptrace_siginfo {
       siginfo_t       psi_siginfo;    /* signal information structure */
       lwpid_t         psi_lwpid;      /* destination LWP of the signal
                                        * value 0 means the whole process
                                        * (route signal to all LWPs) */
} ptrace_siginfo_t;
I've included in the <sys/ptrace.h> header required file to define the siginfo_t type - <sys/siginfo.h>. This makes sure that no existing software will break during build due to a missing type definition.

Interface to monitor vfork(2) operations

Forking a process is one of the fundamental design features of a UNIX operating system. There are two basic types of forks: fork(2) and vfork(2) ones. The fork(2) one is designed to spawn a mostly independent child from parent's memory layout, however this operation is costy. In order to address it and optimize the forking operation there is vfork(2) which shares address space with its parent. From the ptrace(2) point of view, the difference between fork(2) and vfork(2) calls is whether a parent giving birth to its child is suspended waiting on child termination or exec() operation or not.

Currently there is an interface to monitor fork(2) operations - PTRACE_FORK - however NetBSD missed the vfork(2) ones. I've added two new functions: PTRACE_VFORK and PTRACE_VFORK_DONE. The former is supposed to notify why a parent gives a vfork(2)-like birth to its child and later when a child is born with vfork(2) from its parent - perfoming a handshake with two SIGTRAP signals emitted from the kernel. Once the parent is resuming it gets a notification for vfork(2) done.

These events throw SIGTRAP signal with the TRAP_CHLD property. Currently PTRACE_VFORK is a stub and it's planned to be fully implemented later.

Interface to monitor thread operations

A debugger requires an interface to monitor thread birth and termination. This is needed in order to properly track the thread list of a process and ensure that every thread has for example applied watchpoints.

I've added two events:

  • PTRACE_LWP_CREATE - to report thread births,
  • PTRACE_LWP_EXIT - to report thread termination.

This interface reuses the EVENT_MASK and PROCESS_STATE interface. It means that it shares these calls with PTRACE_FORK, PTRACE_VFORK and PTRACE_VFORK_DONE.

To achieve this goal, I've changed the following structure:

typedef struct ptrace_state {
        int     pe_report_event;
        pid_t   pe_other_pid;
} ptrace_state_t;

to

typedef struct ptrace_state {
        int     pe_report_event;
        union {
                pid_t   _pe_other_pid;
                lwpid_t _pe_lwp;
        } _option;
} ptrace_state_t;

#define pe_other_pid    _option._pe_other_pid
#define pe_lwp          _option._pe_lwp
This change keeps the size of ptrace_state_t unchanged as both pid_t and lwpid_t are defined as an int32_t-like integer. New struct form should not break existing software and be source and binary compatible with it.

I've introduced a new SIGTAP type for thread events: TRAP_LWP.

Hardware assisted watchpoints

I've introduced a few changes to the current interface. One of them is allowing to mix single-step operation with enabled hardware assisted watchpoints. The other one was added new extension pw_type to the ptrace_watchpoint structure.

6. Updated doc/TODO.ptrace entries

The current state of TODO.ptrace - after several updates - is as follows:
  • verify ppid of core dump generated with PT_DUMPCORE it must point to the real parent, not tracer
  • adapt OpenBSD regress test (regress/sys/ptrace/ptrace.c) for the ATF context
  • add new ptrace(2) calls to lock (suspend) and unlock LWP within a process
  • add PT_DUMPCORE tests in the ATF framework
  • add ATF tests for PT_WRITE_I and PIOD_WRITE_I - test mprotect restrictions
  • add ATF tests for PIOD_READ_AUXV
  • once the API for hardware watchpoints will stabilize, document it
  • add tests for the procfs interface covering all functions available on the same level as ptrace(2)
  • add support for PT_STEP, PT_GETREGS, PT_SETREGS, PT_GETFPREGS, PT_SETFPREGS in all ports
  • integrate all ptrace(2) features in gdb
  • add ptrace(2) NetBSD support in LLDB
  • add proper implementation of PTRACE_VFORK for vfork(2)-like events
  • remove exect(3) - there is no usecase for it
  • refactor pthread_dbg(3) to only query private pthread_t data, otherwise it duplicates ptrace(2) interface and cannot cover all types of threads
  • add ATF tests for SIGCHLD
  • add ATF tests for PT_SYSCALL and PT_SYSCALLEMU

Features in ELF, DWARF, CTF, DTrace are out of scope for the above list.

7. Future directions

After research and testing the current watchpoint interface, I've realized that it's impossible (impractically complicated) to pretend to have a "safe" watchpoint interface inside the kernel, as the current one isn't safe from undefined behavior even on stock amd64. I've decided to revert this code and introduce PT_GETDBREGS and PT_SETDBREGS restricted to INSECURE secure level mode. The good side of this change is that there is already part of the code needed in the kernel, I have a local draft introducing this interface and it will be easier to integrate with LLDB, as Linux and FreeBSD keep having the same interface.

What has been done in LLDB

I work on the LLDB port inside the pkgsrc-wip repository, in the lldb-netbsd package.

To summarize the changes, there were so far 84 commits in this directory. The overall result is a list of 26 patched or added files. The overall diff's length is 3539 lines.

$ wc -l patches/patch-*                                                               
      12 patches/patch-cmake_LLDBDependencies.cmake
      17 patches/patch-cmake_modules_AddLLDB.cmake
      14 patches/patch-include_lldb_Host_netbsd_HostThreadNetBSD.h
      30 patches/patch-include_lldb_Host_netbsd_ProcessLauncherNetBSD.h
      12 patches/patch-source_CMakeLists.txt
      12 patches/patch-source_Host_CMakeLists.txt
      60 patches/patch-source_Host_common_Host.cpp
      13 patches/patch-source_Host_common_NativeProcessProtocol.cpp
      21 patches/patch-source_Host_netbsd_HostThreadNetBSD.cpp
     175 patches/patch-source_Host_netbsd_ProcessLauncherNetBSD.cpp
      23 patches/patch-source_Host_netbsd_ThisThread.cpp
      24 patches/patch-source_Initialization_SystemInitializerCommon.cpp
     803 patches/patch-source_Plugins_Platform_NetBSD_PlatformNetBSD.cpp
     141 patches/patch-source_Plugins_Platform_NetBSD_PlatformNetBSD.h
      12 patches/patch-source_Plugins_Process_CMakeLists.txt
      13 patches/patch-source_Plugins_Process_NetBSD_CMakeLists.txt
    1392 patches/patch-source_Plugins_Process_NetBSD_NativeProcessNetBSD.cpp
     188 patches/patch-source_Plugins_Process_NetBSD_NativeProcessNetBSD.h
     393 patches/patch-source_Plugins_Process_NetBSD_NativeThreadNetBSD.cpp
      92 patches/patch-source_Plugins_Process_NetBSD_NativeThreadNetBSD.h
      13 patches/patch-tools_lldb-mi_MICmnBase.cpp
      13 patches/patch-tools_lldb-mi_MICmnBase.h
      13 patches/patch-tools_lldb-mi_MIDriver.cpp
      13 patches/patch-tools_lldb-mi_MIUtilString.cpp
      13 patches/patch-tools_lldb-mi_MIUtilString.h
      27 patches/patch-tools_lldb-server_CMakeLists.txt
    3539 total

1. Native Process NetBSD Plugin

I've created the initial code for the Native Process NetBSD Plugin.

  • process resume support (PT_CONTINUE)
  • process step support (PT_STEP) - currently not fully functional
  • functional callback monitor
  • functional process launch operation
  • appropriate ptrace(2) wrapper with logging capabilities
  • initial NativeThreadNetBSD code
  • setup for EVENT_MASK of a traced process (we monitor thread events)
  • preliminary code for other functions, like attach to a process and reading/writing memory of a tracee

2. The MonitorCallback function

The MonitorCallback function supports now the following events:

  • process exit and retrieve status code
  • software breakpoint (TRAP_BRKPT)
  • single step (TRAP_TRACE)
  • process image swich trap (TRAP_EXEC)
  • child traps (TRAP_CHLD) - currently detectable but disabled
  • thread trap (TRAP_LWP)
  • hardware assisted watchpoint trap (TRAP_HWWPT)

3. Other LLDB code, out of the NativeProcessNetBSD Plugin

During this work segment I've completed the following tasks:

  • created initial Native Process Plugin for NetBSD with remote debugging facilities, with skeleton copied from Linux but almost every function used so far had to be rewritten for NetBSD.
  • attached the Native Process Plugin for NetBSD to the build infrastructure.
  • disabled existing code for retrieving and updating Thread Name in NatBSD host plugin as currently not compatible with the LLDB codebase.
  • added NetBSD Process Launcher.
  • support proper native NetBSD Thread ID in Host::GetCurrentThreadID.
  • upgraded Platform NetBSD Plugin for new remote debugging plugin.

4. Automated LLDB Test Results Summary

The number of passing tests increased by 45% between devel/lldb 3.9.1 and lldb-netbsd 2017-01-21.

The above graphs renders test results for:

Example LLDB sessions

It's demo time!

Breakpoint interrupt

In this example, I'm calling a hello world application that is triggering a software breakpoint. It's implemented by embedding int3 call on amd64. The debugger is capable of catching this and resuming till correct process termination.

$ lldb ./int3 
(lldb) target create "./int3"
Current executable set to './int3' (x86_64).
(lldb) r
Hello world!
Process 29578 launched: './int3' (x86_64)
Process 29578 stopped
* thread #1, stop reason = signal SIGTRAP
    frame #0:
(lldb) c
Process 29578 resuming
Process 29578 exited with status = 0 (0x00000000)
(lldb)

Thread monitor interrupt

In this example we set trap on thread events - creation and termination. The executed program incepts a thread and terminates afterwards, this is caught by the MonitorCallback with appropriate thread list update. After the end, program terminates correctly and passes proper exit status to the debugger.

$ lldb ./lwp_create
(lldb) target create "./lwp_create"
Current executable set to './lwp_create' (x86_64).
(lldb) r
Process 27331 launched: './lwp_create' (x86_64)
Hello world!
Process 27331 stopped
* thread #1, stop reason = SIGTRAP has been caught with Process LWP Trap type
    frame #0:
  thread #2, stop reason = SIGTRAP has been caught with Process LWP Trap type
    frame #0:
(lldb) thread list 
Process 27331 stopped
* thread #1: tid = 0x0001, stop reason = SIGTRAP has been caught with Process LWP Trap type
  thread #2: tid = 0x0002, stop reason = SIGTRAP has been caught with Process LWP Trap type
(lldb) c
Process 27331 resuming
Process 27331 stopped
* thread #1, stop reason = SIGTRAP has been caught with Process LWP Trap type
    frame #0:
(lldb) thread list
Process 27331 stopped
* thread #1: tid = 0x0001, stop reason = SIGTRAP has been caught with Process LWP Trap type
(lldb) c
Process 27331 resuming
It works
Process 27331 exited with status = 0 (0x00000000)
(lldb)

Plan for the next milestone

I've listed the following goals for the next milestone.

  • fix conflict with system-wide py-six
  • add support for auxv read operation
  • switch resolution of pid -> path to executable from /proc to sysctl(7)
  • recognize Real-Time Signals (SIGRTMIN-SIGRTMAX)
  • upstream !NetBSDProcessPlugin code
  • switch std::call_once to llvm::call_once
  • add new ptrace(2) interface to lock and unlock threads from execution
  • switch the current PT_WATCHPOINT interface to PT_GETDBREGS and PT_SETDBREGS

This work was sponsored by The NetBSD Foundation.

Previous report "Summary of the ptrace(2) project".

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue to fund projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:

http://netbsd.org/donations/#how-to-donate

Posted at noon on Monday, January 23rd, 2017 Tags: blog

[Update, 2017-01-16 20:48 UTC: All issues have been resolved and NetBSD.org should be functioning normally now.]

As some of you may have noticed, the netbsd.org DNS records are broken right now. Two independent incidents caused this to happen:

  1. UC Berkeley stopped providing secondary DNS service for netbsd.org at adns1.berkeley.edu and adns2.berkeley.edu on a few hours notice, before we had time to update the NS delegations in the ORG. zone. [Update, 2017-01-16 20:48 UTC: The Berkeley network administrators have graciously restored service until we can update our NS delegations. Thanks, Berkeley—for doing this on a US holiday, no less!]
  2. The network hosting our primary nameserver ns.netbsd.de went offline at around 14:00 UTC for reasons unknown. [Update, 2017-01-16 20:48 UTC: The network and ns.netbsd.de are back up.]

We have sent a request to our registrar to update the delegations to make ns.netbsd.org primary and ns.netbsd.de secondary. We asked them to expedite the process, but owing to the weekend, when the humans who process these requests are off work, it may take another day for that to take effect, and another day after that for the cached TTLs on the old NS delegations to Berkeley's nameservers to expire.

As a provisional stop gap, if you need access to the netbsd.org zone and you can prime your DNS cache with manually entered records, you can prime it with the following records to restore service at your site:

netbsd.org.     86400   IN      NS      ns.netbsd.org.
netbsd.org.     86400   IN      NS      ns.netbsd.de.
ns.netbsd.org.  86400   IN      A       199.233.217.200
ns.netbsd.org.  86400   IN      AAAA    2001:470:a085:999::53
  

(Presumably if you are reading this you have some DNS records in the netbsd.org zone cached, but they will eventually expire and you may need some others.)

Posted late Monday afternoon, January 16th, 2017 Tags: blog
The NetBSD Foundation took part in the 2016 Google Summer of Code.

Hrishikesh Goyal worked on the project "Implement Ext4fs support in ReadOnly mode". He tackled two features of an ext4fs implementation: extents and HTree DIR read/write support. His work was committed into the NetBSD source tree in multiple commits.

charles cui worked on adding tests for better "POSIX Test Suite Compliance". This involved porting the Open Posix benchmark suite to NetBSD. Many of the tests showed missing features, and Charles worked with his mentors to improve the results. See his summary for details.

Leonardo Taccari worked on "Split debug symbols for pkgsrc builds". He already blogged about this in much detail on this blog.

A big thank you to Google for sponsoring the students to work on NetBSD, the students for working on the projects, and the mentors that were helping them along!

Posted Saturday afternoon, January 14th, 2017 Tags: blog
Add a comment
Contact | Disclaimer | Copyright © 1994-2017 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
NetBSD® is a registered trademark of The NetBSD Foundation, Inc.