Oct 2017
17 18


This page is a blog mirror of sorts. It pulls in articles from blog's feed and publishes them here (with a feed, too).

Recently, I completed a Kernel ASLR implementation for NetBSD-amd64, making NetBSD the first BSD system to support such a feature. Simply said, KASLR is a feature that randomizes the location of the kernel in memory, making it harder to exploit several classes of vulnerabilities, both locally (privilege escalations) and remotely (remote code executions).

Current design

The current design is based on a specialized kernel called the "prekern", which operates between the bootloader and the kernel itself. The kernel is compiled as a raw library with the GENERIC_KASLR configuration file, while the prekern is compiled as a static binary. When the machine boots, the bootloader jumps into the prekern. The prekern relocates the kernel at a random virtual address (VA), and jumps into it. Finally, the kernel performs some cleanup, and executes normally.

Currently, the kernel is randomized as a single block. That is to say, a random VA is chosen, and the kernel text->rodata->data sections are mapped contiguously starting from there. It has several drawbacks, but it's a first shot.

To complete this implementation, work had to be done at three levels: the bootloader, the prekern and the kernel. I committed several of the kernel and bootloader patches discreetly a few months ago, to pave some way for real changes. In the past few weeks, I changed the low-level x86 layer of the kernel and replaced several hard-coded (and sometimes magic) values by variables, in such a way that the kernel can run with a non-static memory layout. Finally, the last step was committing the prekern itself to the source tree.

Future work

  • Randomize the kernel sections independently, and intertwine them.
  • Modify several kernel entry points not to leak kernel addresses to userland.
  • Randomize the kernel heap too (which is still static for now).
  • Fix a few other things that need some more work.

How to use

All of the patches are now in NetBSD-current. Instructions on how to install and use this implementation can be found here; they are inlined below, and probably won't change in the future.

Make sure you have a v5.11 bootloader installed. If you don't, build and install a new bootloader:

    $ cd /usr/src/sys/arch/i386/stand/boot
    $ make
    # cp biosboot/boot /
Build and install a KASLR kernel:
    $ cd /usr/src
    $ ./build.sh -u kernel=GENERIC_KASLR
    # cp /usr/obj/sys/arch/amd64/compile/GENERIC_KASLR/netbsd /netbsd_kaslr
Finally, build and install a prekern:
    $ cd /usr/src/sys/arch/amd64/stand/prekern
    $ make
    # cp prekern /prekern
Reboot your machine. In the boot prompt, enter:
    > pkboot netbsd_kaslr
The system will boot with no further user interaction. Should you encounter any regression or unexpected behavior, please report it immediately to tech-kern.

Note that you can still boot a static kernel, by typing as usual:

    > boot netbsd


This KASLR implementation will be available starting from NetBSD 9. Once it is stabilized, it may be backported to NetBSD 8. Until then, feel free to test it!

Posted terribly early Thursday morning, October 12th, 2017 Tags: blog

Let me tell you about my experience at EuroBSDcon 2017 in Paris, France. We will see what was presented during the NetBSD developer summit on Friday and then we will give a look to all of the NetBSD and pkgsrc presentations given during the conference session on Saturday and Sunday. Of course, a lot of fun also happened on the "hall track", the several breaks during the conference and the dinners we had together with other *BSD developers and community! This is difficult to describe and I will try to just share some part of that with photographs that we have taken. I can just say that it was a really beautiful experience, I had a great time with others and, after coming back home... ...I miss all of that! :) So, if you have never been in any BSD conferences I strongly suggest you to go to the next ones, so please stay tuned via NetBSD Events. Being there this is probably the only way to understand these feelings!

Thursday (21/09): NetBSD developers dinner

Arriving in Paris via a night train from Italy I literally sleep-walked through Paris getting lost again and again. After getting in touch with other developers we had a dinner together and went sightseeing for a^Wseveral beers!

Friday (22/09): NetBSD developers summit

On Friday morning we met for the NetBSD developers summit kindly hosted by Arolla.

Photograph of the NetBSD develepors summit
From left to right: alnsn, sborrill; abhinav; uwe and leot; christos, cherry, ast and bsiegert; martin and khorben.

The devsummit was moderated by Jörg (joerg) and organized by Jean-Yves (jym).

NetBSD on Google Compute Engine -- Benny Siegert (bsiegert)

After a self-presentation the devsummit presentations session started with the talk presented by Benny (bsiegert) about NetBSD on Google Compute Engine.

Benny first introduced Google Compute Engine (GCE) and then started describing how to run NetBSD on it.

At the moment there are no official NetBSD images and so users need to create their own. However, netbsd-gce script completely automatize this process that:

  • uses Anita to stage an installation in QEMU
  • adjust several tweaks to ensure that networking and storage will work on GCE
  • packs the image into a .tar.gz file

The .tar.gz image then just need to be uploaded to a Cloud Storage bucket, create a GCE image from it and then launch VMs based on that image.

He also discussed about GCE instance metadata, several problems founds and how they were fixed (it's better to use NetBSD 8_BETA or -current!) and some future works.

For more information slides (PDF) of the talk are also available.

Scripting DDB with Forth -- Valery Ushakov (uwe)

Valery (uwe) presented a talk about Scripting DDB with Forth. It was based on a long story and actually the first discussion about it appeared on tech-kern@ mailing list in his Scripting DDB in Forth? thread (ddb(4) is the NetBSD in-kernel debugger).

He showed how one can associate forth commands/conditions with ddb breakpoints. He used "pid divisible by 3" as an example of condition for a breakpoint set in getpid(2) system call:

db{0}> forth
ok : field   create , does> @ + ;
ok #300 field lwp>l_proc
ok #120 field proc>p_pid
ok : getpid   curlwp lwp>l_proc @ proc>p_pid @ ;
ok : checkpid   getpid dup ." > PID IS " . cr   3 mod 0= ;
ok bye
-- STACK: <empty>
db{0}> break sys_getpid_with_ppid
db{0}> command . = checkpid
db{0}> c

...and then on a shell:

# (:)
fatal breakpoint trap in supervisor mode
trap type 1 code 0 eip 0xc090df89 cs 0x8 eflags 0x246 cr2 0xad8ef2c0 ilevel 0 esp 0xc0157fbd
curlwp 0xc2b5c2c0 pid 798 lid 1 lowest kstack 0xdabb42c0
> PID IS 798
0xffffffff   -1
Breakpoint in pid 798.1 (ksh) at        netbsd:sys_getpid_with_ppid: pushl %ebp
db{0}> c
# (:)
fatal breakpoint trap in supervisor mode
trap type 1 code 0 eip 0xc090df89 cs 0x8 eflags 0x246 cr2 0xad8ef2c0 ilevel 0 esp 0xc0157fbd
curlwp 0xc2b5c2c0 pid 823 lid 1 lowest kstack 0xdabb42c0
> PID IS 823
0x00000000  0
Command returned 0

If you are more interested in this presentation I strongly suggest to also give a look to uwe's forth Mercurial repository.

News from the version control front -- Jörg Sonnenberger (joerg)

The third presentation of the devsummit was a presentation about the recent work done by Jörg (joerg) in the VCS conversions.

Jörg started the presentation discussing about the infrastructure used for the CVS -> Fossil -> Git conversion and CVS -> Fossil -> Mercurial conversion.

It's worth also noticing that the Mercurial conversion is fully integrated and is regularly pushed to Bitbucket and src repository pushed some scalability limits to Bitbucket!

Mercurial performance were also compared to the Git ones in details for several operations.

A check list that compared the current status of the NetBSD VCS migration to the FreeBSD VCS wiki one was described and then Jörg discussed the pending work and answered several questions in the Q&A.

For more information please give a look to the joerg's presentation slides (HTML). If you would like to help for the VCS migration please also get in touch with him!

Afternoon discussions and dinner

After the lunch we had several non-scheduled discussions, some time for hacking, etc. We then had a nice dinner together (it was in a restaurant with a very nice waiter who always shouted after every order or after accidently dropping and crashing dishes!, yeah! That's probably a bit weird but I liked that attitude! :)) and then did some sightseeing and had a beer together.

Photograph of the Friday dinner, taken by Christos
From left to right: uwe, bad, ast, leot, martin, abhinav, sborrill, alnsn, spz.

Photograph of the Friday dinner, taken by Abhinav
From left to right: uwe, bad, ast, christos, leot, martin, sborrill, alnsn, spz.

Saturday (23/09): First day of conference session and Social Event

A Modern Replacement for BSD spell(1) -- Abhinav Upadhyay (abhinav)

Abhinav (abhinav) presented his work on the new spell(1) implementation he's working (that isn't just a spell replacement but also a library that can be used by other programs!).

He described the current limitations of old spell(1) (to get an idea please give a look to bin/48684), described the project goals of the new spell(1), additions to /usr/share/dict/words, digged a bit in the implementation and discussed several algorithms used and then provided a performance comparison with other popular free software spell checkers (aspell, hunspell and ispell).

He also showed an interactive demo of the new spell(1) in-action integrated with a shell for auto-completion and spell check.

If you would like to try it please give a look to nbspell Git repository that contains the code and dicts for the new spell(1)!

Video recording (YouTube) of the talk and slides (PDF) are also available!

Portable Hotplugging: NetBSD's uvm_hotplug(9) API development -- Cherry G. Mathew (cherry)

Cherry (cherry) presented recent work done with Santhosh N. Raju (fox) about uvm_hotplug(9).

The talk covered most "behind the scenes" work: how TDD (test driven development) was used, how uvm_hotplug(9) was designed and implemented (with comparisons to the old implementation), interesting edge cases during the development and how atf(7) was used to do performance testing.

It was very interesting to learn how Cherry and Santhosh worked on that and on the conclusion Cherry pointed out the importance of using existing Software Engineering techniques in Systems Programming.

Video recording (YouTube) and slides (PDF) of the talk are also available!

Hardening pkgsrc -- Pierre Pronchery (khorben)

Pierre (khorben) presented a talk about recent pkgsrc security features added in the recent months (and most of them also active on the just released pkgsrc-2017Q3!).

He first introduced how security management and releng is handled in pkgsrc, how to use pkg_admin(1) fetch-pkg-vulnerabilities and audit commands, etc.

Then package signatures (generation, installation) and recent hardening features in pkgsrc were discussed in details, first introducing them and then how pkgsrc handles them:

  • SSP: Stack Smashing Protection (enabled via PKGSRC_USE_SSP in mk.conf)
  • Fortify (enabled via PKGSRC_USE_FORTIFY in mk.conf)
  • Stack check (enabled via PKGSRC_USE_STACK_CHECK in mk.conf)
  • Position-Independent Executables (PIE) (enabled via PKGSRC_MKPIE in mk.conf)
  • RELRO and BIND_NOW (enabled via PKGSRC_USE_RELRO in mk.conf)

Challenges for each hardening features and future works were discussed.

For more information video recording (YouTube) and slides (PDF) of the talk are available. A good introduction and reference for all pkgsrc hardening features is the Hardening packages wiki page.

Reproducible builds on NetBSD -- Christos Zoulas (christos)

Christos (christos) presented the work about reproducible builds on NetBSD.

In his talk he first provided a rationale about reproducible builds (to learn more please give a look to reproducible-builds.org!), he then discussed about the NetBSD (cross) build process, the current status and build variations that are done in the tests.reproducible-builds.org build machines.

Then he provided and described several sources of difference that were present in non-reproducible builds, like file-system timestamps, parallel builds headaches due directory/build order, path normalization, etc. For each of them he also discussed in details how these problems were solved in NetBSD.

In the conclusion the status and possible TODOs were also discussed (please note that both -current and -8 are all built with reproducible flags (-P option of build.sh)!)

Video recording (YouTube) of Christos' talk is available. Apart the resources discussed above a nice introduction to reproducible builds in NetBSD is also the NetBSD fully reproducible builds blog post written by Christos last February!

Social event

The social event on Saturday evening took place on a boat that cruised on the Seine river.

It was a very nice and different way to sightsee Paris, eat and enjoy some drinks and socialize and discuss with other developers and community.

Photograph from the boat, taken by Martin.

Sunday (24/09): Second day of conference session

The school of hard knocks - PT1 -- Sevan Janiyan (sevan)

Sevan (sevan) presented a talk about several notes and lessons learnt whilst running tutorials to introduce NetBSD at several events (OSHUG #46 and OSHUG #57 and #58) and experiences from past events (Chiphack 2013).

He described problems a user may experience and how NetBSD was introduced, in particular trying to avoid the steep learning curve involved when experimenting with operating systems as a first step, exploring documentation/source code, cross-building, scripting in high-level programming languages (Lua) and directly prototyping and getting pragmatic via rump.

Video recording (YouTube) of Sevan's talk and slides (HTML) are available.

The LLDB Debugger on NetBSD -- Kamil Rytarowski (kamil)

Kamil (kamil) presented a talk about the recent LLDB debugger and a lot of other related debuggers (but also non-strictly-related-to-debugging!) works he's doing in the last months.

He first introduced debugging concepts in general, provided several examples and then he started discussing LLDB porting to NetBSD.

He then discussed about ptrace(2) and other introspection interfaces, the several improvements done and tests added for ptrace(2) in atf(7).

He also discussed about tracking LLDB's trunk (if you are more curious please give a look to wip/llvm-git, wip/clang-git, wip/lldb-git packages in pkgsrc-wip!) and about LLVM sanitizers and their current status in NetBSD.

In the conclusion he also discussed various TODOs in these areas.

Video recording (YouTube) and slides (HTML) of Kamil's talk are available. Kamil also regularly write status update blog posts on blog.NetBSD.org and tech-toolchain@ mailing list, so please stay tuned! :)

What's in store for NetBSD 8.0? -- Alistair Crooks (agc)

Alistair (agc) presented a talk about what we will see in NetBSD 8.0.

He discussed about new hardware supported (really "new", not new "old" hardware! Of course also support for VAXstation 4000 TURBOchannel USB and GPIO is actually new hardware as well! :)), LLVM/Clang, virtualization, PGP signing, updated utilities in NetBSD, new networking features (e.g. bouyer's sockcan implementation), u-boot, dtrace(1), improvements and new ports testing, reproducible builds, FDT (Flattened Device Tree) and a lot of other news!

The entire presentation was done using the Socratic method (Q&A) and it was very interactive and nice!

Video recording (YouTube) and slides (PDF) of Alistair's talk are available.

Sunday dinner

After the conference we did some sightseeing in Paris, had a dinner together and then enjoyed some beers!

Photograph of the Sunday dinner, taken by Martin.
On the left side: abhinav, ast, seb, christos
On the right side: leot, Riastradh, uwe, sevan, agc, sborrill

Photograph of the Sunday dinner, taken by Abhinav.
On the left side: martin, ast, seb, christos
On the right side: leot, Riastradh, uwe, sevan, agc, sborrill


It was a very nice weekend and conference. It is worth to mention that EuroBSDcon 2017 was the biggest BSD conference (more than 300 people attended it!).

I would like to thank the entire EuroBSDcon organising committee (Baptiste Daroussin, Antoine Jacoutot, Jean-Sébastien Pédron and Jean-Yves Migeon), EuroBSDcon programme commitee (Antoine Jacoutot, Lars Engels, Ollivier Robert, Sevan Janiyan, Jörg Sonnenberger, Jasper Lievisse Adriaanse and Janne Johansson) and EuroBSDcon Foundation for organizing such a wonderful conference!

I also would like to thank the speakers for presenting very interesting talks, all developers and community that attended the NetBSD devsummit and conference, in particular Jean-Yves and Jörg, for organizing and moderating the devsummit and Arolla that kindly hosted us for the NetBSD devsummit!

A special thanks also to Abhinav (abhinav) and Martin (martin) for photographs and locals Jean-Yves (jym) and Stoned (seb) for helping us in not get lost in Paris' rues! :)

Thank you!

Posted at lunch time on Wednesday, October 11th, 2017 Tags: blog
By vote of The NetBSD Foundation Board of Directors, the officers for the 2017 term are:

President: William J. Coldwell <billc>
Vice President: Jeremy C. Reed <reed>
Secretary: Christos Zoulas <christos>
Treasurer: Christos Zoulas <christos>
Assistant Secretary: Thomas Klausner <wiz>
Assistant Treasurer: Taylor R. Campbell <riastradh>
Posted late Wednesday evening, October 4th, 2017 Tags: blog
This year the annual EuroBSDcon event took place in Paris (September 21st-24th). There were BSD summits, tutorials and talks, including NetBSD & pkgsrc ones.


I came to Paris on Wednesday in order to participate in tutorials:

  • Thursday - DTrace for Developers: no more printfs - by George Neville-Neil
  • Friday - How to untangle your threads from a giant lock in a multiprocessor system - by Taylor R. Campbell

The NetBSD Summit

On Friday there was a NetBSD summit that gathered around 15 developers. I have not participated in it myself because of the tutorial on SMP. There were three presentations:

  • News from the version control front - by joerg,
  • Scripting DDB with Forth - by uwe,
  • NetBSD on Google Compute Engine - by bsiegert.


During Saturday and Sunday there were several NetBSD and pkgsrc related presentations:

  • A Modern Replacement for BSD spell(1) - Abhinav Upadhyay
  • Portable Hotplugging: NetBSD's uvm_hotplug(9) API development - Cherry G. Mathew
  • Hardening pkgsrc - Pierre Pronchery
  • Reproducible builds on NetBSD - Christos Zoulas
  • The school of hard knocks - PT1 - Sevan Janiyan
  • The LLDB Debugger on NetBSD - Kamil Rytarowski
  • What's in store for NetBSD 8.0? - Alistair Crooks

During the closing ceremony there was a speech by Alistair Crooks on behalf of The NetBSD Foundation.


At the conference I presented my work on "The LLDB Debugger on NetBSD":

Plan for the next milestone

Resume porting tsan and msan to NetBSD with a plan to switch back to the LLDB porting.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:


Posted at midnight, October 2nd, 2017 Tags: blog
The slate of nominees was voted for and accepted by the members of the foundation. We'd like to our team of nomcom, voting coordinator, and voting validator for putting together the slate and managing the election process.

We welcome Pierre Pronchery and Makoto Fujiwara to the 2017 Board of Directors. We look forward to working with you!

We appreciate all of the wonderful work that S.P.Zeidler and Erik Berls have done on the board during their time as directors, and are grateful for their excellent service to the foundation.

Thank you to all members participating by nominating candidates and voting on the slate.

Respectfully submitted for The NetBSD Foundation,
William J. Coldwell
Posted terribly early Wednesday morning, September 27th, 2017 Tags: blog
The NetBSD Foundation supports projects that strive to ship the best possible support in developer-oriented software. This is not exclusive to LLVM, but also includes the more traditional GNU toolchain.

Traditionally, developers in distributions like NetBSD merge 3rd party sources upstream once in a while with major release bumps, like switching from GCC 4.8.x to GCC 5.x. The time frame between the releases can be several months or a few years. This appears as a one-time effort from time to time, however with each software revision the code starts to rot on undermaintained targets. This results in local compatibility patches, which are rarely ready or applicable for upstream and thus detached from the development progress. Upstream developers tend to assume that minimal activity from such projects is a result of having no users and not verifying new code on them.

A good way to improve the situation and ensure quality of software that would shorten developers' time and cost, to deepen relations between NetBSD developers and upstream 3rd party software is to attach a build cluster node within the testing infrastructure.

The shortcoming of this approach is that it requires hardware, bandwidth and admin maintenance. The advantage is closer and better support for the NetBSD platform directly from the 3rd party software developers and the immediate detection of regressions, further reducing development time.

After the process of restoration the NetBSD support within the GDB and binutils projects, there is a new member in the GDB's cluster farm that verifies correct build status on NetBSD/amd64. This bot is hosted within The NetBSD Foundation's internal infrastructure.

The immediate follow up is to turn on --enable-targets=all, which will build all the available backends. Only a few more patches are needed to achieve this milestone.

Next steps involve extending this bot to verify other projects within the shared binutils-gdb repository. This includes GNU binutils itself, ld, gold (after adding appropriate platform support), gas, sim and gprof.

The ultimate goal is to enable execution of all tests for each new binutils-gdb commit in the upstream repository. This must be preceded by accomplishing the ongoing contracted task sponsored by The NetBSD Foundation - to port the LLDB debugger to NetBSD, as the LLVM debugger opens a door for new software from the same field.

Posted late Thursday evening, September 14th, 2017 Tags: blog
Joerg Sonnenberger has announced a new set of mirrored repositories.

You can find Mercurial versions of src, pkgsrc and xsrc under


The same rules as for the fossil and github repositories apply, i.e. there may be occasional glitches and if it becomes too bad, they might be recreated from scratch.

See more information in the posted thread to tech-repository.

Posted Friday night, September 1st, 2017 Tags: blog
This month I've finally finished upstreaming NetBSD support in ASan and UBsan. For better coverage of the sanitizers and on user request I've ported libFuzzer and SafeStack. There are mutual dependencies between the compiler-rt features. NetBSD after sorting out msan and tsan shall get all the remaining ones enabled. This is open topic after finishing LLDB. I have also prepared better ground for the coming work on ptrace(2) enhancements with the removal of the filesystem tracing (/proc/#/ctl).


The majority of the work has been done in the LLVM projects.

The developed features are not production ready and they will need productization in the future. There are still issues with paths mismatch ("netbsd" vs "netbsd8.99" vs "netbsd8.99.1") when looking for NetBSD specific support for the compiler-rt features. There is also a need for integration with pkgsrc, as not everything behaves sanely (conflicts with wrappers). The tools are also restricted to be built with the Clang compiler, as GCC support is currently broken. I also noted that the sanitizers behave wrongly in the standalone build (out of the LLVM sources).

I expect to sort out the mentioned problems after finishing LLDB.


There is ongoing discussion with the LLVM community about new JIT API that will be compatible with NetBSD PaX MPROTECT. There have been developed and introduced cleanups in the code (like better error handling templates) in order to prepare a draft of new API.


All local code for ASan has been merged upstream. This includes NetBSD patches in LLVM, Clang and compiler-rt.

All but two (one on i386 version and the other on amd64) tests (check-asan) pass.


Similarly to ASan, UBsan has been fully upstreamed. All tests (check-ubsan) pass.


SafeStack is a software security hardening technique that creates two stacks: one for data that needs to be kept safe, such as return addresses and function pointers; and an unsafe stack for everything else.

With PaX ASLR (Address Space Layout Randomization) and PaX MPROTECT (mprotect(2) restrictions) SafeStack is an excellent candidate for inclusion in pkgsrc.

Core programs could be hardened as well, but the shortcoming of SafeStack for basesystem utilities is pulling in additional dependencies like libpthread on every executable.

Using SafeStack adds marginal overhead.


Citing the project page, LibFuzzer is an in-process, coverage-guided, evolutionary fuzzing engine.

LibFuzzer is linked with the library under test, and feeds fuzzed inputs to the library via a specific fuzzing entry point (aka 'target function'); the fuzzer then tracks which areas of the code are reached, and generates mutations on the corpus of input data in order to maximize the code coverage. The code coverage information for libFuzzer is provided by LLVM's SanitizerCoverage instrumentation.

This functionality still requires more sanitizers to get aboard and is now of restricted functionality.


Part of the TSan code has been upstreamed. However, a functional port isn't finished yet.

The current issues are: proper process memory map handling and NetBSD specific setjmp(3)-like functions support.


I was also working on LSan. This sanitizer already builds and appears to be quite completed, however there is work needed for the implementation of StopTheWorld() function to self-introspect the process and threads. I'm researching a new kernel API for this purpose, but it might wait till the end of LLDB porting.


So far I have not been working on the MSan specific bits. The majority of the code has been upstreamed for this sanitizer in the common sanitizer framework, the proper handling of the NetBSD specific process map is still to be done.


The profile library is used to collect coverage information. It already passes most of the tests, however it's not turned on, as upstream requested additional time to be spent on the issues and it's not a priority right now.

NetBSD kernel

I've removed the filesystem tracing feature.

This is a legacy interface from 4.4BSD, and it was introduced to overcome shortcomings of ptrace(2) at that time, which are no longer relevant (performance). Today /proc/#/ctl offers a narrow subset of ptrace(2) commands and is not applicable for modern applications use beyond simplistic tracing scenarios.

This removal simplified kernel internals. Users are still able to use all the other /proc files.

This change doesn't affect other procfs files and Linux compat features within mount_procfs(8). /proc/#/ctl isn't available on Linux.

Plan for the next milestone

This month I will not work on a new development and I will focus on relax and taking part in EuroBSDCon in Paris. I will speak about the LLDB porting to NetBSD.

Long-term goals are finishing the basis sanitizers (msan, tsan) and switching back to LLDB porting. The sanitizers will be used to develop and debug the LLVM debugger. There is also integration with sanitizers in LLDB.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:


Posted Friday night, September 1st, 2017 Tags: blog

In this blog post series I will discuss about SUBPACKAGES work done during Google Summer of Code 2017.

In this first part I'll briefly introduce what are SUBPACKAGES, why and when can be useful and finally we'll give a quick look to a trivial pkgsrc package that uses them. At the end we'll also dive a bit on parts of the pkgsrc infrastructure that needed to be adjusted for implementing that.


SUBPACKAGES (on some package systems they are known as multi-packages, but this term for pkgsrc is already used by packages that can be built against several versions (e.g. Python, PHP, Ruby packages)) consist in generating multiple binary packages from a single pkgsrc package. For example, from a pkgsrc package - local/frobnitzem - we will see how to generate three separate binary packages: frobnitzem-foo, frobnitzem-bar and frobnitzem-baz.

This can be useful to separate several components of binary packages (and avoid to run the extract and configure phase two times!), for debugpkgs (so that all *.debug files containing debug symbols are contained in a separate -debugpkg package that can be installed only when it is needed), etc..

A simple SUBPACKAGES package: frobnitzem!

To understand how SUBPACKAGES works and can be useful let's start to see an example of it in practice: frobnitzem.

frobnitzem is a trivial package that just install three scripts in ${PREFIX}/bin, let's see it:

% cd pkgsrc/local/frobnitzem
% tree
|-- Makefile
`-- files
    `-- frobnitzem
        |-- frobnitzem-bar
        |-- frobnitzem-baz
        `-- frobnitzem-foo

2 directories, 6 files
% find . -type f | xargs tail -n +1
==> ./Makefile <==
# $NetBSD$

DISTNAME=	frobnitzem-0
DISTFILES=	# empty

MAINTAINER=	leot%NetBSD.org@localhost
HOMEPAGE=	http://netbsd.org/~leot/gsoc2017-diary/
COMMENT=	Simple subpackages example
LICENSE=	public-domain

FILESDIR=	${.CURDIR}/../../local/frobnitzem/files

WRKSRC=		${WRKDIR}/frobnitzem


	${CP} -r ${FILESDIR}/frobnitzem ${WRKDIR}

	${INSTALL_SCRIPT} ${WRKSRC}/frobnitzem-foo ${DESTDIR}${PREFIX}/bin
	${INSTALL_SCRIPT} ${WRKSRC}/frobnitzem-bar ${DESTDIR}${PREFIX}/bin
	${INSTALL_SCRIPT} ${WRKSRC}/frobnitzem-baz ${DESTDIR}${PREFIX}/bin

.include "../../mk/bsd.pkg.mk"

==> ./files/frobnitzem/frobnitzem-bar <==

echo "bar"

==> ./files/frobnitzem/frobnitzem-baz <==

echo "baz"

==> ./files/frobnitzem/frobnitzem-foo <==

echo "foo"

==> ./PLIST <==
@comment $NetBSD$

==> ./DESCR <==
frobnitzem, collection of foo, bar, baz scripts.

(Or, a bit more seriously this is just a very simple package to
test subpackages support!)

Nothing fancy, just three simple scripts, frobnitzem-{bar,baz,foo} that will respectively print to the standard output bar, baz and foo. Let's build and install the frobnitzem package:

% make install
===> Installing dependencies for frobnitzem-0                       
===> Overriding tools for frobnitzem-0                              
===> Extracting for frobnitzem-0  
===> Installing for frobnitzem-0  
===> Installing binary package of frobnitzem-0                      

And now let's try scripts installed as part of the frobnitzem package:

% foreach w (bar baz foo)
... frobnitzem-$w
... end

Okay, as we expected. Despite frobnitzem-{foo,bar,baz} don't do anything particularly useful we can split the frobnitzem-0 package in three separate subpackages: frobnitzem-foo-0, frobnitzem-bar-0 and frobnitzem-baz-0 (they provides different functionalities and can also coexist if they're in separated binary packages).

To do that we need to slighty adjust Makefile, split the PLIST in PLIST.{foo,bar,baz} (one for each separate subpackage), split the DESCR in DESCR.{foo,bar,baz}. So, at the end in local/frobnitzem we'll have:

% tree
|-- DESCR.bar
|-- DESCR.baz
|-- DESCR.foo
|-- Makefile
|-- PLIST.bar
|-- PLIST.baz
|-- PLIST.foo
`-- files
    `-- frobnitzem
        |-- frobnitzem-bar
        |-- frobnitzem-baz
        `-- frobnitzem-foo

2 directories, 10 files

Splitting DESCR and PLIST

DESCR and PLIST splits are straightforward. We just provide a separate DESCR.<spkg> for each subpackage, e.g. for the foo subpackage:

% cat DESCR.foo
frobnitzem, collection of foo, bar, baz scripts.

(Or, a bit more seriously this is just a very simple package to
test subpackages support!)

This package provide the foo functionalities.

Similarly, regarding PLISTs, we just provide a separate PLIST.<spkg> for each subpackage, e.g. for the foo subpackage:

% cat PLIST.foo
@comment $NetBSD$

Makefile changes

In Makefile we'll need to list all SUBPACKAGES hence we'll add the following line as first paragraph:

SUBPACKAGES=	foo bar baz

We'll then need to define a PKGNAME.<spkg> for each subpackages:

PKGNAME.foo=    frobnitzem-foo-0
PKGNAME.bar=    frobnitzem-bar-0
PKGNAME.baz=    frobnitzem-baz-0

...and similarly COMMENT variable should be defined for each subpackage via COMMENT.<spkg>:

COMMENT.foo=    Simple subpackages example (foo)
COMMENT.bar=    Simple subpackages example (bar)
COMMENT.baz=    Simple subpackages example (baz)

To recap here how we have adjusted Makefile, all the other lines rest unchanged:

% sed '/LICENSE/q' < Makefile
# $NetBSD$

SUBPACKAGES=    foo bar baz

DISTNAME=       frobnitzem-0
PKGNAME.foo=    frobnitzem-foo-0
PKGNAME.bar=    frobnitzem-bar-0
PKGNAME.baz=    frobnitzem-baz-0
MASTER_SITES=   # empty
DISTFILES=      # empty
CATEGORIES=     local

MAINTAINER=     leot%NetBSD.org@localhost
HOMEPAGE=       http://netbsd.org/~leot/gsoc2017-diary/
COMMENT.foo=    Simple subpackages example (foo)
COMMENT.bar=    Simple subpackages example (bar)
COMMENT.baz=    Simple subpackages example (baz)
LICENSE=        public-domain

Finally we can install it^Wthem! The usual make install will generate three binary packages (frobnitzem-foo-0.tgz, frobnitzem-bar-0.tgz, frobnitzem-baz-0.tgz) and install all of them:

% make install
===> Installing dependencies for frobnitzem-0
===> Overriding tools for frobnitzem-0
===> Extracting for frobnitzem-0
===> Installing for frobnitzem-0
=> Creating binary package /home/leot/repos/netbsd-github/pkgsrc/packages/All/frobnitzem-foo-0.tgz
=> Creating binary package /home/leot/repos/netbsd-github/pkgsrc/packages/All/frobnitzem-bar-0.tgz
=> Creating binary package /home/leot/repos/netbsd-github/pkgsrc/packages/All/frobnitzem-baz-0.tgz
===> Installing binary package of frobnitzem-foo-0
===> Installing binary package of frobnitzem-bar-0
===> Installing binary package of frobnitzem-baz-0

Now we can try them and use pkg_info(1) to get some information about them:

% frobnitzem-foo
% pkg_info -Fe /usr/pkg/bin/frobnitzem-foo
% pkg_info frobnitzem-baz
Information for frobnitzem-baz-0:

Simple subpackages example (baz)

frobnitzem, collection of foo, bar, baz scripts.

(Or, a bit more seriously this is just a very simple package to
test subpackages support!)

This package provide the baz functionalities.

% pkg_info -L frobnitzem-bar
Information for frobnitzem-bar-0:


So we can see that make install actually installed three different binary packages.

To deinstall all SUBPACKAGES we can run make deinstall in the local/frobnitzem directory (that will remove all subpackages) or we can just manually invoke pkg_delete(1).

An high-level look at how SUBPACKAGES support is implemented

Most of the changes needed are in mk/pkgformat/pkg/ hierarchy (previously known as mk/flavour and then renamed and generalized to other package formats during Anton Panev's Google Summer of Code 2011).

The code in mk/pkgformat/${PKG_FORMAT}/ handle the interaction of pkgsrc with the particular ${PKG_FORMAT}, e.g. for pkg populate meta-data files used by pkg_create(1), install/delete packages via pkg_add(1), and pkg_delete(1), etc.

For more information mk/pkgformat/README is a good introduction to pkgformat hierarchy.

Most of the changes done respect the following template:

.if !empty(SUBPACKAGES)
.  for _spkg_ in ${SUBPACKAGES}
[... code that handles SUBPACKAGES case ...]
.  endfor
.else   # !SUBPACKAGES
[... existing (and usually completely unmodified) code ...]

In particular, in mk/pkgformat/pkg/ targets were adjusted to create/install/deinstall/etc. all subpackages.

Apart mk/pkgformat other changes were needed in mk/install/install.mk in order to adjust the install phase for SUBPACKAGES.

Regarding PLIST.<spkg> handling mk/plist/plist.mk needed some adjustments to honor each PLIST per-subpackage.

mk/bsd.pkg.mk needed to be adjusted too in order to honor several per-subpackage variables (the *.<spkg> ones) and per-subpackage DESCR.<spkg>.


In this first part of this blog post series we have seen what are SUBPACKAGES, when and why they can be useful.

We have then seen a practical example of them taking a very trivial package and learned how to "subpackage-ify" it.

Then we have described - from an high-level perspective - the changes needed to the pkgsrc infrastructure for the SUBPACKAGES features that we have used. If you are more interested in them please give a look to the pkgsrc debugpkg branch that contains all work done described in this blog post.

In the next part we will see how to handle *DEPENDS and buildlink3 inclusion for subpackages.

I would like to thanks Google for organizing Google Summer of Code, the entire The NetBSD Foundation and in particular my mentors Taylor R. Campbell, William J. Coldwell and Thomas Klausner for providing precious guidance during these three months. A special thank you also to Jörg Sonnenberger who provided very useful suggestions. Thank you!

Posted late Thursday afternoon, August 31st, 2017 Tags: blog
In the last month I started with upstream of the code for sanitizers: the common layer and ubsan. I worked also on the elimination of unexpected failures in LLVM and Clang. I've managed to achieve, with a pile of local patches, the number of 0 unexpected bugs within LLVM (check-llvm) and 3 unexpected bugs within Clang (check-clang) (however these ones were caused by hardcoded environment -lstdc++ vs -lc++). The number of failures in sanitizers (check-sanitizer) is also low, it's close to zero.


In order to achieve the goals of testability concerning the LLVM projects, I had to prepare a new pkgsrc-wip package called llvm-all-in-one that contains 12 active LLVM projects within one tree. The set of these projects is composed of: llvm, clang, compiler-rt, libcxx, libcxxabi, libunwind, test-suite, openmp, llgo, lld, lldb, clang-tools-extra. These were required to build and execute test-suites in the LLVM's projects. Ideally the tests should work in standalone packages - built out-of-LLVM-sources - and with GCC/Clang, however the real life is less bright and this forced me to use Clang as the system compiler an all-in-one package in order to develop the work environment with the ability to build and execute unit tests.

There were four threads within LLVM:

  • Broken std::call_once with libstdc++. This is an old and well-known bug, which was usually worked around with a homegrown implementation llvm::call_once. I've discovered that the llvm::call_once workaround isn't sufficient for the whole LLVM functionality, as std::call_once can be called internally inside the libstdc++ libraries - like within the C++11 futures interface. This bug has been solved by Joerg Sonnenberger in the ELF dynamic linker.
  • Unportable shell construct hardcoded in tests ">&". This has been fixed upstream.
  • LLVM JIT. The LLVM Memory generic allocator (or page mapper) was designed to freely map pages with any combination of the protection bits: R,W,X. This approach breaks on NetBSD with PaX MPROTECT and requires redesign of the interfaces. This is the continuation of the past month AllocateRWX and ReleaseRWX compatibility with NetBSD improvements. I've prepared few variations of local patches addressing these issues and it's still open for discussion with upstream. My personal preference is to remove the current API entirely and introduce a newer one with narrowed down functionality to swap between readable (R--), writable (RW-) and executable (R-X) memory pages. This would effectively enforce W^X.
  • Sanitizers support. Right now, I keep the patches locally in order to upstream the common sanitizer code in compiler-rt.

The LLVM JIT API is the last cause of unexpected failures in check-llvm. This breaks MCJIT, ORCJIT and ExecutionEngine libraries and causes around 200 unexpected failures within tests.


I've upstreamed a patch that enables ubsan and asan on Clang's frontend for NetBSD/amd64. This support isn't complete, and requires sanitizers' support code upstreamed to compiler-rt.


The current compiler-rt tasks can be divided into:

  1. upstream sanitizer common code shared with POSIX platforms
  2. upstream sanitizer common code shared with Linux and FreeBSD
  3. upstream sanitizer common code shared with FreeBSD
  4. upstream sanitizer common code specific to NetBSD
  5. build, execute and pass tests for sanitizer common code in check-santizer

This means that ubsan, asan and the rest of the specific sanitizers wait in queue.

All the mentioned tasks are being worked on simultaneously, with a soft goal to finish them one after another from the first to the last one.

The last point with check-sanitizer unveiled so far two generic bugs on NetBSD:

  • Return errno EFAULT instead of EACCES on memory fault with read(2)/write(2)-like syscalls.
These bugs are not strictly real bugs, but they were introducing needless differences with other modern POSIX systems. The fixes were introduced by Christos Zoulas and backported to NetBSD-8.

Plan for the next milestone

I have decided not to open new issues in with the coming month and focus on upstreaming the remaining LLVM code. The roadmap for the next month is to continue working on the goals of the previous months. std::call_once is an example that every delayed bug keeps biting again and again in future.

LLVM 5.0.0 is planned to be released this month (August) and there is a joint motivation with the upstream maintainer to push compatibility fixes for LLVM JIT. There is an option to submit a workaround now and introduce refactoring for the trunk and next version (6.0.0).

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:


Posted at lunch time on Thursday, August 3rd, 2017 Tags: blog
Add a comment
Contact | Disclaimer | Copyright © 1994-2017 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
NetBSD® is a registered trademark of The NetBSD Foundation, Inc.