This month I've finally finished upstreaming NetBSD support in ASan and UBsan. For better coverage of the sanitizers and on user request I've ported libFuzzer and SafeStack. There are mutual dependencies between the compiler-rt features. NetBSD after sorting out msan and tsan shall get all the remaining ones enabled. This is open topic after finishing LLDB. I have also prepared better ground for the coming work on ptrace(2) enhancements with the removal of the filesystem tracing (/proc/#/ctl).

LLVM

The majority of the work has been done in the LLVM projects.

The developed features are not production ready and they will need productization in the future. There are still issues with paths mismatch ("netbsd" vs "netbsd8.99" vs "netbsd8.99.1") when looking for NetBSD specific support for the compiler-rt features. There is also a need for integration with pkgsrc, as not everything behaves sanely (conflicts with wrappers). The tools are also restricted to be built with the Clang compiler, as GCC support is currently broken. I also noted that the sanitizers behave wrongly in the standalone build (out of the LLVM sources).

I expect to sort out the mentioned problems after finishing LLDB.

LLVM JIT

There is ongoing discussion with the LLVM community about new JIT API that will be compatible with NetBSD PaX MPROTECT. There have been developed and introduced cleanups in the code (like better error handling templates) in order to prepare a draft of new API.

ASAN

All local code for ASan has been merged upstream. This includes NetBSD patches in LLVM, Clang and compiler-rt.

All but two (one on i386 version and the other on amd64) tests (check-asan) pass.

UBSAN

Similarly to ASan, UBsan has been fully upstreamed. All tests (check-ubsan) pass.

SafeStack

SafeStack is a software security hardening technique that creates two stacks: one for data that needs to be kept safe, such as return addresses and function pointers; and an unsafe stack for everything else.

With PaX ASLR (Address Space Layout Randomization) and PaX MPROTECT (mprotect(2) restrictions) SafeStack is an excellent candidate for inclusion in pkgsrc.

Core programs could be hardened as well, but the shortcoming of SafeStack for basesystem utilities is pulling in additional dependencies like libpthread on every executable.

Using SafeStack adds marginal overhead.

libFuzzer

Citing the project page, LibFuzzer is an in-process, coverage-guided, evolutionary fuzzing engine.

LibFuzzer is linked with the library under test, and feeds fuzzed inputs to the library via a specific fuzzing entry point (aka 'target function'); the fuzzer then tracks which areas of the code are reached, and generates mutations on the corpus of input data in order to maximize the code coverage. The code coverage information for libFuzzer is provided by LLVM's SanitizerCoverage instrumentation.

This functionality still requires more sanitizers to get aboard and is now of restricted functionality.

TSAN

Part of the TSan code has been upstreamed. However, a functional port isn't finished yet.

The current issues are: proper process memory map handling and NetBSD specific setjmp(3)-like functions support.

LSAN

I was also working on LSan. This sanitizer already builds and appears to be quite completed, however there is work needed for the implementation of StopTheWorld() function to self-introspect the process and threads. I'm researching a new kernel API for this purpose, but it might wait till the end of LLDB porting.

MSAN

So far I have not been working on the MSan specific bits. The majority of the code has been upstreamed for this sanitizer in the common sanitizer framework, the proper handling of the NetBSD specific process map is still to be done.

PROFILE

The profile library is used to collect coverage information. It already passes most of the tests, however it's not turned on, as upstream requested additional time to be spent on the issues and it's not a priority right now.

NetBSD kernel

I've removed the filesystem tracing feature.

This is a legacy interface from 4.4BSD, and it was introduced to overcome shortcomings of ptrace(2) at that time, which are no longer relevant (performance). Today /proc/#/ctl offers a narrow subset of ptrace(2) commands and is not applicable for modern applications use beyond simplistic tracing scenarios.

This removal simplified kernel internals. Users are still able to use all the other /proc files.

This change doesn't affect other procfs files and Linux compat features within mount_procfs(8). /proc/#/ctl isn't available on Linux.

Plan for the next milestone

This month I will not work on a new development and I will focus on relax and taking part in EuroBSDCon in Paris. I will speak about the LLDB porting to NetBSD.

Long-term goals are finishing the basis sanitizers (msan, tsan) and switching back to LLDB porting. The sanitizers will be used to develop and debug the LLVM debugger. There is also integration with sanitizers in LLDB.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:

http://netbsd.org/donations/#how-to-donate

Posted Friday night, September 1st, 2017 Tags:
This month I've finally finished upstreaming NetBSD support in ASan and UBsan. For better coverage of the sanitizers and on user request I've ported libFuzzer and SafeStack. There are mutual dependencies between the compiler-rt features. NetBSD after sorting out msan and tsan shall get all the remaining ones enabled. This is open topic after finishing LLDB. I have also prepared better ground for the coming work on ptrace(2) enhancements with the removal of the filesystem tracing (/proc/#/ctl).

LLVM

The majority of the work has been done in the LLVM projects.

The developed features are not production ready and they will need productization in the future. There are still issues with paths mismatch ("netbsd" vs "netbsd8.99" vs "netbsd8.99.1") when looking for NetBSD specific support for the compiler-rt features. There is also a need for integration with pkgsrc, as not everything behaves sanely (conflicts with wrappers). The tools are also restricted to be built with the Clang compiler, as GCC support is currently broken. I also noted that the sanitizers behave wrongly in the standalone build (out of the LLVM sources).

I expect to sort out the mentioned problems after finishing LLDB.

LLVM JIT

There is ongoing discussion with the LLVM community about new JIT API that will be compatible with NetBSD PaX MPROTECT. There have been developed and introduced cleanups in the code (like better error handling templates) in order to prepare a draft of new API.

ASAN

All local code for ASan has been merged upstream. This includes NetBSD patches in LLVM, Clang and compiler-rt.

All but two (one on i386 version and the other on amd64) tests (check-asan) pass.

UBSAN

Similarly to ASan, UBsan has been fully upstreamed. All tests (check-ubsan) pass.

SafeStack

SafeStack is a software security hardening technique that creates two stacks: one for data that needs to be kept safe, such as return addresses and function pointers; and an unsafe stack for everything else.

With PaX ASLR (Address Space Layout Randomization) and PaX MPROTECT (mprotect(2) restrictions) SafeStack is an excellent candidate for inclusion in pkgsrc.

Core programs could be hardened as well, but the shortcoming of SafeStack for basesystem utilities is pulling in additional dependencies like libpthread on every executable.

Using SafeStack adds marginal overhead.

libFuzzer

Citing the project page, LibFuzzer is an in-process, coverage-guided, evolutionary fuzzing engine.

LibFuzzer is linked with the library under test, and feeds fuzzed inputs to the library via a specific fuzzing entry point (aka 'target function'); the fuzzer then tracks which areas of the code are reached, and generates mutations on the corpus of input data in order to maximize the code coverage. The code coverage information for libFuzzer is provided by LLVM's SanitizerCoverage instrumentation.

This functionality still requires more sanitizers to get aboard and is now of restricted functionality.

TSAN

Part of the TSan code has been upstreamed. However, a functional port isn't finished yet.

The current issues are: proper process memory map handling and NetBSD specific setjmp(3)-like functions support.

LSAN

I was also working on LSan. This sanitizer already builds and appears to be quite completed, however there is work needed for the implementation of StopTheWorld() function to self-introspect the process and threads. I'm researching a new kernel API for this purpose, but it might wait till the end of LLDB porting.

MSAN

So far I have not been working on the MSan specific bits. The majority of the code has been upstreamed for this sanitizer in the common sanitizer framework, the proper handling of the NetBSD specific process map is still to be done.

PROFILE

The profile library is used to collect coverage information. It already passes most of the tests, however it's not turned on, as upstream requested additional time to be spent on the issues and it's not a priority right now.

NetBSD kernel

I've removed the filesystem tracing feature.

This is a legacy interface from 4.4BSD, and it was introduced to overcome shortcomings of ptrace(2) at that time, which are no longer relevant (performance). Today /proc/#/ctl offers a narrow subset of ptrace(2) commands and is not applicable for modern applications use beyond simplistic tracing scenarios.

This removal simplified kernel internals. Users are still able to use all the other /proc files.

This change doesn't affect other procfs files and Linux compat features within mount_procfs(8). /proc/#/ctl isn't available on Linux.

Plan for the next milestone

This month I will not work on a new development and I will focus on relax and taking part in EuroBSDCon in Paris. I will speak about the LLDB porting to NetBSD.

Long-term goals are finishing the basis sanitizers (msan, tsan) and switching back to LLDB porting. The sanitizers will be used to develop and debug the LLVM debugger. There is also integration with sanitizers in LLDB.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:

http://netbsd.org/donations/#how-to-donate

Posted Friday night, September 1st, 2017 Tags:
Joerg Sonnenberger has announced a new set of mirrored repositories.

You can find Mercurial versions of src, pkgsrc and xsrc under

and

The same rules as for the fossil and github repositories apply, i.e. there may be occasional glitches and if it becomes too bad, they might be recreated from scratch.

See more information in the posted thread to tech-repository.

Posted Friday night, September 1st, 2017 Tags:
Joerg Sonnenberger has announced a new set of mirrored repositories.

You can find Mercurial versions of src, pkgsrc and xsrc under

and

The same rules as for the fossil and github repositories apply, i.e. there may be occasional glitches and if it becomes too bad, they might be recreated from scratch.

See more information in the posted thread to tech-repository.

Posted Friday night, September 1st, 2017 Tags:
The NetBSD Foundation supports projects that strive to ship the best possible support in developer-oriented software. This is not exclusive to LLVM, but also includes the more traditional GNU toolchain.

Traditionally, developers in distributions like NetBSD merge 3rd party sources upstream once in a while with major release bumps, like switching from GCC 4.8.x to GCC 5.x. The time frame between the releases can be several months or a few years. This appears as a one-time effort from time to time, however with each software revision the code starts to rot on undermaintained targets. This results in local compatibility patches, which are rarely ready or applicable for upstream and thus detached from the development progress. Upstream developers tend to assume that minimal activity from such projects is a result of having no users and not verifying new code on them.

A good way to improve the situation and ensure quality of software that would shorten developers' time and cost, to deepen relations between NetBSD developers and upstream 3rd party software is to attach a build cluster node within the testing infrastructure.

The shortcoming of this approach is that it requires hardware, bandwidth and admin maintenance. The advantage is closer and better support for the NetBSD platform directly from the 3rd party software developers and the immediate detection of regressions, further reducing development time.

After the process of restoration the NetBSD support within the GDB and binutils projects, there is a new member in the GDB's cluster farm that verifies correct build status on NetBSD/amd64. This bot is hosted within The NetBSD Foundation's internal infrastructure.

The immediate follow up is to turn on --enable-targets=all, which will build all the available backends. Only a few more patches are needed to achieve this milestone.

Next steps involve extending this bot to verify other projects within the shared binutils-gdb repository. This includes GNU binutils itself, ld, gold (after adding appropriate platform support), gas, sim and gprof.

The ultimate goal is to enable execution of all tests for each new binutils-gdb commit in the upstream repository. This must be preceded by accomplishing the ongoing contracted task sponsored by The NetBSD Foundation - to port the LLDB debugger to NetBSD, as the LLVM debugger opens a door for new software from the same field.

Posted late Thursday evening, September 14th, 2017 Tags:
The NetBSD Foundation supports projects that strive to ship the best possible support in developer-oriented software. This is not exclusive to LLVM, but also includes the more traditional GNU toolchain.

Traditionally, developers in distributions like NetBSD merge 3rd party sources upstream once in a while with major release bumps, like switching from GCC 4.8.x to GCC 5.x. The time frame between the releases can be several months or a few years. This appears as a one-time effort from time to time, however with each software revision the code starts to rot on undermaintained targets. This results in local compatibility patches, which are rarely ready or applicable for upstream and thus detached from the development progress. Upstream developers tend to assume that minimal activity from such projects is a result of having no users and not verifying new code on them.

A good way to improve the situation and ensure quality of software that would shorten developers' time and cost, to deepen relations between NetBSD developers and upstream 3rd party software is to attach a build cluster node within the testing infrastructure.

The shortcoming of this approach is that it requires hardware, bandwidth and admin maintenance. The advantage is closer and better support for the NetBSD platform directly from the 3rd party software developers and the immediate detection of regressions, further reducing development time.

After the process of restoration the NetBSD support within the GDB and binutils projects, there is a new member in the GDB's cluster farm that verifies correct build status on NetBSD/amd64. This bot is hosted within The NetBSD Foundation's internal infrastructure.

The immediate follow up is to turn on --enable-targets=all, which will build all the available backends. Only a few more patches are needed to achieve this milestone.

Next steps involve extending this bot to verify other projects within the shared binutils-gdb repository. This includes GNU binutils itself, ld, gold (after adding appropriate platform support), gas, sim and gprof.

The ultimate goal is to enable execution of all tests for each new binutils-gdb commit in the upstream repository. This must be preceded by accomplishing the ongoing contracted task sponsored by The NetBSD Foundation - to port the LLDB debugger to NetBSD, as the LLVM debugger opens a door for new software from the same field.

Posted late Thursday evening, September 14th, 2017 Tags:
The slate of nominees was voted for and accepted by the members of the foundation. We'd like to our team of nomcom, voting coordinator, and voting validator for putting together the slate and managing the election process.

We welcome Pierre Pronchery and Makoto Fujiwara to the 2017 Board of Directors. We look forward to working with you!

We appreciate all of the wonderful work that S.P.Zeidler and Erik Berls have done on the board during their time as directors, and are grateful for their excellent service to the foundation.

Thank you to all members participating by nominating candidates and voting on the slate.



Respectfully submitted for The NetBSD Foundation,
William J. Coldwell
President/Chairperson
Posted terribly early Wednesday morning, September 27th, 2017 Tags:
The slate of nominees was voted for and accepted by the members of the foundation. We'd like to our team of nomcom, voting coordinator, and voting validator for putting together the slate and managing the election process.

We welcome Pierre Pronchery and Makoto Fujiwara to the 2017 Board of Directors. We look forward to working with you!

We appreciate all of the wonderful work that S.P.Zeidler and Erik Berls have done on the board during their time as directors, and are grateful for their excellent service to the foundation.

Thank you to all members participating by nominating candidates and voting on the slate.



Respectfully submitted for The NetBSD Foundation,
William J. Coldwell
President/Chairperson
Posted terribly early Wednesday morning, September 27th, 2017 Tags:
Add a comment
Contact | Disclaimer | Copyright © 1994-2017 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
NetBSD® is a registered trademark of The NetBSD Foundation, Inc.