Nov 2017
S M T W T F S
     
18
19
   

Archives

This page is a blog mirror of sorts. It pulls in articles from blog's feed and publishes them here (with a feed, too).

Since the last update, we've made a number of improvements to the NetBSD Allwinner port. The SUNXI kernel has grown support for 8 new SoCs, and we added many new device drivers to the source repository.

Supported systems

Device driver support

In addition to the countless machine-independent device drivers already in NetBSD, the following Allwinner-specific devices are supported:

Audio codec

The built-in analog audio codec is supported on the following SoCs with the sunxicodec driver: A10, A13, A20, A31, GR8, H2+, H3, and R8.

Ethernet

Ethernet is supported on all applicable Allwinner SoCs. Three ethernet drivers are available:

  • Fast Ethernet MAC (EMAC) as found in A10 and A20 family SoCs
  • Gigabit Ethernet MAC (GMAC) as found in A20, A31, and A80 family SoCs
  • Gigabit Ethernet MAC (EMAC) as found in A64, A83T, H2+, and H3 family SoCs

Framebuffer

Framebuffer console support is available wherever it is supported by U-Boot using the simplefb(4) driver.

Thermal sensors

Thermal sensors are supported on A10, A13, A20, A31, A64, A83T, H2+, and H3 SoCs.

CPU frequency and voltage scaling

On A10, A20, H2+, and H3 SoCs, dynamic CPU frequency and voltage scaling support is available when configured in the device tree. In addition, on H2+ and H3 SoCs, the kernel will automatically detect when the CPU temperature is too high and throttle the CPU frequency and voltage to prevent overheating.

Touch screen

The touch screen controller found in A10, A13, A20, and A31 SoCs is fully supported. The tpctl(8) utility can be used to calibrate the touch screen and has been updated to support standard wsdisplay APIs.

Other drivers

A standard set of devices are supported across all SoCs (where applicable): DMA, GPIO, I2C, interrupt controllers, RTC, SATA, SD/MMC, timers, UART, USB, watchdog, and more.

U-Boot

A framework for U-Boot packages has been added to pkgsrc, and U-Boot packages for many boards already exist.

What now?

There are a few missing features that would be nice to have:

  • Wi-Fi (SDIO). There are a lot of different wireless chips used on these boards, but the majority seem to be either Broadcom or Realtek based. We recently ported OpenBSD's bwfm(4) driver to support the USB version of the Broadcom Wi-Fi controllers, with an expectation that SDIO support will follow at some point in the future.
  • NAND controller. Most boards have eMMC and/or microSD slots, but this would be really useful for the CHIP / CHIP Pro / PocketCHIP family of devices.
  • 64-bit support for sun50i family SoCs
  • Readily available install images. A prototype NetBSD ARM Bootable Images site is available with a limited selection of supported boards.

More information

Posted in the wee hours of Tuesday night, November 8th, 2017 Tags: blog

Since the last update, we've made a number of improvements to the NetBSD Allwinner port. The SUNXI kernel has grown support for 8 new SoCs, and we added many new device drivers to the source repository.

Supported systems

Device driver support

In addition to the countless machine-independent device drivers already in NetBSD, the following Allwinner-specific devices are supported:

Audio codec

The built-in analog audio codec is supported on the following SoCs with the sunxicodec driver: A10, A13, A20, A31, GR8, H2+, H3, and R8.

Ethernet

Ethernet is supported on all applicable Allwinner SoCs. Three ethernet drivers are available:

  • Fast Ethernet MAC (EMAC) as found in A10 and A20 family SoCs
  • Gigabit Ethernet MAC (GMAC) as found in A20, A31, and A80 family SoCs
  • Gigabit Ethernet MAC (EMAC) as found in A64, A83T, H2+, and H3 family SoCs

Framebuffer

Framebuffer console support is available wherever it is supported by U-Boot using the simplefb(4) driver.

Thermal sensors

Thermal sensors are supported on A10, A13, A20, A31, A64, A83T, H2+, and H3 SoCs.

CPU frequency and voltage scaling

On A10, A20, H2+, and H3 SoCs, dynamic CPU frequency and voltage scaling support is available when configured in the device tree. In addition, on H2+ and H3 SoCs, the kernel will automatically detect when the CPU temperature is too high and throttle the CPU frequency and voltage to prevent overheating.

Touch screen

The touch screen controller found in A10, A13, A20, and A31 SoCs is fully supported. The tpctl(8) utility can be used to calibrate the touch screen and has been updated to support standard wsdisplay APIs.

Other drivers

A standard set of devices are supported across all SoCs (where applicable): DMA, GPIO, I2C, interrupt controllers, RTC, SATA, SD/MMC, timers, UART, USB, watchdog, and more.

U-Boot

A framework for U-Boot packages has been added to pkgsrc, and U-Boot packages for many boards already exist.

What now?

There are a few missing features that would be nice to have:

  • Wi-Fi (SDIO). There are a lot of different wireless chips used on these boards, but the majority seem to be either Broadcom or Realtek based. We recently ported OpenBSD's bwfm(4) driver to support the USB version of the Broadcom Wi-Fi controllers, with an expectation that SDIO support will follow at some point in the future.
  • NAND controller. Most boards have eMMC and/or microSD slots, but this would be really useful for the CHIP / CHIP Pro / PocketCHIP family of devices.
  • 64-bit support for sun50i family SoCs
  • Readily available install images. A prototype NetBSD ARM Bootable Images site is available with a limited selection of supported boards.

More information

Posted in the wee hours of Tuesday night, November 8th, 2017 Tags: blog
The past year has been started with bugfixes and the development of regression tests for ptrace(2) and related kernel features, as well as the continuation of bringing LLDB support and LLVM sanitizers (ASan + UBsan and partial TSan + Msan) to NetBSD.
My plan for the next year is to finish implementing TSan and MSan support, followed by a long run of bug fixes for LLDB, ptrace(2), and other related kernel subsystems

TSan

In the past month, I've developed Thread Sanitizer far enough to have a subset of its tests pass on NetBSD, started with addressing breakage related to the memory layout of processes. The reason for this breakage was narrowed down to the current implementation of ASLR, which was too aggressive and which didn't allow enough space to be mapped for Shadow memory. The fix for this was to either force the disabling of ASLR per-process, or globally on the system. The same will certainly happen for MSan executables. After some other corrections, I got TSan to work for the first time ever on October 14th. This was a big achievement, so I've made a snapshot available. Getting the snapshot of execution under GDB was pure hazard.

$ gdb ./a.out                                  
GNU gdb (GDB) 7.12
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64--netbsd".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./a.out...done.
(gdb) r
Starting program: /public/llvm-build/a.out 
[New LWP 2]
==================
WARNING: ThreadSanitizer: data race (pid=1621)
  Write of size 4 at 0x000001475d70 by thread T1:
    #0 Thread1 /public/llvm-build/tsan.c:4:10 (a.out+0x46bf71)

  Previous write of size 4 at 0x000001475d70 by main thread:
    #0 main /public/llvm-build/tsan.c:10:10 (a.out+0x46bfe6)

  Location is global 'Global' of size 4 at 0x000001475d70 (a.out+0x000001475d70)

  Thread T1 (tid=2, running) created by main thread at:
    #0 pthread_create /public/llvm/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:930:3 (a.out+0x412120)
    #1 main /public/llvm-build/tsan.c:9:3 (a.out+0x46bfd1)

SUMMARY: ThreadSanitizer: data race /public/llvm-build/tsan.c:4:10 in Thread1
==================

Thread 2 received signal SIGSEGV, Segmentation fault.

I was able to get the above execution results around 10% of the time (being under a tracer had no positive effect on the frequency of successful executions).

I've managed to hit the following final results for this month, with another set of bugfixes and improvements:

check-tsan:
Expected Passes    : 248
Expected Failures  : 1
Unsupported Tests  : 83
Unexpected Failures: 44

At the end of the month, TSan can now reliably executabe the same (already-working) program every time. The majority of failures are in tests verifying sanitization of correct mutex locking usage.

There are still problems with NetBSD-specific libc and libpthread bootstrap code that conflicts with TSan. Certain functions (pthread_create(3), pthread_key_create(3), _cxa_atexit()) cannot be started early by TSan initialization, and must be deferred late enough for the sanitizer to work correctly.

MSan

I've prepared a scratch support for MSan on NetBSD to help in researching how far along it is. I've also cloned and adapted the existing FreeBSD bits; however, the code still needs more work and isn't functional yet. The number of passed tests (5) is negligible and most likely does not work at all.

The conclusion after this research is that TSan shall be finished first, as it touches similar code.

In the future, there will be likely another round of iterating the system structs and types and adding the missing ones for NetBSD. So far, this part has been done before executing the real MSan code. I've added one missing symbol that was missing and was detected when attempting to link a test program with MSan.

Sanitizers

The GCC team has merged the LLVM sanitizer code, which has resulted in almost-complete support for ASan and UBsan on NetBSD. It can be found in the latest GCC8 snapshot, located in pkgsrc-wip/gcc8snapshot. Though, do note that there is an issue with getting backtraces from libasan.so, which can be worked-around by backtracing ASan events in a debugger. UBsan also passes all GCC regression tests and appears to work fine. The code enabling sanitizers on the GCC/NetBSD frontend will be submitted upstream once the backtracing issue is fixed and I'm satisfied that there are no other problems.

I've managed to upstream a large portion of generic+TSan+MSan code to compiler-rt and reduce local patches to only the ones that are in progress. This deals with any rebasing issues, and allows me to just focus on the delta that is being worked on.

I've tried out the LLDB builds which have TSan/NetBSD enabled, and they built and started fine. However, there were some false positives related to the mutex locking/unlocking code.

Plans for the next milestone

The general goals are to finish TSan and MSan and switch back to LLDB debugging. I plan to verify the impact of the TSan bootstrap initialization on the observed crashes and research the remaining failures.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:

http://netbsd.org/donations/#how-to-donate

Posted in the wee hours of Tuesday night, November 1st, 2017 Tags: blog
The past year has been started with bugfixes and the development of regression tests for ptrace(2) and related kernel features, as well as the continuation of bringing LLDB support and LLVM sanitizers (ASan + UBsan and partial TSan + Msan) to NetBSD.
My plan for the next year is to finish implementing TSan and MSan support, followed by a long run of bug fixes for LLDB, ptrace(2), and other related kernel subsystems

TSan

In the past month, I've developed Thread Sanitizer far enough to have a subset of its tests pass on NetBSD, started with addressing breakage related to the memory layout of processes. The reason for this breakage was narrowed down to the current implementation of ASLR, which was too aggressive and which didn't allow enough space to be mapped for Shadow memory. The fix for this was to either force the disabling of ASLR per-process, or globally on the system. The same will certainly happen for MSan executables. After some other corrections, I got TSan to work for the first time ever on October 14th. This was a big achievement, so I've made a snapshot available. Getting the snapshot of execution under GDB was pure hazard.

$ gdb ./a.out                                  
GNU gdb (GDB) 7.12
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64--netbsd".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./a.out...done.
(gdb) r
Starting program: /public/llvm-build/a.out 
[New LWP 2]
==================
WARNING: ThreadSanitizer: data race (pid=1621)
  Write of size 4 at 0x000001475d70 by thread T1:
    #0 Thread1 /public/llvm-build/tsan.c:4:10 (a.out+0x46bf71)

  Previous write of size 4 at 0x000001475d70 by main thread:
    #0 main /public/llvm-build/tsan.c:10:10 (a.out+0x46bfe6)

  Location is global 'Global' of size 4 at 0x000001475d70 (a.out+0x000001475d70)

  Thread T1 (tid=2, running) created by main thread at:
    #0 pthread_create /public/llvm/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:930:3 (a.out+0x412120)
    #1 main /public/llvm-build/tsan.c:9:3 (a.out+0x46bfd1)

SUMMARY: ThreadSanitizer: data race /public/llvm-build/tsan.c:4:10 in Thread1
==================

Thread 2 received signal SIGSEGV, Segmentation fault.

I was able to get the above execution results around 10% of the time (being under a tracer had no positive effect on the frequency of successful executions).

I've managed to hit the following final results for this month, with another set of bugfixes and improvements:

check-tsan:
Expected Passes    : 248
Expected Failures  : 1
Unsupported Tests  : 83
Unexpected Failures: 44

At the end of the month, TSan can now reliably executabe the same (already-working) program every time. The majority of failures are in tests verifying sanitization of correct mutex locking usage.

There are still problems with NetBSD-specific libc and libpthread bootstrap code that conflicts with TSan. Certain functions (pthread_create(3), pthread_key_create(3), _cxa_atexit()) cannot be started early by TSan initialization, and must be deferred late enough for the sanitizer to work correctly.

MSan

I've prepared a scratch support for MSan on NetBSD to help in researching how far along it is. I've also cloned and adapted the existing FreeBSD bits; however, the code still needs more work and isn't functional yet. The number of passed tests (5) is negligible and most likely does not work at all.

The conclusion after this research is that TSan shall be finished first, as it touches similar code.

In the future, there will be likely another round of iterating the system structs and types and adding the missing ones for NetBSD. So far, this part has been done before executing the real MSan code. I've added one missing symbol that was missing and was detected when attempting to link a test program with MSan.

Sanitizers

The GCC team has merged the LLVM sanitizer code, which has resulted in almost-complete support for ASan and UBsan on NetBSD. It can be found in the latest GCC8 snapshot, located in pkgsrc-wip/gcc8snapshot. Though, do note that there is an issue with getting backtraces from libasan.so, which can be worked-around by backtracing ASan events in a debugger. UBsan also passes all GCC regression tests and appears to work fine. The code enabling sanitizers on the GCC/NetBSD frontend will be submitted upstream once the backtracing issue is fixed and I'm satisfied that there are no other problems.

I've managed to upstream a large portion of generic+TSan+MSan code to compiler-rt and reduce local patches to only the ones that are in progress. This deals with any rebasing issues, and allows me to just focus on the delta that is being worked on.

I've tried out the LLDB builds which have TSan/NetBSD enabled, and they built and started fine. However, there were some false positives related to the mutex locking/unlocking code.

Plans for the next milestone

The general goals are to finish TSan and MSan and switch back to LLDB debugging. I plan to verify the impact of the TSan bootstrap initialization on the observed crashes and research the remaining failures.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:

http://netbsd.org/donations/#how-to-donate

Posted in the wee hours of Tuesday night, November 1st, 2017 Tags: blog
NetBSD participated in the 2017 edition of Google of Summer of Code with 3 students. All of the students finished their projects successfully. The following links report about their activities: Congratulations to the students for finishing their projects successfully, and thanks to Google for sponsoring!
Posted Wednesday afternoon, October 18th, 2017 Tags: blog
NetBSD participated in the 2017 edition of Google of Summer of Code with 3 students. All of the students finished their projects successfully. The following links report about their activities: Congratulations to the students for finishing their projects successfully, and thanks to Google for sponsoring!
Posted Wednesday afternoon, October 18th, 2017 Tags: blog
Recently, I completed a Kernel ASLR implementation for NetBSD-amd64, making NetBSD the first BSD system to support such a feature. Simply said, KASLR is a feature that randomizes the location of the kernel in memory, making it harder to exploit several classes of vulnerabilities, both locally (privilege escalations) and remotely (remote code executions).

Current design

The current design is based on a specialized kernel called the "prekern", which operates between the bootloader and the kernel itself. The kernel is compiled as a raw library with the GENERIC_KASLR configuration file, while the prekern is compiled as a static binary. When the machine boots, the bootloader jumps into the prekern. The prekern relocates the kernel at a random virtual address (VA), and jumps into it. Finally, the kernel performs some cleanup, and executes normally.

Currently, the kernel is randomized as a single block. That is to say, a random VA is chosen, and the kernel text->rodata->data sections are mapped contiguously starting from there. It has several drawbacks, but it's a first shot.

To complete this implementation, work had to be done at three levels: the bootloader, the prekern and the kernel. I committed several of the kernel and bootloader patches discreetly a few months ago, to pave some way for real changes. In the past few weeks, I changed the low-level x86 layer of the kernel and replaced several hard-coded (and sometimes magic) values by variables, in such a way that the kernel can run with a non-static memory layout. Finally, the last step was committing the prekern itself to the source tree.

Future work

  • Randomize the kernel sections independently, and intertwine them.
  • Modify several kernel entry points not to leak kernel addresses to userland.
  • Randomize the kernel heap too (which is still static for now).
  • Fix a few other things that need some more work.

How to use

All of the patches are now in NetBSD-current. Instructions on how to install and use this implementation can be found here; they are inlined below, and probably won't change in the future.

Make sure you have a v5.11 bootloader installed. If you don't, build and install a new bootloader:

    $ cd /usr/src/sys/arch/i386/stand/boot
    $ make
    # cp biosboot/boot /
Build and install a KASLR kernel:
    $ cd /usr/src
    $ ./build.sh -u kernel=GENERIC_KASLR
    # cp /usr/obj/sys/arch/amd64/compile/GENERIC_KASLR/netbsd /netbsd_kaslr
Finally, build and install a prekern:
    $ cd /usr/src/sys/arch/amd64/stand/prekern
    $ make
    # cp prekern /prekern
Reboot your machine. In the boot prompt, enter:
    > pkboot netbsd_kaslr
The system will boot with no further user interaction. Should you encounter any regression or unexpected behavior, please report it immediately to tech-kern.

Note that you can still boot a static kernel, by typing as usual:

    > boot netbsd

Availability

This KASLR implementation will be available starting from NetBSD 9. Once it is stabilized, it may be backported to NetBSD 8. Until then, feel free to test it!

Posted terribly early Thursday morning, October 12th, 2017 Tags: blog
Recently, I completed a Kernel ASLR implementation for NetBSD-amd64, making NetBSD the first BSD system to support such a feature. Simply said, KASLR is a feature that randomizes the location of the kernel in memory, making it harder to exploit several classes of vulnerabilities, both locally (privilege escalations) and remotely (remote code executions).

Current design

The current design is based on a specialized kernel called the "prekern", which operates between the bootloader and the kernel itself. The kernel is compiled as a raw library with the GENERIC_KASLR configuration file, while the prekern is compiled as a static binary. When the machine boots, the bootloader jumps into the prekern. The prekern relocates the kernel at a random virtual address (VA), and jumps into it. Finally, the kernel performs some cleanup, and executes normally.

Currently, the kernel is randomized as a single block. That is to say, a random VA is chosen, and the kernel text->rodata->data sections are mapped contiguously starting from there. It has several drawbacks, but it's a first shot.

To complete this implementation, work had to be done at three levels: the bootloader, the prekern and the kernel. I committed several of the kernel and bootloader patches discreetly a few months ago, to pave some way for real changes. In the past few weeks, I changed the low-level x86 layer of the kernel and replaced several hard-coded (and sometimes magic) values by variables, in such a way that the kernel can run with a non-static memory layout. Finally, the last step was committing the prekern itself to the source tree.

Future work

  • Randomize the kernel sections independently, and intertwine them.
  • Modify several kernel entry points not to leak kernel addresses to userland.
  • Randomize the kernel heap too (which is still static for now).
  • Fix a few other things that need some more work.

How to use

All of the patches are now in NetBSD-current. Instructions on how to install and use this implementation can be found here; they are inlined below, and probably won't change in the future.

Make sure you have a v5.11 bootloader installed. If you don't, build and install a new bootloader:

    $ cd /usr/src/sys/arch/i386/stand/boot
    $ make
    # cp biosboot/boot /
Build and install a KASLR kernel:
    $ cd /usr/src
    $ ./build.sh -u kernel=GENERIC_KASLR
    # cp /usr/obj/sys/arch/amd64/compile/GENERIC_KASLR/netbsd /netbsd_kaslr
Finally, build and install a prekern:
    $ cd /usr/src/sys/arch/amd64/stand/prekern
    $ make
    # cp prekern /prekern
Reboot your machine. In the boot prompt, enter:
    > pkboot netbsd_kaslr
The system will boot with no further user interaction. Should you encounter any regression or unexpected behavior, please report it immediately to tech-kern.

Note that you can still boot a static kernel, by typing as usual:

    > boot netbsd

Availability

This KASLR implementation will be available starting from NetBSD 9. Once it is stabilized, it may be backported to NetBSD 8. Until then, feel free to test it!

Posted terribly early Thursday morning, October 12th, 2017 Tags: blog

Let me tell you about my experience at EuroBSDcon 2017 in Paris, France. We will see what was presented during the NetBSD developer summit on Friday and then we will give a look to all of the NetBSD and pkgsrc presentations given during the conference session on Saturday and Sunday. Of course, a lot of fun also happened on the "hall track", the several breaks during the conference and the dinners we had together with other *BSD developers and community! This is difficult to describe and I will try to just share some part of that with photographs that we have taken. I can just say that it was a really beautiful experience, I had a great time with others and, after coming back home... ...I miss all of that! :) So, if you have never been in any BSD conferences I strongly suggest you to go to the next ones, so please stay tuned via NetBSD Events. Being there this is probably the only way to understand these feelings!

Thursday (21/09): NetBSD developers dinner

Arriving in Paris via a night train from Italy I literally sleep-walked through Paris getting lost again and again. After getting in touch with other developers we had a dinner together and went sightseeing for a^Wseveral beers!

Friday (22/09): NetBSD developers summit

On Friday morning we met for the NetBSD developers summit kindly hosted by Arolla.

Photograph of the NetBSD develepors summit
From left to right: alnsn, sborrill; abhinav; uwe and leot; christos, cherry, ast and bsiegert; martin and khorben.

The devsummit was moderated by Jörg (joerg) and organized by Jean-Yves (jym).

NetBSD on Google Compute Engine -- Benny Siegert (bsiegert)

After a self-presentation the devsummit presentations session started with the talk presented by Benny (bsiegert) about NetBSD on Google Compute Engine.

Benny first introduced Google Compute Engine (GCE) and then started describing how to run NetBSD on it.

At the moment there are no official NetBSD images and so users need to create their own. However, netbsd-gce script completely automatize this process that:

  • uses Anita to stage an installation in QEMU
  • adjust several tweaks to ensure that networking and storage will work on GCE
  • packs the image into a .tar.gz file

The .tar.gz image then just need to be uploaded to a Cloud Storage bucket, create a GCE image from it and then launch VMs based on that image.

He also discussed about GCE instance metadata, several problems founds and how they were fixed (it's better to use NetBSD 8_BETA or -current!) and some future works.

For more information slides (PDF) of the talk are also available.

Scripting DDB with Forth -- Valery Ushakov (uwe)

Valery (uwe) presented a talk about Scripting DDB with Forth. It was based on a long story and actually the first discussion about it appeared on tech-kern@ mailing list in his Scripting DDB in Forth? thread (ddb(4) is the NetBSD in-kernel debugger).

He showed how one can associate forth commands/conditions with ddb breakpoints. He used "pid divisible by 3" as an example of condition for a breakpoint set in getpid(2) system call:

db{0}> forth
ok : field   create , does> @ + ;
ok #300 field lwp>l_proc
ok #120 field proc>p_pid
ok : getpid   curlwp lwp>l_proc @ proc>p_pid @ ;
ok : checkpid   getpid dup ." > PID IS " . cr   3 mod 0= ;
ok bye
-- STACK: <empty>
db{0}> break sys_getpid_with_ppid
db{0}> command . = checkpid
db{0}> c

...and then on a shell:

# (:)
fatal breakpoint trap in supervisor mode
trap type 1 code 0 eip 0xc090df89 cs 0x8 eflags 0x246 cr2 0xad8ef2c0 ilevel 0 esp 0xc0157fbd
curlwp 0xc2b5c2c0 pid 798 lid 1 lowest kstack 0xdabb42c0
> PID IS 798
-- STACK:
0xffffffff   -1
Breakpoint in pid 798.1 (ksh) at        netbsd:sys_getpid_with_ppid: pushl %ebp
db{0}> c
# (:)
fatal breakpoint trap in supervisor mode
trap type 1 code 0 eip 0xc090df89 cs 0x8 eflags 0x246 cr2 0xad8ef2c0 ilevel 0 esp 0xc0157fbd
curlwp 0xc2b5c2c0 pid 823 lid 1 lowest kstack 0xdabb42c0
> PID IS 823
-- STACK:
0x00000000  0
Command returned 0
# 

If you are more interested in this presentation I strongly suggest to also give a look to uwe's forth Mercurial repository.

News from the version control front -- Jörg Sonnenberger (joerg)

The third presentation of the devsummit was a presentation about the recent work done by Jörg (joerg) in the VCS conversions.

Jörg started the presentation discussing about the infrastructure used for the CVS -> Fossil -> Git conversion and CVS -> Fossil -> Mercurial conversion.

It's worth also noticing that the Mercurial conversion is fully integrated and is regularly pushed to Bitbucket and src repository pushed some scalability limits to Bitbucket!

Mercurial performance were also compared to the Git ones in details for several operations.

A check list that compared the current status of the NetBSD VCS migration to the FreeBSD VCS wiki one was described and then Jörg discussed the pending work and answered several questions in the Q&A.

For more information please give a look to the joerg's presentation slides (HTML). If you would like to help for the VCS migration please also get in touch with him!

Afternoon discussions and dinner

After the lunch we had several non-scheduled discussions, some time for hacking, etc. We then had a nice dinner together (it was in a restaurant with a very nice waiter who always shouted after every order or after accidently dropping and crashing dishes!, yeah! That's probably a bit weird but I liked that attitude! :)) and then did some sightseeing and had a beer together.

Photograph of the Friday dinner, taken by Christos
From left to right: uwe, bad, ast, leot, martin, abhinav, sborrill, alnsn, spz.

Photograph of the Friday dinner, taken by Abhinav
From left to right: uwe, bad, ast, christos, leot, martin, sborrill, alnsn, spz.

Saturday (23/09): First day of conference session and Social Event

A Modern Replacement for BSD spell(1) -- Abhinav Upadhyay (abhinav)

Abhinav (abhinav) presented his work on the new spell(1) implementation he's working (that isn't just a spell replacement but also a library that can be used by other programs!).

He described the current limitations of old spell(1) (to get an idea please give a look to bin/48684), described the project goals of the new spell(1), additions to /usr/share/dict/words, digged a bit in the implementation and discussed several algorithms used and then provided a performance comparison with other popular free software spell checkers (aspell, hunspell and ispell).

He also showed an interactive demo of the new spell(1) in-action integrated with a shell for auto-completion and spell check.

If you would like to try it please give a look to nbspell Git repository that contains the code and dicts for the new spell(1)!

Video recording (YouTube) of the talk and slides (PDF) are also available!

Portable Hotplugging: NetBSD's uvm_hotplug(9) API development -- Cherry G. Mathew (cherry)

Cherry (cherry) presented recent work done with Santhosh N. Raju (fox) about uvm_hotplug(9).

The talk covered most "behind the scenes" work: how TDD (test driven development) was used, how uvm_hotplug(9) was designed and implemented (with comparisons to the old implementation), interesting edge cases during the development and how atf(7) was used to do performance testing.

It was very interesting to learn how Cherry and Santhosh worked on that and on the conclusion Cherry pointed out the importance of using existing Software Engineering techniques in Systems Programming.

Video recording (YouTube) and slides (PDF) of the talk are also available!

Hardening pkgsrc -- Pierre Pronchery (khorben)

Pierre (khorben) presented a talk about recent pkgsrc security features added in the recent months (and most of them also active on the just released pkgsrc-2017Q3!).

He first introduced how security management and releng is handled in pkgsrc, how to use pkg_admin(1) fetch-pkg-vulnerabilities and audit commands, etc.

Then package signatures (generation, installation) and recent hardening features in pkgsrc were discussed in details, first introducing them and then how pkgsrc handles them:

  • SSP: Stack Smashing Protection (enabled via PKGSRC_USE_SSP in mk.conf)
  • Fortify (enabled via PKGSRC_USE_FORTIFY in mk.conf)
  • Stack check (enabled via PKGSRC_USE_STACK_CHECK in mk.conf)
  • Position-Independent Executables (PIE) (enabled via PKGSRC_MKPIE in mk.conf)
  • RELRO and BIND_NOW (enabled via PKGSRC_USE_RELRO in mk.conf)

Challenges for each hardening features and future works were discussed.

For more information video recording (YouTube) and slides (PDF) of the talk are available. A good introduction and reference for all pkgsrc hardening features is the Hardening packages wiki page.

Reproducible builds on NetBSD -- Christos Zoulas (christos)

Christos (christos) presented the work about reproducible builds on NetBSD.

In his talk he first provided a rationale about reproducible builds (to learn more please give a look to reproducible-builds.org!), he then discussed about the NetBSD (cross) build process, the current status and build variations that are done in the tests.reproducible-builds.org build machines.

Then he provided and described several sources of difference that were present in non-reproducible builds, like file-system timestamps, parallel builds headaches due directory/build order, path normalization, etc. For each of them he also discussed in details how these problems were solved in NetBSD.

In the conclusion the status and possible TODOs were also discussed (please note that both -current and -8 are all built with reproducible flags (-P option of build.sh)!)

Video recording (YouTube) of Christos' talk is available. Apart the resources discussed above a nice introduction to reproducible builds in NetBSD is also the NetBSD fully reproducible builds blog post written by Christos last February!

Social event

The social event on Saturday evening took place on a boat that cruised on the Seine river.

It was a very nice and different way to sightsee Paris, eat and enjoy some drinks and socialize and discuss with other developers and community.

Photograph from the boat, taken by Martin.

Sunday (24/09): Second day of conference session

The school of hard knocks - PT1 -- Sevan Janiyan (sevan)

Sevan (sevan) presented a talk about several notes and lessons learnt whilst running tutorials to introduce NetBSD at several events (OSHUG #46 and OSHUG #57 and #58) and experiences from past events (Chiphack 2013).

He described problems a user may experience and how NetBSD was introduced, in particular trying to avoid the steep learning curve involved when experimenting with operating systems as a first step, exploring documentation/source code, cross-building, scripting in high-level programming languages (Lua) and directly prototyping and getting pragmatic via rump.

Video recording (YouTube) of Sevan's talk and slides (HTML) are available.

The LLDB Debugger on NetBSD -- Kamil Rytarowski (kamil)

Kamil (kamil) presented a talk about the recent LLDB debugger and a lot of other related debuggers (but also non-strictly-related-to-debugging!) works he's doing in the last months.

He first introduced debugging concepts in general, provided several examples and then he started discussing LLDB porting to NetBSD.

He then discussed about ptrace(2) and other introspection interfaces, the several improvements done and tests added for ptrace(2) in atf(7).

He also discussed about tracking LLDB's trunk (if you are more curious please give a look to wip/llvm-git, wip/clang-git, wip/lldb-git packages in pkgsrc-wip!) and about LLVM sanitizers and their current status in NetBSD.

In the conclusion he also discussed various TODOs in these areas.

Video recording (YouTube) and slides (HTML) of Kamil's talk are available. Kamil also regularly write status update blog posts on blog.NetBSD.org and tech-toolchain@ mailing list, so please stay tuned! :)

What's in store for NetBSD 8.0? -- Alistair Crooks (agc)

Alistair (agc) presented a talk about what we will see in NetBSD 8.0.

He discussed about new hardware supported (really "new", not new "old" hardware! Of course also support for VAXstation 4000 TURBOchannel USB and GPIO is actually new hardware as well! :)), LLVM/Clang, virtualization, PGP signing, updated utilities in NetBSD, new networking features (e.g. bouyer's sockcan implementation), u-boot, dtrace(1), improvements and new ports testing, reproducible builds, FDT (Flattened Device Tree) and a lot of other news!

The entire presentation was done using the Socratic method (Q&A) and it was very interactive and nice!

Video recording (YouTube) and slides (PDF) of Alistair's talk are available.

Sunday dinner

After the conference we did some sightseeing in Paris, had a dinner together and then enjoyed some beers!

Photograph of the Sunday dinner, taken by Martin.
On the left side: abhinav, ast, seb, christos
On the right side: leot, Riastradh, uwe, sevan, agc, sborrill

Photograph of the Sunday dinner, taken by Abhinav.
On the left side: martin, ast, seb, christos
On the right side: leot, Riastradh, uwe, sevan, agc, sborrill

Conclusion

It was a very nice weekend and conference. It is worth to mention that EuroBSDcon 2017 was the biggest BSD conference (more than 300 people attended it!).

I would like to thank the entire EuroBSDcon organising committee (Baptiste Daroussin, Antoine Jacoutot, Jean-Sébastien Pédron and Jean-Yves Migeon), EuroBSDcon programme commitee (Antoine Jacoutot, Lars Engels, Ollivier Robert, Sevan Janiyan, Jörg Sonnenberger, Jasper Lievisse Adriaanse and Janne Johansson) and EuroBSDcon Foundation for organizing such a wonderful conference!

I also would like to thank the speakers for presenting very interesting talks, all developers and community that attended the NetBSD devsummit and conference, in particular Jean-Yves and Jörg, for organizing and moderating the devsummit and Arolla that kindly hosted us for the NetBSD devsummit!

A special thanks also to Abhinav (abhinav) and Martin (martin) for photographs and locals Jean-Yves (jym) and Stoned (seb) for helping us in not get lost in Paris' rues! :)

Thank you!

Posted at lunch time on Wednesday, October 11th, 2017 Tags: blog

Let me tell you about my experience at EuroBSDcon 2017 in Paris, France. We will see what was presented during the NetBSD developer summit on Friday and then we will give a look to all of the NetBSD and pkgsrc presentations given during the conference session on Saturday and Sunday. Of course, a lot of fun also happened on the "hall track", the several breaks during the conference and the dinners we had together with other *BSD developers and community! This is difficult to describe and I will try to just share some part of that with photographs that we have taken. I can just say that it was a really beautiful experience, I had a great time with others and, after coming back home... ...I miss all of that! :) So, if you have never been in any BSD conferences I strongly suggest you to go to the next ones, so please stay tuned via NetBSD Events. Being there this is probably the only way to understand these feelings!

Thursday (21/09): NetBSD developers dinner

Arriving in Paris via a night train from Italy I literally sleep-walked through Paris getting lost again and again. After getting in touch with other developers we had a dinner together and went sightseeing for a^Wseveral beers!

Friday (22/09): NetBSD developers summit

On Friday morning we met for the NetBSD developers summit kindly hosted by Arolla.

Photograph of the NetBSD develepors summit
From left to right: alnsn, sborrill; abhinav; uwe and leot; christos, cherry, ast and bsiegert; martin and khorben.

The devsummit was moderated by Jörg (joerg) and organized by Jean-Yves (jym).

NetBSD on Google Compute Engine -- Benny Siegert (bsiegert)

After a self-presentation the devsummit presentations session started with the talk presented by Benny (bsiegert) about NetBSD on Google Compute Engine.

Benny first introduced Google Compute Engine (GCE) and then started describing how to run NetBSD on it.

At the moment there are no official NetBSD images and so users need to create their own. However, netbsd-gce script completely automatize this process that:

  • uses Anita to stage an installation in QEMU
  • adjust several tweaks to ensure that networking and storage will work on GCE
  • packs the image into a .tar.gz file

The .tar.gz image then just need to be uploaded to a Cloud Storage bucket, create a GCE image from it and then launch VMs based on that image.

He also discussed about GCE instance metadata, several problems founds and how they were fixed (it's better to use NetBSD 8_BETA or -current!) and some future works.

For more information slides (PDF) of the talk are also available.

Scripting DDB with Forth -- Valery Ushakov (uwe)

Valery (uwe) presented a talk about Scripting DDB with Forth. It was based on a long story and actually the first discussion about it appeared on tech-kern@ mailing list in his Scripting DDB in Forth? thread (ddb(4) is the NetBSD in-kernel debugger).

He showed how one can associate forth commands/conditions with ddb breakpoints. He used "pid divisible by 3" as an example of condition for a breakpoint set in getpid(2) system call:

db{0}> forth
ok : field   create , does> @ + ;
ok #300 field lwp>l_proc
ok #120 field proc>p_pid
ok : getpid   curlwp lwp>l_proc @ proc>p_pid @ ;
ok : checkpid   getpid dup ." > PID IS " . cr   3 mod 0= ;
ok bye
-- STACK: <empty>
db{0}> break sys_getpid_with_ppid
db{0}> command . = checkpid
db{0}> c

...and then on a shell:

# (:)
fatal breakpoint trap in supervisor mode
trap type 1 code 0 eip 0xc090df89 cs 0x8 eflags 0x246 cr2 0xad8ef2c0 ilevel 0 esp 0xc0157fbd
curlwp 0xc2b5c2c0 pid 798 lid 1 lowest kstack 0xdabb42c0
> PID IS 798
-- STACK:
0xffffffff   -1
Breakpoint in pid 798.1 (ksh) at        netbsd:sys_getpid_with_ppid: pushl %ebp
db{0}> c
# (:)
fatal breakpoint trap in supervisor mode
trap type 1 code 0 eip 0xc090df89 cs 0x8 eflags 0x246 cr2 0xad8ef2c0 ilevel 0 esp 0xc0157fbd
curlwp 0xc2b5c2c0 pid 823 lid 1 lowest kstack 0xdabb42c0
> PID IS 823
-- STACK:
0x00000000  0
Command returned 0
# 

If you are more interested in this presentation I strongly suggest to also give a look to uwe's forth Mercurial repository.

News from the version control front -- Jörg Sonnenberger (joerg)

The third presentation of the devsummit was a presentation about the recent work done by Jörg (joerg) in the VCS conversions.

Jörg started the presentation discussing about the infrastructure used for the CVS -> Fossil -> Git conversion and CVS -> Fossil -> Mercurial conversion.

It's worth also noticing that the Mercurial conversion is fully integrated and is regularly pushed to Bitbucket and src repository pushed some scalability limits to Bitbucket!

Mercurial performance were also compared to the Git ones in details for several operations.

A check list that compared the current status of the NetBSD VCS migration to the FreeBSD VCS wiki one was described and then Jörg discussed the pending work and answered several questions in the Q&A.

For more information please give a look to the joerg's presentation slides (HTML). If you would like to help for the VCS migration please also get in touch with him!

Afternoon discussions and dinner

After the lunch we had several non-scheduled discussions, some time for hacking, etc. We then had a nice dinner together (it was in a restaurant with a very nice waiter who always shouted after every order or after accidently dropping and crashing dishes!, yeah! That's probably a bit weird but I liked that attitude! :)) and then did some sightseeing and had a beer together.

Photograph of the Friday dinner, taken by Christos
From left to right: uwe, bad, ast, leot, martin, abhinav, sborrill, alnsn, spz.

Photograph of the Friday dinner, taken by Abhinav
From left to right: uwe, bad, ast, christos, leot, martin, sborrill, alnsn, spz.

Saturday (23/09): First day of conference session and Social Event

A Modern Replacement for BSD spell(1) -- Abhinav Upadhyay (abhinav)

Abhinav (abhinav) presented his work on the new spell(1) implementation he's working (that isn't just a spell replacement but also a library that can be used by other programs!).

He described the current limitations of old spell(1) (to get an idea please give a look to bin/48684), described the project goals of the new spell(1), additions to /usr/share/dict/words, digged a bit in the implementation and discussed several algorithms used and then provided a performance comparison with other popular free software spell checkers (aspell, hunspell and ispell).

He also showed an interactive demo of the new spell(1) in-action integrated with a shell for auto-completion and spell check.

If you would like to try it please give a look to nbspell Git repository that contains the code and dicts for the new spell(1)!

Video recording (YouTube) of the talk and slides (PDF) are also available!

Portable Hotplugging: NetBSD's uvm_hotplug(9) API development -- Cherry G. Mathew (cherry)

Cherry (cherry) presented recent work done with Santhosh N. Raju (fox) about uvm_hotplug(9).

The talk covered most "behind the scenes" work: how TDD (test driven development) was used, how uvm_hotplug(9) was designed and implemented (with comparisons to the old implementation), interesting edge cases during the development and how atf(7) was used to do performance testing.

It was very interesting to learn how Cherry and Santhosh worked on that and on the conclusion Cherry pointed out the importance of using existing Software Engineering techniques in Systems Programming.

Video recording (YouTube) and slides (PDF) of the talk are also available!

Hardening pkgsrc -- Pierre Pronchery (khorben)

Pierre (khorben) presented a talk about recent pkgsrc security features added in the recent months (and most of them also active on the just released pkgsrc-2017Q3!).

He first introduced how security management and releng is handled in pkgsrc, how to use pkg_admin(1) fetch-pkg-vulnerabilities and audit commands, etc.

Then package signatures (generation, installation) and recent hardening features in pkgsrc were discussed in details, first introducing them and then how pkgsrc handles them:

  • SSP: Stack Smashing Protection (enabled via PKGSRC_USE_SSP in mk.conf)
  • Fortify (enabled via PKGSRC_USE_FORTIFY in mk.conf)
  • Stack check (enabled via PKGSRC_USE_STACK_CHECK in mk.conf)
  • Position-Independent Executables (PIE) (enabled via PKGSRC_MKPIE in mk.conf)
  • RELRO and BIND_NOW (enabled via PKGSRC_USE_RELRO in mk.conf)

Challenges for each hardening features and future works were discussed.

For more information video recording (YouTube) and slides (PDF) of the talk are available. A good introduction and reference for all pkgsrc hardening features is the Hardening packages wiki page.

Reproducible builds on NetBSD -- Christos Zoulas (christos)

Christos (christos) presented the work about reproducible builds on NetBSD.

In his talk he first provided a rationale about reproducible builds (to learn more please give a look to reproducible-builds.org!), he then discussed about the NetBSD (cross) build process, the current status and build variations that are done in the tests.reproducible-builds.org build machines.

Then he provided and described several sources of difference that were present in non-reproducible builds, like file-system timestamps, parallel builds headaches due directory/build order, path normalization, etc. For each of them he also discussed in details how these problems were solved in NetBSD.

In the conclusion the status and possible TODOs were also discussed (please note that both -current and -8 are all built with reproducible flags (-P option of build.sh)!)

Video recording (YouTube) of Christos' talk is available. Apart the resources discussed above a nice introduction to reproducible builds in NetBSD is also the NetBSD fully reproducible builds blog post written by Christos last February!

Social event

The social event on Saturday evening took place on a boat that cruised on the Seine river.

It was a very nice and different way to sightsee Paris, eat and enjoy some drinks and socialize and discuss with other developers and community.

Photograph from the boat, taken by Martin.

Sunday (24/09): Second day of conference session

The school of hard knocks - PT1 -- Sevan Janiyan (sevan)

Sevan (sevan) presented a talk about several notes and lessons learnt whilst running tutorials to introduce NetBSD at several events (OSHUG #46 and OSHUG #57 and #58) and experiences from past events (Chiphack 2013).

He described problems a user may experience and how NetBSD was introduced, in particular trying to avoid the steep learning curve involved when experimenting with operating systems as a first step, exploring documentation/source code, cross-building, scripting in high-level programming languages (Lua) and directly prototyping and getting pragmatic via rump.

Video recording (YouTube) of Sevan's talk and slides (HTML) are available.

The LLDB Debugger on NetBSD -- Kamil Rytarowski (kamil)

Kamil (kamil) presented a talk about the recent LLDB debugger and a lot of other related debuggers (but also non-strictly-related-to-debugging!) works he's doing in the last months.

He first introduced debugging concepts in general, provided several examples and then he started discussing LLDB porting to NetBSD.

He then discussed about ptrace(2) and other introspection interfaces, the several improvements done and tests added for ptrace(2) in atf(7).

He also discussed about tracking LLDB's trunk (if you are more curious please give a look to wip/llvm-git, wip/clang-git, wip/lldb-git packages in pkgsrc-wip!) and about LLVM sanitizers and their current status in NetBSD.

In the conclusion he also discussed various TODOs in these areas.

Video recording (YouTube) and slides (HTML) of Kamil's talk are available. Kamil also regularly write status update blog posts on blog.NetBSD.org and tech-toolchain@ mailing list, so please stay tuned! :)

What's in store for NetBSD 8.0? -- Alistair Crooks (agc)

Alistair (agc) presented a talk about what we will see in NetBSD 8.0.

He discussed about new hardware supported (really "new", not new "old" hardware! Of course also support for VAXstation 4000 TURBOchannel USB and GPIO is actually new hardware as well! :)), LLVM/Clang, virtualization, PGP signing, updated utilities in NetBSD, new networking features (e.g. bouyer's sockcan implementation), u-boot, dtrace(1), improvements and new ports testing, reproducible builds, FDT (Flattened Device Tree) and a lot of other news!

The entire presentation was done using the Socratic method (Q&A) and it was very interactive and nice!

Video recording (YouTube) and slides (PDF) of Alistair's talk are available.

Sunday dinner

After the conference we did some sightseeing in Paris, had a dinner together and then enjoyed some beers!

Photograph of the Sunday dinner, taken by Martin.
On the left side: abhinav, ast, seb, christos
On the right side: leot, Riastradh, uwe, sevan, agc, sborrill

Photograph of the Sunday dinner, taken by Abhinav.
On the left side: martin, ast, seb, christos
On the right side: leot, Riastradh, uwe, sevan, agc, sborrill

Conclusion

It was a very nice weekend and conference. It is worth to mention that EuroBSDcon 2017 was the biggest BSD conference (more than 300 people attended it!).

I would like to thank the entire EuroBSDcon organising committee (Baptiste Daroussin, Antoine Jacoutot, Jean-Sébastien Pédron and Jean-Yves Migeon), EuroBSDcon programme commitee (Antoine Jacoutot, Lars Engels, Ollivier Robert, Sevan Janiyan, Jörg Sonnenberger, Jasper Lievisse Adriaanse and Janne Johansson) and EuroBSDcon Foundation for organizing such a wonderful conference!

I also would like to thank the speakers for presenting very interesting talks, all developers and community that attended the NetBSD devsummit and conference, in particular Jean-Yves and Jörg, for organizing and moderating the devsummit and Arolla that kindly hosted us for the NetBSD devsummit!

A special thanks also to Abhinav (abhinav) and Martin (martin) for photographs and locals Jean-Yves (jym) and Stoned (seb) for helping us in not get lost in Paris' rues! :)

Thank you!

Posted at lunch time on Wednesday, October 11th, 2017 Tags: blog
Add a comment
Contact | Disclaimer | Copyright © 1994-2017 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
NetBSD® is a registered trademark of The NetBSD Foundation, Inc.