![[NetBSD Logo]](http://www.NetBSD.org/images/NetBSD-smaller.png)
This page is a blog mirror of sorts. It pulls in articles from blog's feed and publishes them here (with a feed, too).
A new SUNXI evbarm kernel has appeared recently in NetBSD -current with support for boards based on the Allwinner H3 system on a chip (SoC). The H3 SoC is a quad-core Cortex-A7 SoC designed primarily for set-top boxes, but has managed to find its way into many single-board computers (SBC). This is one of the first evbarm ports built from the ground up with device tree support, which helps us to use a single kernel config to support many different boards.
To get these boards up and running, first we need to deal with low-level startup code. For the SUNXI kernel this currently lives in sys/arch/evbarm/sunxi/. The purpose of this code is fairly simple; initialize the boot CPU and initialize the MMU so we can jump to the kernel. The initial MMU configuration needs to cover a few things -- early on we need to be able to access the kernel, UART debug console, and the device tree blob (DTB) passed in from U-Boot. We wrap the kernel in a U-Boot header that claims to be a Linux kernel; this is no accident! This tells U-Boot to use the Linux boot protocol when loading the kernel, which ensures that the DTB (loaded by U-Boot) is processed and passed to us in r2.
Once the CPU and MMU are ready, we jump to the generic ARM FDT implementation of initarm in sys/arch/evbarm/fdt/fdt_machdep.c. The first thing this code does is validate and relocate the DTB data. After it has been relocated, we compare the compatible property of the root node in the device tree with the list of ARM platforms compiled into the kernel. The Allwinner sunxi platform code lives in sys/arch/arm/sunxi/sunxi_platform.c. The sunxi platform code provides SoC-specific versions of code needed early at boot. We need to know how to initialize the debug console, spin up application CPUs, reset the board, etc.
Instead of writing H3-specific code for spinning up application CPUs, I took advantage of U-Boot's Power State Coordination Interface implementation. A psci(4) driver was added and the allwinner,sun8i-h3 platform code was modified to use this code to start up all processors.
With a bit of luck, we're now booting and enumerating devices. Apart from a few devices, almost nothing works yet as we are missing a driver for the CCU. The CCU in the Allwinner H3 SoC controls PLLs and most of the clock generation, division, muxing, and gating. Since there are many similarities between Allwinner SoCs, I opted to write generic CCU code and then SoC-specific frontends. The resulting code lives in sys/arch/arm/sunxi/; generic code as sunxi_ccu.c and H3-specific code in sun8i_h3_ccu.c.
Now we have a CCU driver, we can attach a com(4) and have a valid console device.
After this, it's a matter of writing drivers and/or adapting existing code to attach to fdtbus based on the bindings used in the DTB. For cases where we had a compatible driver in the old Allwinner port, I opted to make a copy of the code and FDT-ize it. A few reasons for this -- 1) the old drivers have CCU-specific code with per-SoC ifdefs scattered throughout, 2) I didn't want to break existing kernels, and 3) long term goal is to move the SoCs supported by the old code over to the new code (this process has already started with the Allwinner A31 port).
So what do we get out of this? This is a step towards being able to ship a GENERIC evbarm kernel. I developed the H3 port on two boards, the NanoPi NEO and Orange Pi Plus 2E, but since then users on port-arm@ have been reporting success on many other H3 boards, all from a single kernel config. In addition, I've added support for other Allwinner SoCs (sun8i-a83t, sun6i-a31) to the kernel and have tested booting the same kernel across all 3 SoCs.
Orange Pi Plus 2E boot log is below.
U-Boot SPL 2017.05 (Jul 01 2017 - 17:11:09)
DRAM: 2048 MiB
Trying to boot from MMC1
U-Boot 2017.05 (Jul 01 2017 - 17:11:09 -0300) Allwinner Technology
CPU: Allwinner H3 (SUN8I 1680)
Model: Xunlong Orange Pi Plus 2E
I2C: ready
DRAM: 2 GiB
MMC: SUNXI SD/MMC: 0, SUNXI SD/MMC: 1
In: serial
Out: serial
Err: serial
Net: phy interface7
eth0: ethernet@1c30000
starting USB...
USB0: USB EHCI 1.00
USB1: USB OHCI 1.0
USB2: USB EHCI 1.00
USB3: USB OHCI 1.0
USB4: USB EHCI 1.00
USB5: USB OHCI 1.0
scanning bus 0 for devices... 2 USB Device(s) found
scanning bus 2 for devices... 1 USB Device(s) found
scanning bus 4 for devices... 1 USB Device(s) found
scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot: 0
reading netbsd.ub
6600212 bytes read in 334 ms (18.8 MiB/s)
reading sun8i-h3-orangepi-plus2e.dtb
16775 bytes read in 49 ms (334 KiB/s)
## Booting kernel from Legacy Image at 42000000 ...
Image Name: NetBSD/sunxi 8.99.1
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 6600148 Bytes = 6.3 MiB
Load Address: 40008000
Entry Point: 40008000
Verifying Checksum ... OK
## Flattened Device Tree blob at 43000000
Booting using the fdt blob at 0x43000000
Loading Kernel Image ... OK
Loading Device Tree to 49ff8000, end 49fff186 ... OK
Starting kernel ...
[ Kernel symbol table missing! ]
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017
The NetBSD Foundation, Inc. All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
NetBSD 8.99.1 (SUNXI) #304: Sat Jul 8 11:01:22 ADT 2017
jmcneill@undine.invisible.ca:/usr/home/jmcneill/netbsd/cvs-src/sys/arch/evbarm/compile/obj/SUNXI
total memory = 2048 MB
avail memory = 2020 MB
sysctl_createv: sysctl_create(machine_arch) returned 17
armfdt0 (root)
fdt0 at armfdt0: Xunlong Orange Pi Plus 2E
fdt1 at fdt0
fdt2 at fdt0
cpus0 at fdt0
cpu0 at cpus0: Cortex-A7 r0p5 (Cortex V7A core)
cpu0: DC enabled IC enabled WB disabled EABT branch prediction enabled
cpu0: 32KB/32B 2-way L1 VIPT Instruction cache
cpu0: 32KB/64B 4-way write-back-locking-C L1 PIPT Data cache
cpu0: 512KB/64B 8-way write-through L2 PIPT Unified cache
vfp0 at cpu0: NEON MPE (VFP 3.0+), rounding, NaN propagation, denormals
cpu1 at cpus0
cpu2 at cpus0
cpu3 at cpus0
gic0 at fdt1: GIC
armgic0 at gic0: Generic Interrupt Controller, 160 sources (150 valid)
armgic0: 16 Priorities, 128 SPIs, 7 PPIs, 15 SGIs
fclock0 at fdt2: 24000000 Hz fixed clock
ffclock0 at fdt2: x1 /1 fixed-factor clock
fclock1 at fdt2: 32768 Hz fixed clock
sunxigates0 at fdt2
sunxiresets0 at fdt1
gtmr0 at fdt0: Generic Timer
armgtmr0 at gtmr0: ARMv7 Generic 64-bit Timer (24000 kHz)
armgtmr0: interrupting on irq 27
sunxigpio0 at fdt1: PIO
gpio0 at sunxigpio0: 94 pins
sunxigpio1 at fdt1: PIO
gpio1 at sunxigpio1: 12 pins
sun8ih3ccu0 at fdt1: H3 CCU
fregulator0 at fdt0: vcc3v3
fregulator1 at fdt0: gmac-3v3
fregulator2 at fdt0: vcc3v0
fregulator3 at fdt0: vcc5v0
sunxiusbphy0 at fdt1: USB PHY
/soc/dma-controller@01c02000 at fdt1 not configured
/soc/codec-analog@01f015c0 at fdt1 not configured
/clocks/ir_clk@01f01454 at fdt2 not configured
sunxiemac0 at fdt1: EMAC
sunxiemac0: interrupting on GIC irq 114
rgephy0 at sunxiemac0 phy 0: RTL8169S/8110S/8211 1000BASE-T media interface, rev. 5
rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
rgephy1 at sunxiemac0 phy 1: RTL8169S/8110S/8211 1000BASE-T media interface, rev. 5
rgephy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
psci0 at fdt0: PSCI 0.1
gpioleds0 at fdt0: orangepi:green:pwr orangepi:red:status
gpiokeys0 at fdt0: sw4
sunximmc0 at fdt1: SD/MMC controller
sunximmc0: interrupting on GIC irq 92
sunximmc1 at fdt1: SD/MMC controller
sunximmc1: interrupting on GIC irq 93
sunximmc2 at fdt1: SD/MMC controller
sunximmc2: interrupting on GIC irq 94
ehci0 at fdt1: EHCI
ehci0: interrupting on GIC irq 106
ehci0: 1 companion controller, 1 port
usb0 at ehci0: USB revision 2.0
ohci0 at fdt1: OHCI
ohci0: interrupting on GIC irq 107
ohci0: OHCI version 1.0
usb1 at ohci0: USB revision 1.0
ehci1 at fdt1: EHCI
ehci1: interrupting on GIC irq 108
ehci1: 1 companion controller, 1 port
usb2 at ehci1: USB revision 2.0
ohci1 at fdt1: OHCI
ohci1: interrupting on GIC irq 109
ohci1: OHCI version 1.0
usb3 at ohci1: USB revision 1.0
ehci2 at fdt1: EHCI
ehci2: interrupting on GIC irq 110
ehci2: 1 companion controller, 1 port
usb4 at ehci2: USB revision 2.0
ohci2 at fdt1: OHCI
ohci2: interrupting on GIC irq 111
ohci2: OHCI version 1.0
usb5 at ohci2: USB revision 1.0
/soc/timer@01c20c00 at fdt1 not configured
/soc/watchdog@01c20ca0 at fdt1 not configured
/soc/codec@01c22c00 at fdt1 not configured
com0 at fdt1: ns16550a, working fifo
com0: console
com0: interrupting on GIC irq 32
sunxirtc0 at fdt1: RTC
/soc/ir@01f02000 at fdt1 not configured
cpu3: Cortex-A7 r0p5 (Cortex V7A core)
cpu3: DC enabled IC enabled WB disabled EABT branch prediction enabled
cpu3: 32KB/32B 2-way L1 VIPT Instruction cache
cpu3: 32KB/64B 4-way write-back-locking-C L1 PIPT Data cache
cpu3: 512KB/64B 8-way write-through L2 PIPT Unified cache
vfp3 at cpu3: NEON MPE (VFP 3.0+), rounding, NaN propagation, denormals
cpu1: Cortex-A7 r0p5 (Cortex V7A core)
cpu1: DC enabled IC enabled WB disabled EABT branch prediction enabled
cpu1: 32KB/32B 2-way L1 VIPT Instruction cache
cpu1: 32KB/64B 4-way write-back-locking-C L1 PIPT Data cache
cpu1: 512KB/64B 8-way write-through L2 PIPT Unified cache
vfp1 at cpu1: NEON MPE (VFP 3.0+), rounding, NaN propagation, denormals
cpu2: Cortex-A7 r0p5 (Cortex V7A core)
cpu2: DC enabled IC enabled WB disabled EABT branch prediction enabled
cpu2: 32KB/32B 2-way L1 VIPT Instruction cache
cpu2: 32KB/64B 4-way write-back-locking-C L1 PIPT Data cache
cpu2: 512KB/64B 8-way write-through L2 PIPT Unified cache
vfp2 at cpu2: NEON MPE (VFP 3.0+), rounding, NaN propagation, denormals
sdmmc0 at sunximmc0
sdmmc1 at sunximmc1
sdmmc2 at sunximmc2
uhub0 at usb0: Generic (0000) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
uhub1 at usb2: Generic (0000) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
uhub2 at usb3: Generic (0000) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
uhub3 at usb1: Generic (0000) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
uhub4 at usb4: Generic (0000) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
uhub5 at usb5: Generic (0000) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
ld2 at sdmmc2: <0x15:0x0100:AWPD3R:0x00:0xec19649f:0x000>
sdmmc0: SD card status: 4-bit, C10, U1, V10
ld0 at sdmmc0: <0x27:0x5048:2&DRP:0x07:0x01c828bc:0x109>
ld2: 14910 MB, 7573 cyl, 64 head, 63 sec, 512 bytes/sect x 30535680 sectors
ld0: 15288 MB, 7765 cyl, 64 head, 63 sec, 512 bytes/sect x 31309824 sectors
(manufacturer 0x24c, product 0xf179, standard function interface code 0x7)at sdmmc1 function 1 not configured
ld2: mbr partition exceeds disk size
ld0: 4-bit width, High-Speed/SDR25, 50.000 MHz
ld2: 8-bit width, 52.000 MHz
urtwn0 at uhub0 port 1
urtwn0: Realtek (0xbda) 802.11n NIC (0x8179), rev 2.00/0.00, addr 2
urtwn0: MAC/BB RTL8188EU, RF 6052 1T1R, address e8:de:27:16:0c:81
urtwn0: 1 rx pipe, 2 tx pipes
urtwn0: 11b rates: 1Mbps 2Mbps 5.5Mbps 11Mbps
urtwn0: 11g rates: 1Mbps 2Mbps 5.5Mbps 11Mbps 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
boot device: ld0
root on ld0a dumps on ld0b
root file system type: ffs
kern.module.path=/stand/evbarm/8.99.1/modules
WARNING: clock lost 6398 days
WARNING: using filesystem time
WARNING: CHECK AND RESET THE DATE!
Sat Jul 8 11:05:42 ADT 2017
Starting root file system check:
/dev/rld0a: file system is clean; not checking
Not resizing /: already correct size
swapctl: adding /dev/ld0b as swap device at priority 0
Starting file system checks:
/dev/rld0e: 22 files, 32340 free (8085 clusters)
random_seed: /var/db/entropy-file: Not present
Setting tty flags.
Setting sysctl variables:
ddb.onpanic: 1 -> 1
Starting network.
Hostname: sunxi
IPv6 mode: host
Configuring network interfaces:.
Adding interface aliases:.
Waiting for DAD to complete for statically configured addresses...
Starting dhcpcd.
Starting mdnsd.
Building databases: dev, utmp, utmpx.
Starting syslogd.
Mounting all file systems...
Clearing temporary files.
Updating fontconfig cache: done.
Creating a.out runtime link editor directory cache.
Checking quotas: done.
Setting securelevel: kern.securelevel: 0 -> 1
Starting virecover.
Checking for core dump...
savecore: no core dump
Starting devpubd.
Starting local daemons:.
Updating motd.
Starting ntpd.
Jul 8 11:05:58 sunxi ntpd[595]: ntp_rlimit: Cannot set RLIMIT_STACK: Invalid argument
Starting sshd.
Starting inetd.
Starting cron.
Sat Jul 8 11:06:02 ADT 2017
NetBSD/evbarm (sunxi) (console)
login:
This years pkgsrcCon returned to London once again. It was last held in London back in 2014. The 2014 con was the first pkgsrcCon I attended, I had been working on Darwin/PowerPC fixes for some months and presented on the progress I'd made with a 12" G4 PowerBook. I took away a G4 Mac Mini that day to help spare the PowerBook for use and dedicate a machine for build and testing. The offer of PowerPC hardware donations was repeated at this years con, thanks to jperkin@ who showed up with a backpack full of Mac Minis (more on that later).
Since 2014 we have held cons in Berlin (2015) & Krakow (2016). In Krakow we had talks about a wide range of projects over 2 days, from Haiku Ports to Common Lisp to midipix (building native PE binaries for Windows) and back to the BSDs. I was very pleased to continue the theme of a diverse program this year.
Aside from pkgsrc and NetBSD, we had talks about FreeBSD, OpenBSD, Slackware Linux, and Plan 9.
Things began with a pub gathering on the Friday for the pre-con social, we hung out and chatted till almost midnight on a wide range of topics, such as supporting a system using NFS on MS-DOS, the origins of pdksh, corporate IT, culture and many other topics.
On parting I was asked about the starting time on Saturday as there was some conflicting information. I learnt that the registration email had stated a later start than I had scheduled for & advertised on the website, by 30 minutes.
Lesson learnt: register for your own event!
Not a problem, I still needed to setup a webpage for the live video stream, I could do both when I got back. With some trimming here and there I had a new schedule, I posted that to the pkgsrcCon website and moved to trying to setup a basic web page which contained a snippet of javascript to play a live video stream from Scale Engine.
2+ hours later, it was pointed out that the XSS protection headers on pkgsrc.org breaks the functionality. Thanks to jmcneill@ for debugging and providing a working page.
Saturday started off with Giovanni Bechis speaking about pledge in OpenBSD and adding support to various packages in their ports tree, alnsn@ then spoke about installing packages from a repo hosted on the Tor network.
After a quick coffee break we were back to hear Charles Forsyth speak about how Plan 9 and Inferno dealt with portability, building software and the problem which are avoided by the environment there. This was followed by a very energetic rant by David Spencer from the Slackbuilds project on packaging 3rd party software. Slackbuilds is a packaging system for Slackware Linux, which was inspired by FreeBSD ports.
For the first slot after lunch, agc@ gave a talk on the early history of pkgsrc followed by Thomas Merkel on using vagrant to test pkgsrc changes with ease, locally, using vagrant. khorben@ covered his work on adding security to pkgsrc and bsiegert@ covered the benefits of performing our bulk builds in the cloud and the challenges we currently face.
My talk was about some topics and ideas which had inspired me or caught my attention, and how it could maybe apply to my work.The title of the talk was taken from the name of Andrew Weatherall's Saint Etienne remix, possibly referring to two different styles of track (dub & vocal) merged into one or something else. I meant it in terms of applicability of thoughts and ideas. After me, agc@ gave a second talk on the evolution of the Netflix Open Connect appliance which runs FreeBSD and Vsevolod Stakhov wrapped up the day with a talk about the technical implementation details of the successor to pkg_tools in FreeBSD, called pkg, and how it could be of benefit for pkgsrc.
Netflix confirms it: BSD is not dying #pkgsrcCon
— Benny Siegert (@bentsukun) July 1, 2017
For day 2 we gathered for a hack day at the London Hack Space.
I had burn't some some CD of the most recent macppc builds of NetBSD 8.0_BETA and -current to install and upgrade Mac Minis. I setup the donated G4 minis for everyone in a dual-boot configuration and moved on to taking apart my MacBook Air to inspect the wifi adapter as I wanted to replace it with something which works on FreeBSD. It was not clear from the ifixit teardown photos of cards size, it seemed like a normal mini-PCIe card but it turned out to be far smaller. Thomas had also had the same card in his and we are not alone. Thomas has started putting together a driver for the Broadcom card, the project is still in its early days and lacks support for encrypted networks but hopefully it will appear on review.freebsd.org in the future.
weidi@ worked on fixing SunOS bugs in various packages and later in the night we setup a NetBSD/macppc bulk build environment together on his Mac Mini.
Thomas setup an OpenGrock instance to index the source code of all the software available for packaging in pkgsrc. This helps make the evaluation of changes easier and the scope of impact a little quicker without having to run through a potentially lengthy bulk build with a change in mind to realise the impact.
bsiegert@ cleared his ticket and email backlog for pkgsrc and alnsn@ got NetBSD/evbmips64-eb booting on his EdgeRouter Lite.
#pkgsrcCon hackathon work by @docscream: https://t.co/3GY3TRtdNV code search over everything (still indexing)
— Wiedi (@wied0r) July 2, 2017
On Monday we reconvened at the Hack Space again and worked some more. I started putting together the talks page with the details from Saturday and the the slides which I had received, in preperation for the videos which would come later in the week. By 3pm pkgsrcCon was over. I was pretty exhausted but really pleased to have had a few days of techie fun.
Many thanks to The NetBSD Foundation for purchasing a camera to use for streaming the event and a speedy response all round by the board. The Open Source Specialist Group at BCS, The Chartered Institute for IT and the London Hack Space for hosting us. Scale Engine for providing streaming facility. weidi@ for hosting the recorded videos.
Allan Jude for pointers, Jared McNeill for debugging, NYCBUG and Patrick McEvoy for tips on streaming, the attendees and speakers. This year we had speakers from USA, Italy, Germany and London E2.
Looking forward to pkgsrcCon 2018!
The videos and slides are available here and the Internet Archive.
dogfood
When I have realized that in order to work on the LLVM sanitizers I need to use Clang as the compiler. A part of the compiler-rt (lowlevel LLVM compiler runtime library) has code specifically incompatible with GCC. It was mainly related to intrinsic instructions for atomic functions. I tried to research how much work is needed to port it to GCC. It happened to be non-trivial and I filed a bug on the LLVM bugzilla.
These circumstances made me to switch to Clang as the pkgsrc toolchain. I was using it to test the compilation of small bulks of packages and record build and compiler problems. To save time, I used ccache as my cache for builds.
My options in mk.conf(5):
PKGSRC_COMPILER= ccache clang CCACHE_BASE= /usr/local CCACHE_DIR= /public/ccache_tmp CCACHE_LOGFILE= /tmp/ccache.txt PKG_CC= clang PKG_CXX= clang++ CLANGBASE= /usr/local HAVE_LLVM= yes
It's worth noting that ccache with pkgsrc won't check $HOME/.ccache for configuration, it must be placed in $CCACHE_DIR/ccache.conf.
The documented problems can be summarized as:
- Broken ccache in pkgsrc for C++11 packages.
The pkgsrc framework started supporting C++ languages in the USE_LANGUAGES definition. Packages with newly added USE_LANGUAGES values (such as c++11) were not compiled with ccache because ccache.mk was not yet aware of such values. This broke support of these packages to set these values to be built with ccache. I've corrected it and introduced a new option CCACHE_LOGFILE to more easily track execution of ccache and detect errors. - Broken ccache in pkgsrc for a custom toolchain.
ccache tries finding a real-compiler looking for it in $PATH. When I have built clang within pkgsrc to work on it (installed into/usr/pkg), and I had my main toolchain in/usr/localit was picking the one from/usr/pkgfor new builds and it resulted in cache-misses for new builds. I have installed a fix for it to pass ccache specific PATH to always find the appropriate compiler. - Header <execinfo.h> cannot be included on Clang 5.0.0svn.
For some reason compilers tend to install their own headers that overshadow the system headers. This resulted in build failures in programs including plain <execinfo.h> header (for the backtrace(3) function). This system header used to include <stddef.h> that included our <sys/cdefs.h>... with shadowed <stddef.h> by Clang 5.0.0svn (from $PREFIX/lib/clang/5.0.0/include/stddef.h). Christos Zoulas fixed it by making <execinfo.h> standalone and independent from standard libc headers. - __float128 and GNU libstdc++.
Our basesystem GNU libstdc++ enables __float128 on i386, amd64 and i64 ports. As of now the LLVM equivalent library contains partial support for this type. This results in a problem that affects 3rd party programs in the setup of Clang + libstdc++ detect __float128 support and break because the compiler does not define appropriate global define __FLOAT128__. This issue is still open for discussion on how to solve it for NetBSD. - gforth optimization problems.
Upstream gforth developers ported this FORTH compiler to Clang, and triggered an optimization issue with attempting to needlessly solve a complex internal problem. This results with compilation times of several minutes on a modern CPUs instead of getting the results immediately. The problem has been already reported on the LLVM bugzilla and I have filed a report that it is still valid. - bochs can be built with clang.
A while ago, bochs was buildable only by the GCC compilers and the Clang toolchain was blacklisted. I have verified that this is no longer the case and unmasked the package for compilers other than GCC.
LLVM and Clang testsuites
I have prepared Clang and LLVM testsuites to execute on NetBSD. Correctness of both projects is crucial for LLDB and the LLVM sanitizers to work because their issues resound problems inside programs that depend on them. Originally I have corrected the tests with local patches to build with GCC, and switched later to Clang. I have restructured the packages in pkgsrc-wip in order to execute the test-suite. I have fixed 20 test failures in LLVM implementing AllocateRWX and ReleaseRWX for the NetBSD flavor of PaX MPROTECT. There are still over 200 failures to solve!
It's worth noting that the googletest library (used in a modified version in LLVM and in a regular one in Clang) finally accepted the NetBSD patches.
LLVM asan and ubsan
I expect to get four LLVM sanitizers working in order to move on to LLDB: asan (address sanitizer), ubsan (undefined behavior sanitizer), tsan (thread sanitizer), msan (memory sanitizer). The other ones like dfsan (data-flow sanitizer) or lsan (leak sanitizer) are currently to be skipped. In general, sanitizers are part of the LLDB functionality that I want to get aboard on NetBSD, as there are plugins to integrate them within the debugger. In the current state I require them to debug bugs inside LLDB/NetBSD.
The original work on sanitizers in GCC (with libsanitizer) has been done by Christos Zoulas. GCC libsanitizer is a close sibling of compiler-rt/lib from the LLVM project. I picked up his work and integrated it into compiler-rt and developed the rest (code differences, fixing bugs, Clang/LLVM specific parts in llvm/ and clang/) and I managed to get asan and ubsan to work.
Users should pickup pkgsrc-wip in revision 3e7c52b97b4d6cb8ea69a081409ac818c812c34a and install wip/{llvm,clang,compiler-rt}-netbsd. Clang will be ready for usage:
/usr/pkg/bin/clang -fsanitize=undefined test.cand
/usr/pkg/bin/clang -fsanitize=address test.c
Additional compiler commands that may improve the experience:
-g -O0 -fno-omit-frame-pointer -pie -fPIE
These sanitizers are not production ready and are active under development.
Plan for the next milestone
Roadmap for the next month:
- Finish and send upstream LLVM asan and ubsan support.
- Correct more problems triggered by LLVM and Clang test-suites.
- Resume msan and tsan porting.
This work was sponsored by The NetBSD Foundation.
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
The last year's pkgsrcCon 2016 event took place in Kraków, Poland.
Slides are available on the event site.
Video recordings are stored at https://archive.org/details/pkgsrcCon-2016.
We would like to thank the organizers, sponsors and promotion from the Jagiellonian University, The NetBSD Foundation, Programista Magazyn, OS World, and Subcarpathian BSD User Group.
Born in 1969, he developed a passion for computer science in his youth but chose to study biology. He then attended a bioinformatics training at the Pasteur Institute in 1997 at the end of which he got hired in the scientific IT department. He has continuously worked there since. Not only was he skilled in biology but also in system administration, databases, and programming.
He was introduced to NetBSD in 1999 and quickly became a dedicated user and hacker. He became a NetBSD developer in 2007 for working on Linux/i386 (32 bits) emulation on NetBSD/amd64 (64 bits).
Nicolas leaves a wife and three children.
Memorial note submitted by Marc Baudoin.
Born in 1969, he developed a passion for computer science in his youth but chose to study biology. He then attended a bioinformatics training at the Pasteur Institute in 1997 at the end of which he got hired in the scientific IT department. He has continuously worked there since. Not only was he skilled in biology but also in system administration, databases, and programming.
He was introduced to NetBSD in 1999 and quickly became a dedicated user and hacker. He became a NetBSD developer in 2007 for working on Linux/i386 (32 bits) emulation on NetBSD/amd64 (64 bits).
Nicolas leaves a wife and three children.
Memorial note submitted by Marc Baudoin.
the repository conversion setup for NetBSD CVS -> Fossil -> Git has found a new home. Ironically, on former cvs.NetBSD.org hardware. This provides a somewhat faster conversion cycle as well as removing anoncvs.NetBSD.org from the process. This should avoid occasional problems with incomplete syncs. Two other changes have been applied at the same time:
- The Fossil repositories have moved to using the SHA512 checksums internally. To avoid accidents, a new project code is used. This requires Fossil 2.x.
- The Git repositories map user names to @NetBSD.org addresses (src, xsrc) or @pkgsrc.org addresses (pkgsrc). This allows consolidation on github with user accounts, assuming you have the corresponding addresses as primary or secondary address.
The new locations for the repositories are:
The old conversions will be provided for the near future, but likely stop toward the end of the month.
A special thanks goes to Petra and Christos from the admin team for the assistance with the machine setup and low-level CVS clean-ups.
Joerg
-- source https://mail-index.netbsd.org/tech-repository/2017/06/10/msg000637.html
the repository conversion setup for NetBSD CVS -> Fossil -> Git has found a new home. Ironically, on former cvs.NetBSD.org hardware. This provides a somewhat faster conversion cycle as well as removing anoncvs.NetBSD.org from the process. This should avoid occasional problems with incomplete syncs. Two other changes have been applied at the same time:
- The Fossil repositories have moved to using the SHA512 checksums internally. To avoid accidents, a new project code is used. This requires Fossil 2.x.
- The Git repositories map user names to @NetBSD.org addresses (src, xsrc) or @pkgsrc.org addresses (pkgsrc). This allows consolidation on github with user accounts, assuming you have the corresponding addresses as primary or secondary address.
The new locations for the repositories are:
The old conversions will be provided for the near future, but likely stop toward the end of the month.
A special thanks goes to Petra and Christos from the admin team for the assistance with the machine setup and low-level CVS clean-ups.
Joerg
-- source https://mail-index.netbsd.org/tech-repository/2017/06/10/msg000637.html
It turned out that more bugs were unveiled and this is not the final report on LLDB.
The good
Besides the greater enhancements this month I performed a cleanup in the ATF ptrace(2) tests again. Additionally I have managed to unbreak the LLDB Debug build and to eliminate compiler warnings in the NetBSD Native Process Plugin.
It is worth noting that LLVM can run tests on NetBSD again, the patch in gtest/LLVM has been installed by Joerg Sonnenberg and a more generic one has been submitted to the upstream googletest repository. There was also an improvement in ftruncate(2) on the LLVM side (authored by Joerg).
Since LLD (the LLVM linker) is advancing rapidly, it improved support for NetBSD and it can link a functional executable on NetBSD. I submitted a patch to stop crashing it on startup anymore. It was nearly used for linking LLDB/NetBSD and it spotted a real linking error... however there are further issues that need to be addressed in the future. Currently LLD is not part of the mainline LLDB tasks - it's part of improving the work environment. This linker should reduce the linking time - compared to GNU linkers - of LLDB by a factor of 3x-10x and save precious developer time. As of now LLDB linking can take minutes on a modern amd64 machine designed for performance.
Kernel correctness
I have researched (in pkgsrc-wip) initial support for multiple threads in the NetBSD Native Process Plugin. This code revealed - when running the LLDB regression test-suite - new kernel bugs. This unfortunately affects the usability of a debugger in a multithread environment in general and explains why GDB was never doing its job properly in such circumstances.
One of the first errors was asserting kernel panic with PT_*STEP, when a debuggee has more than a single thread. I have narrowed it down to lock primitives misuse in the do_ptrace() kernel code. The fix has been committed.
LLDB and userland correctness
LLDB introduced support for kevent(2) and it contains the following function:
Status MainLoop::RunImpl::Poll() {
in_events.resize(loop.m_read_fds.size());
unsigned i = 0;
for (auto &fd : loop.m_read_fds)
EV_SET(&in_events[i++], fd.first, EVFILT_READ, EV_ADD, 0, 0, 0);
num_events = kevent(loop.m_kqueue, in_events.data(), in_events.size(),
out_events, llvm::array_lengthof(out_events), nullptr);
if (num_events < 0)
return Status("kevent() failed with error %d\n", num_events);
return Status();
}
It works on FreeBSD and MacOSX, however it broke on NetBSD.
Culprit line:
EV_SET(&in_events[i++], fd.first, EVFILT_READ, EV_ADD, 0, 0, 0);
FreeBSD defined EV_SET() as a macro this way:
#define EV_SET(kevp_, a, b, c, d, e, f) do { \
struct kevent *kevp = (kevp_); \
(kevp)->ident = (a); \
(kevp)->filter = (b); \
(kevp)->flags = (c); \
(kevp)->fflags = (d); \
(kevp)->data = (e); \
(kevp)->udata = (f); \
} while(0)
NetBSD version was different:
#define EV_SET(kevp, a, b, c, d, e, f) \
do { \
(kevp)->ident = (a); \
(kevp)->filter = (b); \
(kevp)->flags = (c); \
(kevp)->fflags = (d); \
(kevp)->data = (e); \
(kevp)->udata = (f); \
} while (/* CONSTCOND */ 0)
This resulted in heap damage, as keyp was incremented every time a value was
assigned to (keyp)->.
Without GCC asan and ubsan tools, finding this bug would be much more time consuming, as the random memory corruption was affecting unrelated lambda function in a different part of the code.
To use the GCC sanitizers with packages from pkgsrc, on NetBSD-current, one has to use one or both of these lines:
_WRAP_EXTRA_ARGS.CXX+= -fno-omit-frame-pointer -O0 -g -ggdb -U_FORTIFY_SOURCE -fsanitize=address -fsanitize=undefined -lasan -lubsan CWRAPPERS_APPEND.cxx+= -fno-omit-frame-pointer -O0 -g -ggdb -U_FORTIFY_SOURCE -fsanitize=address -fsanitize=undefined -lasan -lubsan
While there, I have fixed another - generic - bug in the LLVM headers. The class Triple constructor hadn't initialized the SubArch field, which upsetting the GCC address sanitizer. It was triggered in LLDB in the following code:
void ArchSpec::Clear() {
m_triple = llvm::Triple();
m_core = kCore_invalid;
m_byte_order = eByteOrderInvalid;
m_distribution_id.Clear();
m_flags = 0;
}
I have filed a patch for review to address this.
The bad
Unfortunately this is not the full story and there is further mandatory work.
LLDB acceleration
The EV_SET() bug broke upstream LLDB over a month ago, and during this period the debugger was significantly accelerated and parallelized. It is difficult to declare it definitely, but it might be the reason why the tracer's runtime broke due to threading desynchronization. LLDB behaves differently when run standalone, under ktruss(1) and under gdb(1) - the shared bug is that it always fails in one way or another, which isn't trivial to debug.
The ugly
There are also unpleasant issues at the core of the Operating System.
Kernel troubles
Another bug with single-step functions that affects another aspect of correctness - this time with reliable execution of a program - is that processes die in non-deterministic ways when single-stepped. My current impression is that there is no appropriate translation between process and thread (LWP) states under a debugger.
These issues are sibling problems to unreliable PT_RESUME and PT_SUSPEND.
In order to be able to appropriately address this, I have diligently studied this month the Solaris Internals book to get a better image of the design of the NetBSD kernel multiprocessing, which was modeled after this commercial UNIX.
Plan for the next milestone
The current troubles can be summarized as data races in the kernel and at the same time in LLDB. I have decided to port the LLVM sanitizers, as I require the Thread Sanitizer (tsan). Temporarily I have removed the code for tracing processes with multiple threads to hide the known kernel bugs and focus on the LLDB races.
Unfortunately LLDB is not easily bisectable (build time of the LLVM+Clang+LLDB stack, number of revisions), therefore the debugging has to be performed on the most recent code from upstream trunk.
This work was sponsored by The NetBSD Foundation.
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
It turned out that more bugs were unveiled and this is not the final report on LLDB.
The good
Besides the greater enhancements this month I performed a cleanup in the ATF ptrace(2) tests again. Additionally I have managed to unbreak the LLDB Debug build and to eliminate compiler warnings in the NetBSD Native Process Plugin.
It is worth noting that LLVM can run tests on NetBSD again, the patch in gtest/LLVM has been installed by Joerg Sonnenberg and a more generic one has been submitted to the upstream googletest repository. There was also an improvement in ftruncate(2) on the LLVM side (authored by Joerg).
Since LLD (the LLVM linker) is advancing rapidly, it improved support for NetBSD and it can link a functional executable on NetBSD. I submitted a patch to stop crashing it on startup anymore. It was nearly used for linking LLDB/NetBSD and it spotted a real linking error... however there are further issues that need to be addressed in the future. Currently LLD is not part of the mainline LLDB tasks - it's part of improving the work environment. This linker should reduce the linking time - compared to GNU linkers - of LLDB by a factor of 3x-10x and save precious developer time. As of now LLDB linking can take minutes on a modern amd64 machine designed for performance.
Kernel correctness
I have researched (in pkgsrc-wip) initial support for multiple threads in the NetBSD Native Process Plugin. This code revealed - when running the LLDB regression test-suite - new kernel bugs. This unfortunately affects the usability of a debugger in a multithread environment in general and explains why GDB was never doing its job properly in such circumstances.
One of the first errors was asserting kernel panic with PT_*STEP, when a debuggee has more than a single thread. I have narrowed it down to lock primitives misuse in the do_ptrace() kernel code. The fix has been committed.
LLDB and userland correctness
LLDB introduced support for kevent(2) and it contains the following function:
Status MainLoop::RunImpl::Poll() {
in_events.resize(loop.m_read_fds.size());
unsigned i = 0;
for (auto &fd : loop.m_read_fds)
EV_SET(&in_events[i++], fd.first, EVFILT_READ, EV_ADD, 0, 0, 0);
num_events = kevent(loop.m_kqueue, in_events.data(), in_events.size(),
out_events, llvm::array_lengthof(out_events), nullptr);
if (num_events < 0)
return Status("kevent() failed with error %d\n", num_events);
return Status();
}
It works on FreeBSD and MacOSX, however it broke on NetBSD.
Culprit line:
EV_SET(&in_events[i++], fd.first, EVFILT_READ, EV_ADD, 0, 0, 0);
FreeBSD defined EV_SET() as a macro this way:
#define EV_SET(kevp_, a, b, c, d, e, f) do { \
struct kevent *kevp = (kevp_); \
(kevp)->ident = (a); \
(kevp)->filter = (b); \
(kevp)->flags = (c); \
(kevp)->fflags = (d); \
(kevp)->data = (e); \
(kevp)->udata = (f); \
} while(0)
NetBSD version was different:
#define EV_SET(kevp, a, b, c, d, e, f) \
do { \
(kevp)->ident = (a); \
(kevp)->filter = (b); \
(kevp)->flags = (c); \
(kevp)->fflags = (d); \
(kevp)->data = (e); \
(kevp)->udata = (f); \
} while (/* CONSTCOND */ 0)
This resulted in heap damage, as keyp was incremented every time a value was
assigned to (keyp)->.
Without GCC asan and ubsan tools, finding this bug would be much more time consuming, as the random memory corruption was affecting unrelated lambda function in a different part of the code.
To use the GCC sanitizers with packages from pkgsrc, on NetBSD-current, one has to use one or both of these lines:
_WRAP_EXTRA_ARGS.CXX+= -fno-omit-frame-pointer -O0 -g -ggdb -U_FORTIFY_SOURCE -fsanitize=address -fsanitize=undefined -lasan -lubsan CWRAPPERS_APPEND.cxx+= -fno-omit-frame-pointer -O0 -g -ggdb -U_FORTIFY_SOURCE -fsanitize=address -fsanitize=undefined -lasan -lubsan
While there, I have fixed another - generic - bug in the LLVM headers. The class Triple constructor hadn't initialized the SubArch field, which upsetting the GCC address sanitizer. It was triggered in LLDB in the following code:
void ArchSpec::Clear() {
m_triple = llvm::Triple();
m_core = kCore_invalid;
m_byte_order = eByteOrderInvalid;
m_distribution_id.Clear();
m_flags = 0;
}
I have filed a patch for review to address this.
The bad
Unfortunately this is not the full story and there is further mandatory work.
LLDB acceleration
The EV_SET() bug broke upstream LLDB over a month ago, and during this period the debugger was significantly accelerated and parallelized. It is difficult to declare it definitely, but it might be the reason why the tracer's runtime broke due to threading desynchronization. LLDB behaves differently when run standalone, under ktruss(1) and under gdb(1) - the shared bug is that it always fails in one way or another, which isn't trivial to debug.
The ugly
There are also unpleasant issues at the core of the Operating System.
Kernel troubles
Another bug with single-step functions that affects another aspect of correctness - this time with reliable execution of a program - is that processes die in non-deterministic ways when single-stepped. My current impression is that there is no appropriate translation between process and thread (LWP) states under a debugger.
These issues are sibling problems to unreliable PT_RESUME and PT_SUSPEND.
In order to be able to appropriately address this, I have diligently studied this month the Solaris Internals book to get a better image of the design of the NetBSD kernel multiprocessing, which was modeled after this commercial UNIX.
Plan for the next milestone
The current troubles can be summarized as data races in the kernel and at the same time in LLDB. I have decided to port the LLVM sanitizers, as I require the Thread Sanitizer (tsan). Temporarily I have removed the code for tracing processes with multiple threads to hide the known kernel bugs and focus on the LLDB races.
Unfortunately LLDB is not easily bisectable (build time of the LLVM+Clang+LLDB stack, number of revisions), therefore the debugging has to be performed on the most recent code from upstream trunk.
This work was sponsored by The NetBSD Foundation.
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can: