May 2018
S M T W T F S
   
23 24
   

Archives

This page is a blog mirror of sorts. It pulls in articles from blog's feed and publishes them here (with a feed, too).

Pinebook is an affordable 64-bit ARM notebook. Today we're going to take a look at the kernel output at startup and talk about what hardware support is available on NetBSD.

Loaded initial symtab at 0xffffffc0007342c0, strtab at 0xffffffc0007e3208, # entries 29859
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017,
2018 The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California.  All rights reserved.

NetBSD 8.99.17 (GENERIC64) #272: Mon May 21 07:25:46 ADT 2018
	jmcneill@persona.local:/Users/jmcneill/netbsd/cvs-src/sys/arch/evbarm/compile/obj/GENERIC64
total memory = 2007 MB
avail memory = 1942 MB

Pinebook comes with 2GB RAM standard. A small amount of this is reserved by the kernel and framebuffer.

timecounter: Timecounters tick every 10.000 msec
armfdt0 (root)
fdt0 at armfdt0: Pine64 Pinebook
fdt1 at fdt0
fdt2 at fdt0

NetBSD uses flattened device-tree (FDT) to enumerate devices on all Allwinner based SoCs. On a running system, you can inspect the device tree using the ofctl(8) utility:

$ ofctl -p /sound  
[Caching 111 nodes and 697 properties]
compatible              73696d70 6c652d61 7564696f 2d636172   "simple-audio-car
            0010:       6400.... ........ ........ ........   d"
name                    736f756e 6400.... ........ ........   "sound"
simple-audio-card,format
                        69327300 ........ ........ ........   i2s.
simple-audio-card,frame-master
                        0000001c ........ ........ ........   ....
simple-audio-card,name  53554e35 30492041 7564696f 20436172   SUN50I Audio Car
            0010:       6400.... ........ ........ ........   d.
cpus0 at fdt0
cpu0 at cpus0: Cortex-A53 r0p4 (Cortex V8-A core)
cpu0: CPU Affinity 0-0-0-0
cpu0: IC enabled, DC enabled, EL0/EL1 stack Alignment check enabled
cpu0: Cache Writeback Granule 16B, Exclusives Reservation Granule 16B
cpu0: Dcache line 64, Icache line 64
cpu0: L1 32KB/64B 2-way read-allocate VIPT Instruction cache
cpu0: L1 32KB/64B 4-way write-back read-allocate write-allocate PIPT Data cache
cpu0: L2 512KB/64B 16-way write-back read-allocate write-allocate PIPT Unified cache
cpu0: revID=0x80, PMCv3, 4k table, 64k table, 16bit ASID
cpu0: auxID=0x11120, FP, CRC32, SHA1, SHA256, AES+PMULL, NEON, rounding, NaN propagation, denormals, 32x64bitRegs, Fused Multiply-Add

Pinebook's Allwinner A64 processor is based on the ARM Cortex-A53. It is designed to run at frequencies up to 1.2GHz.

cpufreqdt0 at cpu0

Dynamic voltage and frequency scaling (DVFS) is supported for the CPU. More on this in a bit...

cpu1 at cpus0: disabled (uniprocessor kernel)
cpu2 at cpus0: disabled (uniprocessor kernel)
cpu3 at cpus0: disabled (uniprocessor kernel)

The A64 is a quad core design. NetBSD's aarch64 pmap does not yet support SMP, so three cores are disabled for now.

gic0 at fdt1: GIC
armgic0 at gic0: Generic Interrupt Controller, 224 sources (215 valid)
armgic0: 16 Priorities, 192 SPIs, 7 PPIs, 16 SGIs

The interrupt controller is a standard ARM GIC-400 design.

fclock0 at fdt0: 24000000 Hz fixed clock
fclock1 at fdt0: 32768 Hz fixed clock
fclock2 at fdt0: 16000000 Hz fixed clock

Fixed clock input signals.

gtmr0 at fdt0: Generic Timer
gtmr0: interrupting on GIC irq 27
armgtmr0 at gtmr0: ARMv7 Generic 64-bit Timer (24000 kHz)
timecounter: Timecounter "armgtmr0" frequency 24000000 Hz quality 500

Standard ARMv7/ARMv8 architected timer.

sun50ia64ccu0 at fdt1: A64 CCU
sun50ia64rccu0 at fdt1: A64 PRCM CCU

Clock drivers for managing PLLs, module clock dividers, clock gating, software resets, etc. Information about the clock tree is exported in the hw.clk sysctl namespace (root access required to read these values).

# sysctl hw.clk.sun50ia64ccu0.mmc2
hw.clk.sun50ia64ccu0.mmc2.rate = 200000000
hw.clk.sun50ia64ccu0.mmc2.parent = pll_periph0_2x
hw.clk.sun50ia64ccu0.mmc2.parent_domain = sun50ia64ccu0
sunxinmi0 at fdt1: R_INTC

NMI interrupt controller. This is wired to the POK (power on key) pin on the power management IC. In NetBSD, a long (>1.5s) press of the power button on Pinebook triggers a power down event.

sunxigpio0 at fdt1: PIO
gpio0 at sunxigpio0: 103 pins
sunxigpio0: interrupting on GIC irq 43
sunxigpio1 at fdt1: PIO
gpio1 at sunxigpio1: 13 pins
sunxigpio1: interrupting on GIC irq 77

GPIO and multi-function I/O pin support.

sunxipwm0 at fdt1: PWM

Pulse-width modulation driver. This is used on Pinebook to control the brightness of the display backlight.

sunxiusbphy0 at fdt1: USB PHY

USB 2.0 PHY, responsible for powering up and initialization of USB ports.

sunxirsb0 at fdt1: RSB
sunxirsb0: interrupting on GIC irq 71
iic0 at sunxirsb0: I2C bus

Allwinner "Reduced Serial Bus" is an I2C-like interface used to communicate with power management ICs.

axppmic0 at iic0 addr 0x3a3: AXP803
axpreg0 at axppmic0: dldo1
axpreg1 at axppmic0: dldo2
axpreg2 at axppmic0: dldo3
axpreg3 at axppmic0: dldo4
axpreg4 at axppmic0: eldo1
axpreg5 at axppmic0: eldo2
axpreg6 at axppmic0: eldo3
axpreg7 at axppmic0: fldo1
axpreg8 at axppmic0: fldo2
axpreg9 at axppmic0: dcdc1
axpreg10 at axppmic0: dcdc2
axpreg11 at axppmic0: dcdc3
axpreg12 at axppmic0: dcdc4
axpreg13 at axppmic0: dcdc5
axpreg14 at axppmic0: dcdc6
axpreg15 at axppmic0: aldo1
axpreg16 at axppmic0: aldo2
axpreg17 at axppmic0: aldo3

The X-Powers AXP803 power management IC (PMIC) controls power outputs and battery charging. Power, charging, and battery status is exported to userland with the envsys(4) API:

$ envstat -d axppmic0
                   Current  CritMax  WarnMax  WarnMin  CritMin  Unit
   ACIN present:      TRUE
   VBUS present:     FALSE
battery present:      TRUE
       charging:     FALSE
   charge state:    NORMAL
battery percent:       100                                      none
sun6idma0 at fdt1: DMA controller (8 channels)
sun6idma0: interrupting on GIC irq 82

8-channel programmable DMA controller. Used by the audio driver.

/soc/syscon@1c00000 at fdt1 not configured

A miscellaneous register set of registers. The Allwinner A64 ethernet driver uses these resources to configure the PHY interface. Not used on Pinebook.

ausoc0 at fdt0: SUN50I Audio Card
a64acodec0 at fdt1: A64 Audio Codec (analog part)
sunxii2s0 at fdt1: Audio Codec (digital part)
sun8icodec0 at fdt1: Audio Codec
sun8icodec0: interrupting on GIC irq 60

Analog audio codec. On Allwinner A64 this functionality is spread across multiple modules. The ausoc driver is responsible for talking to these drivers and exposing a single audio device. Initialization is deferred until all drivers are ready, so the audio device is attached below.

psci0 at fdt0: PSCI 0.2

ARM "Power State Coordination Interface". NetBSD uses this to implement reset, power off, and (for multiprocessor kernels) starting secondary CPUs.

gpiokeys0 at fdt0: Lid Switch

Pinebook has a Hall effect sensor wired to a GPIO to signal whether the lid is open or closed. On NetBSD state change events are sent to powerd, as well as broadcast to other drivers in the kernel. For example, the backlight driver will automatically turn off the display when the lid is closed.

pwmbacklight0 at fdt0: PWM Backlight <0 30 40 50 60 70 80 90 100>

The backlight brightness on Pinebook is controlled by the PWM controller. These controls are exported via sysctl:

$ sysctl hw.pwmbacklight0
hw.pwmbacklight0.levels = 0 30 40 50 60 70 80 90 100
hw.pwmbacklight0.level = 80

In addition, the framebuffer driver will pass WSDISPLAYIO_SVIDEO ioctl requests through to the backlight driver, so screen blanking in Xorg will work.

sunximmc0 at fdt1: SD/MMC controller
sunximmc0: interrupting on GIC irq 92
sunximmc1 at fdt1: SD/MMC controller
sunximmc1: interrupting on GIC irq 93
sunximmc2 at fdt1: SD/MMC controller
sunximmc2: interrupting on GIC irq 94

SD/MMC controllers. On Pinebook, they are used for the following functions:

  • #0: SD card slot
  • #1: SDIO Wi-Fi
  • #2: Onboard eMMC

motg0 at fdt1: USB OTG
motg0: interrupting on GIC irq 103
usb0 at motg0: USB revision 2.0

USB OTG controller. NetBSD supports this in host mode only.

ehci0 at fdt1: EHCI
ehci0: interrupting on GIC irq 104
ehci0: EHCI version 1.0
ehci0: 1 companion controller, 1 port
usb1 at ehci0: USB revision 2.0
ohci0 at fdt1: OHCI
ohci0: interrupting on GIC irq 105
ohci0: OHCI version 1.0
usb2 at ohci0: USB revision 1.0
ehci1 at fdt1: EHCI
ehci1: interrupting on GIC irq 106
ehci1: EHCI version 1.0
ehci1: 1 companion controller, 1 port
usb3 at ehci1: USB revision 2.0
ohci1 at fdt1: OHCI
ohci1: interrupting on GIC irq 107
ohci1: OHCI version 1.0
usb4 at ohci1: USB revision 1.0

USB 2.0 host controllers.

com0 at fdt1: ns16550a, working fifo
com0: interrupting on GIC irq 32

Serial console.

sunxirtc0 at fdt1: RTC

Real-time clock. Note that the Pinebook does not include a battery backed RTC.

sunxiwdt0 at fdt1: Watchdog
sunxiwdt0: default watchdog period is 16 seconds

The watchdog timer can be enabled with the wdogctl(8) utility:

# wdogctl
Available watchdog timers:
        sunxiwdt0, 16 second period
sunxithermal0 at fdt1: Thermal sensor controller
sunxithermal0: interrupting on GIC irq 63
sunxithermal0: cpu: alarm 85C hyst 58C shut 106C
sunxithermal0: gpu1: alarm 85C hyst 58C shut 106C
sunxithermal0: gpu2: alarm 85C hyst 58C shut 106C

CPU and GPU temperatures are exported to userland with the envsys(4) API:

$ envstat -d sunxithermal0
                     Current  CritMax  WarnMax  WarnMin  CritMin  Unit
  CPU temperature:    54.000                                      degC
GPU temperature 1:    52.000                                      degC
GPU temperature 2:    52.000                                      degC

In addition, the thermal sensor driver coordinates with the DVFS driver to automatically throttle CPU frequency and voltage when temperatures exceed alarm thresholds.

genfb0 at fdt2: Simple Framebuffer (1366x768 32-bpp @ 0xbe000000)
genfb0: switching to framebuffer console
genfb0: framebuffer at 0xbe000000, size 1366x768, depth 32, stride 5464
wsdisplay0 at genfb0 kbdmux 1: console (default, vt100 emulation)
wsmux1: connecting to wsdisplay0
wsdisplay0: screen 1-3 added (default, vt100 emulation)

NetBSD will automatically use a framebuffer configured by U-Boot using the simplefb protocol. Since the simple frame buffer driver uses genfb(4), it is supported by Xorg out of the box.

ausoc0: codec: sun8icodec0, cpu: sunxii2s0, aux: a64acodec0
audio0 at ausoc0: full duplex, playback, capture, mmap, independent
ausoc0: Virtual format configured - Format SLINEAR, precision 16, channels 2, frequency 48000
ausoc0: Latency: 256 milliseconds

Analog playback and capture are supported on Pinebook. In addition, headphone jack sensing is supported to automatically mute the internal speakers when headphones are plugged in. The following mixer controls are available:

$ mixerctl -a
outputs.master=192,192
outputs.mute=off
outputs.source=line
inputs.line=192,192
inputs.headphones=128,128
record.line=0,0
record.mic=96,96
record.mic.preamp=off
record.mic2=96,96
record.mic2.preamp=off
record.agc=96,96
record.source=
spkr0 at audio0: PC Speaker (synthesized)
wsbell at spkr0 not configured

Pseudo PC speaker driver (not currently enabled in GENERIC64 kernel).

timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
Lid Switch: lid opened.
cpufreqdt0: 1152.000 MHz, 1300000 uV
cpufreqdt0: 1104.000 MHz, 1260000 uV
cpufreqdt0: 1008.000 MHz, 1200000 uV
cpufreqdt0: 816.000 MHz, 1080000 uV
cpufreqdt0: 648.000 MHz, 1040000 uV
cpufreqdt0: 408.000 MHz, 1000000 uV

Dynamic voltage and frequency scaling is supported. The desired frequency can be set manually with sysctl, or automatically by installing sysutils/estd from pkgsrc.

# sysctl machdep.cpu
machdep.cpu.frequency.target = 1152
machdep.cpu.frequency.current = 1152
machdep.cpu.frequency.available = 1152 1104 1008 816 648 408
sdmmc0 at sunximmc0
sdmmc1 at sunximmc1
sdmmc2 at sunximmc2
uhub0 at usb0: NetBSD (0000) MOTG root hub (0000), class 9/0, rev 2.00/1.00, addr 1
uhub0: 1 port with 1 removable, self powered
uhub1 at usb1: NetBSD (0000) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
uhub1: 1 port with 1 removable, self powered
uhub2 at usb2: NetBSD (0000) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
uhub2: 1 port with 1 removable, self powered
uhub3 at usb3: NetBSD (0000) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1
uhub3: 1 port with 1 removable, self powered
uhub4 at usb4: NetBSD (0000) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1
uhub4: 1 port with 1 removable, self powered
IPsec: Initialized Security Association Processing.
(manufacturer 0x24c, product 0xb703, standard function interface code 0x7) at sdmmc1 function 1 not configured

SDIO Wi-Fi is not yet supported.

ld2 at sdmmc2: <0x88:0x0103:NCard :0x00:0x30601836:0x000>
ld2: 14800 MB, 7517 cyl, 64 head, 63 sec, 512 bytes/sect x 30310400 sectors
ld2: 8-bit width, HS200, 64 MB cache, 200.000 MHz

Pinebook comes with 16GB of built-in eMMC storage. NetBSD drives this in High Speed HS200 mode, which can support up to 200MB/s at 200MHz.

uhub5 at uhub3 port 1: vendor 05e3 (0x5e3) USB2.0 Hub (0x608), class 9/0, rev 2.00/88.32, addr 2
uhub5: single transaction translator
uhub5: 4 ports with 1 removable, self powered
uhidev0 at uhub5 port 1 configuration 1 interface 0
uhidev0: HAILUCK CO.,LTD (0x258a) USB KEYBOARD (0x0c), rev 1.10/1.00, addr 3, iclass 3/1
ukbd0 at uhidev0: 8 Variable keys, 6 Array codes
wskbd0 at ukbd0: console keyboard, using wsdisplay0
uhidev1 at uhub5 port 1 configuration 1 interface 1
uhidev1: HAILUCK CO.,LTD (0x258a) USB KEYBOARD (0x0c), rev 1.10/1.00, addr 3, iclass 3/0
uhidev1: 9 report ids
ums0 at uhidev1 reportid 1: 5 buttons, W and Z dirs
wsmouse0 at ums0 mux 0
uhid0 at uhidev1 reportid 2: input=1, output=0, feature=0
uhid1 at uhidev1 reportid 3: input=3, output=0, feature=0
uhid2 at uhidev1 reportid 5: input=0, output=0, feature=5
uhid3 at uhidev1 reportid 6: input=0, output=0, feature=1039
uhid4 at uhidev1 reportid 9: input=1, output=0, feature=0

The Pinebook's keyboard and touchpad are connected internally to a USB controller.

The sleep (Fn+Esc), home (Fn+F1), volume down (Fn+F3), volume up (Fn+F4), and mute (Fn+F5) keys on the keyboard are mapped to uhid(4) devices. These can be used with usbhidaction(1):

# cat /etc/usbhidaction.conf 
Consumer:Consumer_Control.Consumer:Volume_Up                    1
        mixerctl -n -w outputs.master++
Consumer:Consumer_Control.Consumer:Volume_Down                  1
        mixerctl -n -w outputs.master--
Consumer:Consumer_Control.Consumer:Mute                         1
        mixerctl -n -w outputs.mute++
Consumer:Consumer_Control.Consumer:AC_Home                      1
        /etc/powerd/scripts/hotkey_button AC_Home pressed
Generic_Desktop:System_Control.Generic_Desktop:System_Sleep     1
        /etc/powerd/scripts/sleep_button System_Sleep pressed
# /usr/bin/usbhidaction -c /etc/usbhidaction.conf -f /dev/uhid0 -i -p /var/run/usbhidaction-uhid0.pid 
# /usr/bin/usbhidaction -c /etc/usbhidaction.conf -f /dev/uhid1 -i -p /var/run/usbhidaction-uhid1.pid 
urtwn0 at uhub5 port 2
urtwn0: Realtek (0x7392) 802.11n WLAN Adapter (0x7811), rev 2.00/2.00, addr 4
urtwn0: MAC/BB RTL8188CUS, RF 6052 1T1R, address 80:1f:02:94:40:12
urtwn0: 1 rx pipe, 2 tx pipes
urtwn0: 11b rates: 1Mbps 2Mbps 5.5Mbps 11Mbps
urtwn0: 11g rates: 1Mbps 2Mbps 5.5Mbps 11Mbps 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps

Externally connected USB Wi-Fi adapter (not included).

uvideo0 at uhub5 port 3 configuration 1 interface 0: Image Processor (0x90c) USB 2.0 PC Cam (0x37c), rev 2.00/0.05, addr 5
video0 at uvideo0: Image Processor (0x90c) USB 2.0 PC Cam (0x37c), rev 2.00/0.05, addr 5

The built-in video camera is supported using standard V4L2 APIs.

$ videoctl -a
info.cap.card=Image Processor (0x90c) USB 2.0
info.cap.driver=uvideo
info.cap.bus_info=usb:00000009
info.cap.version=8.99.17
info.cap.capabilities=0x5000001
info.format.0=YUYV
info.input.0=Camera
info.input.0.type=baseband
info.standard.0=webcam
boot device: ld2
root on ld2a dumps on ld2b

NetBSD supports a wide variety of Allwinner boards. The wiki contains more information about supported Allwinner SoCs and peripherals.

Posted at lunch time on Monday, May 21st, 2018 Tags: blog

I've recently been gifted a fancy laptop - a Dell XPS 15 9550.
I want to run NetBSD on it and have it run well, and I've set aside time to achieve this.
Things work surprisingly well out of the box - touchscreen and touchpad work, but need to be calibrated better. 4K video is displayed, but without graphical acceleration, which is somewhat slow. I could adjust the resolution to be lower, but haven't tried yet.

These are some of the lessons I learned from doing this, being new to drivers and networking.

rtsx - SD card reader

This one was simple and straightforward.
The PCI ID of my SD is 10ec:525a, a realtek PCI device with ID 0x525a.
Normal SD card readers are supported by a generic driver, but mine wasn't.
I've been told that when SD cards don't present themselves as the generic device, it is for good reason.

Adding the PCI device ID to sys/dev/pci/pcidevs would have revealed the adjacent entries:

product REALTEK RTS5229		0x5229	RTS5229 PCI-E Card Reader
product REALTEK RTS5249		0x5249	RTS5249 PCI-E Card Reader
product REALTEK RTS525A		0x525A	RTS525A PCI-E Card Reader <- my addition
product REALTEK RTL8402		0x5286	RTL8402 PCI-E Card Reader

These are all supported by rtsx(4), which is a driver originally from OpenBSD.
Looking at the driver in OpenBSD, it has diverged since, and indeed someone had added RTS525A support.

I've initially tried to synchronize the driver, but it seemed beyond my capabilities, and just went to steal the support.
Found the individual commit and it was kinda ugly, there are conditionals for variants in some functions.
I've added the RTS525A in the matching conditionals.

It would have been nice to make it less ugly, but I'm only adding a bit of fuel to the fire.
I've received hints that drivers for hardware from some manufacturers is going to be ugly either way.

I saw that "read-only detect" is not supported. Curious if I could added, I wondered what happens if you try to write when the SD card's read-only switch is held. Apparently this is a suggestion for software, I could read back the contents I wrote after detach. perhaps not having the support isn't too bad.

bwfm - wireless networking

Previously, Broadcom wifi in a laptop would have resulted in sadness, followed by disassembly and hopeful replacement, unless the manufacturer has chosen to whitelist wireless cards, in which case I am doomed to a USB wifi card.
Thanks to bluerise@openbsd, jmcneill and Broadcom for open sourcing their drivers, we now have a port of the linux brcmfmac driver, for Broadcom FullMAC wireless devices. Mine is one.

These devices do a lot of 802.11 weight lifting on their own. Unfortunately that means that if they hadn't had their firmware updated, vulnerabilities are likely, such as KRACK. Previously jmcneill had made it work for the USB variant, and even made it use some of the kernel's 802.11 code, allowing to avoid some vulnerabilities.

He did not port the PCI variant, added later in OpenBSD. I've set out to steal this code.

First and foremost, I need the ability to express 'please build the pci variant code', or to add it to the pci config.
The files I have for the driver are:

sys/dev/ic/{bwfm.c, bwfmvar.h, bwfmreg.h}
sys/dev/pci/if_bwfm_pci.{c,h}
The USB variant already exists, it uses:
sys/dev/ic/{bwfm.c, bwfmvar.h, bwfmreg.h}
sys/dev/usb/if_bwfm_usb.{c,h}
ic is something shared among pci, usb, sdio code. the code for it is already built due to being added to sys/conf/files.
I only need to add the PCI code.

The file for this is sys/dev/pci/files.pci. Fortunately it has plenty of examples, and here is what I added:

# Broadcom FullMAC USB wireless adapter
attach	bwfm at pci with bwfm_pci: firmload
file	dev/pci/if_bwfm_pci.c		bwfm_pci

I've had to adapt APIs - some OpenBSD ones not existing in NetBSD. Fortunately there are lots of drivers to look around. I've made a very simple if_rxr_* and made it static. MCLGETI I copied from another driver, again static.
My first attempt at if_rxr_* (however simple) didn't seem to work. I rewrote it and made sure to KASSERT assumptions I assume are correct.
I was hoping to find an existing "NetBSD-ish" API for this, but I'm not sure there is something quite like it, and it is pretty simple. But it was a good opportunity to learn more about NetBSD APIs.

Once I got it to compile and not panic on attach, I had a problem. sending a message to the device failed to get the appropriate response, and attach failed.
I've spent a long time reading the initialization sequence, in OpenBSD and in linux brcmfmac, but couldn't find anything. I was ready to give up, but then I decided to diff my file to OpenBSD, and saw that in an attempt to calm type warnings (fatal by default, as NetBSD builds with -Werror), I made some unnecessary and incorrect changes.
I've switched to building without -Werror (that means passing -V NOGCCERROR=1 to build.sh, or adding to mk.conf) and reverted this changes, and then it attached cleanly.
In retrospect, I shouldn't have made unnecessary changes as a beginner. -V NOGCCERROR=1 is there. Device work first, cosmetic warning fixes second.

After it had attached, I tried to connect. I hoped wpa_supplicant -d would tell why it failed, but it only said that the connection times out.
Like many drivers in the *BSD world, bwfm has a BWFM_DEBUG and a debug printf that is controllable. I turned it on to get these messages from bwfm_rx_event:

bwfm0: newstate 0 -> 1
bwfm0: newstate 1 -> 1
bwfm_rx_event: buf 0xffffe404b97f9010 len 354 datalen 280 code 69 status 8 reason 0
bwfm_rx_event: buf 0xffffe404b97f9010 len 354 datalen 280 code 69 status 8 reason 0
bwfm_rx_event: buf 0xffffe404b97f9010 len 362 datalen 288 code 69 status 8 reason 0
bwfm_rx_event: buf 0xffffe404b97f9010 len 354 datalen 280 code 69 status 8 reason 0
bwfm_rx_event: buf 0xffffe404b97f9010 len 354 datalen 280 code 69 status 8 reason 0
bwfm_rx_event: buf 0xffffe404b97f9010 len 354 datalen 280 code 69 status 8 reason 0
bwfm_rx_event: buf 0xffffe404b97f9010 len 362 datalen 288 code 69 status 8 reason 0
bwfm_rx_event: buf 0xffffe404b97f9010 len 354 datalen 280 code 69 status 8 reason 0
bwfm_rx_event: buf 0xffffe404b97f9010 len 86 datalen 12 code 69 status 0 reason 0
bwfm_rx_event: buf 0xffffe404b9837010 len 74 datalen 0 code 26 status 0 reason 0
bwfm0: newstate 1 -> 2
bwfm_rx_event: buf 0xffffe404b9837010 len 82 datalen 8 code 0 status 3 reason 0
bwfm0: newstate 2 -> 1

To decipher the messages, I picked up what the various numbers means. Deciphered, it says:

802.11 state INIT -> SCAN
802.11 state SCAN -> SCAN
received messages from the driver:
code 69 (scan result)
...
code 26 (scan end)
802.11 state SCAN -> AUTH
code 0 (SET_SSID) status 3 (NO NETWORKS)
802.11 state AUTH -> SCAN

Authenticating is failing to find a matching network. But why?
Now experienced with the previous experience ("I've found my problem! it was me!"), I went back to some more attempts to sync the driver, which changed the SSID-setting code.

With this corrected, the driver worked as expected.

In total, it would have been easier to start with a working driver and adapting it to make some changes, like having the USB variant or testing on OpenBSD, but I managed anyway, it just took longer.
In two weeks I will receive the USB variant, but the driver is already working for me. I had trouble installing OpenBSD on the same machine.

I'll clean up my changes and post it for review soon.

Thanks to jak, pgoyette, rkujawa, phone for various fixes.
Thanks to maxv for fixing a bug affecting my use of a network stack API.
Thanks to bluerise@openbsd and jmcneill for the prior driver work.
Thanks to Broadcom for providing open source drivers under a friendly license.

remaining work - graphical acceleration.

NetBSD's x86 graphical acceleration is mostly from Linux, from linux 3.15.
This predates support for my Skylake GPU and nVidia graphics on my machine, and also the ability to dual-GPU effectively.
I'll try to sync the drivers and made previous attempts to learn about graphics, but I can't promise anything.

Posted late Thursday evening, May 3rd, 2018 Tags: blog
For the past month I've been mostly working on improving the kernel code in the ptrace(2) API. Additionally, I've prepared support for reading NetBSD/aarch64 core(5) files.

LLDB

I've rebased my old patches for handling NetBSD core(5) files to recent LLDB from HEAD (7.0svn). I had to implement the support for aarch64, as requested by Joerg Sonnenberger for debugging NetBSD/arm64 bugs. The NetBSD case is special here, as we set general purpose registers in 128bit containers, while other OSes prefer 64-bit ones. I had to add a local kludge to cast the interesting 64-bit part of the value.

I've generated a local prebuilt toolchain with lldb prepared for a NetbBSD/amd64 host and shared with developers.

Debug Registers

I've improved the security of Debug Registers, with the following changes:

  • x86_dbregs_setup_initdbstate() stores the initial status of Debug Registers, I've explicitly disabled some of their bits for further operation. This register context is used when initializing Debug Registers for a userland thread. This is a paranoid measure used just in case a bootloader uses them and leaves the bits lit up.
  • I've added a new sysctl(3) switch security.models.extensions.user_set_dbregs: "Whether unprivileged users may set CPU Debug Registers". The Debug Registers use privileged operations and we decided to add another check, disabling them by default. There is no regress in the elementary functionality and a user can still read the Debug Register context.
  • This new sysctl(3) functionality has been covered by ATF checks in existing tests (i386 and amd64 for now).

fork(2) and vfork(2)

I've pushed various changes in the kernel subsystems and ATF userland tests for the ptrace(2) functionality:

  • I've integrated all fork(2) and vfork(2) tests into the shared function body. They tests PTRACE_FORK, PTRACE_VFORK and PTRACE_VFORK_DONE. There are now eight fork1..fork8 and eight vfork1..vfork8 tests (total 16). I'm planning to cover several remaining corner case scenarios in the forking code.
  • I've removed an unused variable from linux_sys_clone(), which calls fork1(9). This provides an opportunity to refactor the generic code.
  • I've refactored start_init() setting inside it the initproc pointer. This eliminates the remaining user of rnewprocp in the fork1(9) API.
  • I've removed the rnewprocp argument from fork1(9). This argument in the current implementation could cause use-after-free in corner cases.
  • PTRACE_VFORK has been implemented and made functional.

Security hardening

I've prohibited calling PT_TRACE_ME for a process that is either initproc (PID1) or a direct child of PID1.

I've prohibited calling PT_ATTACH from initproc. This shouldn't happen in normal circumstances, but just in case this would lead to invalid branches in the kernel.

With the above alternations, I've removed a bug causing a creation of a process that is not debuggable. It's worth to note that this bug still exists in other popular kernels. A simple reproducer for pre-8.0 and other OSes using ptrace(2) (Linux, most BSDs ...):

    $ cat antidebug.c
    #include <sys/types.h>
    #include <sys/ptrace.h>
    
    #include <stdlib.h>
    #include <unistd.h>
    #include <stdio.h>
    #include <errno.h>
    
    int
    main(int argc, char **argv)
    {
            pid_t child;
            int rv;
            int n = 0;
    
            child = fork();
            if (child == 0) {
                    while (getppid() != 1)
                            continue;
                    rv = ptrace(PT_TRACE_ME, 0, 0, 0);
                    if (rv != 0)
                            abort();
                    printf("Try to detach to me with a debugger!! ");
                    printf("haha My PID is %d\n", getpid());
                    while (1) {
                            printf("%d\n", n++);
                            sleep(1);
                    }
            }
            exit(0);
    }    

Additionally it's no longer valid calling and returning success PT_TRACE_ME when a process is already traced.

These security changes are covered by new ATF ptrace(2) tests:

  1. traceme_pid1_parent - Assert that a process cannot mark its parent a debugger twice.
  2. traceme_twice - Verify that PT_TRACE_ME is not allowed when our parent is PID1.

Other ptrace(2) enhancements

A list of other changes:

  • I've refactored part of the existing ATF ptrace(2) tests, focusing on code deduplication and covering more corner cases. I'm now especially testing PT_TRACE_ME existing scenarios with signals from various categories: SIGSTOP, SIGCONT, SIGKILL, SIGABRT, SIGHUP. Some signals are non-maskable ones, some generate core(5) files etc. They might use different kernel paths and I'm now covering them in a more uniform and hopefully more maintainable form.
  • In the kernel, I've removed unused branch in the proc_stop_done() and sigswitch() functions. This functionality was used in deprecated filesystem tracing feature (/proc). It prevented emitting child signals to a debugging program, namely with the SIGCHLD signal.

    The modern solution to perform tracing without signals in a debugger is to spawn a debugging server and outsource the tracing functionality to it. This is done in software like gdb-server, lldb-server etc.

  • I've removed a stray XXX comment from I and D read/write operations, and added a proper explanation:

    +        /*
    +        * The I and D separate address space has been inherited from PDP-11.
    +        * The 16-bit UNIX started with a single address space per program,
    +        * but was extended to two 16-bit (2 x 64kb) address spaces.
    +        *
    +        * We no longer maintain this feature in maintained architectures, but
    +        * we keep the API for backward compatiblity. Currently the I and D
    +        * operations are exactly the same and not distinguished in debuggers.
    +        */
    
  • I've improved the WCOREDUMP() checker in the ATF tests, casting it to either 1 or 0. This helps to handler correctly the tests generating core(5) files.
  • Improve of the proc_stoptrace() function. proc_stoptrace() is dedicated for emitting a syscall trap for a debugger, either on entry or exit of the system function routine.

    Changes:

    • Change an if() branch of an invalid condition of being traced by initproc (PID1) to KASSERT(9).
    • Assert that the current process has set appropriate flags (PSL_TRACED and PSL_SYSCALL).
    • Use ktrpoint(KTR_PSIG) and ktrpsig()/e_ktrpsig() in order to register the emitted signal for the ktrace(1) event debugging.

Summary

A critical Problem Report kern/51630 regarding lack of PTRACE_VFORK implementation has been fixed. This means that there are no other unimplemented API calls, but there are still bugs in the existing ones.

With fixes and addition of new test cases, as of today we are passing 961 ptrace(2) tests and skipping 1 (out of 1018 total).

Plan for the next milestone

Cover the remaining forking corner-cases in the context of debuggers with new ATF tests and fix the remaining bugs.

The first step is to implement proper support for handling PT_TRACE_ME-traced scenarios from a vfork(2)ed child. Next I plan to keep covering the corner cases of the forking code and finish this by removal of subtle bugs that are still left in the code since the SMP'ification.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:

http://netbsd.org/donations/#how-to-donate

Posted late Tuesday night, May 2nd, 2018 Tags: blog

Google Summer of Code logo We are very happy to announce The NetBSD Foundation Google Summer of Code 2018 projects:

We have started the community bonding period where students get in touch with mentors and community and get more confident with documentation and the code base. The coding period will start from May 14 until August 6.

Please welcome all our students and a big good luck to students and mentors!

A big thank you also to Google and The NetBSD Foundation Organization Administrators!

Looking forward to a nice Google Summer of Code!

Posted mid-morning Tuesday, May 1st, 2018 Tags: blog

The NetBSD Project is pleased to announce NetBSD 8.0 RC 1, the first release candidate for the upcoming NetBSD 8.0 release.

25 years and a few days after the first official NetBSD release (NetBSD 0.8 on April 19, 1993) we are now quickly approaching the first final release from the netbsd-8 branch that has been in the work for more most of a year now.

The official RC1 announcement list these major changes compared to older releases:

  • USB stack rework, USB3 support added
  • In-kernel audio mixer
  • Reproducible builds
  • PaX MPROTECT (W^X) memory protection enforced by default on some architectures with fine-grained memory protection and suitable ELF formats: i386, amd64, evbarm, landisk, pmax
  • PaX ASLR enabled by default on:
    i386, amd64, evbarm, landisk, pmax, sparc64
  • MKPIE (position independent executables) by default for userland on: i386, amd64, arm, m68k, mips, sh3, sparc64
  • added can(4), a socket layer for CAN busses
  • added ipsecif(4) for route-based VPNs
  • made part of the network stack MP-safe
  • NET_MPSAFE kernel option is required to try
  • WAPBL stability and performance improvements

Specific to i386 and amd64 CPUs:
  • Meltdown mitigation: SVS (separate virtual address spaces)
  • Spectre mitigation (support in gcc, used by default for kernels)
  • SMAP support
  • (U)EFI bootloader

Various new drivers:
  • nvme(4) for modern solid state disks
  • iwm(4), a driver for Intel Wireless devices (AC7260, AC7265, AC3160...)
  • ixg(4): X540, X550 and newer device support.
  • ixv(4): Intel 10G Ethernet virtual function driver.
  • bta2dpd - new Bluetooth Advanced Audio Distribution Profile daemon

Many evbarm kernels now use FDT (flat device tree) information (loadable at boot time from an external file) for device configuration, the number of kernels has decreased but the numer of boards has vastly increased.

Lots of updates to 3rd party software included:
  • GCC 5.5 with support for Address Sanitizer and Undefined Behavior Sanitizer
  • GDB 7.12
  • GNU binutils 2.27
  • Clang/LLVM 3.8.1
  • OpenSSH 7.6
  • OpenSSL 1.0.2k
  • mdocml 1.14.1
  • acpica 20170303
  • ntp 4.2.8p11-o
  • dhcpcd 7.0.3
  • Lua 5.3.4

The NetBSD developers and the release engineering team have spent a lot of effort to make sure NetBSD 8.0 will be a superb release, but we have not yet fixed most of the accompanying documentation. So the included release notes and install documents will be updated before the final release, and also the above list of major items may lack important things.

Get NetBSD 8.0 RC1 from our CDN (provided by fastly) or one of the ftp mirrors.

Complete source and binaries for NetBSD are available for download at many sites around the world. A list of download sites providing FTP, AnonCVS, and other services may be found at http://www.NetBSD.org/mirrors/.

Please test RC1, so we can make the final release the best one ever so far. We are looking forward to your feedback. Please send-pr any bugs or mail us at releng at NetBSD.org for more general comments.

Posted at noon on Wednesday, April 25th, 2018 Tags: blog
I've spent a month on fixes and debugging issues around the tracing facilities in the kernel.

Distribution cleanup

As planned in the previous month, I've performed cleanup in the distribution:

  • Removal of unneeded PT_SET_SIGMASK and PT_GET_SIGMASK, followed by pullup to NetBSD-8.
  • regnsub(3) and regasub(3) API change has been abandoned as it causes backward compatibility bootstrap breakage from older distributions. Altering it could require additional changes in libtre, but it has been abandoned too. At the end I've finally removed just the USE_LIBTRE build option, replacing the libc's regex(3) implementation with an alternative library from libtre/agrep, because it no longer builds as a regex-replacement and the upstream development stalled few years ago.
  • Backport of _UC_MACHINE_FP() to NetBSD-8 has been finished, this means that we can use the same code in NetBSD-8 and NetBSD-current in the sources of sanitizers.

Improvements in the ATF ptrace(2) tests

I've performed the following operations in the ATF ptrace(2) and related tests, in the following commit order:

  • Correct all ATF failures in t_ptrace_x86_wait.h (debug registers)
  • ATF: Correct a race bug in attach2 (t_ptrace_wait*)
  • ATF: Reenable attach2 in t_ptrace_wait*
  • Add a new function in ATF t_ptrace_wait*: await_zombie_raw()
  • ATF t_ptrace_wait*: Disable debug messages in msg.h
  • ATF: Add new test race1 in t_ptrace_wait*
  • Add new ATF tests: kernel/t_zombie

Zombie detection race

I've detected that the operation polling for a single process status - using sysctl(3) - can report the same process twice, for the first time as an alive process and for the second time as a zombie. This used to break the ATF ptrace(2) tests where we have a polling function, detecting the state change of a dying process.

I've prepared an applied a fix for this case and the bug detected in ATF ptrace(2) tests is now gone. I've included additional regression tests as noted below, to catch the race in a dedicated t_zombie test-suite.

There is also a controversy here, as it's not specified by POSIX what happens when we are polling an unrelated process, that is not a child of its parent. This operation can fail and we the previous approach we could report the same process twice, while with the newer one reporting twice is much more unlikely on the cost of missing the process at times.

My preference is that a process between the state transition of alive->dying->dead and dead->zombie, can disappear for a while. I find this metaphor more natural rather than observing two entities that one is dying and the other is a zombie. There is no perfect solution to this process watch and just following the narrow cases requested with POSIX seems to be the proper solution and it de facto assumes such races.

Bohrbug in X86 Debug Registers

The release engineering machines were observing occasional failures with the X86 Debug Registers in the ptrace(2) ATF test-suite. This was appearing once a while, approximately quarterly, on both amd64 and i386 machines.

I've missed them previously in my original work a year ago, as this happened to be caused by the fact that this bug is not reproducible on newer (quicker? more cores?) Intel CPUs. This failure was reproduced only in software-emulation (slow!) qemu and on Intel Core 2 Duo.

An attempt to execute the same X86 Debug Register test on Intel i7 for 100M times does not make this crash to pop up even a single time.

I've spent the majority of the past month on researching this bug and it happened to be a bohrbug, this means that adding almost any debug code anywhere makes it disappear completely or make reproducible siginificantly less often.

It's not worth the space to describe the process of understanding this bug more closely, step by step - it's worth to mention the current state of the understanding of mine. We can observe a repeated syscall for the _lwp_kill(2) operation (executed with raise(SIGSTOP)), with the same trap frame (or very similar). I still don't know how is it possible.

I've finally added support to the NetBSD kernel - in my local copy of the sources - for Bochs-style debug console/protocol (ISA port 0xe9). This allows me to log internal qemu debug and NetBSD kernel messages into a single buffer and output to a single file. I need this property as matching two logging buffers isn't trivial, especially since just the logging of interrupt frames in qemu can quickly generate hundreds of megabytes of text. Assuming that I've reproduced the bohrbug after 20k executions of the same test and there is a lot of noise from unrelated kernel/hardware traps, it's a useful property. I'm planning to cleanup this code and submit to the NetBSD sources.

For completeness I had to patch the qemu isa-debugcon (Bochs-style debug console) source code to log into the same buffer as the internal qemu debug messages... sometimes bugs require non-conventional approaches to research them. Ideally I would like to have a rewind/record feature in qemu together with the gdb-server emulation, but we are still not there.

ptrace(2) status and plans

Now there is the X86 Debug Register race. Also, the following tests still marked as expected failures:

  • eventmask3
  • resume1
  • signal3, signal5, signal6, signal7, signal8, signal9
  • suspend1, suspend2
  • vfork1

This shows that the remaining major problems are in:

  • vfork(2)
  • signals
  • threads

Once I will squash the X86 Debug Register race, I plan to work on these bugs in this oder: vfork handling bugs, signals and threads.

I have in mind addition of a lot of a lot of tests verifying correct signal processing in traced programs.

In this iteration of ptrace(2) hardening, I plan to skip Machine-Dependent extensions (like AVXv2 registers) and extending core(5) files with additional features (for example we might want to know exact ABI or FPU hardware layout).

Plan for the next milestone

Keep working on the bug reproducible with X86 Debug Registers and switch to the remaining ptrace(2) failures.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:

http://netbsd.org/donations/#how-to-donate

Posted late Sunday night, April 2nd, 2018 Tags: blog

In recent years, no one in the releng team has had adequate time to devote to release engineering work. To address this shortcoming, the foundation has agreed to pay me part time to carry out the following duties:

  • process pullup tickets
  • write and update release notes
  • coordinate with the security team
  • plan for future releases and release milestones
  • identify and document critical bugs preventing a release
  • create documents/web-/wikipages describing the current status of releases/release branches and keep them up to date
  • identify missing pullups and push developers to submit pullup requests

While this does not automatically provide developer time (still subject to volunteer time and motivation), it is however still a major step forward towards handling the release cycle, which is (currently) taking too long, in a professional manner.

We hope to

  • Speed up the release process for all pending releases on all active branches, as specified by the releng team and documented on the releng wiki page(s)
  • Make the release process more transparent by providing up to date and easily accessible status documents
  • Create better releases by making sure critical bugs get identified and fixed (if possible)

To help the other issue mentioned above, we plan to offer (small) bug bounties for bugs identified as release show-stoppers.

Posted mid-morning Sunday, March 25th, 2018 Tags: blog

The NetBSD Project is pleased to announce NetBSD 7.1.2, the second security/bugfix update of the NetBSD 7.1 release branch. It represents a selected subset of fixes deemed important for security or stability reasons. If you are running an earlier release of NetBSD, we strongly suggest updating to 7.1.2.

For more details, please see the release notes.

Complete source and binaries for NetBSD are available for download at many sites around the world. A list of download sites providing FTP, AnonCVS, and other services may be found at http://www.NetBSD.org/mirrors/.

Posted terribly early Monday morning, March 19th, 2018 Tags: blog
I've managed to unbreak the LLDB debugger as much as possible with the current kernel and hit problems with ptrace(2) that are causing issues with further work on proper NetBSD support. Meanwhile, I've upstreamed all the planned NetBSD patches to sanitizers and helped other BSDs to gain better or initial support.

LLDB

Since the last time I worked on LLDB, we have introduced many changes to the kernel interfaces (most notably related to signals) that apparently fixed some bugs in Go and introduced regressions in ptrace(2). Part of the regressions were noted by the existing ATF tests. However, the breakage was only marked as a new problem to resolve. For completeness, the ptrace(2) code was also cleaned up by Christos Zoulas, and we fixed some bugs with compat32.

I've fixed a crash in *NetBSD::Factory::Launch(), triggered on startup of the lldb-server application.

Here is the commit message:

We cannot call process_up->SetState() inside
the NativeProcessNetBSD::Factory::Launch
function because it triggers a NULL pointer
deference.

The generic code for launching a process in:
GDBRemoteCommunicationServerLLGS::LaunchProcess
sets the m_debugged_process_up pointer after
a successful call to  m_process_factory.Launch().
If we attempt to call process_up->SetState()
inside a platform specific Launch function we
end up dereferencing a NULL pointer in
NativeProcessProtocol::GetCurrentThreadID().

Use the proper call process_up->SetState(,false)
that sets notify_delegates to false.

Differential Revision: D42868

I've synchronized the logging interfaces in PlatformNetBSD.cpp with Linux, switching to a more generic and modern API and thus reducing the unneeded code difference with this OS.

Differential Revision: D42912

I've submitted a patch to fix recognition of NetBSD images (programs and userland core(5) files). This code is still pending a review and now marked as "Changes Planned", because I was requested to ship tests and I feel more comfortable shipping tests with a more functional debugger.

Differential Revision: D42870

The immediate kernel tracing bug is generating invalid signals, SIGSTOP instead of SIGTRAP and they are apparently occurring under abnormal conditions. This is the reason why I decided to return to ptrace(2) and correct all the problems.

The abnormal breakage looks like this:

Process 1369 stopped
* thread #1, stop reason = signal SIGSTOP
    frame #0: 0x00007f7efc000770 ld.elf_so`.rtld_start
ld.elf_so`.rtld_start:
->  0x7f7efc000770 <+0>: subq   $0x10, %rsp
    0x7f7efc000774 <+4>: movq   %rsp, %r12
    0x7f7efc000777 <+7>: pushq  %rbx
    0x7f7efc000778 <+8>: andq   $-0x10, %rsp

I can step few instructions, but after stepping through a few indeterministic number of them the process is killed and lldb-server detaches abnormally.

Process 1369 stopped
* thread #1, stop reason = instruction step over
    frame #0: 0x00007f7efc000774 ld.elf_so`.rtld_start + 4
ld.elf_so`.rtld_start:
->  0x7f7efc000774 <+4>:  movq   %rsp, %r12
    0x7f7efc000777 <+7>:  pushq  %rbx
    0x7f7efc000778 <+8>:  andq   $-0x10, %rsp
    0x7f7efc00077c <+12>: leaq   0x21087d(%rip), %rax      ; _GLOBAL_OFFSET_TABLE_
(lldb)  
 <  16> send packet: $vCont;s:0001#b3
Process 1369 exited with status = -1 (0xffffffff) lost connection

My observation is that without fixing the kernel we won't make much more progress.

Sanitizers

I suspended development of new features in sanitizers last month, but I was still in the process of upstreaming of local patches. This process was time-consuming as it required rebasing patches, adding dedicated tests, and addressing all other requests and comments from the upstream developers.

A fairly complete list of changes that landed upstream:

  • Add new interceptors: strlcpy(3) and strlcat(3)
  • Add new NetBSD interceptors: devname(3), devname_r(3)
  • Handle NetBSD symbol mangling devname -> __devname50
  • Correct a bug in GetArgsAndEnv() for NetBSD
  • Add new interceptor: lstat(2)
  • Add NetBSD syscall hooks skeleton in sanitizers
  • Prevent recursive MSan interceptors in fgets(3)
  • Prevent recursive MSan interceptors in strftime(3) like functions
  • Teach sanitizer about NetBSD specific ioctl(2) calls
  • Enable syscall-specific functions in TSan/NetBSD
  • Enable test/asan for NetBSD
  • Implement a large part of NetBSD syscalls of netbsd_syscall_hooks.h
  • Add initial XRay support for NetBSD
  • Recognize all NetBSD architectures in UBSan
  • Stop intercepting forkpty(3) and openpty(3) on NetBSD
  • Add new interceptor: fgetln(3)
  • Add new interceptor: strmode(3)
  • Correct ctype(3) functions with NLS on NetBSD
  • Skip two more ioctl interceptors for NetBSD
  • Add new interceptors: getttyent(3) family
  • Add new interceptors: getprotoent(3) family
  • Add new interceptors: getnetent(3) family
  • Disable ASan exceptions on NetBSD
  • Stop linking sanitized applications with -lutil and -lkvm on NetBSD
  • Handle the NetBSD case in ToolChain::getOSLibName()
  • Skip two more ioctl interceptors for NetBSD
  • Mark the textdomain.cc test as unsupported on BSDs

I'm not counting hot fixes, as some changes were triggering build or test issues on !NetBSD hosts. Thankfully all these issues were addressed quickly. The final result is a reduction of local delta size of almost 1MB to less than 100KB (1205 lines of diff). The remaining patches are rescheduled for later, mostly because they depend on extra work with cross-OS tests and prior integration of sanitizers with the basesystem distribution. I didn't want to put extra work here in the current state of affairs and, I've registered as a mentor for Google Summer of Code for the NetBSD Foundation and prepared Software Quality improvement tasks in order to outsource part of the labour.

Userland changes

Part of the work landed the basesystem tree. Here is a list:

  • Install GCC (gcc.old/) headers for Sanitizers
  • Install GCC (gcc) headers for Sanitizers
  • Introduce _UC_MACHINE_FP() as a macro
  • Stop installing dbregs.h
  • Add new tests in lib/libc/sys/t_ucontext

I've also improved documentation for some of the features of NetBSD, described in man-pages. These pieces of information were sometimes wrong or incomplete, and this makes covering the NetBSD system with features such as sanitizers harder as there is a mismatch between the actual code and the documented code.

Some pieces of software also require better namespacing support, these days mostly for the POSIX standard. I've fixed few low-hanging fruits there and requested pullups to NetBSD-8(BETA).

I thank the developers for improving the landed code in order to ship the best solutions for users.

mdnsd - Multicast and Unicast DNS daemon

I've been debugging the connectivity issues between lldb client and lldb-server. I've observed a dying connection for one particular message (these programs communicate using the GDB remote protocol, with LLDB extensions): qHostInfo. This message emitted by a client asks the server about the Host Information.

The communication looks like this:

$ ./lldb
(lldb) log enable gdb-remote packets
(lldb) process connect connect://localhost:1234
                 <   1> send packet: +
                 history[1] tid=0x0001 <   1> send packet: +
                 <  19> send packet: $QStartNoAckMode#b0
                 <   1> read packet: +
                 <   6> read packet: $OK#9a
                 <   1> send packet: +
                 <  41> send packet: $qSupported:xmlRegisters=i386,arm,mips#12
                 < 124> read packet: $PacketSize=20000;QStartNoAckMode+;QThreadSuffixSupported+;QListThreadsInStopReply+;qEcho+;QPassSignals+;qXfer:auxv:read+#be
                 <  26> send packet: $QThreadSuffixSupported#e4
                 <   6> read packet: $OK#9a
                 <  27> send packet: $QListThreadsInStopReply#21
                 <   6> read packet: $OK#9a
                 <  13> send packet: $qHostInfo#9b

The communication is now hanging and data is no longer being received by the client.

I was debugging the server, the client side, and even tapping the wire to test if there was ongoing communication and to find the lost answer. The answer looks like this and is 400-500 octets long (originally sent as a single line, divided into 100-octet rows):

$triple:7838365f36342d756e6b6e6f776e2d6e6574627364382e39392e3132;ptrsize:8;watchpoint_exceptions_rec
eived:after;endian:little;os_version:8.99.12;os_build:30383939303031323030;os_kernel:4e6574425344203
82e39392e3132202847454e45524943292023303a20576564204665622032382030383a30363a33332043455420323031382
020726f6f744063686965667465633a2f7075626c69632f6e65746273642d726f6f742f7379732f617263682f616d6436342
f636f6d70696c652f47454e45524943;hostname:6c6f63616c686f7374;#88

We can decode the string using e.g. radare2 tools:

$ rax2 -s 7838365f36342d756e6b6e6f776e2d6e6574627364382e39392e3132;echo
x86_64-unknown-netbsd8.99.12

The debugging took a while, and after finding no bugs on the client and server side I've finally detected the root cause of the problem: mdnsd. An upgrade in the development branch broke mdnsd and it couldn't resolve the hostname anymore. LLDB was calling the following algorithm on the server side:

#include <stdio.h>
#include <unistd.h>
#include <netdb.h>
#include <limits.h>

int
main(int argc, char **argv)
{
  char hostname[PATH_MAX];

  hostname[sizeof(hostname) - 1] = '\0';
  if (gethostname(hostname, sizeof(hostname) - 1) == 0) {
    printf("gethostname done\n");
    struct hostent *h = gethostbyname(hostname);
    printf("gethostbyname done\n");
    if (h)
      printf("h: '%s'\n", h->h_name);
    else
      printf("!h: '%s'\n", hostname);
  } else {
    printf("gethostname error\n");
  }

  return 0;
}

The gethostbyname(3) operation was taking 10 seconds, instead of returning a proper result almost immediately. I've verified that LLDB expects a response in 1 second and GDB within 2 seconds. This was a good sign that something was broken on the NetBSD side. Thanks to the excellent ktruss(1) tool I tracked down the root cause quickly and with feedback provided by more experienced networking engineers we concluded that mdnsd was broken.

I've found a workaround, defining my host in /etc/hosts and assuring that /etc/nsswitch.conf lists files before dns & mdnsd for the hosts option.

The mdnsd problem has been reported to developers and was quickly fixed by Christos Zoulas.

The name resolution with mdnsd is quick and correct again:

$ time getent hosts rugged.local
192.168.0.241     rugged.local
    0.03s real     0.00s user     0.00s system

BSD collaboration in LLVM

A One-man-show in human activity is usually less fun and productive than collaboration in a team. This is also true in software development. Last month I was helping as a reviewer to port LLVM features to FreeBSD and when possible to OpenBSD. This included MSan/FreeBSD, libFuzzer/FreeBSD, XRay/FreeBSD and UBSan/OpenBSD.

I've landed most of the submitted and reviewed code to the mainstream LLVM tree.

Part of the code also verified the correctness of NetBSD routes in the existing porting efforts and showed new options for improvement. This is the reason why I've landed preliminary XRay/NetBSD code and added missing NetBSD bits to ToolChain::getOSLibName(). The latter produced setup issues with the prebuilt LLVM toolchain, as the directory name with compiler-rt goodies were located in a path like ./lib/clang/7.0.0/lib/netbsd8.99.12 with a varying OS version. This could stop working after upgrades, so I've simplified it to "netbsd", similar to FreeBSD and Solaris.

Prebuilt toolchain for testers

I've prepared a build of Clang/LLVM with LLDB and compiler-rt features prebuilt on NetBSD/amd64 v. 8.99.12:

llvm-clang-compilerrt-lldb-7.0.0beta_2018-02-28.tar.bz2

Plan for the next milestone

With the approaching NetBSD 8.0 release I plan to finish backporting a few changes there from HEAD:

  • Remove one unused feature from ptrace(2), PT_SET_SIGMASK & PT_GET_SIGMASK. I've originally introduced these operations with criu/rr-like software in mind, but they are misusing or even abusing ptrace(2) and are not regular process debuggers. I plan to remove this operation from HEAD and backport this to NetBSD-8(BETA), before the release, so no compat will be required for this call. Future ports of criu/rr should involve dedicated kernel support for such requirements.
  • Finish the backport of _UC_MACHINE_FP() to NetBSD-8. This will allow use of the same code in sanitizers in HEAD and NetBSD-8.0.
  • By popular demand, improve the regnsub(3) and regasub(3) API, adding support for more or less substitutions than 10.

Once done, I will return to ptrace(2) debugging and corrections.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:

http://netbsd.org/donations/#how-to-donate

Posted late Thursday afternoon, March 1st, 2018 Tags: blog
An update on the recent security affairs and how they are, or were, handled on NetBSD

DEFCON 25 and 34c3

Ilja Van Sprundel presented at Defcon 25 (July 2017) and 34c3 (December 2017) the results of his audit of the BSD kernels.

The issues affecting NetBSD were fixed overnight in the NetBSD-current branch, and were propagated to the stable branches within a month. Kernels from NetBSD-6 and NetBSD-7 built after August 23rd 2017 had all the necessary fixes.

Some reports published recently suggest that the stable branches remained vulnerable for months, and that NetBSD was lagging behind; that is simply not true.

In Ilja Van Sprundel’s report, NetBSD was criticized for having too much legacy and buggy code. Several proactive measures were taken, within a month again, to clean up the system. These measures were:

  • TCP_COMPAT_42 was removed.
  • COMPAT_FREEBSD was disabled.
  • COMPAT_SVR4 and COMPAT_SVR4_32 were disabled on all architectures.
  • COMPAT_IBCS2 was disabled on all architectures but Vax.
  • COMPAT_SVR4 support for i386 was removed.
  • COMPAT_IBCS2 support for i386 was removed.
  • VM86 was removed.

Several of these changes were propagated to the stable branches. Since, several additional improvements were made to further externalize some parts of the kernel, in such a way that features can be taken out of the system by default, but still be loaded as kernel modules dynamically when they are needed. This aims, of course, at reducing the attack surface in the base system.

Due to the limited human resources available in security-team@, Security Advisories generally take time to be issued. A Security Advisory for the reported problems had not been issued in time, and it was decided not to issue one. The Security Team will continue working on more recent security issues.

Meltdown and Spectre

The counter-measure for Meltdown, called SVS (Separate Virtual Space), is being developed. It was first committed on January 7th 2018, and has now reached a stable state. It is available only on x86 64bit (amd64) for now, this architecture being our primary target.

A significant effort is required to back-port SVS to the stable branches: many improvements were made in the amd64 port (better security and performance) since the last release, and they will have to be, at some point, back-ported too.

Regarding Spectre, Intel and AMD have issued microcode updates. In the case of Intel, the new microcode adds several MSRs, that the OS can tune to disable branch prediction. Given that NetBSD supports microcode updates, it is possible to install a new microcode; however, no option is available yet to tune the aforementioned MSRs.

It is not clear whether the fixes proposed by Intel and AMD are sufficiently reliable. Recent reports suggest that some CPUs have started misbehaving when running with the new microcodes. Therefore, the fix for Spectre is expected to take a little more time to produce than that of Meltdown.

Posted early Monday morning, February 5th, 2018 Tags: blog
Add a comment
Contact | Disclaimer | Copyright © 1994-2018 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
NetBSD® is a registered trademark of The NetBSD Foundation, Inc.