Jan 2017
S M T W T F S
21
22
       

Archives

This page is a blog mirror of sorts. It pulls in articles from blog's feed and publishes them here (with a feed, too).

[Update, 2017-01-16 20:48 UTC: All issues have been resolved and NetBSD.org should be functioning normally now.]

As some of you may have noticed, the netbsd.org DNS records are broken right now. Two independent incidents caused this to happen:

  1. UC Berkeley stopped providing secondary DNS service for netbsd.org at adns1.berkeley.edu and adns2.berkeley.edu on a few hours notice, before we had time to update the NS delegations in the ORG. zone. [Update, 2017-01-16 20:48 UTC: The Berkeley network administrators have graciously restored service until we can update our NS delegations. Thanks, Berkeley—for doing this on a US holiday, no less!]
  2. The network hosting our primary nameserver ns.netbsd.de went offline at around 14:00 UTC for reasons unknown. [Update, 2017-01-16 20:48 UTC: The network and ns.netbsd.de are back up.]

We have sent a request to our registrar to update the delegations to make ns.netbsd.org primary and ns.netbsd.de secondary. We asked them to expedite the process, but owing to the weekend, when the humans who process these requests are off work, it may take another day for that to take effect, and another day after that for the cached TTLs on the old NS delegations to Berkeley's nameservers to expire.

As a provisional stop gap, if you need access to the netbsd.org zone and you can prime your DNS cache with manually entered records, you can prime it with the following records to restore service at your site:

netbsd.org.     86400   IN      NS      ns.netbsd.org.
netbsd.org.     86400   IN      NS      ns.netbsd.de.
ns.netbsd.org.  86400   IN      A       199.233.217.200
ns.netbsd.org.  86400   IN      AAAA    2001:470:a085:999::53
  

(Presumably if you are reading this you have some DNS records in the netbsd.org zone cached, but they will eventually expire and you may need some others.)

Posted late Monday afternoon, January 16th, 2017 Tags: blog
The NetBSD Foundation took part in the 2016 Google Summer of Code.

Hrishikesh Goyal worked on the project "Implement Ext4fs support in ReadOnly mode". He tackled two features of an ext4fs implementation: extents and HTree DIR read/write support. His work was committed into the NetBSD source tree in multiple commits.

charles cui worked on adding tests for better "POSIX Test Suite Compliance". This involved porting the Open Posix benchmark suite to NetBSD. Many of the tests showed missing features, and Charles worked with his mentors to improve the results. See his summary for details.

Leonardo Taccari worked on "Split debug symbols for pkgsrc builds". He already blogged about this in much detail on this blog.

A big thank you to Google for sponsoring the students to work on NetBSD, the students for working on the projects, and the mentors that were helping them along!

Posted Saturday afternoon, January 14th, 2017 Tags: blog

The first release candidate of NetBSD 7.1 is now available for download at:

http://cdn.NetBSD.org/pub/NetBSD/NetBSD-7.1_RC1/

Those of you who prefer to build from source can continue to follow the netbsd-7 branch or use the netbsd-7-1-RC1 tag.

There have been quite a lot of changes since 7.0. See src/doc/CHANGES-7.1 for the full list.

Please help us out by testing 7.1_RC1. We love any and all feedback. Report problems through the usual channels (submit a PR or write to the appropriate list). More general feedback is welcome at releng@NetBSD.org.

Posted early Monday morning, January 9th, 2017 Tags: blog
In short, we are already in a good state with the existing ptrace(2) interfaces, as most necessary functions in LLDB are representable by existing NetBSD specific interfaces. We can fully implement core LLDB functionality without further extensions in ptrace(2). During this project dozen of bugs were investigated & fixed and several hundreds of ATF tests added. The major addition is newly added support for hardware assisted watchpoints API for ptrace(2) on amd64 and preliminary code for i386 and XEN.

What has been done

1. Verified basic ptrace(2) functionality:

  • debugger cannot attach to PID0 (as root and user)
  • debugger cannot attach to PID1 (as user)
  • debugger cannot attach to PID1 with sercurelevel >= 1 (as root and user)
  • debugger cannot attach to self
  • debugger cannot attach to another process unless the process's root directory is at or below the tracing process's root

2. Verified the full matrix of combinations of wait(2) and ptrace(2) in the following test-cases

  • tracee can emit PT_TRACE_ME for parent and raise SIGSTOP followed by _exit(2)
  • tracee can emit PT_TRACE_ME for parent and raise SIGSTOP followed by _exit(2), with perent sending SIGINT and catching this singal only once with a signal handler and without termination of tracee
  • tracee can emit PT_TRACE_ME for parent and raise SIGSTOP, with perent sending SIGINT and terminating the child without signal handler
  • tracee can emit PT_TRACE_ME for parent and raise SIGSTOP, with perent sending SIGINT and terminating the child without signal handler
  • tracee can emit PT_TRACE_ME for parent and raise SIGCONT, and parent reports it as process stopped
  • assert that tracer sees process termination earlier than the parent
  • assert that any tracer sees process termination earlier than its parent
  • assert that tracer parent can PT_ATTACH to its child
  • assert that tracer child can PT_ATTACH to its parent
  • assert that tracer sees its parent when attached to tracer (check getppid(2))
  • assert that tracer sees its parent when attached to tracer (check sysctl(7) and struct kinfo_proc2)
  • assert that tracer sees its parent when attached to tracer (check /proc/curproc/status 3rd column)
  • verify that empty EVENT_MASK is preserved
  • verify that PTRACE_FORK in EVENT_MASK is preserved
  • verify that fork(2) is intercepted by ptrace(2) with EVENT_MASK set to PTRACE_FORK
  • verify that fork(2) is not intercepted by ptrace(2) with empty EVENT_MASK
  • verify that vfork(2) is intercepted by ptrace(2) with EVENT_MASK set to PTRACE_VFORK [currently EVENT_VFORK not implemented]
  • verify that vfork(2) is not intercepted by ptrace(2) with empty EVENT_MASK [currently failing as EVENT_VFORK not implemented]
  • verify PT_IO with PIOD_READ_D and len = sizeof(uint8_t)
  • verify PT_IO with PIOD_READ_D and len = sizeof(uint16_t)
  • verify PT_IO with PIOD_READ_D and len = sizeof(uint32_t)
  • verify PT_IO with PIOD_READ_D and len = sizeof(uint64_t)
  • verify PT_IO with PIOD_WRITE_D and len = sizeof(uint8_t)
  • verify PT_IO with PIOD_WRITE_D and len = sizeof(uint16_t)
  • verify PT_IO with PIOD_WRITE_D and len = sizeof(uint32_t)
  • verify PT_IO with PIOD_WRITE_D and len = sizeof(uint64_t)
  • verify PT_READ_D called once
  • verify PT_READ_D called twice
  • verify PT_READ_D called three times
  • verify PT_READ_D called four times
  • verify PT_WRITE_D called once
  • verify PT_WRITE_D called twice
  • verify PT_WRITE_D called three times
  • verify PT_WRITE_D called four times
  • verify PT_IO with PIOD_READ_D and PIOD_WRITE_D handshake
  • verify PT_IO with PIOD_WRITE_D and PIOD_READ_D handshake
  • verify PT_READ_D with PT_WRITE_D handshake
  • verify PT_WRITE_D with PT_READ_D handshake
  • verify PT_IO with PIOD_READ_I and len = sizeof(uint8_t)
  • verify PT_IO with PIOD_READ_I and len = sizeof(uint16_t)
  • verify PT_IO with PIOD_READ_I and len = sizeof(uint32_t)
  • verify PT_IO with PIOD_READ_I and len = sizeof(uint64_t)
  • verify PT_READ_I called once
  • verify PT_READ_I called twice
  • verify PT_READ_I called three times
  • verify PT_READ_I called four times
  • verify plain PT_GETREGS call without further steps
  • verify plain PT_GETREGS call and retrieve PC
  • verify plain PT_GETREGS call and retrieve SP
  • verify plain PT_GETREGS call and retrieve INTRV
  • verify PT_GETREGS and PT_SETREGS calls without changing regs
  • verify plain PT_GETFPREGS call without further steps
  • verify PT_GETFPREGS and PT_SETFPREGS calls without changing regs
  • verify single PT_STEP call
  • verify PT_STEP called twice
  • verify PT_STEP called three times
  • verify PT_STEP called four times
  • verify that PT_CONTINUE with SIGKILL terminates child
  • verify that PT_KILL terminates child
  • verify basic LWPINFO call for single thread (PT_TRACE_ME)
  • verify basic LWPINFO call for single thread (PT_ATTACH from tracer)

3. Documentation of ptrace(2)

  • documented PT_SET_EVENT_MASK, PT_GET_EVENT_MASK and PT_GET_PROCESS_STATE
  • updated and fixed documentation of PT_DUMPCORE
  • documented PT_GETXMMREGS and PT_SETXMMREGS (i386 port specific)
  • documented PT_GETVECREGS and PT_SETVECREGS (ppc ports specific)
  • other tweaks and cleanups in the documentation

4. exect(3) - execve(2) wrapper with tracing capabilities

Researched its usability and added ATF test, it's close to be marked for removal - it's marked as broken as it is on all ports... this call was inherited from BSD4.2 (VAX) and was never useful since the inception as it is enabling singlestepping before calling execve(2) and tracing libc calls before switching to new process image.

5. pthread_dbg(3) - POSIX threads debugging library documentation

  • added documentation for the library in man-page
  • upstreamed to the mandoc project to recognize the pthread_dbg(3) library
  • document td_close(3) close connection to a threaded process
  • document td_map_pth2thr(3) convert a pthread_t to a thread handle
  • document td_open(3) make connection to a threaded process
  • document td_thr_getname(3) get the user-assigned name of a thread
  • document td_thr_info(3) get information on a thread
  • document td_thr_iter(3) iterate over the threads in the process

6. pthread_dbg(3) - pthread debug library - t_dummy tests

  • assert that dummy lookup functions stops td_open(3)
  • assert that td_open(3) for basic proc_{read,write,lookup} works
  • asserts that calling td_open(3) twice for the same process fails

7. pthread_dbg(3) - pthread debug library - test of features

  • assert that td_thr_iter(3) call without extra logic works
  • assert that td_thr_iter(3) call is executed for each thread once
  • assert that for each td_thr_iter(3) call td_thr_info(3) is valid
  • assert that for each td_thr_iter(3) call td_thr_getname(3) is valid
  • assert that td_thr_getname(3) handles shorter buffer parameter and the result is properly truncated
  • assert that pthread_t can be translated with td_map_pth2thr(3) to td_thread_t -- and assert earlier that td_thr_iter(3) call is valid
  • assert that pthread_t can be translated with td_map_pth2thr(3) to td_thread_t -- and assert later that td_thr_iter() call is valid
  • assert that pthread_t can be translated with td_map_pth2thr(3) to td_thread_t -- compare thread's name of pthread_t and td_thread_t
  • assert that pthread_t can be translated with td_map_pth2thr(3) to td_thread_t -- assert that thread is in the TD_STATE_RUNNING state

8. pthread_dbg(3) - pthread debug library - code fixes

  • fix pt_magic (part of pthread_t) read in td_thr_info(3)
  • correct pt_magic reads in td_thr_{getname,suspend,resume}(3)
  • fix pt_magic read in td_map_pth2thr(3)
  • always set trailing '\0' in td_thr_getname(3) to compose valid ASCIIZ string
  • kill SA thread states (TD_STATE_*) in pthread_dbg and add TD_STATE_DEAD
  • obsolete thread_type in td_thread_info_st in pthread_dbg.h

This library was designed for Scheduler Activation, and this feature was removed in NetBSD 5.0.

9. wait(2) family tests

  • test that wait6(2) handled stopped/continued process loop
  • test whether wait(2)-family of functions return error and set ECHILD for lack of children
  • test whether wait(2)-family of functions return error and set ECHILD for lack of children, with WNOHANG option verifying that error is still signaled and errno set

10. Debug Registers assisted watchpoints:

  • fix rdr6() function on amd64
  • add accessors for available x86 Debug Registers (DR0-DR3, DR6, DR7)
  • switch x86 CPU Debug Register types from vaddr_t to register_t
  • torn down KSTACK_CHECK_DR0, i386-only feature to detect stack overflow

12. i386 port tests

  • call PT_GETREGS and iterate over General Purpose registers

13. amd64 port tests

  • call PT_GETREGS and iterate over General Purpose registers
  • call PT_COUNT_WATCHPOINTS and assert four available watchpoints
  • call PT_COUNT_WATCHPOINTS and assert four available watchpoints, verify that we can read these watchpoints
  • call PT_COUNT_WATCHPOINTS and assert four available watchpoints, verify that we can read and write unmodified these watchpoints
  • call PT_COUNT_WATCHPOINTS and test code trap with watchpoint 0
  • call PT_COUNT_WATCHPOINTS and test code trap with watchpoint 1
  • call PT_COUNT_WATCHPOINTS and test code trap with watchpoint 2
  • call PT_COUNT_WATCHPOINTS and test code trap with watchpoint 3
  • call PT_COUNT_WATCHPOINTS and test write trap with watchpoint 0
  • call PT_COUNT_WATCHPOINTS and test write trap with watchpoint 1
  • call PT_COUNT_WATCHPOINTS and test write trap with watchpoint 2
  • call PT_COUNT_WATCHPOINTS and test write trap with watchpoint 3
  • call PT_COUNT_WATCHPOINTS and test write trap with watchpoint 0
  • call PT_COUNT_WATCHPOINTS and test rw trap with watchpoint 0 on data read
  • call PT_COUNT_WATCHPOINTS and test rw trap with watchpoint 1 on data read
  • call PT_COUNT_WATCHPOINTS and test rw trap with watchpoint 2 on data read
  • call PT_COUNT_WATCHPOINTS and test rw trap with watchpoint 3 on data read

14. Other minor improvements

  • document in wtf(7) PCB process control block
  • fix cpu_switchto(9) prototype in a comment in RICSV

15. Add support for hardware assisted watchpoints/breakpoints API in ptrace(2)

Add new ptrace(2) calls:
  1. PT_COUNT_WATCHPOINTS - count the number of available hardware watchpoints
  2. PT_READ_WATCHPOINT - read struct ptrace_watchpoint from the kernel state
  3. PT_WRITE_WATCHPOINT - write new struct ptrace_watchpoint state, this includes enabling and disabling watchpoints
The ptrace_watchpoint structure contains MI and MD parts:
typedef struct ptrace_watchpoint {
	int		pw_index;	/* HW Watchpoint ID (count from 0) */
	lwpid_t		pw_lwpid;	/* LWP described */
	struct mdpw	pw_md;		/* MD fields */
} ptrace_watchpoint_t;
For example amd64 defines MD as follows:
struct mdpw {
	void	*md_address;
	int	 md_condition;
	int	 md_length;
};

These calls are protected with the __HAVE_PTRACE_WATCHPOINTS guard.

Tested on amd64, initial support added for i386 and XEN.

16. Reported bugs:

  • PR kern/51596 (ptrace(2): raising SIGCONT in a child results with WIFCONTINUED and WIFSTOPPED true in the parent)
  • PR kern/51600 (Tracer must detect zombie before child's parent)
  • PR standards/51603 WIFCONTINUED()=true always implies WIFSTOPPED()=true
  • PR standards/51606 wait(2) with WNOHANG does not return errno ECHILD for lack of children
  • PR kern/51621: PT_ATTACH from a parent is unreliable [false positive]
  • PR kern/51624: Tracee process cannot see its appropriate parent when debugged by a tracer
  • PR kern/51630 ptrace(2) command PT_SET_EVENT_MASK: option PTRACE_VFORK unsupported
  • PR lib/51633 tests/lib/libpthread_dbg/t_dummy unreliable [false positive]
  • PR lib/51635 td_thr_iter in seems broken [false positive]
  • PR lib/51636: It's not possible to run under gdb(1) programs using the pthread_dbg library
  • PR kern/51649: PRIxREGISTER and PTRACE_REG_* mismatch
  • PR kern/51685 (ptrace(2): Signal does not set PL_EVENT_SIGNAL in (struct ptrace_lwpinfo.)pl_event)
  • PR port-amd64/51700 exect(3) misdesigned and hangs

.... and several critical ones not reported and fixed directly by the NetBSD team.

Credit for Christos Zoulas and K. Robert Elz for helping with the mentioned bugs.

17. Added doc/TODO.ptrace entries

  • verify ppid of core dump generated with PT_DUMPCORE it must point to the real parent, not tracer
  • adapt OpenBSD regress test (regress/sys/ptrace/ptrace.c) for the ATF context
  • add new ptrace(2) calls to lock (suspend) and unlock LWP within a process
  • add PT_DUMPCORE tests in the ATF framework
  • add ATF tests for PT_WRITE_I and PIOD_WRITE_I - test mprotect restrictions
  • add ATF tests for PIOD_READ_AUXV
  • add tests for the procfs interface covering all functions available on the same level as ptrace(2)
  • add support for PT_STEP, PT_GETREGS, PT_SETREGS, PT_GETFPREGS, PT_SETFPREGS in all ports
  • integrate all ptrace(2) features in gdb
  • add ptrace(2) NetBSD support in LLDB
  • add support for detecting equivalent events to PTRACE_O_TRACEEXEC, PTRACE_O_TRACECLONE, PTRACE_O_TRACEEXIT from Linux
  • exect(3) rething or remove -- maybe PT_TRACE_ME + PTRACE_O_TRACEEXEC?
  • refactor pthread_dbg(3) to only query private pthread_t data, otherwise it duplicates ptrace(2) interface and cannot cover all types of threads

Features in ELF, DWARF, CTF, DTrace are out of scope of the above list.

Plan for the coming weeks

My initial goal is to copy the Linux Process Plugin and add minimal functional support for NetBSD in LLDB. It will be followed with running and passing some tests from the lldb-server test-suite.

This is a shift from the original plan about porting FreeBSD Process Plugin to NetBSD, as the FreeBSD one is lacking remote debugging support and it needs to be redone from scratch.

I'm going to fork wip/lldb-git (as for git snapshot from 16 Dec 2016) for the purpose of this task to wip/lldb-netbsd and work there.

Next steps after finishing this task are to sync up with Pavel from the LLDB team after New Year. The NetBSD Process Plugin will be used as a reference to create new Common Process Plugin shared between Linux and (Net)BSD.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue to fund projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:

http://netbsd.org/donations/#how-to-donate

Posted Sunday afternoon, December 18th, 2016 Tags: blog

The NetBSD Project is pleased to announce NetBSD 7.0.2, the second security/bugfix update of the NetBSD 7.0 release branch. It represents a selected subset of fixes deemed important for security or stability reasons. If you are running an earlier release of NetBSD, we strongly suggest updating to 7.0.2.

For more details, please see the release notes.

Complete source and binaries for NetBSD are available for download at many sites around the world. A list of download sites providing FTP, AnonCVS, SUP, and other services may be found at http://www.NetBSD.org/mirrors/.

Posted in the wee hours of Wednesday night, October 27th, 2016 Tags: blog
Greetings,


The NetBSD Foundation has initiated a contract with Kamil Rytarowski
 to complete the following during it:

   1.  Add missing interfaces in ptrace(2), mostly sync it with
       the FreeBSD capabilities, add ATF tests, add documentation.

   2.  Develop process plugin in LLDB based on the FreeBSD code in
       LLDB and make it functional (start passing at least some tests).

   3.  Revamp the process plugin in LLDB for new remote debugging
       capabilities (in order to get it accepted and merged upstream),
       pass more tests.

   4.  LLDB x86 support, pass more of the standard LLDB tests
       and import LLDB to the NetBSD base. Add some ATF LLDB basic
       functionality tests to the tree. The original tests are
       unreliable and generate false positives.

   5.  Develop missing features needed for .NET (POSIX robust
       mutexes), add ATF tests.

   6.  Develop missing features for VirtualBox as host, including
       needing sigevent(2) on par with POSIX and SIGEV_KEVENT, and
       other real-time AIO related interfaces as needed.

   7.  Port Apple's Swift programming language. Enhance .NET port to
       validate new interface and correct more issues as needed.

   8.  Improve VirtualBox host support. Make it build first by disabling
       missing features of providing empty facades for them.

   9.  Implement CDROM, floppy, NIC support for NetBSD in VBox as host.

  10.  Make VirtualBox runnable at least with a restricted feature set, ship
        it in pkgsrc, and submit it upstream.


The NetBSD Foundation will continue to work diligently with the
community to fund projects furthering specific key and quality
improvements to the NetBSD projects.  We have a list of projects at
http://wiki.netbsd.org/projects/ as well as welcome other proposals
to move our flag forward to next releases!

Thank you to Kamil for committing to it and we all look forward to it!

The NetBSD Foundation is a non-profit organization and welcomes any
donations to help us continue to fund projects and services to the
open-source community.  Please consider visiting the following URL,
and chip in what you can:

		http://netbsd.org/donations/#how-to-donate





			Submitted for The NetBSD Foundation,
				William J. Coldwell (billc@)
				President, Chairperson

20161012: edited to fix #10 to be the correct wording.
Posted Wednesday night, October 12th, 2016 Tags: blog
Please join us in welcoming the new Directors to the NetBSD Foundation Board:

Michael van Elst, Taylor R. Campbell, Thomas Klausner

We'd like to sincerely thank the departing board members for their service during their term:

Matthew Sporleder, SAITOH Masanobu, Christos Zoulas

Christos will be remaining as Secretary and Treasurer for the Foundation.


Your 2016-2017 Directors of the Foundation are:
Erik Berls
Taylor R. Campbell
William J. Coldwell
Michael van Elst
Thomas Klausner
Jeremy C. Reed
S. P. Zeidler

The current office holders of the Foundation are:
President: William J. Coldwell
Vice President: Jeremy C. Reed
Secretary: Christos Zoulas
Treasurer: Christos Zoulas

Thank you to all of the developers that nominated and voted, the NomCom, the Voting Administrator and Voting Validator.

Respectfully submitted for the Board of Directors,
William J. Coldwell

Posted late Thursday afternoon, October 6th, 2016 Tags: blog

Google Summer of Code 2016 is now over. We have polished the code and documentation and submitted the final term evaluation on 23rd of August 2016. The mentors evaluated us in the following week.

If you are impatient (and this time impatience is definitely a virtue!) please take a look to all The NetBSD Foundation GSoC 2016 projects' code submissions!

Introduction

In part 1 we have learned what happens under the hood when we split debugging symbols: how debugging information is stored/stripped off, the relevant ELF sections involved, various tools to dump that information, etc..

In particular, we have studied what happens on NetBSD when we set the MKDEBUG* flags.

With this background we can finally try to implement this functionality in the pkgsrc infrastructure. In this blog post we will first give a practical look at PKG_DEBUGDATA and then we will analyze the code that I have written in the first mid-term of the GSoC to accomplish that.

PKG_DEBUGDATA in action

Before digging in the implementation of debugdata functionality let's see what it produces!

In part 1 we took as example a very simple program that just prints out the lyrics of Ten Green Bottles song. After packaging it as local/green-bottles let's just build it without any special pkgsrc system variables set:

$ cd pkgsrc/local/green-bottles
$ make install
[...]
$ green-bottles
ten green bottles hanging on the wall
[...]
$ pkg_info -L green-bottles
Information for green-bottles-0:

Files:
/usr/pkg/bin/green-bottles
$ gdb `which green-bottles`
GNU gdb (GDB) 7.10.1
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
[...]
Reading symbols from /usr/pkg/bin/green-bottles...(no debugging symbols found)...done.
(gdb) quit

That's expected because we did not set any special CFLAGS nor INSTALL_UNSTRIPPED.

Before setting PKG_DEBUGDATA to "yes" let's inspect its documentation via the help target:

$ make help topic=PKG_DEBUGDATA
===> mk/bsd.debugdata.mk (keywords: INSTALL_UNSTRIPPED PKG_DEBUGLEVEL PKG_DEBUGDATA debug):
# This Makefile fragment is included by bsd.pkg.mk and implements the
# logic needed to strip debug symbols off the program/libraries.
#
# User-settable variables:
#
# PKG_DEBUGDATA
#       If "yes", install stripped debug symbols for all programs and shared
#       libraries. Please notice that if it is defined INSTALL_UNSTRIPPED will
#       be also defined internally.
#
# PKG_DEBUGLEVEL
#       Used to control the granularity of the debug information. Can be
#       "small", "default", "detailed".
#       TODO: the name of this variable should be changed because it can be
#       TODO: easily confused with PKG_DEBUG_LEVEL!
#
# See also:
#       INSTALL_UNSTRIPPED
#

Now that we have finally an idea of what PKG_DEBUGDATA does, let's set it and reinstall the green-bottles package:

$ make clean deinstall
[...]
$ PKG_DEBUGDATA=yes make install
[...]
===> Installing for green-bottles-0
[...]
=> Stripping debug symbols
=> Installing debug data
[...]
=> Checking for missing debug data in green-bottles-0
[...]

Apart from the various phases we can see three extra messages that are now printed out: stripping of debug symbols, installing of debug data and a check for missing debug data.

If we inspect the files installed we can now also see green-bottles.debug:

$ pkg_info -L green-bottles
Information for green-bottles-0:

Files:
/usr/pkg/bin/green-bottles.debug
/usr/pkg/bin/green-bottles

Indeed if we now try to run it through gdb(1):

$ gdb `which green-bottles`
GNU gdb (GDB) 7.10.1
Copyright (C) 2015 Free Software Foundation, Inc.
[...]
Reading symbols from /usr/pkg/bin/green-bottles...Reading symbols from /usr/pkg/bin/green-bottles.debug...done.
done.
(gdb) b main
Breakpoint 1 at 0xad0: file green-bottles.c, line 29.
(gdb) run
Starting program: /usr/pkg/bin/green-bottles

Breakpoint 1, main () at green-bottles.c:29
29      {
(gdb)
[...]

green-bottles was installed as usual without requiring any changes from MAINTAINERs, in fact the various .debug files are generated, installed and added to the PLIST dynamically.

A look at mk/bsd.debugdata.mk and mk/check/check-debugdata.mk implementation

All the functionalities that implement stripping of the debug data from installed programs and libraries are implemented entirely in bsd.debugdata.mk.

check/check-debugdata.mk checks that all programs and libraries have the debug data correctly stripped into their corresponding *.debug files.

mk/bsd.debugdata.mk

The first 23 lines of bsd.debugdata.mk contain a comment that is also accessible via the help target. In particular it describes all the user-settable variables:

  • PKG_DEBUGDATA: if "yes" it turns on debugdata functionality to generate the *.debug files and then install them as part of the package.
  • PKG_DEBUGLEVEL: useful to control the granularity of the debug information, i.e. the -g level flag passed to the compiler. It can be "small", "default" and "detailed".

Then _VARGROUPS and _USER_VARS.<vargroup> are set accordingly. These are used by the show-all target (if you are more curious regarding that please just try make show-all and/or give a look to mk/misc/show.mk).

After various definitions of some used variables (should be self-explainable) _FIND_DEBUGGABLEDATA_ERE is defined as an Extended Regular Expression that is used to match potential files that can be stripped. _FIND_DEBUGGABLEDATA_ERE is then used to limit the file list generated via _FIND_DEBUGGABLEDATA_FILELIST_CMD

Lines 63-86 pass the appropriate debug flags and debug level to the compiler.

Then _PLIST_DEBUGDATA is added to the PLIST_SRC_DFLT variable. PLIST_SRC_DFLT contains all the default PLISTs that usually resides in pkgsrc/category/PLIST* (e.g. PLIST, PLIST.common, PLIST.NetBSD, etc.). If you are more interested in how it works, please give a look to mk/plist/plist.mk. In this way all the *.debug files listed in _PLIST_DEBUGDATA are dynamically appended to the package PLIST.

In lines 90-111 generate-strip-debugdata target is defined. It basically just does the operations we have explored in part 1 with little adjustments for pkgsrc:

  • Check if the current file is strippable via objdump(1)
  • If the current file is strippable objcopy --only-keep-debug program|library program.debug|library.debug is invoked in order to populate the .debug file.
  • If the current file is strippable objcopy --strip-debug -p -R .gnu_debuglink --add-gnu-debuglink=program.debug|library.debug program|library is invoked to delete the debug information from the current file (program or library) and to add the .gnu_debuglink ELF section to it so that the debugger knows where to pick the corresponding .debug file.
  • Append the .debug path and filename to the _PLIST_DEBUGDATA file.

To accomplish that RUN is used to run all shell commands without printing them. Please note that when RUN is used all shell commands should be terminated with a semicolon. For more information regarding RUN please take a look at make help topic=RUN.

In lines 113-122 install-strip-debugdata is defined. It just installs all *.debug files via INSTALL_DATA. install-strip-debugdata target will then be invoked after the post-install phase.

mk/check/check-debugdata.mk

First 26 lines of check-debugdata.mk contain a comment for the pkgsrc online documentation. It describes the following user-settable variables:

  • CHECK_DEBUGDATA: if "yes" it enables debugdata checks (by default most checks are usually enabled only if PKG_DEVELOPER is "yes")

...and the following package-settable variables:

  • CHECK_DEBUGDATA_SKIP: list of shell patterns that should be excluded from the check.
  • CHECK_DEBUGDATA_SUPPORTED: whether the check should be enabled for the package or not.

Like what it was done for bsd.debugdata.mk, _VARGROUPS and _{USER,PKG}_VARS.check-debugdata are defined accordingly.

Similarly to what was done for _FIND_DEBUGGABLEDATA_ERE and _FIND_DEBUGGABLEDATA_FILELIST_CMD in bsd.debugdata.mk, _CHECK_DEBUGDATA_ERE and _CHECK_DEBUGDATA_FILELIST_CMD are defined.

In lines 54-103 _check-debugdata target is defined. _check-debugdata performs the following checks, respectively:

  • Check that the program/library is readable.
  • If the program/library is a file format recognizable by objdump(1) check if it has a .gnu_debuglink ELF section.
  • Check that the respective .debug file of the program/library is readable.
  • Print a warning message if the .debug file does not contain a .debug_info ELF section.

Other minor changes in mk/*

In order to instruct pkgsrc to pick up bsd.debugdata.mk and checks/check-debugdata.mk, nothing intrusive though:

Conclusion

In this blog post we have learned what PKG_DEBUGDATA does via a practical example.

Then we have examinated how bsd.debugdata.mk and checks/check-debugdata.mk are implemented.

PKG_DEBUGDATA is completely agnostic if a package uses GNU configure, cmake, etc., and it is expected to work without any changes. I have only tested it with simple packages also containing libraries and the ones that uses libtool but I have still not tested it in the wild and some minor adjustements can be probably needed.

Thanks again to Google for organizing Google Summer of Code, The NetBSD Foundation and in particular my mentors David Maxwell, Jöerg Sonnenberger, Taylor R. Campbell, Thomas Klausner and William J. Coldwell.

References

Posted late Wednesday evening, September 14th, 2016 Tags: blog

When I got my Sun T1000 machine, it came with a ~80 GB hard disk - good enough for a NetBSD installation, but a bit challenged when you want to use logical domains. Time to expand disk space, or maybe make it faster? But these 1U server machines do not offer a lot of room for extensions, and it is sometimes tricky to get hold of the official extension options nowadays.

So I had fun with disks and modern replacements again... (after the scsi2sd adventures).

The T1000 were offered in single 3.5" disk configurations (this is what I got) or with two 2.5" disks. The mainboard has two sata (sas?) connectors. But the disk tray in my machine is not usable for mounting two 2.5" HDs.

But: the machine has a spare PCIe slot:
PCIe slot on the back of a T1000
and we support NVMe, so why not put a M.2 module in an adapter card in there? I tried and failed. I learned that the T1000's PCIe x4 slot is PCIe version 1, but modern NVMe modules do not fancy working in anything older than PCIe version 3.

Too bad, but whatever. I switched things around, my amd64 desktop machine got a faster "SSD" (i.e. the NVMe), another machine got the SSD freed from there and I ended up with a spare 128 GB sata SSD.

Now the T1000 had a sata connector free, but no way to place the disk and no power connector for it. The mainboard had a legacy 5/12V jack and the single disk used a strange power/sata connector:

SATA and power connector for the single disk in my T1000

but of course that could be replaced easily. I cut a Y-sata power split and one legacy 5/12V power connector from an old (broken) PSU and soldered a dual sata power connector cable that connected to the jack on the mainboard.

I found a small cavity in the at the front of the machine in the sheet separating PSU and disk from the fans and mainboard. Big enough to push a sata power connector and a sata connector through! So I just glued the SSD at the bottom of the chasis in front of the fans (it should be low enough to not harm airflow significantly):
SSD on bottom of chasis

It was a bit tricky to fit the custom power cabling and the standard sata cable alongside the disk, and I hope the disk will not get too hot to unsolder the connectors ;-)
Custom power cabling for the SSD
Hardware service technicians would hate this, as usually the T1000 offers really easy and fast hard disk replacement, but I suppose no one but me will ever deal with hardware failure on this particular machine.

Now thats it, fast base system on SSD and plenty of space for other LDOMs data on the second disk:

mpt0 at pci3 dev 2 function 0: Symbios Logic SAS1064 (rev. 0x02)
mpt0: interrupting at ivec 7c0
mpt0: Phy 0: Link Status Unknown
mpt0: Phy 0: Link Status Unknown
scsibus0 at mpt0: 63 targets, 8 luns per target
scsibus0: waiting 2 seconds for devices to settle...
sd0 at scsibus0 target 0 lun 0:  disk fixed
sd0: 111 GB, 114474 cyl, 16 head, 127 sec, 512 bytes/sect x 234441648 sectors
sd0: tagged queueing
sd1 at scsibus0 target 1 lun 0:  disk fixed
sd1: 1863 GB, 1907730 cyl, 16 head, 127 sec, 512 bytes/sect x 3907029168 sectors
sd1: tagged queueing

Unfortunately the machine is louder than hell, and even though the rack is in our basement, the sound level is still irritating elsewhere in the house (massive stone walls, old style german construction). My wife officially hates the T1000.

Now that the hardware is ready, it is time to get it fully supported and import the LDOM management/utilities. I need to find time and help Palle!

Posted at noon on Thursday, July 7th, 2016 Tags: blog

NetBSD is happy to announce a generous setup provided by Fastly to give us CDN services. We are live with cdn.NetBSD.org for downloading iso files, binary packages, and anything else that you would find on ftp.NetBSD.org.

nycdn is using nyftp.NetBSD.org as an origin so you can use it to download build snapshots and other useful stuff you would otherwise find on nyftp. (NetBSD-daily for example)


We have already changed some default download links (downloads and pkgsrc) and some pkgsrc files, so you might already be using the cdn without knowing it.

You can also change your PKG_PATH from ftp://ftp.netbsd.org to http://cdn.netbsd.org

HTTPS also works, but not IPv6


If you are not familiar, a CDN is a globally distributed set of caching proxy servers which makes downloading files faster when they are hot in the cache.

Posted at lunch time on Tuesday, July 5th, 2016 Tags: blog
Add a comment
Contact | Disclaimer | Copyright © 1994-2017 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
NetBSD® is a registered trademark of The NetBSD Foundation, Inc.