Jun 2016
S M T W T F S
     
26 27    

Archives

This page is a blog mirror of sorts. It pulls in articles from blog's feed and publishes them here (with a feed, too).

For the 10th time The NetBSD Foundation was selected for the GSoC 2016!

Now that we're near the first mid-term evaluation and have written the code during these weeks it's also the right time to start writing some reports regarding our projects in this series of blog posts.

About Split debug symbols for pkgsrc builds GSoC project

As part of Split debug symbols for pkgsrc builds GSoC project I'm working to provide support for pkgsrc packages for splitted packages that just contain debug symbols for their correspondent package (e.g. for the foo-0.1.2.tgz package there will be a corresponding foo-0.1.2-debugpkg.tgz package that just contains stripped debug symbols of all the former binaries and libraries installed by foo-0.1.2).

If you're more curious and you would like to know more information about it please take a look to the proposal.

Introduction

In this blog post we will learn how debug information are stored and stripped off from the programs and/or libraries. We will first write a simple program and a Makefile to analyze what MKDEBUG* flags in NetBSD do. Then we will take a look more in depth to how everything is implemented in the various src/share/*.mk files and at the end we will give a look to related works already implemented in RPM and dpkg.

A pretty long list of references is also provided for the most curiouses readers!

A quick introduction to ELF and how debug information are stored/stripped off

In order to become familiar with ELF format a good starting point are Object file and Executable and Linkable Format pages from Wikipedia, the free encyclopedia.

Trying to describe ELF format is not easy in short terms so, it is strongly suggested to read the nice article series written by Eric Youngdale for Linux Journal: The ELF Object File Format: Introduction and The ELF Object File Format by Dissection. Please note that these two resources should be enough to completely understand this blog post!

After reading the above resources we have just learned that every programs and libraries in NetBSD (and several other Unix-like operating systems) uses the ELF format. There are four types of ELF object files:

  • executable
  • relocatable
  • shared
  • core

For more information regarding them please give a look to elf(5).

We are interested to understand what happens when we compile the programs/libraries with debugging options (basically the -g option).

NetBSD already supports everything out of the box and so we can quickly start looking at it just writing a simple Makefile and a program that will print the lyrics of the famous Ten Green Bottles song! To avoid all the hassle of providing (multiple times!) the right flags to the compiler and manually invoke the right tool we can just write a very simple Makefile that will do everything for us:

$ cat green-bottles/Makefile
#	$NetBSD$

NOMAN=	# defined

PROG=	green-bottles

.include <bsd.prog.mk>

Now that we have the Makefile we can start writing the green-bottles PROGram (please note that all the green bottles accidentally fall were properly recycled during the writing of this article):

$ cat green-bottles/green-bottles.c 
#include <stdio.h>

void
sing_green_bottles(int n)
{
	const char *numbers[] = { "no more", "one", "two", "three", "four", "five",
	    "six", "seven", "eight", "nine", "ten" };

	if ((1 <= n) && (n <= 10)) {
		printf("%s green bottle%s hanging on the wall\n",
		    numbers[n], n > 1 ? "s" : "");
		printf("%s green bottle%s hanging on the wall\n",
		    numbers[n], n > 1 ? "s" : "");
		printf("and if %s green bottle should accidentally fall,\n",
		    n > 2 ? "one" : "that");
		printf("there'll be %s green bottles hanging on the wall.\n",
		    numbers[n - 1]);
	}

	return;
}


/*
 * Sing the famous `Ten Green Bottles' song.
 */
int
main(void)
{
	int i;

	for (i = 10; i > 0; i--) {
		sing_green_bottles(i);
	}

	return 0;
}

OK! Now everything is ready and if we just invoke make(1) we'll build the program. However, we would like to inspect what's happening behind the scenes, so we'll look at each steps. Please note that right now it is not important that you'll understand everything because we'll look at what make(1) magic do in more details later.

First, we compile the C program to generate the relocatable object file, i.e. green-bottles.o:

$ cd green-bottles/
$ make green-bottles.o
#   compile  green-bottles/green-bottles.o
gcc -O2 -fPIE    -std=gnu99   -Werror     -c    green-bottles.c
ctfconvert -g -L VERSION green-bottles.o

Let's see what file(1) says regarding it:

$ file green-bottles.o
green-bottles.o: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped

In order to get more information we can use readelf(1) tool provided by the binutils (GNU binary utilities), e.g. via readelf -h (the -h option is used to just print the file headers, if you would like to get more information you can use the -a option instead):

$ readelf -h green-bottles.o
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              REL (Relocatable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          2816 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           0 (bytes)
  Number of program headers:         0
  Size of section headers:           64 (bytes)
  Number of section headers:         17
  Section header string table index: 13

We can see the 17 sections always via readelf (-S option). Now let's recompile it but via the debugging options turned on:

$ make green-bottles.o MKDEBUG=yes
#   compile  green-bottles/green-bottles.o
gcc -O2 -fPIE  -g   -std=gnu99   -Werror     -c    green-bottles.c
ctfconvert -g -L VERSION -g green-bottles.o

If we are careful we can see that unlike the previous make incantation now the -g option is passed to the compiler... Let's see if we can inspect that via readelf:

$ readelf -h green-bottles.o
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              REL (Relocatable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          6424 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           0 (bytes)
  Number of program headers:         0
  Size of section headers:           64 (bytes)
  Number of section headers:         29
  Section header string table index: 25

We can note several differences compared to the previous relocatable file compiled without MKDEBUG:

  • Start of section headers (previously 2816, now 6424)
  • Number of section headers (previously 17, now 29)
  • Section header string table index (previously 13, now 25)

If we compare the sections between the two relocatable files (tips: using: readelf -WS green-bottles.o | sed -nEe 's/^ \[ *([0-9]+)\] ([^ ]*) .*/\2/p' is a possible way to do it) we can observe the following new ELF sections:

  • .debug_info: contains main DWARF DIEs (Debugging Information Entry)
  • .debug_abbrev: contains abbreviations used in .debug_info section
  • .debug_loc: contains location expressions
  • .debug_aranges: contains a table for lookup by addresses of program entities (i.e. data objects, types, functions)
  • .debug_ranges: contains address ranges referenced by DIEs
  • .debug_line: contains line number program
  • .debug_str: contains all strings referenced by .debug_info
  • other .rela.debug_*

It's time to finally build the program:

$ make green-bottles
rm -f .gdbinit
touch .gdbinit
#      link  green-bottles/green-bottles
gcc     -pie  -shared-libgcc      -o green-bottles  green-bottles.o  -Wl,-rpath-link,/lib  -L=/lib
ctfmerge -t -g -L VERSION -o green-bottles green-bottles.o

We can observe:

$ readelf -h green-bottles
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x730
  Start of program headers:          64 (bytes into file)
  Start of section headers:          6448 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         7
  Size of section headers:           64 (bytes)
  Number of section headers:         31
  Section header string table index: 27

...and for its counterpart compiled via MKDEBUG=yes:

$ readelf -h green-bottles
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x730
  Start of program headers:          64 (bytes into file)
  Start of section headers:          8304 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         7
  Size of section headers:           64 (bytes)
  Number of section headers:         38
  Section header string table index: 34

Not so surprisingly the number of the 7 extra sections are exactly the .debug_* ones!

Now that it's clear the difference between the program compiled with/without -g option let's see what happen when the debug symbols are stripped off the program:

$ make green-bottles.debug MKDEBUG=yes
#    create  green-bottles/green-bottles.debug
(  objcopy --only-keep-debug green-bottles green-bottles.debug  && objcopy --strip-debug -p -R .gnu_debuglink  --add-gnu-debuglink=green-bottles.debug green-bottles  ) || (rm -f green-bottles.debug; false)

We can try to describe what happened with an image:

green-bottles and green-bottles.debug ELF sections

The first objcopy(1) incantation generate the green-bottles.debug file. The second objcopy(1) incantation strip the debug symbols off green-bottles (now that they're stored in green-bottles.debug they are no more needed) and add the .gnu_debuglink ELF section to it.

Let's quickly look them via file(1):

$ file green-bottles green-bottles.debug
green-bottles:       ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /usr/libexec/ld.elf_so, for NetBSD 7.99.29, not stripped
green-bottles.debug: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter *empty*, for NetBSD 7.99.29, not stripped

Using readelf we can note that now green-bottles has 32 sections and green-bottles.debug has 38 sections. green-bottles has one extra section that was added by the objcopy(1) incantation, let's see it:

$ readelf -x '.gnu_debuglink' green-bottles

Hex dump of section '.gnu_debuglink':
  0x00000000 67726565 6e2d626f 74746c65 732e6465 green-bottles.de
  0x00000010 62756700 90b06f1c                   bug...o.

The .gnu_debuglink section contain the basename(3) of the .debug file and its CRC32. The .gnu_debuglink section is used to properly pick the correct .debug file from the DEBUGDIR directory (we'll see how it will work later when we will invoke the GNU debugger).

Regarding the sections in the .debug file all of them are preserved but several have no data, we can check that by invoking:

$ readelf `seq -f '-x %g' 0 37` green-bottles.debug
$ readelf `seq -f '-x %g' 0 31` green-bottles

...and comparing their respective output.

Now that everything should be clearer we can just try to invoke it through gdb(1) and see what happens:

$ gdb ./green-bottles
GNU gdb (GDB) 7.10.1
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64--netbsd".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./green-bottles...Reading symbols from /tmp/green-bottles/green-bottles.debug...done.
done.
(gdb) b main
Breakpoint 1 at 0xac0: file green-bottles.c, line 29.
(gdb) b sing_green_bottles
Breakpoint 2 at 0x940: file green-bottles.c, line 5.
(gdb) run
Starting program: /tmp/green-bottles/green-bottles

Breakpoint 1, main () at green-bottles.c:29
29      {
(gdb) n
32              for (i = 10; i > 0; i--) {
(gdb) n
33                      sing_green_bottles(i);
(gdb) print i
$1 = 10
(gdb) cont
Continuing.

Breakpoint 2, sing_green_bottles (n=10) at green-bottles.c:5
5       {
(gdb) bt
#0  sing_green_bottles (n=10) at green-bottles.c:5
#1  0x00000000b7802ad7 in main () at green-bottles.c:33
[... we can now looks and debug it as we wish! ...]

So we can see that the green-bottles.debug file is loaded from the same directory where green-bottles program was present (in our case /tmp/green-bottles/ but if a corresponding file .debug is not found gdb look for it in the DEBUGDIR, i.e. /usr/libdata/debug/; e.g. for /usr/bin/yes it will look for debug symbols in /usr/libdata/debug//usr/bin/yes.debug). This is the same for all other programs and libraries.

A look to what MKDEBUG and MKDEBUGLIB do

NetBSD already provides MKDEBUG and MKDEBUGLIB mk.conf(5) variables to achieve the separation of the debug symbols. They respectively split symbols from programs and libraries.

The implementation to do that is in src/share/mk/bsd.prog.mk (for programs) and src/share/mk/bsd.lib.mk (for libraries). Several global variables used are defined in src/share/mk/bsd.own.mk.

bsd.prog.mk

In bsd.prog.mk:58 if MKDEBUG is defined and not "no" [sic] the -g flag is added to CFLAGS.

In bsd.prog.mk:310 the internal __progdebuginstall make target is defined to install the .debug file for the respective program. It is then called from bsd.prog.mk:589 and bsd.prog.mk:604 (respectively for MKUPDATE == "no" and MKUPDATE != "no", please note the dependency operators ! vs : for the two cases).

In bsd.prog.mk:437 _PROGDEBUG.${_P} is defined as ${PROGNAME.${_P}}.debug, inside a for loop. ${_P} is just an element of the ${PROGS} and ${PROGS_CXX} lists. E.g.: for src/bin/echo echo is the PROG value. bsd.prog.mk turns single-program PROG and PROG_CXX variable into the multi-word PROGS and PROGS_CXX variables.

In bsd.prog.mk:545 there is the most important part. After checking if _PROGDEBUG.${_P} is defined a ${_PROGDEBUG.${_P}} target is defined and ${OBJCOPY} is invoked two times. In the first incantation the ${_PROGDEBUG.${_P}} file (containing the strip debug symbols) is generated for ${_P}. The second incantation is needed to get rid of (now no more needed) debug symbols from ${_P} and --add-gnu-debuglink add a .gnu_debuglink section to ${_P} containing the filename of the ${_PROGDEBUG.${_P}}; e.g. for echo it will be echo.debug (plus the CRC32 of echo.debug - padded as needed). Regarding other options used by ${OBJCOPY} we should note the -p option needed to preserve dates and -R is added in order to be sure to update the .gnu_debuglink section.

For a gentler introduction and to understand why these steps are needed please read (gdb.info)Separate Debug Files (you can just use info(1), i.e. info '(gdb.info)Separate Debug Files').

bsd.lib.mk

The logic and objcopy(1) incantation are similar to the ones used in bsd.prog.mk. The most interesting part is in bsd.lib.mk:622. Apart the *.debug files if MKDEBUGLIB is defined and not "no" [sic] also *_g.a archives are created for the respective libraries archives (although they are stored directly in the several lib/ directories not in /usr/libdata/debug/).

bsd.own.mk

In bsd.own.mk various DEBUG* variables are defined:

  • DEBUGDIR: where *.debug files are stored. Please notice that this is also the place where debugging symbols are looked (for more information please give a look to objcopy(1))
  • DEBUGGRP: the -g option passed to install(1) for installing debug symbols
  • DEBUGOWN: the -o option passed to install(1) for installing debug symbols
  • DEBUGMODE: the -m option passed to install(1) for installing debug symbols

Related works

dpkg

The Debian Developer's Reference written by the Developer's Reference Team has a Best practices for debug packages (section 6.7.9). The logic used is more or less the same of the one used by src/share/mk in NetBSD and described above.

After a quick inspection of dh_strip (part of debhelper package) some interesting ideas to look further are:

  • the file(1) logic used in testfile() subroutine
  • handling of non-C/C++ programming languages: OCaml native code shared libraries (*.cmxs) and nodejs binaries (*.node)

RPM

The Fedora Project Wiki contains some interesting tips, in particular regarding most common issues that happens in stripping debugging symbols in the Packaging:Debuginfo page. Some of the logic is handled in find-debuginfo.sh.

Another interesting resource is the Releases/FeatureBuildId page. The page discusses what Red Hat have done regarding using the .note.gnu.build-id section and why have done them.

(Yet another) interesting idea adopted by Fedora developers is the Features/MiniDebugInfo. More information regarding MiniDebugInfo are also present in (gdb.info)MiniDebugInfo. Please note that this is not completely related to stripping debugging symbols (indeed the MiniDebugInfo is directly stored in program/library!) but can be considered in order to provide better .core (both in the pkgsrc and NetBSD cases).

Mark J. Wielaard presented in FOSDEM 2016 a talk that summarizes many of the thematics discussed in this diary. Abstract, video recording and more resources are available in the FOSDEM website correspective event page: Where are your symbols, debuginfo and sources?. Apart his talk a very interesting reading is his blog post regarding the talk. In the blog post there are a lot of interesting information, all worth to be taken in consideration also for the pkgsrc case.

Conclusion

In this blog post we have learned what's happening when we use MKDEBUG* mk.conf(5) variables and how everything works.

We have also gave a quick look to other related works, in particular RPM and dpkg package managers.

If you are curious on what I'm doing right now and you would like to also look at the code you can give a look to the git pkgsrc repository repository fork in the debugpkg branch.

Apart the several references discussed above if you would like to learn more about several aspects that wasn't discussed there... Introduction to the DWARF Debugging Format written by Michael Eager is a good starting point for DWARF (debugging data format); you can also use objdump -g to show these information in the *.debug files. Regarding GDB a gentle introduction to it is Using GNU's GDB Debugger by Peter Jay Salzman.

I would like to thanks Google for organizing Google Summer of Code and The NetBSD Foundation, without them I would not be able to work on this project!

A particular and big thank you goes to my mentors David Maxwell, Jöerg Sonnenberger, Taylor R. Campbell, Thomas Klausner and William J. Coldwell for the invaluable help, guidance and feedbacks they're providing!

References

Posted late Wednesday evening, June 22nd, 2016 Tags: blog

The pkgsrc team has prepared the 50th release of their package management system, with the 2016Q1 version. It's infrequent event, as the 100th release will be held after 50 quarters.

The NetBSD team has prepared series of interviews with the authors. The next one is with Ryo ONODERA, a Japanese developer maintaining large C++ packages.

Hi Ryo, please introduce yourself.

Hi,
I am hobbyist.
I maintain some large C++ packages, however my knowledge about recent C++ is poor.
My current home work is to learn modern C++.

First of all, congratulations on the 50th release of pkgsrc! How do you feel about this anniversary?

I am very glad to commit some changes for this remarkable 50th release.
I feel 50 releases is very long history.
The pkgsrc should be improved for next 100th anniversary.

What are the main benefits of the pkgsrc system?

The pkgsrc helps building from source.
Building from source is interesting and I have learned many things from it. It is worth for experiencing.

I like up-to-date software.
And I believe many people like latest software.
Recently security concerns us.
Sharing the recipe for latest software is getting worth, I believe.

Where and how do you use pkgsrc?

My laptop and home NAS run NetBSD/amd64.
And of course I uses pkgsrc.
And some day-job FreeBSD/amd64 servers also use pkgsrc.

What are the pkgsrc projects you are currently working on?

I am user of Mozilla Firefox (pkgsrc/www/firefox) and LibreOffice (pkgsrc/misc/libreoffice).
I need latest ones. And I will keep them up-to-date.

If you analyze the current state of pkgsrc, which improvements and changes do you wish for the future?

I feel Fortran support is not so powerful. This should be improved.
And Chromium browser should be ported to NetBSD.
If I setup great machine, I will try to do it.

Do you have any practical tips to share with the pkgsrc users?

Good /etc/mk.conf or /usr/pkg/etc/mk.conf improve your pkgsrc experience.
At least,
WRKOBJDIR=/usr/tmp/pkgsrc
and
MASTER_SITE_OVERRIDE=http://ftp.XX.netbsd.org/pub/pkgsrc/distfiles/
should improve your pkgsrc experience.

What's the best way to start contributing to pkgsrc and what needs to be done?

Updating simple package is good start point. For example, Font package or simple C software.
I recommend to get commit bit for pkgsrc-wip.
Committing your idea to the repository is invaluable experience.

Do you plan to participate in the upcoming pkgsrcCon 2016 in Kraków (1-3 July)?

Sadly, I cannot do long distance travel...
If video streaming is provided, I will watch it carefully!

Thank you.
Please use and contribute the pkgsrc!

Posted terribly early Wednesday morning, June 8th, 2016 Tags: blog

The pkgsrc team has prepared the 50th release of their package management system, with the 2016Q1 version. It's infrequent event, as the 100th release will be held after 50 quarters.

The NetBSD team has prepared series of interviews with the authors. The next one is with Jonathan Perkin, a developer in the Joyent team.

Hi Jonathan, please introduce yourself.

Hello! Thirty-something, married with 4 kids. Obviously this means life is usually pretty busy! I work as a Software Engineer for Joyent, where we provide SmartOS zones (also known as "containers" these days) running pkgsrc. This means I am in the privileged position of getting paid to work full-time on what for many years has been my hobby.

First of all, congratulations on the 50th release of pkgsrc! How do you feel about this anniversary?

I've been involved in pkgsrc since 2001, which was a few years before we started the quarterly releases. Back then and during the early 2000s there was a significant amount of work going into the pkgsrc infrastructure to give it all the amazing features that we have today, but that often meant the development branch had some rough edges while those features were still being integrated across the tree.

The quarterly releases gave users confidence in building from a stable release without unexpected breakages, and also helped developers to schedule any large changes at the appropriate time.

At Joyent we make heavy use of the quarterly releases, producing new SmartOS images for each branch, so for example our 16.1.x images are based on pkgsrc-2016Q1, and so on.

Reaching the 50th release makes me feel old! It also makes me feel proud that we've come a long way, yet still have people who want to be involved and continue to develop both the infrastructure and packages.

I'd also like to highlight the fantastic work of the pkgsrc releng team, who work to ensure the releases are kept up-to-date until the next one is released. They do a great job and we benefit a lot from their work.

What are the main benefits of the pkgsrc system?

For me the big one is portability. This is what sets it apart from all other package managers (and, increasingly, software in general), not just because it runs on so many platforms but because it is such a core part of the infrastructure and has been constantly developed and refined over the years. We are now up to 23 distinct platforms, not counting different distributions, and adding support for new ones is relatively easy thanks to the huge amount of work which has gone into the infrastructure.

The other main benefit for me is the buildlink infrastructure and various quality checks we have. As someone who distributes binary packages to end users, it is imperative that those packages work as expected on the target environment and don't have any embarrassing bugs. The buildlink system ensures (amongst other things) that dependencies are correct and avoids many issues around build host pollution. We then have a number of QA scripts which analyse the generated package and ensure that the contents are accurate, RPATHs are correct, etc. It's not perfect and there are more tests we could write, but these catch a lot of mistakes that would otherwise go undetected until a user submits a bug report.

Others for me are unprivileged support, signed packages, multi-version support, pbulk, and probably a lot of other things I've forgotten and take for granted!

Where and how do you use pkgsrc?

As mentioned above I work on pkgsrc for SmartOS. We are probably one of the biggest users of pkgsrc in the world, shipping over a million package downloads per year and rising to our users, not including those distributed as part of our images or delivered from mirrors. This is where I spend the majority of my time working on pkgsrc, and it is all performed remotely on a number of zones running in the Joyent Public Cloud. The packages we build are designed to run not just on SmartOS but across all illumos distributions, and so I also have an OmniOS virtual machine locally where I test new releases before announcing them.

As an OS X user, I also use pkgsrc on my MacBook. This is generally where I perform any final tests before committing changes to pkgsrc so that I'm reasonably confident they are correct, but I also install a bunch of different packages from pkgsrc (mutt, ffmpeg, nodejs, jekyll, pstree etc) for my day-to-day work. I also have a number of Mac build servers in my loft and at the Joyent offices in San Francisco where I produce the binary OS X packages we offer which are starting to become popular among users looking for an alternative to Homebrew or MacPorts.

Finally, I have a few Linux machines also running in the Joyent Public Cloud which I have configured for continuous bulk builds of pkgsrc trunk. These help me to test any infrastructure changes I'm working on to ensure that they are portable and correct.

On all of these machines I have written infrastructure to perform builds inside chroots, ensuring a consistent environment and allowing me to work on multiple things simultaneously. They all have various tools installed (git, pkgvi, pkglint, etc) to aid my personal development workflow. We then make heavy use of GitHub and Jenkins to manage automatic builds when pushing to various branches.

What are the pkgsrc projects you are currently working on?

One of my priorities over the past year has been on performance. We build a lot of packages (over 40,000 per branch, and we support up to 4 branches simultaneously), and when the latest OpenSSL vulnerability hits it's critical to get a fix out to users as quickly as possible. We're now at the stage where, with a couple of patches, we can complete a full bulk build in under 3 hours. There is still a lot of room for improvement though, so recently I've been looking at slibtool (a libtool replacement written in C) and supporting dash (a minimal POSIX shell which is faster than bash).

There are also a few features we've developed at Joyent that I continue to maintain, such as our multiarch work (which combines 32-bit and 64-bit builds into a single package), additional multi-version support for MySQL and Percona, SMF support, and a bunch of other patches which aren't yet ready to be integrated.

I'm also very keen on getting new users into pkgsrc and turning them into developers, so a lot of my time has been spent on making pkgsrc more accessible, whether that's via our pkgbuild image (which gives users a ready-made pkgsrc development environment) or the developer guides I've written, or maintaining our https://pkgsrc.joyent.com/ website. There's lots more to do in this area though to ensure users of all abilities can contribute meaningfully.

Most of my day-to-day work though is general bug fixing and updating packages, performing the quarterly release builds, and maintaining our build infrastructure.

If you analyze the current state of pkgsrc, which improvements and changes do you wish for the future?

More users and developers! I am one of only a small handful of people who are paid to work on pkgsrc, the vast majority of the work is done by our amazing volunteer community. By its very nature pkgsrc requires constant effort updating existing packages and adding new ones. This is something that will never change and if anything the demand is accelerating, so we need to ensure that we continue to train up and add new developers if we are to keep up.

We need more documentation, more HOWTO guides, simpler infrastructure, easier patch submission, faster and less onerous on-boarding of developers, more bulk builds, more development machines. Plenty to be getting on with!

Some technical changes I'd like to see are better upgrade support, launchd support, integration of a working alternative pkg backend e.g. IPS, bmake IPC (so we don't need to recompute the same variables over and over), and many more!

Do you have any practical tips to share with the pkgsrc users?

Separate your build and install environments, so e.g. build in chroots or in a VM then deploy the built packages to your target. Trying to update a live install is the source of many problems, and there are few things more frustrating than having your development environment be messed up by an upgrade which fails part-way through.

For brand new users, document your experience and tell us what works and what sucks. Many of us have been using pkgsrc for many many years, and have lost your unique ability to identify problems, inconsistencies, and bad documentation.

If you run into problems, connect to Freenode IRC #pkgsrc, and we'll try to help you out. Hang out there even if you aren't having problems!

Finally, if you like pkgsrc, tell your friends, write blog posts, post to Hacker News etc. It's amazing to me how unknown pkgsrc is despite being around for so long, and how many people love it when they discover it.

More users leads to more developers, which leads to improved pkgsrc, which leads to more users, which...

What's the best way to start contributing to pkgsrc and what needs to be done?

Pick something that interests you and just start working on it. The great thing about pkgsrc is that there are open tasks for any ability, from documentation fixes all the way through adding packages to rewriting large parts of the infrastructure.

When you have something to contribute, don't worry about whether it's perfect or how you are to deliver it. Just make it available and let us know via PR, pull request, or just mail, and we can take it from there.

Do you plan to participate in the upcoming pkgsrcCon 2016 in Kraków (1-3 July)?

I am hoping to. If so I usually give a talk on what we've been working on at Joyent over the past year, and will probably do the same.

Posted terribly early Tuesday morning, June 7th, 2016 Tags: blog

The pkgsrc team has prepared the 50th release of their package management system, with the 2016Q1 version. It's infrequent event, as the 100th release will be held after 50 quarters.

The NetBSD team has prepared series of interviews with the authors. The next one is with Benny Siegert, a developer active in the release engineering team.

Hi Benny, please introduce yourself.

I came to pkgsrc to my work on MirBSD. MirBSD only had a handful of developers, so there was not enough manpower to maintain our own ports tree -- not that we didn't try! In the end, we decided to support pkgsrc, and I joined the NetBSD project as a developer, in an amazingly quick and painless process.

My dayjob is as an SRE at Google; luckily, Google allows me to use my 20% time to work on pkgsrc. Working in this job has changed my perspective on computing. I try to apply some of the SRE principles (automate repetitive work, discipline in bug tracking, etc.) to my work in pkgsrc.

First of all, congratulations on the 50th release of pkgsrc! How do you feel about this anniversary?

Wow, 50 releases already! I find it remarkable how pkgsrc has continued on a stable growth trajectory all these years. And together, we have built one of the best and most advanced package collections.

What are the main benefits of the pkgsrc system?

pkgsrc runs on almost any platform that you are likely to use, from NetBSD, other BSDs, commercial Unixes, Linux and Mac OS. Whatever the platform, you have the same huge choice of up-to-date packages. You can install them with a single command. That's pretty compelling.

Where and how do you use pkgsrc?

These days, I mostly use pkgsrc on NetBSD and Mac OS X. On the Mac, pkgsrc may not be the most popular package collection but it still works amazingly well. (By the way, I applaud the team behind saveosx.org for making an effort to make pkgsrc more widely known among Mac users.)

What are the pkgsrc projects you are currently working on?

By accident, I ended up being the maintainer of the pkgsrc stable branch :) I am the one who handles most of the security updates to the stable release.

As a fan of the Go programming language (and a contributor to the project), I work on making software written in Go easy to use in pkgsrc. There is infrastructure (go-package.mk) for packaging Go software easily.

If you analyze the current state of pkgsrc, which improvements and changes do you wish for the future?

I would love to have more modern tooling. Gnats for bugs and CVS for the repository are both outdated. But this is an ongoing discussion.

I would also like to have a more rigorous handling of security fixes. The vulnerability DB is great and kept very well; on the other hand actually fixing the vulnerabilities is sometimes neglected, particularly for packages that not many people use.

Do you have any practical tips to share with the pkgsrc users?

- If you are on a machine where you do not have root access (such as a shared Linux machine), you can bootstrap pkgsrc in unprivileged mode. This way, everything builds and installs without needing to use root rights.

- Read up on "pkg_admin audit" and use it regularly, to find when you have packages with security problems installed.

What's the best way to start contributing to pkgsrc and what needs to be done?

pkgsrc-wip has a really low barrier to entry. Try to make your own package for something simple and put it in wip.

Look in pkgsrc/doc/TODO, it contains some suggestions for things you may want to work on. There is also a long list of suggested package updates in there, you can send a PR with patch for these.

Finally, if you run "pkg_admin audit", as I suggested above, and discover that pkgsrc does not contain a fix for a given vulnerability, you can try to find a patch and submit it via PR. I would be more than happy to apply it :)

Do you plan to participate in the upcoming pkgsrcCon 2016 in Kraków (1-3 July)?

pkgsrcCon is a fantastic conference. I am not 100% sure yet if I can make it but I will try to.

Posted terribly early Monday morning, June 6th, 2016 Tags: blog

The pkgsrc team has prepared the 50th release of their package management system, with the 2016Q1 version. It's infrequent event, as the 100th release will be held after 50 quarters.

The NetBSD team has prepared series of interviews with the authors. The 3rd one is with Thomas Klausner, a developer well known for his maintainership of the pkgsrc-wip project.

First of all, congratulations on the 50th release of pkgsrc! How do you feel about this anniversary?

I'm very glad that pkgsrc that so many people find pkgsrc useful and like working on it, both on pkgsrc itself and pkgsrc-wip.

What are the main benefits of the pkgsrc system?

Get packages installed on your system, and keep them up-to-date, and don't worry about the underlying operating system.

Where and how do you use pkgsrc?

I use pkgsrc on my desktop machine at home and on various servers.

What are the pkgsrc projects you are currently working on?

Currently I have no single big project. I regularly try to keep a couple of hundred packages up-to-date, and to feed back patches upstream, so more software builds out-of-the-box. This also has the advantages of making updates easier, and spreading awareness of pkgsrc and NetBSD. Recently I've been more focusing on pushing upstream the pkgsrc patches for firefox.

If you analyze the current state of pkgsrc, which improvements and changes do you wish for the future?

A recent issue is that we need to add a framework for PaX security features; luckily, this is already implemented and just needs merging. Longer term I think our binary package tools could use some love and fresh code.

Do you have any practical tips to share with the pkgsrc users?

If you build your packages yourself, then use pkgtools/mksandbox to create sandboxes and run bulk builds inside them using pkgtools/pbulk. It makes life in general and updates in particular so much easier!

What's the best way to start contributing to pkgsrc and what needs to be done?

The easiest way is to start using pkgsrc on your own machines. Once you feel comfortable with that you'll probably notice that you want to update a package, or add a new one. If you reach that point, visit http://pkgsrc.org/wip/ to get commit access to the pkgsrc wip repository.

If you need help, contact the pkgsrc-users mailing list or visit us in #pkgsrc on freenode!

Do you plan to participate in the upcoming pkgsrcCon 2016 in Kraków (1-3 July)?

Definitely! I'm looking forward to meeting other pkgsrc developers again or for the first time, and to many interesting talks.

Cheers,
Thomas

Posted terribly early Thursday morning, June 2nd, 2016 Tags: blog

The pkgsrc team has prepared the 50th release of their package management system, with the 2016Q1 version. It's infrequent event, as the 100th release will be held after 50 quarters.

The NetBSD team has prepared series of interviews with the authors. The 2nd one is with Sevan Janiyan, a developer well known for his bulk builds for several platforms.

Hi Sevan, please introduce yourself.

Hello,

I'm Sevan Janiyan. A sysadmin from England and a NetBSD developer, working on the pkgsrc packaging system. My areas of focus are pkgsrc security and Darwin (PowerPC) but as I enjoy running many operating systems I manage builds on a variety of them across different CPU architectures. I was a user of pkgsrc for many years but only started working on pkgsrc itself early 2014 when I obtained a PowerBook from a friend and started fixing issues on OS X Tiger/PowerPC. I was invited to become a member of TNF in 2015.

First of all, congratulations on the 50th release of pkgsrc! How do you feel about this anniversary?

It's fantastic to see a tool that has evolved over time, it's extremely flexible and has grown support for numerous operating systems during this period. Not only can it serve as the native packaging system for most OS' but thanks to its multi platform support, it can be used to save a lot of effort in implementing a packaging system on a project when a set of packages need to be built and deployed in a consistent manner across multiple operating systems.

What are the main benefits of the pkgsrc system?

The ability the provide a single API for dealing with multiple environments & toolchains, for example translating a setting to the relevant flags to compliment the compiler in use.

Unprivileged mode allows a user to build the tools they require in a location which executables are permitted and the user has write access e.g your home directory when the partition it resides on is not mounted noexec ;-)

The buildlink framework - it provides the ability to detect components available on the host operating system & allows a user the choice whether to build against such a component or to opt for a version provided by pkgsrc itself.

Where and how do you use pkgsrc?

pkgsrc is an essential part of my sysadmin tool belt, on one hand I rely on it to obtain a set of packages on a system without touching the systems packaging system. This is for example on a customer system where I either have not been given root access or I do not wish to install a package on a system with a big dependency list for my personal use.

The packages in pkgsrc generally see very little in terms of local changes, besides tweaks to ensure package integrates into the system. This encourages interaction from developers with projects to upstream changes. This can be a benefit when debugging software issues where it is not certain if the issue exists in the version of software package from OS vendor or there is a bug in the software project upstream.

The ability to bootstrap multiple instances of the packaging system under different prefixes permits a user to install multiple and conflicting versions of software in isolated locations on a system.

As a use case, I worked on a project troubleshooting a clients varnish instance. The issues they were experiencing was specific to the version packaged for their OS by vendor. This was varnish 3.x and 4.1.0 had just been released, we decided to evaluate varnish 4.1.0 but as there were changes to the configuration language some development would need to by carried out, to adapt things to the new syntax. To reduce downtime the instance of varnish installed using the native package manager was left untouched and continued running, pkgsrc was bootstrapped in a separate location and varnish was installed from there. The development work to bring the config up to date happened with the new version of varnish from pkgsrc listening on a different port, but running alongside the original 3.x instance. Switching between the two instances was just a matter of changing ports to forward traffic to on the front-end web servers. Unfortunately 4.1.0 release had some bugs, so we considered trying 4.0.x. Another instance of pkgsrc was bootstrapped & v4.0.x was installed, again running on another port. This instance was brought up alongside the other two instance and started receiving traffic, at this point it was trivial to evaluate behaviour across 3 different versions of a piece of software running on a single host.

What are the pkgsrc projects you are currently working on?

I've worked very little in the tree recently. I continue to run the bulk builds of pkgsrc-current on a variety of systems and have recently begun making the packages generated from some of these builds for people to be able evaluate pkgsrc. At present there are packages for OS X Tiger (PowerPC), Debian Linux (amd64 & armv7), FreeBSD (amd64) and OmniOS published on https://files.venture37.com/pkgsrc/packages

Debian/armv7
http://mail-index.netbsd.org/pkgsrc-users/2016/03/22/msg023192.html

Debian/amd64
http://mail-index.netbsd.org/pkgsrc-users/2016/03/26/msg023209.html

FreeBSD/amd64
http://mail-index.netbsd.org/pkgsrc-users/2016/03/29/msg023222.html

If you analyze the current state of pkgsrc, which improvements and changes do you wish for the future?

A better mechanism for running the bulkbuilds or to grok how to setup the parallel builds.

Do you have any practical tips to share with the pkgsrc users?

As I mentioned with the varnish example, you can save a considerable amount of effort when required to run multiple instances of conflicting components by using pkgsrc. As everything isolated to a given prefix, specified during bootstrap, it's trivial to run multiple versions of Ruby for example without any conflict.

You can leverage this to experiment with changes which could be deemed volatile using other means.

What's the best way to start contributing to pkgsrc and what needs to be done?

Pick an OS from list, try to bootstrap pkgsrc on it, try to install some packages. If the process failed at any stage, file a bug report even better if the report includes a patch.

Rinse, repeat :-)

If you have resources to attempt larger builds, follow the steps to setup a bulk build environment and run a build. The results ideally should be published on a public webserver & the report posted to the pkgsrc-bulk list, developers and maintainers use these reports to see problems and it could serve as a starting point for a potential contributor as a list of things that need to be fixed.

Bulktracker also makes use of these reports to build a picture of the status of packages and their impact across multiple platforms.

Do you plan to participate in the upcoming pkgsrcCon 2016 in Kraków (1-3 July)?

Yes, see you there

Sevan

Posted terribly early Wednesday morning, June 1st, 2016 Tags: blog

The pkgsrc team has prepared the 50th release of their package management system, with the 2016Q1 version. It's infrequent event, as the 100th release will be held after 50 quarters.

The NetBSD team has prepared series of interviews with the authors. We started with Joerg Sonnenberger, a developer well known for his LLVM on NetBSD contribution.

Hi Joerg, please introduce yourself.

Hello Kamil,

I'm a software developer with a background in mathematics. Since I am doing Open Source development primarily for fun, my interests are changing over time and cover many different topics.

First of all, congratulations on the 50th release of pkgsrc! How do you feel about this anniversary?

The pkgsrc quarterly branches were one of the reasons for me to start looking at pkgsrc -- a bit more than a decade ago. As such, my 40th branch has passed recently as well, so it is a bit personal as well. I'm looking back proudly to what we have achieved, but also a bit sad to some of the faces and names that disappeared from the community over that time.

What are the main benefits of the pkgsrc system?

For me, the biggest advantage of pkgsrc is to provide me with the freedom to have a consistent environment across different platforms. Having to deal with BSD, Solaris and Linux flavours, it is very important to be able to sit down and get a consistent set of tools, consistent patches for programs to depend on etc.

Where and how do you use pkgsrc?

I'm using it on my own NetBSD and SmartOS systems as well as for providing software images for my customers at work.

What are the pkgsrc projects you are currently working on?

After that time focusing on bug hunting for the clang builds, I've recently returned to working on pbulk again. There are a number of areas where the distributed build can be improved. My main goal at the moment is to reduce the required coupling of master and slave. Sharing the distfiles is more challenging than it seems on first sight. NFS or null mounts are one option, but often, different packages sharing the same tarball are built in parallel. On an update, they will try to fetch the same file concurrently and often fail badly. The receipt I ended up using for the NetBSD 7.0 bulk builds is a low-memory virtual machine with nginx distribution the distfiles to all build slaves via HTTP(S). If a new file is obtained from elsewhere, it will be uploaded back to my local mirror via PUT. This has been quite reliable so far, but I still have to document the various steps involved for others. The new part will be better handling of the binary packages themselve. Using something like PUT again makes it easier to provide signatures without exposing the key material to all builds.

If you analyze the current state of pkgsrc, which improvements and changes do you wish for the future?

The biggest new term wish is that Leonardo's Summer of Code project will be successful and we finally get a proper infrastructure for building multiple packages in one step. The removal of non-DESTDIR support was kind of a show stopper for doing this sanely and that one is finally done. It didn't even take 10 years to reach that point From an infrastructural point of view, I want to tackle the shell fragment library topic again soon and also work on improving the build performance. The cwrapper support has been pending for a while, someone (else) has to sit down and fix the USE_CROSSBASE handling. I think that's the only real fallout we have left in the tree for making it the default, with all the performance boosts it can provide. More resources for builds is a general wish, but that also often requires porting to new hardware and the like.

Do you have any practical tips to share with the pkgsrc users?

The only practical tip is to aggressively use virtual machines or chroots for building whenever you can't just use binary packages. I.e. if you can't use pre-made binary packages, make your own.

What's the best way to start contributing to pkgsrc and what needs to be done?

Adopt a package, keep it updated and care for it and its environment. Everything tends to start from that. While pkglint is a handy tool for improving your own contributions, especially in pkgsrc-wip, it often helps even more to chat with a regular contributor whenever more challenging problems happen. Unlike many Linux packaging system, we try very hard to make simple things easy and compact. Many problems have different solutions and I sometimes look back to my own changes from years ago and find better solutions now. It is hard to put this kind of experience into documentation. None had the resources for training Watson yet.

Do you plan to participate in the upcoming pkgsrcCon 2016 in Kraków (1-3 July)?

I plan to come, I just don't know if I have time to prepare a presentation for it

Joerg

Posted terribly early Tuesday morning, May 31st, 2016 Tags: blog

The NetBSD Project is pleased to announce NetBSD 7.0.1, the first security/bugfix update of the NetBSD 7.0 release branch. It represents a selected subset of fixes deemed important for security or stability reasons. If you are running an earlier release of NetBSD, we strongly suggest updating to 7.0.1.

For more details, please see the release notes.

Complete source and binaries for NetBSD are available for download at many sites around the world. A list of download sites providing FTP, AnonCVS, SUP, and other services may be found at http://www.NetBSD.org/mirrors/.

Posted early Saturday morning, May 28th, 2016 Tags: blog

Recently on a few mailing lists a discussion about creating a simple turbo-channel USB adapter came up. I have an old Alpha machine with turbo channel and no on-board USB, so considered ordering one - but wanted to know upfront whether that alpha machine still worked.

I started with updating the installation on the alpha to -current, and ran out of disk space!

No big deal, it still had the original ~1GB disk (RZ26) in it, and a full NetBSD install nowadays needs slightly more. I have plenty of unused SCSI disks lying around, but when looking for 50pin ones, I found I have none left.

Another machine I was looking at for a different root disk solution was an old mac68k (Quadra 640 AV). I had replaced the disk it came with by an 8GB one, but it draws too much power for the internal PSU, so I had to move it to an external SCSI enclosure, which is inconvenient.

Additionally both disks made quite some noise and probably would not last a lot longer. So I started looking for "cheap", silent and low-power alternatives.

I found the SCSI2SD project, which looked promising and had a firmware update earlier this year. I ordered a few boards from idead studio and started testing.

The solution is mostly plug & play, a configuration utility allows updating the firmware and configuring the devices.

I did several bonnie++ runs to get an idea of the speed achievable. All machines are not fast, and disk access has always been slow. To get an overall idea, I also tested on a faster alpha (still with slow scsi disks). Then I moved both test machines over to "root on sd card" and tested again. The results look like this:

Test Sequential Output Sequential Input Random Seeks Sequential Create Random Create
Per Chr Block Rewrite Per Chr Block K/s CPU Create Read Delete Create Read Delete
K/s CPU K/s CPU K/s CPU K/s CPU K/s CPU /s CPU /s CPU /s CPU /s CPU /s CPU /s CPU
Ref: DS20 3436 35 3460 6 2935 5 9565 94 19494 285.7 16 3 878 81 31793 99 2939 54 940 84 1351 96 203 10
Alpha HD 906 92 2146 41 980 24 947 93 2178 23 78.5 14 36 95 926 72 290 90 37 96 39 97 25 34
Alpha SD 548 77 897 19 450 9 490 97 1140 9 24.9 4 85 89 1245 75 437 84 90 93 119 96 29 16
Mac HD 98 95 737 89 373 91 94 96 647 89 14.1 47 22 93 502 75 122 81 23 94 31 97 14 43
Mac SD 95 95 646 89 339 92 93 96 636 79 14.5 75 22 90 473 75 124 85 23 92 32 97 13 46

So even for machines with very slow hard disk access, there is a (relatively small?) performance hit with the SCSI2SD solution. Whether it is too bad for your application, depends on various factors - all of the affected machines are very slow overall anyway.

Just a few days ago a new hardware revision of the SCSI2SD cards has been announced which is supposed to improve performance in the future (when the firmware has caught up), so it may be worth to wait (or get the newer hardware) if performance is important.

I have a VAX machine that could go with this solution as well, but results are expected to be similar (or worse) than for the slow alpha, as my vax is pretty fast (for a vax) and has a decent SCSI subsystem.

For reference, below are the full dmesg from the fast reference and both test machines after switching to SD root, and the sniplet for the original hard disk

AlphaServer DS20

"Fast" reference machine
file system: FFSv2, log

Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016
The NetBSD Foundation, Inc. All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.

NetBSD 7.99.29 (GENERIC-$Revision: 1.368 $) #7: Mon May 23 20:44:04 CEST 2016
martin@martins.aprisoft.de:/ssd/src/sys/arch/alpha/compile/GENERIC
AlphaServer DS20 500 MHz, s/n AY94910150
8192 byte page size, 1 processor.
total memory = 1024 MB
(2712 KB reserved for PROM, 1021 MB used by NetBSD)
avail memory = 994 MB
timecounter: Timecounters tick every 0.976 msec
Kernelized RAIDframe activated
mainbus0 (root)
cpu0 at mainbus0: ID 0 (primary), 21264-4
cpu0: Architecture extensions: 0x303
tsc0 at mainbus0: 21272 Core Logic Chipset, Cchip rev 0
tsc0: 8 Dchips, 2 memory buses of 32 bytes
tsc0: arrays present: 512MB, 512MB, 0MB, 0MB, Dchip 0 rev 1
tsp0 at tsc0
pci0 at tsp0 bus 0
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
sio0 at pci0 dev 5 function 0: vendor 1080 product c693 (rev. 0x00)
cypide0 at pci0 dev 5 function 1: Cypress 82C693 IDE Controller (rev. 0x00)
cypide0: bus-master DMA support present, but unused (registers at unsafe address 0x10000)
cypide0: primary channel wired to compatibility mode
cypide0: primary channel interrupting at isa irq 14
atabus0 at cypide0 channel 0
cypide1 at pci0 dev 5 function 2: Cypress 82C693 IDE Controller (rev. 0x00)
cypide1: hardware does not support DMA
cypide1: primary channel wired to compatibility mode
cypide1: secondary channel interrupting at isa irq 15
atabus1 at cypide1 channel 0
ohci0 at pci0 dev 5 function 3: vendor 1080 product c693 (rev. 0x00)
ohci0: interrupting at isa irq 10
ohci0: OHCI version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
isa0 at sio0
lpt0 at isa0 port 0x3bc-0x3bf irq 7
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com0: console
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
pckbc0 at isa0 port 0x60-0x64
attimer0 at isa0 port 0x40-0x43
pcppi0 at isa0 port 0x61
midi0 at pcppi0: PC speaker
spkr0 at pcppi0
isabeep0 at pcppi0
fdc0 at isa0 port 0x3f0-0x3f7 irq 6 drq 2
mcclock0 at isa0 port 0x70-0x71: mc146818 compatible time-of-day clock
attimer0: attached to pcppi0
tsp1 at tsc0
pci1 at tsp1 bus 0
pci1: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
tlp0 at pci1 dev 7 function 0: Macronix MX98713 Ethernet, pass 0.0
tlp0: interrupting at dec 6600 irq 47
tlp0: Ethernet address 00:40:05:50:ee:9b
tlp0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX
isp0 at pci1 dev 8 function 0: QLogic 1020 Fast Wide SCSI HBA
isp0: interrupting at dec 6600 irq 43
isp1 at pci1 dev 9 function 0: QLogic 1020 Fast Wide SCSI HBA
isp1: interrupting at dec 6600 irq 39
tsciic0 at tsc0
iic0 at tsciic0: I2C bus
timecounter: Timecounter "clockinterrupt" frequency 1024 Hz quality 0
timecounter: Timecounter "PCC" frequency 499882560 Hz quality 1000
scsibus0 at isp0: 16 targets, 8 luns per target
scsibus1 at isp1: 16 targets, 8 luns per target
scsibus0: waiting 2 seconds for devices to settle...
scsibus1: waiting 2 seconds for devices to settle...
uhub0 at usb0: vendor 1080 OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
sd0 at scsibus0 target 0 lun 0: <COMPAQ, BB00911CA0, 3B05> disk fixed
sd0: 8678 MB, 5273 cyl, 20 head, 168 sec, 512 bytes/sect x 17773524 sectors
sd0: sync (50.00ns offset 8), 16-bit (40.000MB/s) transfers, tagged queueing
sd1 at scsibus0 target 1 lun 0: <COMPAQ, BB00912301, B016> disk fixed
sd1: 8678 MB, 5273 cyl, 20 head, 168 sec, 512 bytes/sect x 17773524 sectors
sd1: sync (50.00ns offset 8), 16-bit (40.000MB/s) transfers, tagged queueing
sd2 at scsibus0 target 2 lun 0: <COMPAQ, BB00911CA0, 3B05> disk fixed
sd2: 8678 MB, 5273 cyl, 20 head, 168 sec, 512 bytes/sect x 17773524 sectors
sd2: sync (50.00ns offset 8), 16-bit (40.000MB/s) transfers, tagged queueing
fd0 at fdc0 drive 0: 1.44MB, 80 cyl, 2 head, 18 sec
sd3 at scsibus0 target 3 lun 0: <COMPAQ, BB00912301, B016> disk fixed
sd3: 8678 MB, 5273 cyl, 20 head, 168 sec, 512 bytes/sect x 17773524 sectors
sd3: sync (50.00ns offset 8), 16-bit (40.000MB/s) transfers, tagged queueing
cd0 at scsibus1 target 5 lun 0: <DEC, RRD47 (C) DEC, 1206> cdrom removable
raid0: RAID Level 1
raid0: Components: /dev/sd0a /dev/sd1a
raid0: Total Sectors: 17773440 (8678 MB)
raid1: RAID Level 1
raid1: Components: /dev/sd2a /dev/sd3a
raid1: Total Sectors: 17773440 (8678 MB)
root on raid0a dumps on raid0b
root file system type: ffs
kern.module.path=/stand/alpha/7.99.29/modules

Alpha DEC 3000

with SCSI2SD

Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016
The NetBSD Foundation, Inc. All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.

NetBSD 7.99.29 (GENERIC-$Revision: 1.368 $) #3: Thu May 5 09:17:11 CEST 2016
martin@martins.aprisoft.de:/ssd/src/sys/arch/alpha/compile/GENERIC
DEC 3000 - M300, 150MHz, s/n
8192 byte page size, 1 processor.
total memory = 98304 KB
(2048 KB reserved for PROM, 96256 KB used by NetBSD)
avail memory = 82352 KB
timecounter: Timecounters tick every 0.976 msec
Kernelized RAIDframe activated
mainbus0 (root)
cpu0 at mainbus0: ID 0 (primary), 21064-1
tcasic0 at mainbus0
tc0 at tcasic0: 12.5 MHz clock
sfb0 at tc0 slot 6 offset 0x2000000: 1280x1024, 8bpp
wsdisplay1 at sfb0 kbdmux 1
wsmux1: connecting to wsdisplay1
ioasic0 at tc0 slot 5 offset 0x0: slow mode
le0 at ioasic0 offset 0xc0000: address 08:00:2b:3c:93:27
le0: 32 receive buffers, 8 transmit buffers
zsc0 at ioasic0 offset 0x100000
vsms0 at zsc0 channel 0
wsmouse0 at vsms0 mux 0
zstty0 at zsc0 channel 1 (console i/o)
zsc1 at ioasic0 offset 0x180000
lkkbd0 at zsc1 channel 0
wskbd0 at lkkbd0 mux 1
wskbd0: connecting to wsdisplay1
zsc1: channel 1 not configured
mcclock0 at ioasic0 offset 0x200000: mc146818 compatible time-of-day clock
bba0 at ioasic0 offset 0x240000
audio0 at bba0: full duplex, playback, capture, mmap
tcds0 at tc0 slot 4 offset 0x0: TurboChannel Dual SCSI (baseboard)
asc0 at tcds0 chip 0: NCR53C94, 25MHz, SCSI ID 0
scsibus0 at asc0: 8 targets, 8 luns per target
timecounter: Timecounter "clockinterrupt" frequency 1024 Hz quality 0
timecounter: Timecounter "PCC" frequency 150006528 Hz quality 1000
lkkbd0: no keyboard
scsibus0: waiting 2 seconds for devices to settle...
sd0 at scsibus0 target 1 lun 0: <codesrc, SCSI2SD, 1.0> disk fixed
sd0: 3712 MB, 473 cyl, 255 head, 63 sec, 512 bytes/sect x 7603200 sectors
sd0: async, 8-bit transfers
root on sd0a dumps on sd0b
root file system type: ffs
kern.module.path=/stand/alpha/7.99.29/modules

with original RZ26

sd0 at scsibus0 target 3 lun 0: disk fixed
sd0: 1001 MB, 2570 cyl, 14 head, 57 sec, 512 bytes/sect x 2050860 sectors
sd0: sync (200.00ns offset 15), 8-bit (5.000MB/s) transfers, tagged queueing

Mac Quadra 640AV

File system: FFSv1, log

with SCSI2SD

Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016
The NetBSD Foundation, Inc. All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.

NetBSD 7.99.29 (MAC-BETH) #18: Tue May 10 05:40:16 CEST 2016
martin@night-owl.duskware.de:/usr/src/sys/arch/mac68k/compile/MAC-BETH
Apple Macintosh Centris 660AV (68040)
cpu: delay factor 800
fpu: mc68040
total memory = 45056 KB
avail memory = 40248 KB
timecounter: Timecounters tick every 16.666 msec
mrg: 'Quadra AV ROMs' ROM glue, tracing off, debug off, silent traps
mainbus0 (root)
obio0 at mainbus0
esp0 at obio0 addr 0: address 0x2da000: NCR53C96, 25MHz, SCSI ID 7
scsibus0 at esp0: 8 targets, 8 luns per target
adb0 at obio0
intvid0 at obio0 @ 50100600: CIVIC video subsystem
intvid0: 640 x 480, 65536 color
macfb0 at intvid0
wsdisplay0 at macfb0 (kbdmux ignored)
mc0 at obio0: address 08:00:07:46:74:0a
zsc0 at obio0 chip type 0
zsc0 channel 0: d_speed 9600 DCD clk 0 CTS clk 0
zstty0 at zsc0 channel 0 (console i/o)
zsc0 channel 1: d_speed 9600 DCD clk 0 CTS clk 0
zstty1 at zsc0 channel 1
nubus0 at mainbus0
timecounter: Timecounter "clockinterrupt" frequency 60 Hz quality 0
timecounter: Timecounter "VIA1 T2" frequency 783360 Hz quality 100
scsibus0: waiting 2 seconds for devices to settle...
adb0 (direct, Cuda): 2 targets
aed0 at adb0 addr 0: ADB Event device
akbd0 at adb0 addr 2: keyboard II (ISO layout)
wskbd0 at akbd0 (mux ignored)
ams0 at adb0 addr 3: 1-button, 100 dpi mouse
wsmouse0 at ams0 (mux ignored)
sd0 at scsibus0 target 0 lun 0: <codesrc, SCSI2SD, 4.6> disk fixed
sd0: 15104 MB, 1925 cyl, 255 head, 63 sec, 512 bytes/sect x 30934016 sectors
sd0: async, 8-bit transfers
cd0 at scsibus0 target 3 lun 0: <SONY, CD-ROM CDU-8003A, 1.9a> cdrom removable
cd0: async, 8-bit transfers
boot device: sd0
root on sd0a dumps on sd0b
root file system type: ffs
kern.module.path=/stand/mac68k/7.99.29/modules

with hard disk

sd0 at scsibus0 target 0 lun 0: disk fixed
sd0: 8761 MB, 14384 cyl, 3 head, 415 sec, 512 bytes/sect x 17942584 sectors
sd0: async, 8-bit transfers, tagged queueing
Posted late Friday afternoon, May 27th, 2016 Tags: blog

I was asked to prepare some stats on pkgsrc's growth, as part of the celebrations of pkgsrc-2016Q1, pkgsrc's fiftieth quarterly release.

This mail from 2004 says:

By my calculations, at the end of December 2003, there were 4310 packages in the NetBSD Packages Collection, up from 4170 the previous month, a rise of only 140. We also tagged a new branch for pkgsrc, which is being actively maintained - the branch is called "pkgsrc-2003Q4". As the name implies, we will be branching pkgsrc on a regular basis from now on, and maintaining the branch.

We had previously cut pkgsrc releases to coincide with NetBSD releases, and for some other events, but pkgsrc-2003Q4 can be seen as the first branch where pkgsrc was really its own entity. It can be checked out with the "pkgsrc-2003Q4" tag, and browsed at:

We keep a complete log of changes we have made to pkgsrc since its inception. At the lowest level, pkgsrc-2003Q4 had 3059 pkgsrc entries, whilst 2016Q1 had 14311 pkgsrc entries. Due to different versions of some languages, like python, perl, ruby and php, the number of packages is a much higher number. For instance, 2003Q4 had 4310 packages, and 2016Q1 had 17154. We have also provided a histogram of pkgsrc activity from its inception until now.

pkgsrc-adds/updates/deletes

To see how pkgsrc has spread across platforms, 2003Q4 supported 7 platforms, 2016Q1 supports 23 platforms.

In addition, pkgsrc has now grown its own conference. The first pkgsrccon was in Vienna in 2004 -- see the original 2004 pkgsrccon information and this year's pkgsrccon is in Krakow -- see the 2016 pkgsrccon information We also produced a map of pkgsrccon venues for those interested in the history and geography of pkgsrccon.

pkgsrccon-sites

Alistair Crooks
for the pkgsrc team
Sun May 8 11:26:15 PDT 2016

Posted late Monday evening, May 9th, 2016 Tags: blog
Add a comment
Contact | Disclaimer | Copyright © 1994-2016 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
NetBSD® is a registered trademark of The NetBSD Foundation, Inc.