I was asked to prepare some stats on pkgsrc's growth, as part of the celebrations of pkgsrc-2016Q1, pkgsrc's fiftieth quarterly release.

This mail from 2004 says:

By my calculations, at the end of December 2003, there were 4310 packages in the NetBSD Packages Collection, up from 4170 the previous month, a rise of only 140. We also tagged a new branch for pkgsrc, which is being actively maintained - the branch is called "pkgsrc-2003Q4". As the name implies, we will be branching pkgsrc on a regular basis from now on, and maintaining the branch.

We had previously cut pkgsrc releases to coincide with NetBSD releases, and for some other events, but pkgsrc-2003Q4 can be seen as the first branch where pkgsrc was really its own entity. It can be checked out with the "pkgsrc-2003Q4" tag, and browsed at:

We keep a complete log of changes we have made to pkgsrc since its inception. At the lowest level, pkgsrc-2003Q4 had 3059 pkgsrc entries, whilst 2016Q1 had 14311 pkgsrc entries. Due to different versions of some languages, like python, perl, ruby and php, the number of packages is a much higher number. For instance, 2003Q4 had 4310 packages, and 2016Q1 had 17154. We have also provided a histogram of pkgsrc activity from its inception until now.

pkgsrc-adds/updates/deletes

To see how pkgsrc has spread across platforms, 2003Q4 supported 7 platforms, 2016Q1 supports 23 platforms.

In addition, pkgsrc has now grown its own conference. The first pkgsrccon was in Vienna in 2004 -- see the original 2004 pkgsrccon information and this year's pkgsrccon is in Krakow -- see the 2016 pkgsrccon information We also produced a map of pkgsrccon venues for those interested in the history and geography of pkgsrccon.

pkgsrccon-sites

Alistair Crooks
for the pkgsrc team
Sun May 8 11:26:15 PDT 2016

Posted late Monday evening, May 9th, 2016 Tags:

Recently on a few mailing lists a discussion about creating a simple turbo-channel USB adapter came up. I have an old Alpha machine with turbo channel and no on-board USB, so considered ordering one - but wanted to know upfront whether that alpha machine still worked.

I started with updating the installation on the alpha to -current, and ran out of disk space!

No big deal, it still had the original ~1GB disk (RZ26) in it, and a full NetBSD install nowadays needs slightly more. I have plenty of unused SCSI disks lying around, but when looking for 50pin ones, I found I have none left.

Another machine I was looking at for a different root disk solution was an old mac68k (Quadra 640 AV). I had replaced the disk it came with by an 8GB one, but it draws too much power for the internal PSU, so I had to move it to an external SCSI enclosure, which is inconvenient.

Additionally both disks made quite some noise and probably would not last a lot longer. So I started looking for "cheap", silent and low-power alternatives.

I found the SCSI2SD project, which looked promising and had a firmware update earlier this year. I ordered a few boards from idead studio and started testing.

The solution is mostly plug & play, a configuration utility allows updating the firmware and configuring the devices.

I did several bonnie++ runs to get an idea of the speed achievable. All machines are not fast, and disk access has always been slow. To get an overall idea, I also tested on a faster alpha (still with slow scsi disks). Then I moved both test machines over to "root on sd card" and tested again. The results look like this:

Test Sequential Output Sequential Input Random Seeks Sequential Create Random Create
Per Chr Block Rewrite Per Chr Block K/s CPU Create Read Delete Create Read Delete
K/s CPU K/s CPU K/s CPU K/s CPU K/s CPU /s CPU /s CPU /s CPU /s CPU /s CPU /s CPU
Ref: DS20 3436 35 3460 6 2935 5 9565 94 19494 285.7 16 3 878 81 31793 99 2939 54 940 84 1351 96 203 10
Alpha HD 906 92 2146 41 980 24 947 93 2178 23 78.5 14 36 95 926 72 290 90 37 96 39 97 25 34
Alpha SD 548 77 897 19 450 9 490 97 1140 9 24.9 4 85 89 1245 75 437 84 90 93 119 96 29 16
Mac HD 98 95 737 89 373 91 94 96 647 89 14.1 47 22 93 502 75 122 81 23 94 31 97 14 43
Mac SD 95 95 646 89 339 92 93 96 636 79 14.5 75 22 90 473 75 124 85 23 92 32 97 13 46

So even for machines with very slow hard disk access, there is a (relatively small?) performance hit with the SCSI2SD solution. Whether it is too bad for your application, depends on various factors - all of the affected machines are very slow overall anyway.

Just a few days ago a new hardware revision of the SCSI2SD cards has been announced which is supposed to improve performance in the future (when the firmware has caught up), so it may be worth to wait (or get the newer hardware) if performance is important.

I have a VAX machine that could go with this solution as well, but results are expected to be similar (or worse) than for the slow alpha, as my vax is pretty fast (for a vax) and has a decent SCSI subsystem.

For reference, below are the full dmesg from the fast reference and both test machines after switching to SD root, and the sniplet for the original hard disk

AlphaServer DS20

"Fast" reference machine
file system: FFSv2, log

Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016
The NetBSD Foundation, Inc. All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.

NetBSD 7.99.29 (GENERIC-$Revision: 1.368 $) #7: Mon May 23 20:44:04 CEST 2016
martin@martins.aprisoft.de:/ssd/src/sys/arch/alpha/compile/GENERIC
AlphaServer DS20 500 MHz, s/n AY94910150
8192 byte page size, 1 processor.
total memory = 1024 MB
(2712 KB reserved for PROM, 1021 MB used by NetBSD)
avail memory = 994 MB
timecounter: Timecounters tick every 0.976 msec
Kernelized RAIDframe activated
mainbus0 (root)
cpu0 at mainbus0: ID 0 (primary), 21264-4
cpu0: Architecture extensions: 0x303
tsc0 at mainbus0: 21272 Core Logic Chipset, Cchip rev 0
tsc0: 8 Dchips, 2 memory buses of 32 bytes
tsc0: arrays present: 512MB, 512MB, 0MB, 0MB, Dchip 0 rev 1
tsp0 at tsc0
pci0 at tsp0 bus 0
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
sio0 at pci0 dev 5 function 0: vendor 1080 product c693 (rev. 0x00)
cypide0 at pci0 dev 5 function 1: Cypress 82C693 IDE Controller (rev. 0x00)
cypide0: bus-master DMA support present, but unused (registers at unsafe address 0x10000)
cypide0: primary channel wired to compatibility mode
cypide0: primary channel interrupting at isa irq 14
atabus0 at cypide0 channel 0
cypide1 at pci0 dev 5 function 2: Cypress 82C693 IDE Controller (rev. 0x00)
cypide1: hardware does not support DMA
cypide1: primary channel wired to compatibility mode
cypide1: secondary channel interrupting at isa irq 15
atabus1 at cypide1 channel 0
ohci0 at pci0 dev 5 function 3: vendor 1080 product c693 (rev. 0x00)
ohci0: interrupting at isa irq 10
ohci0: OHCI version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
isa0 at sio0
lpt0 at isa0 port 0x3bc-0x3bf irq 7
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com0: console
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
pckbc0 at isa0 port 0x60-0x64
attimer0 at isa0 port 0x40-0x43
pcppi0 at isa0 port 0x61
midi0 at pcppi0: PC speaker
spkr0 at pcppi0
isabeep0 at pcppi0
fdc0 at isa0 port 0x3f0-0x3f7 irq 6 drq 2
mcclock0 at isa0 port 0x70-0x71: mc146818 compatible time-of-day clock
attimer0: attached to pcppi0
tsp1 at tsc0
pci1 at tsp1 bus 0
pci1: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
tlp0 at pci1 dev 7 function 0: Macronix MX98713 Ethernet, pass 0.0
tlp0: interrupting at dec 6600 irq 47
tlp0: Ethernet address 00:40:05:50:ee:9b
tlp0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX
isp0 at pci1 dev 8 function 0: QLogic 1020 Fast Wide SCSI HBA
isp0: interrupting at dec 6600 irq 43
isp1 at pci1 dev 9 function 0: QLogic 1020 Fast Wide SCSI HBA
isp1: interrupting at dec 6600 irq 39
tsciic0 at tsc0
iic0 at tsciic0: I2C bus
timecounter: Timecounter "clockinterrupt" frequency 1024 Hz quality 0
timecounter: Timecounter "PCC" frequency 499882560 Hz quality 1000
scsibus0 at isp0: 16 targets, 8 luns per target
scsibus1 at isp1: 16 targets, 8 luns per target
scsibus0: waiting 2 seconds for devices to settle...
scsibus1: waiting 2 seconds for devices to settle...
uhub0 at usb0: vendor 1080 OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
sd0 at scsibus0 target 0 lun 0: <COMPAQ, BB00911CA0, 3B05> disk fixed
sd0: 8678 MB, 5273 cyl, 20 head, 168 sec, 512 bytes/sect x 17773524 sectors
sd0: sync (50.00ns offset 8), 16-bit (40.000MB/s) transfers, tagged queueing
sd1 at scsibus0 target 1 lun 0: <COMPAQ, BB00912301, B016> disk fixed
sd1: 8678 MB, 5273 cyl, 20 head, 168 sec, 512 bytes/sect x 17773524 sectors
sd1: sync (50.00ns offset 8), 16-bit (40.000MB/s) transfers, tagged queueing
sd2 at scsibus0 target 2 lun 0: <COMPAQ, BB00911CA0, 3B05> disk fixed
sd2: 8678 MB, 5273 cyl, 20 head, 168 sec, 512 bytes/sect x 17773524 sectors
sd2: sync (50.00ns offset 8), 16-bit (40.000MB/s) transfers, tagged queueing
fd0 at fdc0 drive 0: 1.44MB, 80 cyl, 2 head, 18 sec
sd3 at scsibus0 target 3 lun 0: <COMPAQ, BB00912301, B016> disk fixed
sd3: 8678 MB, 5273 cyl, 20 head, 168 sec, 512 bytes/sect x 17773524 sectors
sd3: sync (50.00ns offset 8), 16-bit (40.000MB/s) transfers, tagged queueing
cd0 at scsibus1 target 5 lun 0: <DEC, RRD47 (C) DEC, 1206> cdrom removable
raid0: RAID Level 1
raid0: Components: /dev/sd0a /dev/sd1a
raid0: Total Sectors: 17773440 (8678 MB)
raid1: RAID Level 1
raid1: Components: /dev/sd2a /dev/sd3a
raid1: Total Sectors: 17773440 (8678 MB)
root on raid0a dumps on raid0b
root file system type: ffs
kern.module.path=/stand/alpha/7.99.29/modules

Alpha DEC 3000

with SCSI2SD

Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016
The NetBSD Foundation, Inc. All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.

NetBSD 7.99.29 (GENERIC-$Revision: 1.368 $) #3: Thu May 5 09:17:11 CEST 2016
martin@martins.aprisoft.de:/ssd/src/sys/arch/alpha/compile/GENERIC
DEC 3000 - M300, 150MHz, s/n
8192 byte page size, 1 processor.
total memory = 98304 KB
(2048 KB reserved for PROM, 96256 KB used by NetBSD)
avail memory = 82352 KB
timecounter: Timecounters tick every 0.976 msec
Kernelized RAIDframe activated
mainbus0 (root)
cpu0 at mainbus0: ID 0 (primary), 21064-1
tcasic0 at mainbus0
tc0 at tcasic0: 12.5 MHz clock
sfb0 at tc0 slot 6 offset 0x2000000: 1280x1024, 8bpp
wsdisplay1 at sfb0 kbdmux 1
wsmux1: connecting to wsdisplay1
ioasic0 at tc0 slot 5 offset 0x0: slow mode
le0 at ioasic0 offset 0xc0000: address 08:00:2b:3c:93:27
le0: 32 receive buffers, 8 transmit buffers
zsc0 at ioasic0 offset 0x100000
vsms0 at zsc0 channel 0
wsmouse0 at vsms0 mux 0
zstty0 at zsc0 channel 1 (console i/o)
zsc1 at ioasic0 offset 0x180000
lkkbd0 at zsc1 channel 0
wskbd0 at lkkbd0 mux 1
wskbd0: connecting to wsdisplay1
zsc1: channel 1 not configured
mcclock0 at ioasic0 offset 0x200000: mc146818 compatible time-of-day clock
bba0 at ioasic0 offset 0x240000
audio0 at bba0: full duplex, playback, capture, mmap
tcds0 at tc0 slot 4 offset 0x0: TurboChannel Dual SCSI (baseboard)
asc0 at tcds0 chip 0: NCR53C94, 25MHz, SCSI ID 0
scsibus0 at asc0: 8 targets, 8 luns per target
timecounter: Timecounter "clockinterrupt" frequency 1024 Hz quality 0
timecounter: Timecounter "PCC" frequency 150006528 Hz quality 1000
lkkbd0: no keyboard
scsibus0: waiting 2 seconds for devices to settle...
sd0 at scsibus0 target 1 lun 0: <codesrc, SCSI2SD, 1.0> disk fixed
sd0: 3712 MB, 473 cyl, 255 head, 63 sec, 512 bytes/sect x 7603200 sectors
sd0: async, 8-bit transfers
root on sd0a dumps on sd0b
root file system type: ffs
kern.module.path=/stand/alpha/7.99.29/modules

with original RZ26

sd0 at scsibus0 target 3 lun 0: disk fixed
sd0: 1001 MB, 2570 cyl, 14 head, 57 sec, 512 bytes/sect x 2050860 sectors
sd0: sync (200.00ns offset 15), 8-bit (5.000MB/s) transfers, tagged queueing

Mac Quadra 640AV

File system: FFSv1, log

with SCSI2SD

Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016
The NetBSD Foundation, Inc. All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.

NetBSD 7.99.29 (MAC-BETH) #18: Tue May 10 05:40:16 CEST 2016
martin@night-owl.duskware.de:/usr/src/sys/arch/mac68k/compile/MAC-BETH
Apple Macintosh Centris 660AV (68040)
cpu: delay factor 800
fpu: mc68040
total memory = 45056 KB
avail memory = 40248 KB
timecounter: Timecounters tick every 16.666 msec
mrg: 'Quadra AV ROMs' ROM glue, tracing off, debug off, silent traps
mainbus0 (root)
obio0 at mainbus0
esp0 at obio0 addr 0: address 0x2da000: NCR53C96, 25MHz, SCSI ID 7
scsibus0 at esp0: 8 targets, 8 luns per target
adb0 at obio0
intvid0 at obio0 @ 50100600: CIVIC video subsystem
intvid0: 640 x 480, 65536 color
macfb0 at intvid0
wsdisplay0 at macfb0 (kbdmux ignored)
mc0 at obio0: address 08:00:07:46:74:0a
zsc0 at obio0 chip type 0
zsc0 channel 0: d_speed 9600 DCD clk 0 CTS clk 0
zstty0 at zsc0 channel 0 (console i/o)
zsc0 channel 1: d_speed 9600 DCD clk 0 CTS clk 0
zstty1 at zsc0 channel 1
nubus0 at mainbus0
timecounter: Timecounter "clockinterrupt" frequency 60 Hz quality 0
timecounter: Timecounter "VIA1 T2" frequency 783360 Hz quality 100
scsibus0: waiting 2 seconds for devices to settle...
adb0 (direct, Cuda): 2 targets
aed0 at adb0 addr 0: ADB Event device
akbd0 at adb0 addr 2: keyboard II (ISO layout)
wskbd0 at akbd0 (mux ignored)
ams0 at adb0 addr 3: 1-button, 100 dpi mouse
wsmouse0 at ams0 (mux ignored)
sd0 at scsibus0 target 0 lun 0: <codesrc, SCSI2SD, 4.6> disk fixed
sd0: 15104 MB, 1925 cyl, 255 head, 63 sec, 512 bytes/sect x 30934016 sectors
sd0: async, 8-bit transfers
cd0 at scsibus0 target 3 lun 0: <SONY, CD-ROM CDU-8003A, 1.9a> cdrom removable
cd0: async, 8-bit transfers
boot device: sd0
root on sd0a dumps on sd0b
root file system type: ffs
kern.module.path=/stand/mac68k/7.99.29/modules

with hard disk

sd0 at scsibus0 target 0 lun 0: disk fixed
sd0: 8761 MB, 14384 cyl, 3 head, 415 sec, 512 bytes/sect x 17942584 sectors
sd0: async, 8-bit transfers, tagged queueing
Posted late Friday afternoon, May 27th, 2016 Tags:

The NetBSD Project is pleased to announce NetBSD 7.0.1, the first security/bugfix update of the NetBSD 7.0 release branch. It represents a selected subset of fixes deemed important for security or stability reasons. If you are running an earlier release of NetBSD, we strongly suggest updating to 7.0.1.

For more details, please see the release notes.

Complete source and binaries for NetBSD are available for download at many sites around the world. A list of download sites providing FTP, AnonCVS, SUP, and other services may be found at http://www.NetBSD.org/mirrors/.

Posted early Saturday morning, May 28th, 2016 Tags:

The pkgsrc team has prepared the 50th release of their package management system, with the 2016Q1 version. It's infrequent event, as the 100th release will be held after 50 quarters.

The NetBSD team has prepared series of interviews with the authors. We started with Joerg Sonnenberger, a developer well known for his LLVM on NetBSD contribution.

Hi Joerg, please introduce yourself.

Hello Kamil,

I'm a software developer with a background in mathematics. Since I am doing Open Source development primarily for fun, my interests are changing over time and cover many different topics.

First of all, congratulations on the 50th release of pkgsrc! How do you feel about this anniversary?

The pkgsrc quarterly branches were one of the reasons for me to start looking at pkgsrc -- a bit more than a decade ago. As such, my 40th branch has passed recently as well, so it is a bit personal as well. I'm looking back proudly to what we have achieved, but also a bit sad to some of the faces and names that disappeared from the community over that time.

What are the main benefits of the pkgsrc system?

For me, the biggest advantage of pkgsrc is to provide me with the freedom to have a consistent environment across different platforms. Having to deal with BSD, Solaris and Linux flavours, it is very important to be able to sit down and get a consistent set of tools, consistent patches for programs to depend on etc.

Where and how do you use pkgsrc?

I'm using it on my own NetBSD and SmartOS systems as well as for providing software images for my customers at work.

What are the pkgsrc projects you are currently working on?

After that time focusing on bug hunting for the clang builds, I've recently returned to working on pbulk again. There are a number of areas where the distributed build can be improved. My main goal at the moment is to reduce the required coupling of master and slave. Sharing the distfiles is more challenging than it seems on first sight. NFS or null mounts are one option, but often, different packages sharing the same tarball are built in parallel. On an update, they will try to fetch the same file concurrently and often fail badly. The receipt I ended up using for the NetBSD 7.0 bulk builds is a low-memory virtual machine with nginx distribution the distfiles to all build slaves via HTTP(S). If a new file is obtained from elsewhere, it will be uploaded back to my local mirror via PUT. This has been quite reliable so far, but I still have to document the various steps involved for others. The new part will be better handling of the binary packages themselve. Using something like PUT again makes it easier to provide signatures without exposing the key material to all builds.

If you analyze the current state of pkgsrc, which improvements and changes do you wish for the future?

The biggest new term wish is that Leonardo's Summer of Code project will be successful and we finally get a proper infrastructure for building multiple packages in one step. The removal of non-DESTDIR support was kind of a show stopper for doing this sanely and that one is finally done. It didn't even take 10 years to reach that point From an infrastructural point of view, I want to tackle the shell fragment library topic again soon and also work on improving the build performance. The cwrapper support has been pending for a while, someone (else) has to sit down and fix the USE_CROSSBASE handling. I think that's the only real fallout we have left in the tree for making it the default, with all the performance boosts it can provide. More resources for builds is a general wish, but that also often requires porting to new hardware and the like.

Do you have any practical tips to share with the pkgsrc users?

The only practical tip is to aggressively use virtual machines or chroots for building whenever you can't just use binary packages. I.e. if you can't use pre-made binary packages, make your own.

What's the best way to start contributing to pkgsrc and what needs to be done?

Adopt a package, keep it updated and care for it and its environment. Everything tends to start from that. While pkglint is a handy tool for improving your own contributions, especially in pkgsrc-wip, it often helps even more to chat with a regular contributor whenever more challenging problems happen. Unlike many Linux packaging system, we try very hard to make simple things easy and compact. Many problems have different solutions and I sometimes look back to my own changes from years ago and find better solutions now. It is hard to put this kind of experience into documentation. None had the resources for training Watson yet.

Do you plan to participate in the upcoming pkgsrcCon 2016 in Kraków (1-3 July)?

I plan to come, I just don't know if I have time to prepare a presentation for it

Joerg

Posted terribly early Tuesday morning, May 31st, 2016 Tags:
Add a comment
Contact | Disclaimer | Copyright © 1994-2017 The NetBSD Foundation, Inc. ALL RIGHTS RESERVED.
NetBSD® is a registered trademark of The NetBSD Foundation, Inc.