Using pbulk to create a pkgsrc binary repository

pkgtools/pbulk package consists in a set of tools designed to ease mass-building of pkgsrc packages, and the creation your own pkgsrc binaries repository.

Its setup needs a bit of work, here is how to prepare and run your bulk-build box. In this article we will only consider a one-node machine.

This documentation is based on the The pkgsrc Guide.

Prerequisites

These are the prerequisites needed by pbulk:

• A pkgsrc source tree
• Possibly a src source tree, only some rare packages needs it. If you choose not to checkout src, simply create a /usr/src directory so mksandbox (see below) doesn't complain about non-existing directory.
• Possibly (not mandatory) a tool like misc/screen or misc/tmux as the full build process can take a very long time.

For example, to prepare a pkgsrc-2011Q3 bulk build:

# cd /usr
# cvs -d anoncvs@anoncvs.netbsd.org:/cvsroot co -rpkgsrc-2011Q3 pkgsrc

Avoid automatic update of pkgsrc tree (cron or such), if you're in the middle of a build, it could lead to unfortunate results.

Prepare a chroot

In order to isolate the bulk build, it is advised you run all the operations within a chroot. Running pbulk on your real environment would wipe all of your installed packages, and would modify your base system with lots of directories, users and groups you don't need.

Fortunately, a tool called mksandbox will simplify this process. mksandbox is located in the pkgtools/mksandbox package, and it is called like this:

# mksandbox [optional flags] /path/to/sandbox

For example, to create a sandbox in /home/bulk without the X11 system, run:

# mksandbox --without-x /home/bulk

This command will prepare and mount most of the needed directories, and will place a shell script on top of the sandbox filesystem called sandbox. This script is used to mount/umount your sandbox. It is a good idea to add /var/spool to the list of directories mounted as read/write in your sandbox so the email report is actually sent. Simply add:

/var/spool /var/spool rw \

to the list of directories in the sandbox script. sandbox script use is really straightforward:

# /path/to/your/sandbox/sandbox umount

Unmounts the sandbox

# /path/to/your/sandbox/sandbox mount

Mounts the sandbox

Prepare the pbulk environment

Now that our sandbox is available and mounted, we will chroot to it by executing the sandbox script without any parameter specified:

# /path/to/your/sandbox/sandbox

Create pbulk user (a Bourne style shell must be used)

# useradd -s /bin/sh pbulk

In a file, set the preferences to use when building packages (mk.conf fragment). Here is a sample mk.conf.frag file:

SKIP_LICENSE_CHECK=             yes
ALLOW_VULNERABLE_PACKAGES=      yes

PKG_DEVELOPER?=         yes

WRKDIR=                 /tmp/work

# site specific changes

PKG_OPTIONS.irssi=      perl inet6
PKG_OPTIONS.mplayer=    oss

DSPAM_STORAGE_DRIVER=   mysql
PKG_OPTIONS.dspam+=     graphs
PKG_OPTIONS.dovecot=    ssl ldap dovecot-sieve dovecot-managesieve
PKG_OPTIONS.nagios-nrpe=ssl tcpwrappers

X11_TYPE=               modular

Deploy and configure pbulk tools

# sh /usr/pkgsrc/mk/pbulk/pbulk.sh -n -c mk.conf.frag

pbulk configuration file is /usr/pbulk/etc/pbulk.conf. You may want to review and customize some parameters like "base_url" and "report_recipients".

Also, in order to avoid hangs, it might be a good idea to add the following to the top of pbulk.conf

ulimit -t 1800 # set the limit on CPU time (in seconds)
ulimit -v 2097152 # limits process address space

Running the build

Now that everything's in place, we can fire up the build from the chroot using the following command:

# /usr/pbulk/bin/bulkbuild

It is recommended to run the build inside a tool like misc/screen or misc/tmux as it will take a lot of time.

If the build is stopped, it is possible to restart it by invoking:

# /usr/pbulk/bin/bulkbuild-restart

Hints

If you'd like to rebuild a single package, use the bulkbuild-rebuild command followed by the package name.