Using pbulk to create a pkgsrc binary repository

pkgtools/pbulk package consists in a set of tools designed to ease mass-building of pkgsrc packages, and the creation your own pkgsrc binaries repository.

Its setup needs a bit of work, here is how to prepare and run your bulk-build box. In this article we will only consider a one-node machine.

This documentation is based on the The pkgsrc Guide.

Prerequisites

These are the prerequisites needed by pbulk:

For example, to prepare a pkgsrc-2011Q3 bulk build:

# cd /usr
# cvs -d anoncvs@anoncvs.netbsd.org:/cvsroot co -rpkgsrc-2011Q3 pkgsrc

Avoid automatic update of pkgsrc tree (cron or such), if you're in the middle of a build, it could lead to unfortunate results.

Prepare a chroot

In order to isolate the bulk build, it is advised you run all the operations within a chroot. Running pbulk on your real environment would wipe all of your installed packages, and would modify your base system with lots of directories, users and groups you don't need.

Fortunately, a tool called mksandbox will simplify this process. mksandbox is located in the pkgtools/mksandbox package, and it is called like this:

# mksandbox [optionnal flags] /path/to/sandbox

For example, to create a sandbox in /home/bulk without the X11 system, run:

# mksandbox --without-x /home/bulk

This command will prepare and mount most of the needed directories, and will place a shell script on top of the sandbox filesystem called sandbox. This script is used to mount/umount your sandbox. It is a good idea to add /var/spool to the list of directories mounted as read/write in your sandbox so the email report is actually sent. Simply add:

/var/spool /var/spool rw \

to the list of directories in the sandbox script. sandbox script use is really straightforward:

# /path/to/your/sandbox/sandbox umount

Unmounts the sandbox

# /path/to/your/sandbox/sandbox mount

Mounts the sandbox

Prepare the pbulk environment

Now that our sandbox is available and mounted, we will chroot to it:

# chroot /home/bulk /bin/sh

Create pbulk user

# useradd bulk

Prepare configuration for packages build (mk.conf fragment). Here's sample mk.conf.frag file:

SKIP_LICENSE_CHECK=             yes
ALLOW_VULNERABLE_PACKAGES=      yes

PKG_DEVELOPER?=         yes

# site specific changes

PKG_OPTIONS.irssi=      perl inet6
PKG_OPTIONS.mplayer=    oss

DSPAM_STORAGE_DRIVER=   mysql
PKG_OPTIONS.dspam+=     graphs
PKG_OPTIONS.dovecot=    ssl ldap dovecot-sieve dovecot-managesieve
PKG_OPTIONS.nagios-nrpe=ssl tcpwrappers

X11_TYPE=               modular

Deploy and configure pbulk tools

# sh /usr/pkgsrc/mk/pbulk/pbulk.sh -n -c mk.conf.frag

pbulk configuration file is /usr/pbulk/etc/pbulk.conf. You may want to review and customize some parameters like "base_url" and "report_recipients".

Also, in order to avoid hangs, it might be a good idea to add the following to the top of pbulk.conf

ulimit -t 1800 # set the limit on CPU time (in seconds)
ulimit -v 2097152 # limits process address space

Running the build

Now that everything's in place, we can fire up the build from the chroot using the following command:

# /usr/pbulk/bin/bulkbuild

It is recommended to run the build inside a tool like misc/screen or misc/tmux as it will take a lot of time.

If the build is stopped, it is possible to restart it by invoking:

# /usr/pbulk/bin/bulkbuild-restart

Hints

If you'd like to rebuild a single package, use the bulkbuild-rebuild command followed by the package name.

Add a comment