NetBSD Wiki (prerelease)/tutorials/pkgsrc/ pbulk

Using pbulk to create a pkgsrc binary repository

pkgtools/pbulk package consists in a set of tools designed to ease mass-building of pkgsrc packages, and the creation your own pkgsrc binaries repository.

Its setup needs a bit of work, here is how to prepare and run your bulk-build box. In this article we will only consider a one-node machine.

This documentation is based on the pbulk-HOWTO.

Prerequisites

These are the prerequisites needed by pbulk:

For example, to prepare a pkgsrc-2011Q3 bulk build:

# cd /usr
# cvs -d anoncvs@anoncvs.netbsd.org:/cvsroot co -rpkgsrc-2011Q3 pkgsrc

Avoid automatic update of pkgsrc tree (cron or such), if you're in the middle of a build, it could lead to unfortunate results.

Prepare a chroot

In order to isolate the bulk build, it is advised you run all the operations within a chroot. Running pbulk on your real environment would wipe all of your installed packages, and would modify your base system with lots of directories, users and groups you don't need.

Fortunately, a tool called mksandbox will simplify this process. mksandbox is located in the pkgtools/mksandbox package, and it is called like this:

# mksandbox [optionnal flags] /path/to/sandbox

For example, to create a sandbox in /home/bulk without the X11 system, run:

# mksandbox --without-x /home/bulk

This command will prepare and mount most of the needed directories, and will place a shell script on top of the sandbox filesystem called sandbox. This script is used to mount/umount your sandbox. It is a good idea to add /var/spool to the list of directories mounted as read/write in your sandbox so the email report is actually sent. Simply add:

/var/spool /var/spool rw \

to the list of directories in the sandbox script. sandbox script use is really straightforward:

# /path/to/your/sandbox/sandbox umount

Unmounts the sandbox

# /path/to/your/sandbox/sandbox mount

Mounts the sandbox

Prepare the pbulk environment

Now that our sandbox is available and mounted, we will chroot to it:

# chroot /home/bulk /bin/ksh

As the pbulk-HOWTO suggests, we will create some helpers directories to keep things clean:

# mkdir /bulklog # bulk meta data and log files of each package
# mkdir /scratch # WRKOBJDIR
# mkdir /distfiles # DISTDIR
# mkdir /packages # PACKAGES

The /scratch directory must be writable by a bulk user defined below in /usr/pkg_bulk/etc/pbulk.conf as the build is done using DESTDIR.

# useradd bulk
# chown bulk /scratch

It is now time to bootstrap (yes, even on NetBSD) a pbulk environment so the latter is not wiped out when a new bulk build is started:

# cd /usr/pkgsrc/bootstrap
# ./bootstrap --prefix=/usr/pkg_bulk --pkgdbdir=/usr/pkg_bulk/.pkgdb
# rm -rf work

Add the line:

PACKAGES=   /scratch/pbulk_packages

to pbulk's prefix mk.conf (/usr/pkg_bulk/etc/mk.conf).

Then install pbulk itself on this newly created separate prefix:

# cd /usr/pkgsrc/pkgtools/pbulk
# env PATH=/usr/pkg_bulk/bin:/usr/pkg_bulk/sbin:${PATH} bmake install

Configuring pbulk

pbulk configuration file is /usr/pkg_bulk/etc/pbulk.conf, here are some parameters you should modify to customize the run to your needs:

base_url= # should point to the URL where the txt report will be
master_mode=no # this will not be a distributed build
# [...]
report_recipients="your@email.com"
# [...]
#bootstrapkit= must be commented on NetBSD !
unprivileged_user=bulk # used for unprivileged user-destdir builds
# [...]
bulklog=/bulklog
packages=/packages
prefix=/usr/pkg
pkgsrc=/usr/pkgsrc
pkgdb=/var/db/pkg
varbase=/var
# [...]
make=/usr/bin/make # must be modified on NetBSD !

Also, in order to avoid hangs, it might be a good idea to add the following to the top of pbulk.conf

ulimit -t 1800 # set the limit on CPU time (in seconds)
ulimit -v 2097152 # limits process address space

Configuring the build

Last but not least, you should configure your /etc/mk.conf (in the chroot !). Here's an example of what we use at NetBSDfr:

WRKOBJDIR=              /scratch
PKGSRCDIR=              /usr/pkgsrc
DISTDIR=                /distfiles
PACKAGES=               /packages

FAILOVER_FETCH=         yes

SKIP_LICENSE_CHECK=             yes
ALLOW_VULNERABLE_PACKAGES=      yes

PKG_DEVELOPER?=         yes

# site specific changes

PKG_OPTIONS.irssi=      perl inet6
PKG_OPTIONS.mplayer=    oss

DSPAM_STORAGE_DRIVER=   mysql
PKG_OPTIONS.dspam+=     graphs
PKG_OPTIONS.dovecot=    ssl ldap dovecot-sieve dovecot-managesieve
PKG_OPTIONS.nagios-nrpe=ssl tcpwrappers

X11_TYPE=               modular

Running the build

Now that everything's in place, we can fire up the build from the chroot using the following command:

# /usr/pkg_bulk/bin/bulkbuild

It is recommended to run the build inside a tool like misc/screen or misc/tmux as it will take a lot of time.

If the build is stopped, it is possible to restart it by invoking:

# /usr/pkg_bulk/bin/bulkbuild-restart

Hints

If you'd like to restart the build to retry building some failing packages, empty /meta/error (don't delete it):

# printf "" > /meta/error

If you'd like to rebuild a single package, use the bulkbuild-rebuild command followed by the package name.

Add a comment