Return to pbulk.mdwn CVS log

Up to [NetBSD Developer Wiki] / wikisrc / tutorials / pkgsrc

missing LF

## Using pbulk to create a pkgsrc binary repository

*pkgtools/pbulk* package consists in a set of tools designed to ease
mass-building of *pkgsrc* packages, and the creation your own *pkgsrc* binaries
repository.

Its setup needs a bit of work, here is how to prepare and run your bulk-build
box. In this article we will only consider a one-node machine.

This documentation is based on the [pbulk-HOWTO](http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/doc/HOWTO-pbulk?rev=1.7&content-type=text/x-cvsweb-markup).

### Prerequisites

These are the prerequisites needed by *pbulk*:

* A *pkgsrc* source tree
* A *src* source tree
* Possibly (not mandatory) a *xsrc* source tree
* Possibly (not mandatory) a tool like *misc/screen* as the full build process can takes a **very** long time.

For example, let's say you'd want to regularly build *pkgsrc-2011Q3* packages for *NetBSD 5.1* on your build box, you would prepare the following:

	# cd /usr
	# cvs -d anoncvs@anoncvs.netbsd.org:/cvsroot co -rnetbsd-5-1
	# cvs -d anoncvs@anoncvs.netbsd.org:/cvsroot co -rpkgsrc-2011Q3

And get sure those are always up-to-date by adding the following to a *crontab*:

	30 1 * * * cd /usr/pkgsrc && /usr/bin/cvs up -Pd >/dev/null 2>&1
	00 2 * * * cd /usr/src && /usr/bin/cvs up -Pd >/dev/null 2>&1

### Prepare a chroot

In order to isolate the bulk build, it is advised you run all the operations
within a *chroot*. Running *pbulk* on your real environment would wipe all
of your installed packages, and would modify your base system with lots of
directories, users and groups you don't need.

Fortunately, a tool called *mksandbox* will simplify this process. *mksandbox*
is located in *mk/bulk/mksandbox*, and it is called like this:

	# mk/bulk/mksandbox [optionnal flags] /path/to/sandbox

For example, to create a sandbox in */home/bulk* without the X11 system, run:

	# mk/bulk/mksandbox --without-x /home/bulk

This command will prepare and mount most of the needed directories, and will
place a shell script on top of the sandbox filesystem called *sandbox*. This
script is used to *mount*/*umount* your sandbox. It is a good idea to add
*/var/spool* to the list of directories mounted as read/write in your sandbox
so the email report is actually sent. Simply add:

	/var/spool /var/spool rw \

to the list of directories in the *sandbox* script. *sandbox* script use is
really straightforward:

	# /path/to/your/sandbox/sandbox umount

Unmounts the sandbox

	# /path/to/your/sandbox/sandbox mount

Mounts the sandbox

### Prepare the pbulk environment

Now that our sandbox is available and mounted, we will chroot to it:

	# chroot /home/bulk /bin/ksh

As the [pbulk-HOWTO](http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/doc/HOWTO-pbulk?rev=1.7&content-type=text/x-cvsweb-markup) suggests, we will create some helpers directories to keep things clean:

	# mkdir /bulklog # bulk meta data and log files of each package
	# mkdir /scratch # WRKOBJDIR
	# mkdir /distfiles # DISTDIR
	# mkdir /packages # PACKAGES

The */scratch* directory must be writable by a *bulk* user defined below in */usr/pkg_bulk/etc/pbulk.conf* as the build is done using DESTDIR.

	# useradd bulk
	# chown bulk /scratch

It is now time to bootstrap (yes, even on NetBSD) a *pbulk* environment so the latter is not wiped out when a new bulk build is started:

	# cd /usr/pkgsrc/bootstrap
	# ./bootstrap --prefix=/usr/pkg_bulk --pkgdbdir=/usr/pkg_bulk/.pkgdb
	# rm -rf work

Then install *pbulk* itself on this newly created separate prefix:

	# cd /usr/pkgsrc/pkgtools/pbulk
	# env PATH=/usr/pkg_bulk/bin:/usr/pkg_bulk/sbin:${PATH} bmake install

### Configuring pbulk

*pbulk* configuration file is */usr/pkg_bulk/etc/pbulk.conf*, here are
some parameters you should modify to customize the run to your needs:

	base_url= # should point to the URL where the txt report will be
	master_mode=no # this will not be a distributed build
	# [...]
	report_recipients="your@email.com"
	# [...]
	#bootstrapkit= must be commented on NetBSD !
	unprivileged_user=bulk # used for unprivileged user-destdir builds
	# [...]
	bulklog=/bulklog
	packages=/packages
	prefix=/usr/pkg
	pkgsrc=/usr/pkgsrc
	pkgdb=/var/db/pkg
	varbase=/var
	# [...]
	make=/usr/bin/make # must be modified on NetBSD !

### Configuring the build

Last but not least, you should configure your */etc/mk.conf* (in the *chroot* !).
Here's an example of what we use at [NetBSDfr](http://www.netbsdfr.org/):

	WRKOBJDIR=              /scratch
	PKGSRCDIR=              /usr/pkgsrc
	DISTDIR=                /distfiles
	PACKAGES=               /packages

	FAILOVER_FETCH=         yes

	X11_TYPE=               modular

	SKIP_LICENSE_CHECK=             yes
	ALLOW_VULNERABLE_PACKAGES=      yes

	PKG_DEVELOPER?=         yes

	PKG_OPTIONS.irssi=      perl inet6
	PKG_OPTIONS.mplayer=    oss

	DSPAM_STORAGE_DRIVER=   mysql
	PKG_OPTIONS.dspam+=     graphs
	PKG_OPTIONS.dovecot=    ssl ldap dovecot-sieve dovecot-managesieve
	PKG_OPTIONS.nagios-nrpe=ssl tcpwrappers

### Running the build

Now that everything's in place, we can fire up the build from the *chroot* using
the following command:

	# /usr/pkg_bulk/bin/bulkbuild

It is recommended to run the build inside a tool like *misc/screen* or
*misc/tmux* as it will take a lot of time.

If the build is stopped, it is possible to restart it by invoking:

	# /usr/pkg_bulk/bin/bulkbuild-restart

### Hints

If you'd like to restart the build to retry building some failing packages, **empty** */meta/error* (don't delete it):

	# cat > /meta/error << EOF
	EOF

If you'd like to rebuild a single package, use the *bulkbuild-rebuild* command followed by the package name.