Distributing pkgsrc builds across different OSes

You may want to use several machines to speed up your pkgsrc builds, but as those computers are not running NetBSD, you may think they are useless as build-helpers. This is where enters NetBSD's cross-compiling system in conjuction with distcc.

A classic scenario

GNU/Linux is a very common OS you probably have on your network, in this tutorial, we will focus on how to use a Debian GNU/Linux system as a distcc node, but this procedure can apply to almost any pkgsrc-supported platform.

First things first, your GNU/Linux machine must have the following packages installed

Building the cross-compiling chain

Once done, download NetBSD's source tree. For example, if your target is a NetBSD 5.1.2 system

# pwd
# cvs -d anoncvs@anoncvs.fr.netbsd.org:/cvsroot co -rnetbsd-5-1-2-RELEASE src

We will then use the build.sh script in order to build the tools needed for cross-compiling

# cd src
# ./build.sh -m amd64 tools

Do not use the -u flag, as we need to configure nbcompat for the tools to build correctly.

Once finished, you should have a directory like


On the src directory. This is where the cross-compiling toolkit resides.

Installing and configuring distcc

It is mandatory to configure distcc so it uses our cross-comping tools to build binaries for another platform. On a Debian system, this is done in /etc/default/distcc

# [...]
# Allowed networks
# IP where distcc will listen
# Here's the real trick, supersede $PATH so the firsts binaries
# distcc will look for are NetBSD's ones

After that, simply start distcc

/etc/init.d/distcc start

Testing the setup

On a NetBSD machine located in distcc's allowed network, add the following to /etc/mk.conf

PKGSRC_COMPILER=ccache distcc gcc
DISTCC_HOSTS= localhost

And fire up the make command in a pkgsrc subdirectory containing a C-based package. You should see something like this in helper's /var/log/distccd.log file

distccd[20634] (dcc_job_summary) client: COMPILE_OK exit:0 sig:0 core:0 ret:0 time:57ms gcc /usr/pkgsrc/misc/tmux/work/.ccache-cache/tmp/unvis.tmp.bulkor.18298.i
distccd[20734] (dcc_job_summary) client: COMPILE_OK exit:0 sig:0 core:0 ret:0 time:172ms gcc /usr/pkgsrc/misc/tmux/work/.ccache-cache/tmp/imsg-buffe.tmp.bulkor.20945.i
distccd[20586] (dcc_job_summary) client: COMPILE_OK exit:0 sig:0 core:0 ret:0 time:40ms gcc /usr/pkgsrc/misc/tmux/work/.ccache-cache/tmp/strtonum.tmp.bulkor.4879.i

Of course, you may want to adjust the MAKE_JOBS according to the number of nodes your build-cluster has.

pbulk and distcc

If you intend to use distributed build while running pbulk, you must add the following to the /etc/mk.conf of your sandbox

.for DISTCCDEPS in devel/ccache sysutils/checkperms pkgtools/digest devel/distcc devel/popt devel/libtool-base lang/f2c devel/gmake
.   if ${PKGPATH} == ${DISTCCDEPS}
.   endif

Or the scanning phase will end up with circular dependencies

Cyclic dependency for package:

Enjoy your faster builds!

Add a comment