cross compile distcc

Distributing pkgsrc builds across different OSes

You may want to use several machines to speed up your pkgsrc builds, but as those computers are not running NetBSD, you may think they are useless as build-helpers. This is where enters NetBSD's cross-compiling system in conjunction with distcc.

A classic scenario

GNU/Linux is a very common OS you probably have on your network, in this tutorial, we will focus on how to use a Debian GNU/Linux system as a distcc node, but this procedure can apply to almost any pkgsrc-supported platform.

First things first, your GNU/Linux machine must have the following packages installed

gcc
g++
zlib1g-dev
ncurses-base

Building the cross-compiling chain

Once done, download NetBSD's source tree. For example, if your target is a NetBSD 5.1.2 system

# pwd
/home/netbsd
# cvs -d anoncvs@anoncvs.fr.netbsd.org:/cvsroot co -rnetbsd-5-1-2-RELEASE src

We will then use the build.sh script in order to build the tools needed for cross-compiling

# cd src
# ./build.sh -m amd64 tools

Do not use the -u flag, as we need to configure nbcompat for the tools to build correctly.

Once finished, you should have a directory like

tooldir.Linux-2.6.32-5-xen-amd64-x86_64

On the src directory. This is where the cross-compiling toolkit resides.

Installing and configuring distcc

It is mandatory to configure distcc so it uses our cross-comping tools to build binaries for another platform. On a Debian system, this is done in /etc/default/distcc

STARTDISTCC="true"
# [...]
# Allowed networks
ALLOWEDNETS="127.0.0.1 192.168.0.0/24"
# IP where distcc will listen
LISTENER="192.168.0.7"
###
# Here's the real trick, supersede $PATH so the firsts binaries
# distcc will look for are NetBSD's ones
###
PATH=/home/netbsd/src/tooldir.Linux-2.6.32-5-xen-amd64-x86_64/x86_64--netbsd/bin:$PATH

After that, simply start distcc

/etc/init.d/distcc start

Testing the setup

On a NetBSD machine located in distcc's allowed network, add the following to /etc/mk.conf

PKGSRC_COMPILER=ccache distcc gcc
MAKE_JOBS=4
DISTCC_HOSTS=192.168.0.7 localhost

And fire up the make command in a pkgsrc subdirectory containing a C-based package. You should see something like this in helper's /var/log/distccd.log file

distccd[20634] (dcc_job_summary) client: 192.168.0.2:64006 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:57ms gcc /usr/pkgsrc/misc/tmux/work/.ccache-cache/tmp/unvis.tmp.bulkor.18298.i
distccd[20734] (dcc_job_summary) client: 192.168.0.2:64005 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:172ms gcc /usr/pkgsrc/misc/tmux/work/.ccache-cache/tmp/imsg-buffe.tmp.bulkor.20945.i
distccd[20586] (dcc_job_summary) client: 192.168.0.2:64004 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:40ms gcc /usr/pkgsrc/misc/tmux/work/.ccache-cache/tmp/strtonum.tmp.bulkor.4879.i

Of course, you may want to adjust the MAKE_JOBS according to the number of nodes your build-cluster has.

pbulk and distcc

If you intend to use distributed build while running pbulk, you must add the following to the /etc/mk.conf of your sandbox

.for DISTCCDEPS in devel/ccache sysutils/checkperms pkgtools/digest devel/distcc devel/popt devel/libtool-base lang/f2c devel/gmake
.   if ${PKGPATH} == ${DISTCCDEPS}
IGNORE_DISTCC=  yes
IGNORE_CCACHE=  yes
.   endif
.endfor

Or the scanning phase will end up with circular dependencies

Cyclic dependency for package:
ccache-3.1.4nb1
checkperms-1.11
digest-20111104
distcc-3.1nb1
popt-1.16nb1
libtool-base-2.2.6bnb5
f2c-20100903
ccache-3.1.4nb1