Speeding up pkgsrc builds with ccache and distcc

Building an important amount of packages with pkgsrc can take a very long time. Two helper softwares can speed up operations significantly: ccache and distcc.


From package's DESCR:

ccache is a compiler cache. It acts as a caching pre-processor to C/C++ compilers, using the -E compiler switch and a hash to detect when a compilation can be satisfied from cache. This often results in a 5 to 10 times speedup in common compilations.

Using ccache in pkgsrc is very simple, just add the following line to your /etc/mk.conf:

PKGSRC_COMPILER=    ccache gcc

Declaring ccache as a compiler in mk.conf will make it a dependency for every package to be built.


From package's DESCR:

distcc is a program to distribute compilation of C or C++ code across several machines on a network. distcc should always generate the same results as a local compile, is simple to install and use, and is often two or more times faster than a local compile.

We will setup distcc with two hosts called hostA and hostB. First, install the software on both machines:

# cd /usr/pkgsrc/devel/distcc && make install clean
# cp /usr/pkg/share/examples/rc.d/distccd /etc/rc.d

Configure some parameters in order to allow hostA and hostB to use each other's distcc instances. hostA's IP address is, hostB's IP address is

hostA$ grep distcc /etc/rc.conf
distccd_flags="--allow --allow --listen --log-file=/home/distcc/distccd.log"

hostB$ grep distcc /etc/rc.conf
distccd_flags="--allow --allow --listen --log-file=/home/distcc/distccd.log"

Instead of sending logs to syslog, we will use a custom logfile located in distcc's user home directory:

# mkdir /home/distcc && chown distcc /home/distcc

We can then fire up distcc on both hosts:

# /etc/rc.d/distccd start

In order to use hostnames instead of their IP addresses, add them to both /etc/hosts:

# tail -2 /etc/hosts hostA hostB

And finally tell pkgsrc to use distcc along with ccache by adding these lines to /etc/mk.conf on both machines:

PKGSRC_COMPILER=    ccache distcc gcc
DISTCC_HOSTS=       hostA hostB
MAKE_JOBS=      4

Here we define MAKE_JOBS to 4 because we are using two single-CPU hosts. The recommended value for MAKE_JOBS is number of CPUs*2 to avoid idle time.


To see distcc in action, simply watch the /home/distcc/distccd.log file while you are building a package:

$ tail -f /home/distcc/distccd.log
distccd[5218] (dcc_job_summary) client: COMPILE_OK exit:0 sig:0 core:0 ret:0 time:175ms gcc lockfile.c
distccd[8292] (dcc_job_summary) client: COMPILE_OK exit:0 sig:0 core:0 ret:0 time:222ms gcc counters.c
distccd[27779] (dcc_job_summary) client: COMPILE_OK exit:0 sig:0 core:0 ret:0 time:3009ms gcc ccache.c
distccd[27779] (dcc_job_summary) client: COMPILE_OK exit:0 sig:0 core:0 ret:0 time:152ms gcc compopt.c

If you update the PKGSRC_COMPILER value to include ccache as the wiki states, it does use it, but it creates a new .ccache directory under each packages work subdirectory. So, any use of "make clean" will remove that cache directory and you will loose your work. (using ccache --print-config and/or -s will make what's going on clearer)

If you set


to where root's global cache is, it will persist. If you upgrade your compilers, you may want to run

ccache --clear

to remove old cached files before you rebuild your packages.

This thread http://mail-index.netbsd.org/pkgsrc-users/2009/08/31/msg010604.html has some additional information.

Comment by waddell late Sunday night, December 14th, 2015
Add a comment