Return to build_ccache_distcc.mdwn CVS log

Up to [NetBSD Developer Wiki] / wikisrc / pkgsrc

Annotation of wikisrc/pkgsrc/build_ccache_distcc.mdwn, revision 1.1

1.1     ! imil        1: ## Speeding up pkgsrc builds with ccache and distcc
        !             2: 
        !             3: Building an important amount of packages with *pkgsrc* can take a very long time. Two helper softwares can speed up operations significantly: [ccache](http://ccache.samba.org/) and [distcc](http://code.google.com/p/distcc/).
        !             4: 
        !             5: ### ccache
        !             6: 
        !             7: From package's DESCR:
        !             8: 
        !             9: *ccache is a compiler cache. It acts as a caching pre-processor
        !            10: to C/C++ compilers, using the -E compiler switch and a hash to
        !            11: detect when a compilation can be satisfied from cache. This
        !            12: often results in a 5 to 10 times speedup in common compilations.*
        !            13: 
        !            14: Using *ccache* in *pkgsrc* is very simple, just add the following line to your */etc/mk.conf*:
        !            15: 
        !            16:        PKGSRC_COMPILER=        ccache gcc
        !            17: 
        !            18: Declaring *ccache* as a compiler in *mk.conf* will make it a dependency for every package to be built.
        !            19: 
        !            20: ### distcc
        !            21: 
        !            22: From package's DESCR:
        !            23: 
        !            24: *distcc is a program to distribute compilation of C or C++ code across
        !            25: several machines on a network. distcc should always generate the same
        !            26: results as a local compile, is simple to install and use, and is often
        !            27: two or more times faster than a local compile.*
        !            28: 
        !            29: We will setup *distcc* with two hosts called hostA and hostB. First, install the software on both machines:
        !            30: 
        !            31:        # cd /usr/pkgsrc/devel/distcc && make install clean
        !            32:        # cp /usr/pkg/share/examples/rc.d/distccd /etc/rc.d
        !            33: 
        !            34: Configure some parameters in order to allow hostA and hostB to use each other's *distcc* instances. hostA's IP address is 192.168.1.1, hostB's IP address is 192.168.1.2:
        !            35: 
        !            36:        hostA$ grep distcc /etc/rc.conf
        !            37:        distccd=YES
        !            38:        distccd_flags="--allow 192.168.1.0/24 --allow 127.0.0.1 --listen 192.168.1.1 --log-file=/home/distcc/distccd.log"
        !            39: 
        !            40:        hostB$ grep distcc /etc/rc.conf
        !            41:        distccd=YES
        !            42:        distccd_flags="--allow 192.168.1.0/24 --allow 127.0.0.1 --listen 192.168.1.2 --log-file=/home/distcc/distccd.log"
        !            43: 
        !            44: Instead of sending logs to *syslog*, we will use a custom logfile located in *distcc*'s user home directory:
        !            45: 
        !            46:        # mkdir /home/distcc && chown distcc /home/distcc
        !            47: 
        !            48: We can then fire up *distcc* on both hosts:
        !            49: 
        !            50:        # /etc/rc.d/distccd start
        !            51: 
        !            52: In order to use hostnames instead of their IP addresses, add them to both */etc/hosts*:
        !            53: 
        !            54:        # grep ^host /etc/hosts
        !            55:        192.168.1.1     hostA
        !            56:        192.168.1.2     hostB
        !            57: 
        !            58: And finally tell *pkgsrc* to use *distcc* along with *ccache* by adding these lines to */etc/mk.conf* on both machines:
        !            59: 
        !            60:        PKGSRC_COMPILER=        ccache distcc gcc
        !            61:        DISTCC_HOSTS=           hostA hostB
        !            62:        MAKE_JOBS=              4
        !            63: 
        !            64: Here we define *MAKE_JOBS* to 4 because we are using two single-CPU hosts. The recommended value for *MAKE_JOBS* is *number of CPUs\*2* to avoid idle time.
        !            65: 
        !            66: ### Testing
        !            67: 
        !            68: To see *distcc* in action, simply watch the */home/distcc/distccd.log* file while you are building a package:
        !            69: 
        !            70:        $ tail -f /home/distcc/distccd.log
        !            71:        distccd[5218] (dcc_job_summary) client: 192.168.67.3:64865 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:175ms gcc lockfile.c
        !            72:        distccd[8292] (dcc_job_summary) client: 192.168.67.3:64864 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:222ms gcc counters.c
        !            73:        distccd[27779] (dcc_job_summary) client: 192.168.67.3:64881 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:3009ms gcc ccache.c
        !            74:        distccd[27779] (dcc_job_summary) client: 192.168.67.3:64863 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:152ms gcc compopt.c