Annotation of wikisrc/tutorials/continuous_building_and_testing_netbsd_with_buildbot.mdwn, revision 1.2

1.1       wiki        1: [[!meta title="Continuous NetBSD release building and testing "]]
                      2: 
1.2     ! wiki        3: **Contents**
        !             4: 
1.1       wiki        5: [[!toc]]
                      6: 
                      7: I decided to bought new computer and setup a continuous building service on it to always have fresh built binaries ready for usage and to be able to test my changes more easily. As continuous integration service I chose [buildbot](www.buildbot.net). Buildbot as package was located in pkgsrc wip and I have updated it to it's latest version 0.8.2 and introduced new package for building buildbot slave program.
                      8: 
                      9: 
                     10: # NetBSD setup
                     11: 
                     12: I have created new Logical Volume and mounted it to /usr/devel directory. After that I have created new user called buildbot. 
                     13: 
                     14:     As root:
                     15:     useradd -m buildbot # Add buildbot user
                     16: 
                     17:        # Create Buildbot directories
                     18:     mkdir /usr/devel/buildbot
                     19:     mkdir /usr/devel/buildslave
                     20: 
                     21:     chown buildbot /usr/devel/buildbot /usr/devel/buildslave
                     22: 
                     23:        # Install Buildbot daemon and Buildbot slave
                     24:        cd /usr/pkgsrc/wip/buildbot
                     25:        make install
                     26: 
                     27:        cd /usr/pkgsrc/wip/buildslave
                     28:        make install
                     29: 
                     30: For My build setup it was needed to change /etc/login.conf and add builbot user to build group
                     31: 
                     32:        builder:\
                     33:                :datasize-cur=1024M:\
                     34:                :datasize-max=infinity:\
                     35:                :maxproc-max=1000:\
                     36:                :maxproc-cur=1000:\
                     37:                :openfiles-cur=1024:
                     38:        
                     39:        usermod -L builder buildbot
                     40: 
                     41: # Buildbot setup
                     42: 
                     43: ## Buildmaster Setup
                     44: 
                     45: First we need to initialise new buildmaster directory 
                     46: 
                     47:        As buildbot user:
                     48:        cd /usr/devel
                     49:        buildbot create-master /usr/devel/buildmaster
                     50: 
                     51: I have decided that for builds I would like to run maximum 2 parallel builds on one build slave client and I want to have system build once a day. After setup I used this configuration file to get buildmaster working.
                     52: 
                     53: 
                     54:        # -*- python -*-
                     55:        # ex: set syntax=python:
                     56:        
                     57:        c = BuildmasterConfig = {}
                     58: 
                     59:        from buildbot.buildslave import BuildSlave
                     60:        c['slaves'] = [BuildSlave("bot1name", "bot1pass", max_builds=2)]
                     61: 
                     62:        c['slavePortnum'] = 9989
                     63: 
                     64:        from buildbot.changes.pb import PBChangeSource
                     65:        c['change_source'] = PBChangeSource()
                     66: 
                     67:        from buildbot.scheduler import Scheduler
                     68:        c['schedulers'] = []
                     69: 
                     70:        from buildbot.schedulers import timed
                     71:        s = timed.Nightly(name='daily',
                     72:                       builderNames=["buildbot-netbsd-vanilla-i386", "buildbot-netbsd-vanilla-amd64"],  
                     73:                                               hour=13,
                     74:                                       minute=0)
                     75: 
                     76:        c['schedulers'] = [s]
                     77: 
                     78:        cvsroot = ":pserver:anoncvs@anoncvs.netbsd.org:/cvsroot"
                     79:        cvsmodule = "src"
                     80: 
                     81:        from buildbot.process import factory
                     82:        from buildbot.steps.source import CVS
                     83:        from buildbot.steps.shell import Compile
                     84:        from buildbot.steps.shell import ShellCommand
                     85:        from buildbot.steps.shell import Test
                     86:        from buildbot.steps.python_twisted import Trial
                     87: 
                     88:        f1 = factory.BuildFactory()
                     89:        f1.addStep(CVS(cvsroot=cvsroot, cvsmodule=cvsmodule, login="", mode="update"))
                     90:        f1.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","i386"]))
                     91:        f1.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","i386"], 
                     92:                                         warningPattern="^WWarning: "))
                     93:        f1.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","i386"]))
                     94: 
                     95:        f2 = factory.BuildFactory()
                     96:        f2.addStep(CVS(cvsroot=cvsroot, cvsmodule=cvsmodule, login="", mode="update"))
                     97:        f2.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","amd64"]))
                     98:        f2.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","amd64"], 
                     99:                                         warningPattern="^WWarning: "))
                    100:        f2.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","amd64"]))
                    101: 
                    102:        f3 = factory.BuildFactory()
                    103:        f3.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/getdev_src.sh", "dev-i386"]))
                    104:        f3.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","dev-i386"]))
                    105:        f3.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","dev-i386"],
                    106:                                         warningPattern="^WWarning: "))
                    107:        f3.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","dev-i386"]))
                    108: 
                    109: 
                    110:        f4 = factory.BuildFactory()
                    111:        f4.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/getdev_src.sh", "dev-amd64"]))
                    112:        f4.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","dev-amd64"]))
                    113:        f4.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","dev-amd64"],
                    114:                                         warningPattern="^WWarning: "))
                    115:        f4.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","dev-amd64"]))
                    116: 
                    117: 
                    118:        b1 = {'name': "buildbot-netbsd-vanilla-i386",
                    119:              'slavename': "bot1name",
                    120:              'builddir': "full",
                    121:              'factory': f1,
                    122:              }
                    123: 
                    124:        b2 = {'name': "buildbot-netbsd-vanilla-amd64",
                    125:              'slavename': "bot1name",
                    126:              'builddir': "full-amd64",
                    127:              'factory': f2,
                    128:              }
                    129: 
                    130:        b3 = {'name': "buildbot-netbsd-development-tree-i386",
                    131:              'slavename': "bot1name", 
                    132:              'builddir': "dev-i386",
                    133:              'factory': f3,
                    134:              }
                    135: 
                    136:        b4 = {'name': "buildbot-netbsd-development-tree-amd64",
                    137:              'slavename': "bot1name",
                    138:              'builddir': "dev-amd64",
                    139:              'factory': f4,
                    140:              }
                    141: 
                    142:        c['builders'] = [b1, b2, b3, b4]
                    143:        c['status'] = []
                    144: 
                    145:        from buildbot.status import html
                    146:        from buildbot.status.web.authz import Authz
                    147: 
                    148:        authz = Authz(
                    149:                 forceBuild=True,
                    150:                         forceAllBuilds=True, 
                    151:                         stopBuild=True,
                    152:                         stopAllBuilds=True,
                    153:                         cancelPendingBuild=True)
                    154:        c['status'].append(html.WebStatus(http_port=8010, authz=authz))
                    155: 
                    156:        c['projectName'] = "NetBSD development daily builds"
                    157:        c['projectURL'] = "www.netbsd.org/~haad/builds/"
                    158: 
                    159:        c['buildbotURL'] = "http://musasi.haad.chillisys.com:8010/"
                    160: 
                    161: For easier builds and testing I have added 3 scripts to _buildbot-basedir_/bin/
                    162: 
                    163: * clean.sh
                    164: * build.sh
                    165: * test.sh
                    166: 
                    167: From master.cfg file you can see that all of these scripts are called with one argument which is basically {prefix}-{arch} where 
                    168: 
                    169: * {prefix} -> prefix name for different builds for same arch (developmant tree, vanilla tree, some netbsd branch etc.)
                    170: * {arch} -> architecture name used for build
                    171: 
                    172: I have implemented one version of script for every architecture built by my buildbot server.
                    173: 
                    174:        Clean script is used to clean build dir on slave before build starts.
                    175: 
                    176:        #!/bin/sh
                    177: 
                    178:        dirname=$1
                    179:        # get arch from dirname
                    180:        arch=$(echo $dirname | cut -f 2 -d\-);
                    181: 
                    182:        buildslave_dir="/usr/devel/buildslave/"
                    183:        obj_dir="${buildslave_dir}/obj/${dirname}"
                    184: 
                    185:        echo "Removing build artefacts from ${obj_dir}/*"
                    186:        rm -rf ${obj_dir}/*
                    187: 
                    188:        Build builds whole system 
                    189: 
                    190:        #!/bin/sh
                    191: 
                    192:        dirname=$1
                    193:        j_flag=9
                    194:        arch=$(echo $dirname | cut -f 2 -d\-);
                    195: 
                    196:        buildslave_dir="/usr/devel/buildslave/"
                    197:        obj_dir="${buildslave_dir}/obj/${dirname}"
                    198: 
                    199:        echo "Building NetBSD tree for architecture ${arch} in $(pwd), objdir ${obj_dir}"
                    200:        ./build.sh -V GMAKE_J_ARGS="-j ${j_flag}" -O ${obj_dir} -T ${obj_dir}/tooldir -Uu -m ${arch} -j ${j_flag} release
                    201: 
                    202:        Test will later run anita and run regression test suite from it
                    203:        #!/bin/sh
                    204: 
                    205:        dirname=$1
                    206:        arch=$(echo $dirname | cut -f 2 -d\-);
                    207: 
                    208:        buildslave_dir="/usr/devel/buildslave/"
                    209:        obj_dir="${buildslave_dir}/obj/${dirname}"
                    210:        anita_dir="${buildslave_dir}/anita/${dirname}"
                    211:        release_dir="${obj_dir}/releasedir/${arch}/"
                    212: 
                    213:        echo "Running tests on ${arch}"
                    214:        anita --workdir ${anita_dir} test ${release_dir}
                    215: ## Buildslave setup
                    216: 
                    217: Buildslave is actual buildbot cluster worker which does worked scheduled to him by buildbot. Command used to created buildslave contains buildir [buildbot hostname:port] buildslave name password. Name and password must match those in master.cfg file. 
                    218: 
                    219:        As buildbot user:
                    220:        buildslave create-slave /usr/devel/buildslave localhost:9989 buildslave buildslavepass
                    221: 
                    222: # Anita
                    223: 
                    224: ## Anita integration with buildbot

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb