Annotation of wikisrc/tutorials/continuous_building_and_testing_netbsd_with_buildbot.mdwn, revision 1.1

1.1     ! wiki        1: [[!meta title="Continuous NetBSD release building and testing "]]
        !             2: 
        !             3: [[!toc]]
        !             4: 
        !             5: I decided to bought new computer and setup a continuous building service on it to always have fresh built binaries ready for usage and to be able to test my changes more easily. As continuous integration service I chose [buildbot](www.buildbot.net). Buildbot as package was located in pkgsrc wip and I have updated it to it's latest version 0.8.2 and introduced new package for building buildbot slave program.
        !             6: 
        !             7: 
        !             8: # NetBSD setup
        !             9: 
        !            10: I have created new Logical Volume and mounted it to /usr/devel directory. After that I have created new user called buildbot. 
        !            11: 
        !            12:     As root:
        !            13:     useradd -m buildbot # Add buildbot user
        !            14: 
        !            15:        # Create Buildbot directories
        !            16:     mkdir /usr/devel/buildbot
        !            17:     mkdir /usr/devel/buildslave
        !            18: 
        !            19:     chown buildbot /usr/devel/buildbot /usr/devel/buildslave
        !            20: 
        !            21:        # Install Buildbot daemon and Buildbot slave
        !            22:        cd /usr/pkgsrc/wip/buildbot
        !            23:        make install
        !            24: 
        !            25:        cd /usr/pkgsrc/wip/buildslave
        !            26:        make install
        !            27: 
        !            28: For My build setup it was needed to change /etc/login.conf and add builbot user to build group
        !            29: 
        !            30:        builder:\
        !            31:                :datasize-cur=1024M:\
        !            32:                :datasize-max=infinity:\
        !            33:                :maxproc-max=1000:\
        !            34:                :maxproc-cur=1000:\
        !            35:                :openfiles-cur=1024:
        !            36:        
        !            37:        usermod -L builder buildbot
        !            38: 
        !            39: # Buildbot setup
        !            40: 
        !            41: ## Buildmaster Setup
        !            42: 
        !            43: First we need to initialise new buildmaster directory 
        !            44: 
        !            45:        As buildbot user:
        !            46:        cd /usr/devel
        !            47:        buildbot create-master /usr/devel/buildmaster
        !            48: 
        !            49: I have decided that for builds I would like to run maximum 2 parallel builds on one build slave client and I want to have system build once a day. After setup I used this configuration file to get buildmaster working.
        !            50: 
        !            51: 
        !            52:        # -*- python -*-
        !            53:        # ex: set syntax=python:
        !            54:        
        !            55:        c = BuildmasterConfig = {}
        !            56: 
        !            57:        from buildbot.buildslave import BuildSlave
        !            58:        c['slaves'] = [BuildSlave("bot1name", "bot1pass", max_builds=2)]
        !            59: 
        !            60:        c['slavePortnum'] = 9989
        !            61: 
        !            62:        from buildbot.changes.pb import PBChangeSource
        !            63:        c['change_source'] = PBChangeSource()
        !            64: 
        !            65:        from buildbot.scheduler import Scheduler
        !            66:        c['schedulers'] = []
        !            67: 
        !            68:        from buildbot.schedulers import timed
        !            69:        s = timed.Nightly(name='daily',
        !            70:                       builderNames=["buildbot-netbsd-vanilla-i386", "buildbot-netbsd-vanilla-amd64"],  
        !            71:                                               hour=13,
        !            72:                                       minute=0)
        !            73: 
        !            74:        c['schedulers'] = [s]
        !            75: 
        !            76:        cvsroot = ":pserver:anoncvs@anoncvs.netbsd.org:/cvsroot"
        !            77:        cvsmodule = "src"
        !            78: 
        !            79:        from buildbot.process import factory
        !            80:        from buildbot.steps.source import CVS
        !            81:        from buildbot.steps.shell import Compile
        !            82:        from buildbot.steps.shell import ShellCommand
        !            83:        from buildbot.steps.shell import Test
        !            84:        from buildbot.steps.python_twisted import Trial
        !            85: 
        !            86:        f1 = factory.BuildFactory()
        !            87:        f1.addStep(CVS(cvsroot=cvsroot, cvsmodule=cvsmodule, login="", mode="update"))
        !            88:        f1.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","i386"]))
        !            89:        f1.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","i386"], 
        !            90:                                         warningPattern="^WWarning: "))
        !            91:        f1.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","i386"]))
        !            92: 
        !            93:        f2 = factory.BuildFactory()
        !            94:        f2.addStep(CVS(cvsroot=cvsroot, cvsmodule=cvsmodule, login="", mode="update"))
        !            95:        f2.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","amd64"]))
        !            96:        f2.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","amd64"], 
        !            97:                                         warningPattern="^WWarning: "))
        !            98:        f2.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","amd64"]))
        !            99: 
        !           100:        f3 = factory.BuildFactory()
        !           101:        f3.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/getdev_src.sh", "dev-i386"]))
        !           102:        f3.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","dev-i386"]))
        !           103:        f3.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","dev-i386"],
        !           104:                                         warningPattern="^WWarning: "))
        !           105:        f3.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","dev-i386"]))
        !           106: 
        !           107: 
        !           108:        f4 = factory.BuildFactory()
        !           109:        f4.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/getdev_src.sh", "dev-amd64"]))
        !           110:        f4.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","dev-amd64"]))
        !           111:        f4.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","dev-amd64"],
        !           112:                                         warningPattern="^WWarning: "))
        !           113:        f4.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","dev-amd64"]))
        !           114: 
        !           115: 
        !           116:        b1 = {'name': "buildbot-netbsd-vanilla-i386",
        !           117:              'slavename': "bot1name",
        !           118:              'builddir': "full",
        !           119:              'factory': f1,
        !           120:              }
        !           121: 
        !           122:        b2 = {'name': "buildbot-netbsd-vanilla-amd64",
        !           123:              'slavename': "bot1name",
        !           124:              'builddir': "full-amd64",
        !           125:              'factory': f2,
        !           126:              }
        !           127: 
        !           128:        b3 = {'name': "buildbot-netbsd-development-tree-i386",
        !           129:              'slavename': "bot1name", 
        !           130:              'builddir': "dev-i386",
        !           131:              'factory': f3,
        !           132:              }
        !           133: 
        !           134:        b4 = {'name': "buildbot-netbsd-development-tree-amd64",
        !           135:              'slavename': "bot1name",
        !           136:              'builddir': "dev-amd64",
        !           137:              'factory': f4,
        !           138:              }
        !           139: 
        !           140:        c['builders'] = [b1, b2, b3, b4]
        !           141:        c['status'] = []
        !           142: 
        !           143:        from buildbot.status import html
        !           144:        from buildbot.status.web.authz import Authz
        !           145: 
        !           146:        authz = Authz(
        !           147:                 forceBuild=True,
        !           148:                         forceAllBuilds=True, 
        !           149:                         stopBuild=True,
        !           150:                         stopAllBuilds=True,
        !           151:                         cancelPendingBuild=True)
        !           152:        c['status'].append(html.WebStatus(http_port=8010, authz=authz))
        !           153: 
        !           154:        c['projectName'] = "NetBSD development daily builds"
        !           155:        c['projectURL'] = "www.netbsd.org/~haad/builds/"
        !           156: 
        !           157:        c['buildbotURL'] = "http://musasi.haad.chillisys.com:8010/"
        !           158: 
        !           159: For easier builds and testing I have added 3 scripts to _buildbot-basedir_/bin/
        !           160: 
        !           161: * clean.sh
        !           162: * build.sh
        !           163: * test.sh
        !           164: 
        !           165: From master.cfg file you can see that all of these scripts are called with one argument which is basically {prefix}-{arch} where 
        !           166: 
        !           167: * {prefix} -> prefix name for different builds for same arch (developmant tree, vanilla tree, some netbsd branch etc.)
        !           168: * {arch} -> architecture name used for build
        !           169: 
        !           170: I have implemented one version of script for every architecture built by my buildbot server.
        !           171: 
        !           172:        Clean script is used to clean build dir on slave before build starts.
        !           173: 
        !           174:        #!/bin/sh
        !           175: 
        !           176:        dirname=$1
        !           177:        # get arch from dirname
        !           178:        arch=$(echo $dirname | cut -f 2 -d\-);
        !           179: 
        !           180:        buildslave_dir="/usr/devel/buildslave/"
        !           181:        obj_dir="${buildslave_dir}/obj/${dirname}"
        !           182: 
        !           183:        echo "Removing build artefacts from ${obj_dir}/*"
        !           184:        rm -rf ${obj_dir}/*
        !           185: 
        !           186:        Build builds whole system 
        !           187: 
        !           188:        #!/bin/sh
        !           189: 
        !           190:        dirname=$1
        !           191:        j_flag=9
        !           192:        arch=$(echo $dirname | cut -f 2 -d\-);
        !           193: 
        !           194:        buildslave_dir="/usr/devel/buildslave/"
        !           195:        obj_dir="${buildslave_dir}/obj/${dirname}"
        !           196: 
        !           197:        echo "Building NetBSD tree for architecture ${arch} in $(pwd), objdir ${obj_dir}"
        !           198:        ./build.sh -V GMAKE_J_ARGS="-j ${j_flag}" -O ${obj_dir} -T ${obj_dir}/tooldir -Uu -m ${arch} -j ${j_flag} release
        !           199: 
        !           200:        Test will later run anita and run regression test suite from it
        !           201:        #!/bin/sh
        !           202: 
        !           203:        dirname=$1
        !           204:        arch=$(echo $dirname | cut -f 2 -d\-);
        !           205: 
        !           206:        buildslave_dir="/usr/devel/buildslave/"
        !           207:        obj_dir="${buildslave_dir}/obj/${dirname}"
        !           208:        anita_dir="${buildslave_dir}/anita/${dirname}"
        !           209:        release_dir="${obj_dir}/releasedir/${arch}/"
        !           210: 
        !           211:        echo "Running tests on ${arch}"
        !           212:        anita --workdir ${anita_dir} test ${release_dir}
        !           213: ## Buildslave setup
        !           214: 
        !           215: Buildslave is actual buildbot cluster worker which does worked scheduled to him by buildbot. Command used to created buildslave contains buildir [buildbot hostname:port] buildslave name password. Name and password must match those in master.cfg file. 
        !           216: 
        !           217:        As buildbot user:
        !           218:        buildslave create-slave /usr/devel/buildslave localhost:9989 buildslave buildslavepass
        !           219: 
        !           220: # Anita
        !           221: 
        !           222: ## Anita integration with buildbot

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb