Contents
I decided to bought new computer and setup a continuous building service on it to always have fresh built binaries ready for usage and to be able to test my changes more easily. As continuous integration service I chose buildbot. Buildbot as package was located in pkgsrc wip and I have updated it to it's latest version 0.8.2 and introduced new package for building buildbot slave program.
NetBSD setup
I have created new Logical Volume and mounted it to /usr/devel directory. After that I have created new user called buildbot.
As root:
useradd -m buildbot # Add buildbot user
# Create Buildbot directories
mkdir /usr/devel/buildbot
mkdir /usr/devel/buildslave
chown buildbot /usr/devel/buildbot /usr/devel/buildslave
# Install Buildbot daemon and Buildbot slave
cd /usr/pkgsrc/wip/buildbot
make install
cd /usr/pkgsrc/wip/buildslave
make install
For My build setup it was needed to change /etc/login.conf and add builbot user to build group
builder:\
:datasize-cur=1024M:\
:datasize-max=infinity:\
:maxproc-max=1000:\
:maxproc-cur=1000:\
:openfiles-cur=1024:
usermod -L builder buildbot
Buildbot setup
Buildmaster Setup
First we need to initialise new buildmaster directory
As buildbot user:
cd /usr/devel
buildbot create-master /usr/devel/buildmaster
I have decided that for builds I would like to run maximum 2 parallel builds on one build slave client and I want to have system build once a day. After setup I used this configuration file to get buildmaster working.
# -*- python -*-
# ex: set syntax=python:
c = BuildmasterConfig = {}
from buildbot.buildslave import BuildSlave
c['slaves'] = [BuildSlave("bot1name", "bot1pass", max_builds=2)]
c['slavePortnum'] = 9989
from buildbot.changes.pb import PBChangeSource
c['change_source'] = PBChangeSource()
from buildbot.scheduler import Scheduler
c['schedulers'] = []
from buildbot.schedulers import timed
s = timed.Nightly(name='daily',
builderNames=["buildbot-netbsd-vanilla-i386", "buildbot-netbsd-vanilla-amd64"],
hour=13,
minute=0)
c['schedulers'] = [s]
cvsroot = ":pserver:anoncvs@anoncvs.netbsd.org:/cvsroot"
cvsmodule = "src"
from buildbot.process import factory
from buildbot.steps.source import CVS
from buildbot.steps.shell import Compile
from buildbot.steps.shell import ShellCommand
from buildbot.steps.shell import Test
from buildbot.steps.python_twisted import Trial
f1 = factory.BuildFactory()
f1.addStep(CVS(cvsroot=cvsroot, cvsmodule=cvsmodule, login="", mode="update"))
f1.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","i386"]))
f1.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","i386"],
warningPattern="^WWarning: "))
f1.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","i386"]))
f2 = factory.BuildFactory()
f2.addStep(CVS(cvsroot=cvsroot, cvsmodule=cvsmodule, login="", mode="update"))
f2.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","amd64"]))
f2.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","amd64"],
warningPattern="^WWarning: "))
f2.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","amd64"]))
f3 = factory.BuildFactory()
f3.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/getdev_src.sh", "dev-i386"]))
f3.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","dev-i386"]))
f3.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","dev-i386"],
warningPattern="^WWarning: "))
f3.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","dev-i386"]))
f4 = factory.BuildFactory()
f4.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/getdev_src.sh", "dev-amd64"]))
f4.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","dev-amd64"]))
f4.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","dev-amd64"],
warningPattern="^WWarning: "))
f4.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","dev-amd64"]))
b1 = {'name': "buildbot-netbsd-vanilla-i386",
'slavename': "bot1name",
'builddir': "full",
'factory': f1,
}
b2 = {'name': "buildbot-netbsd-vanilla-amd64",
'slavename': "bot1name",
'builddir': "full-amd64",
'factory': f2,
}
b3 = {'name': "buildbot-netbsd-development-tree-i386",
'slavename': "bot1name",
'builddir': "dev-i386",
'factory': f3,
}
b4 = {'name': "buildbot-netbsd-development-tree-amd64",
'slavename': "bot1name",
'builddir': "dev-amd64",
'factory': f4,
}
c['builders'] = [b1, b2, b3, b4]
c['status'] = []
from buildbot.status import html
from buildbot.status.web.authz import Authz
authz = Authz(
forceBuild=True,
forceAllBuilds=True,
stopBuild=True,
stopAllBuilds=True,
cancelPendingBuild=True)
c['status'].append(html.WebStatus(http_port=8010, authz=authz))
c['projectName'] = "NetBSD development daily builds"
c['projectURL'] = "www.netbsd.org/~haad/builds/"
c['buildbotURL'] = "http://musasi.haad.chillisys.com:8010/"
For easier builds and testing I have added 3 scripts to buildbot-basedir/bin/
- clean.sh
- build.sh
- test.sh
From master.cfg file you can see that all of these scripts are called with one argument which is basically {prefix}-{arch} where
- {prefix} -> prefix name for different builds for same arch (developmant tree, vanilla tree, some netbsd branch etc.)
- {arch} -> architecture name used for build
I have implemented one version of script for every architecture built by my buildbot server.
Clean script is used to clean build dir on slave before build starts.
#!/bin/sh
dirname=$1
# get arch from dirname
arch=$(echo $dirname | cut -f 2 -d\-);
buildslave_dir="/usr/devel/buildslave/"
obj_dir="${buildslave_dir}/obj/${dirname}"
echo "Removing build artefacts from ${obj_dir}/*"
rm -rf ${obj_dir}/*
Build builds whole system
#!/bin/sh
dirname=$1
j_flag=9
arch=$(echo $dirname | cut -f 2 -d\-);
buildslave_dir="/usr/devel/buildslave/"
obj_dir="${buildslave_dir}/obj/${dirname}"
echo "Building NetBSD tree for architecture ${arch} in $(pwd), objdir ${obj_dir}"
./build.sh -V GMAKE_J_ARGS="-j ${j_flag}" -O ${obj_dir} -T ${obj_dir}/tooldir -Uu -m ${arch} -j ${j_flag} release
Test will later run anita and run regression test suite from it
#!/bin/sh
dirname=$1
arch=$(echo $dirname | cut -f 2 -d\-);
buildslave_dir="/usr/devel/buildslave/"
obj_dir="${buildslave_dir}/obj/${dirname}"
anita_dir="${buildslave_dir}/anita/${dirname}"
release_dir="${obj_dir}/releasedir/${arch}/"
echo "Running tests on ${arch}"
anita --workdir ${anita_dir} test ${release_dir}
Buildslave setup
Buildslave is actual buildbot cluster worker which does worked scheduled to him by buildbot. Command used to created buildslave contains buildir [buildbot hostname:port] buildslave name password. Name and password must match those in master.cfg file.
As buildbot user:
buildslave create-slave /usr/devel/buildslave localhost:9989 buildslave buildslavepass