Contents

  1. NetBSD setup
  2. Buildbot setup
  3. Anita

I decided to bought new computer and setup a continuous building service on it to always have fresh built binaries ready for usage and to be able to test my changes more easily. As continuous integration service I chose buildbot. Buildbot as package was located in pkgsrc wip and I have updated it to it's latest version 0.8.2 and introduced new package for building buildbot slave program.

NetBSD setup

I have created new Logical Volume and mounted it to /usr/devel directory. After that I have created new user called buildbot.

As root:
useradd -m buildbot # Add buildbot user

# Create Buildbot directories
mkdir /usr/devel/buildbot
mkdir /usr/devel/buildslave

chown buildbot /usr/devel/buildbot /usr/devel/buildslave

# Install Buildbot daemon and Buildbot slave
cd /usr/pkgsrc/wip/buildbot
make install

cd /usr/pkgsrc/wip/buildslave
make install

For My build setup it was needed to change /etc/login.conf and add builbot user to build group

builder:\
    :datasize-cur=1024M:\
    :datasize-max=infinity:\
    :maxproc-max=1000:\
    :maxproc-cur=1000:\
    :openfiles-cur=1024:

usermod -L builder buildbot

Buildbot setup

Buildmaster Setup

First we need to initialise new buildmaster directory

As buildbot user:
cd /usr/devel
buildbot create-master /usr/devel/buildmaster

I have decided that for builds I would like to run maximum 2 parallel builds on one build slave client and I want to have system build once a day. After setup I used this configuration file to get buildmaster working.

# -*- python -*-
# ex: set syntax=python:

c = BuildmasterConfig = {}

from buildbot.buildslave import BuildSlave
c['slaves'] = [BuildSlave("bot1name", "bot1pass", max_builds=2)]

c['slavePortnum'] = 9989

from buildbot.changes.pb import PBChangeSource
c['change_source'] = PBChangeSource()

from buildbot.scheduler import Scheduler
c['schedulers'] = []

from buildbot.schedulers import timed
s = timed.Nightly(name='daily',
               builderNames=["buildbot-netbsd-vanilla-i386", "buildbot-netbsd-vanilla-amd64"],  
                       hour=13,
                       minute=0)

c['schedulers'] = [s]

cvsroot = ":pserver:anoncvs@anoncvs.netbsd.org:/cvsroot"
cvsmodule = "src"

from buildbot.process import factory
from buildbot.steps.source import CVS
from buildbot.steps.shell import Compile
from buildbot.steps.shell import ShellCommand
from buildbot.steps.shell import Test
from buildbot.steps.python_twisted import Trial

f1 = factory.BuildFactory()
f1.addStep(CVS(cvsroot=cvsroot, cvsmodule=cvsmodule, login="", mode="update"))
f1.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","i386"]))
f1.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","i386"], 
                 warningPattern="^WWarning: "))
f1.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","i386"]))

f2 = factory.BuildFactory()
f2.addStep(CVS(cvsroot=cvsroot, cvsmodule=cvsmodule, login="", mode="update"))
f2.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","amd64"]))
f2.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","amd64"], 
                 warningPattern="^WWarning: "))
f2.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","amd64"]))

f3 = factory.BuildFactory()
f3.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/getdev_src.sh", "dev-i386"]))
f3.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","dev-i386"]))
f3.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","dev-i386"],
                                 warningPattern="^WWarning: "))
f3.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","dev-i386"]))


f4 = factory.BuildFactory()
f4.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/getdev_src.sh", "dev-amd64"]))
f4.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","dev-amd64"]))
f4.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","dev-amd64"],
                                 warningPattern="^WWarning: "))
f4.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","dev-amd64"]))


b1 = {'name': "buildbot-netbsd-vanilla-i386",
      'slavename': "bot1name",
      'builddir': "full",
      'factory': f1,
      }

b2 = {'name': "buildbot-netbsd-vanilla-amd64",
      'slavename': "bot1name",
      'builddir': "full-amd64",
      'factory': f2,
      }

b3 = {'name': "buildbot-netbsd-development-tree-i386",
      'slavename': "bot1name", 
      'builddir': "dev-i386",
      'factory': f3,
      }

b4 = {'name': "buildbot-netbsd-development-tree-amd64",
      'slavename': "bot1name",
      'builddir': "dev-amd64",
      'factory': f4,
      }

c['builders'] = [b1, b2, b3, b4]
c['status'] = []

from buildbot.status import html
from buildbot.status.web.authz import Authz

authz = Authz(
         forceBuild=True,
         forceAllBuilds=True, 
         stopBuild=True,
         stopAllBuilds=True,
         cancelPendingBuild=True)
c['status'].append(html.WebStatus(http_port=8010, authz=authz))

c['projectName'] = "NetBSD development daily builds"
c['projectURL'] = "www.netbsd.org/~haad/builds/"

c['buildbotURL'] = "http://musasi.haad.chillisys.com:8010/"

For easier builds and testing I have added 3 scripts to buildbot-basedir/bin/

From master.cfg file you can see that all of these scripts are called with one argument which is basically {prefix}-{arch} where

I have implemented one version of script for every architecture built by my buildbot server.

Clean script is used to clean build dir on slave before build starts.

#!/bin/sh

dirname=$1
# get arch from dirname
arch=$(echo $dirname | cut -f 2 -d\-);

buildslave_dir="/usr/devel/buildslave/"
obj_dir="${buildslave_dir}/obj/${dirname}"

echo "Removing build artefacts from ${obj_dir}/*"
rm -rf ${obj_dir}/*

Build builds whole system 

#!/bin/sh

dirname=$1
j_flag=9
arch=$(echo $dirname | cut -f 2 -d\-);

buildslave_dir="/usr/devel/buildslave/"
obj_dir="${buildslave_dir}/obj/${dirname}"

echo "Building NetBSD tree for architecture ${arch} in $(pwd), objdir ${obj_dir}"
./build.sh -V GMAKE_J_ARGS="-j ${j_flag}" -O ${obj_dir} -T ${obj_dir}/tooldir -Uu -m ${arch} -j ${j_flag} release

Test will later run anita and run regression test suite from it
#!/bin/sh

dirname=$1
arch=$(echo $dirname | cut -f 2 -d\-);

buildslave_dir="/usr/devel/buildslave/"
obj_dir="${buildslave_dir}/obj/${dirname}"
anita_dir="${buildslave_dir}/anita/${dirname}"
release_dir="${obj_dir}/releasedir/${arch}/"

echo "Running tests on ${arch}"
anita --workdir ${anita_dir} test ${release_dir}

Buildslave setup

Buildslave is actual buildbot cluster worker which does worked scheduled to him by buildbot. Command used to created buildslave contains buildir [buildbot hostname:port] buildslave name password. Name and password must match those in master.cfg file.

As buildbot user:
buildslave create-slave /usr/devel/buildslave localhost:9989 buildslave buildslavepass

Anita

Anita integration with buildbot

Add a comment