File:  [NetBSD Developer Wiki] / wikisrc / tutorials / continuous_building_and_testing_netbsd_with_buildbot.mdwn
Revision 1.2: download - view: text, annotated - select for diffs
Sat Feb 16 12:35:36 2013 UTC (7 years, 9 months ago) by wiki
Branches: MAIN
CVS tags: HEAD
web commit by jdf: Add `**Contents**` prior to TOC

    1: [[!meta title="Continuous NetBSD release building and testing "]]
    2: 
    3: **Contents**
    4: 
    5: [[!toc]]
    6: 
    7: I decided to bought new computer and setup a continuous building service on it to always have fresh built binaries ready for usage and to be able to test my changes more easily. As continuous integration service I chose [buildbot](www.buildbot.net). Buildbot as package was located in pkgsrc wip and I have updated it to it's latest version 0.8.2 and introduced new package for building buildbot slave program.
    8: 
    9: 
   10: # NetBSD setup
   11: 
   12: I have created new Logical Volume and mounted it to /usr/devel directory. After that I have created new user called buildbot. 
   13: 
   14:     As root:
   15:     useradd -m buildbot # Add buildbot user
   16: 
   17: 	# Create Buildbot directories
   18:     mkdir /usr/devel/buildbot
   19:     mkdir /usr/devel/buildslave
   20: 
   21:     chown buildbot /usr/devel/buildbot /usr/devel/buildslave
   22: 
   23: 	# Install Buildbot daemon and Buildbot slave
   24: 	cd /usr/pkgsrc/wip/buildbot
   25: 	make install
   26: 
   27: 	cd /usr/pkgsrc/wip/buildslave
   28: 	make install
   29: 
   30: For My build setup it was needed to change /etc/login.conf and add builbot user to build group
   31: 
   32: 	builder:\
   33: 		:datasize-cur=1024M:\
   34: 		:datasize-max=infinity:\
   35: 		:maxproc-max=1000:\
   36: 		:maxproc-cur=1000:\
   37: 		:openfiles-cur=1024:
   38: 	
   39: 	usermod -L builder buildbot
   40: 
   41: # Buildbot setup
   42: 
   43: ## Buildmaster Setup
   44: 
   45: First we need to initialise new buildmaster directory 
   46: 
   47: 	As buildbot user:
   48: 	cd /usr/devel
   49: 	buildbot create-master /usr/devel/buildmaster
   50: 
   51: I have decided that for builds I would like to run maximum 2 parallel builds on one build slave client and I want to have system build once a day. After setup I used this configuration file to get buildmaster working.
   52: 
   53: 
   54: 	# -*- python -*-
   55: 	# ex: set syntax=python:
   56: 	
   57: 	c = BuildmasterConfig = {}
   58: 
   59: 	from buildbot.buildslave import BuildSlave
   60: 	c['slaves'] = [BuildSlave("bot1name", "bot1pass", max_builds=2)]
   61: 
   62: 	c['slavePortnum'] = 9989
   63: 
   64: 	from buildbot.changes.pb import PBChangeSource
   65: 	c['change_source'] = PBChangeSource()
   66: 
   67: 	from buildbot.scheduler import Scheduler
   68: 	c['schedulers'] = []
   69: 
   70: 	from buildbot.schedulers import timed
   71: 	s = timed.Nightly(name='daily',
   72: 	               builderNames=["buildbot-netbsd-vanilla-i386", "buildbot-netbsd-vanilla-amd64"],  
   73: 					       hour=13,
   74: 	        		       minute=0)
   75: 
   76: 	c['schedulers'] = [s]
   77: 
   78: 	cvsroot = ":pserver:anoncvs@anoncvs.netbsd.org:/cvsroot"
   79: 	cvsmodule = "src"
   80: 
   81: 	from buildbot.process import factory
   82: 	from buildbot.steps.source import CVS
   83: 	from buildbot.steps.shell import Compile
   84: 	from buildbot.steps.shell import ShellCommand
   85: 	from buildbot.steps.shell import Test
   86: 	from buildbot.steps.python_twisted import Trial
   87: 
   88: 	f1 = factory.BuildFactory()
   89: 	f1.addStep(CVS(cvsroot=cvsroot, cvsmodule=cvsmodule, login="", mode="update"))
   90: 	f1.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","i386"]))
   91: 	f1.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","i386"], 
   92: 					 warningPattern="^WWarning: "))
   93: 	f1.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","i386"]))
   94: 
   95: 	f2 = factory.BuildFactory()
   96: 	f2.addStep(CVS(cvsroot=cvsroot, cvsmodule=cvsmodule, login="", mode="update"))
   97: 	f2.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","amd64"]))
   98: 	f2.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","amd64"], 
   99: 					 warningPattern="^WWarning: "))
  100: 	f2.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","amd64"]))
  101: 
  102: 	f3 = factory.BuildFactory()
  103: 	f3.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/getdev_src.sh", "dev-i386"]))
  104: 	f3.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","dev-i386"]))
  105: 	f3.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","dev-i386"],
  106: 	                                 warningPattern="^WWarning: "))
  107: 	f3.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","dev-i386"]))
  108: 
  109: 
  110: 	f4 = factory.BuildFactory()
  111: 	f4.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/getdev_src.sh", "dev-amd64"]))
  112: 	f4.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","dev-amd64"]))
  113: 	f4.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","dev-amd64"],
  114: 	                                 warningPattern="^WWarning: "))
  115: 	f4.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","dev-amd64"]))
  116: 
  117: 
  118: 	b1 = {'name': "buildbot-netbsd-vanilla-i386",
  119: 	      'slavename': "bot1name",
  120: 	      'builddir': "full",
  121: 	      'factory': f1,
  122: 	      }
  123: 
  124: 	b2 = {'name': "buildbot-netbsd-vanilla-amd64",
  125: 	      'slavename': "bot1name",
  126: 	      'builddir': "full-amd64",
  127: 	      'factory': f2,
  128: 	      }
  129: 
  130: 	b3 = {'name': "buildbot-netbsd-development-tree-i386",
  131: 	      'slavename': "bot1name", 
  132: 	      'builddir': "dev-i386",
  133: 	      'factory': f3,
  134: 	      }
  135: 
  136: 	b4 = {'name': "buildbot-netbsd-development-tree-amd64",
  137: 	      'slavename': "bot1name",
  138: 	      'builddir': "dev-amd64",
  139: 	      'factory': f4,
  140: 	      }
  141: 
  142: 	c['builders'] = [b1, b2, b3, b4]
  143: 	c['status'] = []
  144: 
  145: 	from buildbot.status import html
  146: 	from buildbot.status.web.authz import Authz
  147: 
  148: 	authz = Authz(
  149: 	         forceBuild=True,
  150: 			 forceAllBuilds=True, 
  151: 			 stopBuild=True,
  152: 			 stopAllBuilds=True,
  153: 			 cancelPendingBuild=True)
  154: 	c['status'].append(html.WebStatus(http_port=8010, authz=authz))
  155: 
  156: 	c['projectName'] = "NetBSD development daily builds"
  157: 	c['projectURL'] = "www.netbsd.org/~haad/builds/"
  158: 
  159: 	c['buildbotURL'] = "http://musasi.haad.chillisys.com:8010/"
  160: 
  161: For easier builds and testing I have added 3 scripts to _buildbot-basedir_/bin/
  162: 
  163: * clean.sh
  164: * build.sh
  165: * test.sh
  166: 
  167: From master.cfg file you can see that all of these scripts are called with one argument which is basically {prefix}-{arch} where 
  168: 
  169: * {prefix} -> prefix name for different builds for same arch (developmant tree, vanilla tree, some netbsd branch etc.)
  170: * {arch} -> architecture name used for build
  171: 
  172: I have implemented one version of script for every architecture built by my buildbot server.
  173: 
  174: 	Clean script is used to clean build dir on slave before build starts.
  175: 
  176: 	#!/bin/sh
  177: 
  178: 	dirname=$1
  179: 	# get arch from dirname
  180: 	arch=$(echo $dirname | cut -f 2 -d\-);
  181: 
  182: 	buildslave_dir="/usr/devel/buildslave/"
  183: 	obj_dir="${buildslave_dir}/obj/${dirname}"
  184: 
  185: 	echo "Removing build artefacts from ${obj_dir}/*"
  186: 	rm -rf ${obj_dir}/*
  187: 
  188: 	Build builds whole system 
  189: 
  190: 	#!/bin/sh
  191: 
  192: 	dirname=$1
  193: 	j_flag=9
  194: 	arch=$(echo $dirname | cut -f 2 -d\-);
  195: 
  196: 	buildslave_dir="/usr/devel/buildslave/"
  197: 	obj_dir="${buildslave_dir}/obj/${dirname}"
  198: 
  199: 	echo "Building NetBSD tree for architecture ${arch} in $(pwd), objdir ${obj_dir}"
  200: 	./build.sh -V GMAKE_J_ARGS="-j ${j_flag}" -O ${obj_dir} -T ${obj_dir}/tooldir -Uu -m ${arch} -j ${j_flag} release
  201: 
  202: 	Test will later run anita and run regression test suite from it
  203: 	#!/bin/sh
  204: 
  205: 	dirname=$1
  206: 	arch=$(echo $dirname | cut -f 2 -d\-);
  207: 
  208: 	buildslave_dir="/usr/devel/buildslave/"
  209: 	obj_dir="${buildslave_dir}/obj/${dirname}"
  210: 	anita_dir="${buildslave_dir}/anita/${dirname}"
  211: 	release_dir="${obj_dir}/releasedir/${arch}/"
  212: 
  213: 	echo "Running tests on ${arch}"
  214: 	anita --workdir ${anita_dir} test ${release_dir}
  215: ## Buildslave setup
  216: 
  217: Buildslave is actual buildbot cluster worker which does worked scheduled to him by buildbot. Command used to created buildslave contains buildir [buildbot hostname:port] buildslave name password. Name and password must match those in master.cfg file. 
  218: 
  219: 	As buildbot user:
  220: 	buildslave create-slave /usr/devel/buildslave localhost:9989 buildslave buildslavepass
  221: 
  222: # Anita
  223: 
  224: ## Anita integration with buildbot

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb