Annotation of wikisrc/tutorials/continuous_building_and_testing_netbsd_with_buildbot.mdwn, revision 1.2
1.1 wiki 1: [[!meta title="Continuous NetBSD release building and testing "]]
2:
1.2 ! wiki 3: **Contents**
! 4:
1.1 wiki 5: [[!toc]]
6:
7: I decided to bought new computer and setup a continuous building service on it to always have fresh built binaries ready for usage and to be able to test my changes more easily. As continuous integration service I chose [buildbot](www.buildbot.net). Buildbot as package was located in pkgsrc wip and I have updated it to it's latest version 0.8.2 and introduced new package for building buildbot slave program.
8:
9:
10: # NetBSD setup
11:
12: I have created new Logical Volume and mounted it to /usr/devel directory. After that I have created new user called buildbot.
13:
14: As root:
15: useradd -m buildbot # Add buildbot user
16:
17: # Create Buildbot directories
18: mkdir /usr/devel/buildbot
19: mkdir /usr/devel/buildslave
20:
21: chown buildbot /usr/devel/buildbot /usr/devel/buildslave
22:
23: # Install Buildbot daemon and Buildbot slave
24: cd /usr/pkgsrc/wip/buildbot
25: make install
26:
27: cd /usr/pkgsrc/wip/buildslave
28: make install
29:
30: For My build setup it was needed to change /etc/login.conf and add builbot user to build group
31:
32: builder:\
33: :datasize-cur=1024M:\
34: :datasize-max=infinity:\
35: :maxproc-max=1000:\
36: :maxproc-cur=1000:\
37: :openfiles-cur=1024:
38:
39: usermod -L builder buildbot
40:
41: # Buildbot setup
42:
43: ## Buildmaster Setup
44:
45: First we need to initialise new buildmaster directory
46:
47: As buildbot user:
48: cd /usr/devel
49: buildbot create-master /usr/devel/buildmaster
50:
51: I have decided that for builds I would like to run maximum 2 parallel builds on one build slave client and I want to have system build once a day. After setup I used this configuration file to get buildmaster working.
52:
53:
54: # -*- python -*-
55: # ex: set syntax=python:
56:
57: c = BuildmasterConfig = {}
58:
59: from buildbot.buildslave import BuildSlave
60: c['slaves'] = [BuildSlave("bot1name", "bot1pass", max_builds=2)]
61:
62: c['slavePortnum'] = 9989
63:
64: from buildbot.changes.pb import PBChangeSource
65: c['change_source'] = PBChangeSource()
66:
67: from buildbot.scheduler import Scheduler
68: c['schedulers'] = []
69:
70: from buildbot.schedulers import timed
71: s = timed.Nightly(name='daily',
72: builderNames=["buildbot-netbsd-vanilla-i386", "buildbot-netbsd-vanilla-amd64"],
73: hour=13,
74: minute=0)
75:
76: c['schedulers'] = [s]
77:
78: cvsroot = ":pserver:anoncvs@anoncvs.netbsd.org:/cvsroot"
79: cvsmodule = "src"
80:
81: from buildbot.process import factory
82: from buildbot.steps.source import CVS
83: from buildbot.steps.shell import Compile
84: from buildbot.steps.shell import ShellCommand
85: from buildbot.steps.shell import Test
86: from buildbot.steps.python_twisted import Trial
87:
88: f1 = factory.BuildFactory()
89: f1.addStep(CVS(cvsroot=cvsroot, cvsmodule=cvsmodule, login="", mode="update"))
90: f1.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","i386"]))
91: f1.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","i386"],
92: warningPattern="^WWarning: "))
93: f1.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","i386"]))
94:
95: f2 = factory.BuildFactory()
96: f2.addStep(CVS(cvsroot=cvsroot, cvsmodule=cvsmodule, login="", mode="update"))
97: f2.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","amd64"]))
98: f2.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","amd64"],
99: warningPattern="^WWarning: "))
100: f2.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","amd64"]))
101:
102: f3 = factory.BuildFactory()
103: f3.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/getdev_src.sh", "dev-i386"]))
104: f3.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","dev-i386"]))
105: f3.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","dev-i386"],
106: warningPattern="^WWarning: "))
107: f3.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","dev-i386"]))
108:
109:
110: f4 = factory.BuildFactory()
111: f4.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/getdev_src.sh", "dev-amd64"]))
112: f4.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/clean.sh","dev-amd64"]))
113: f4.addStep(Compile(command=["/usr/devel/buildbot/bin/build.sh","dev-amd64"],
114: warningPattern="^WWarning: "))
115: f4.addStep(ShellCommand(command=["/usr/devel/buildbot/bin/test.sh","dev-amd64"]))
116:
117:
118: b1 = {'name': "buildbot-netbsd-vanilla-i386",
119: 'slavename': "bot1name",
120: 'builddir': "full",
121: 'factory': f1,
122: }
123:
124: b2 = {'name': "buildbot-netbsd-vanilla-amd64",
125: 'slavename': "bot1name",
126: 'builddir': "full-amd64",
127: 'factory': f2,
128: }
129:
130: b3 = {'name': "buildbot-netbsd-development-tree-i386",
131: 'slavename': "bot1name",
132: 'builddir': "dev-i386",
133: 'factory': f3,
134: }
135:
136: b4 = {'name': "buildbot-netbsd-development-tree-amd64",
137: 'slavename': "bot1name",
138: 'builddir': "dev-amd64",
139: 'factory': f4,
140: }
141:
142: c['builders'] = [b1, b2, b3, b4]
143: c['status'] = []
144:
145: from buildbot.status import html
146: from buildbot.status.web.authz import Authz
147:
148: authz = Authz(
149: forceBuild=True,
150: forceAllBuilds=True,
151: stopBuild=True,
152: stopAllBuilds=True,
153: cancelPendingBuild=True)
154: c['status'].append(html.WebStatus(http_port=8010, authz=authz))
155:
156: c['projectName'] = "NetBSD development daily builds"
157: c['projectURL'] = "www.netbsd.org/~haad/builds/"
158:
159: c['buildbotURL'] = "http://musasi.haad.chillisys.com:8010/"
160:
161: For easier builds and testing I have added 3 scripts to _buildbot-basedir_/bin/
162:
163: * clean.sh
164: * build.sh
165: * test.sh
166:
167: From master.cfg file you can see that all of these scripts are called with one argument which is basically {prefix}-{arch} where
168:
169: * {prefix} -> prefix name for different builds for same arch (developmant tree, vanilla tree, some netbsd branch etc.)
170: * {arch} -> architecture name used for build
171:
172: I have implemented one version of script for every architecture built by my buildbot server.
173:
174: Clean script is used to clean build dir on slave before build starts.
175:
176: #!/bin/sh
177:
178: dirname=$1
179: # get arch from dirname
180: arch=$(echo $dirname | cut -f 2 -d\-);
181:
182: buildslave_dir="/usr/devel/buildslave/"
183: obj_dir="${buildslave_dir}/obj/${dirname}"
184:
185: echo "Removing build artefacts from ${obj_dir}/*"
186: rm -rf ${obj_dir}/*
187:
188: Build builds whole system
189:
190: #!/bin/sh
191:
192: dirname=$1
193: j_flag=9
194: arch=$(echo $dirname | cut -f 2 -d\-);
195:
196: buildslave_dir="/usr/devel/buildslave/"
197: obj_dir="${buildslave_dir}/obj/${dirname}"
198:
199: echo "Building NetBSD tree for architecture ${arch} in $(pwd), objdir ${obj_dir}"
200: ./build.sh -V GMAKE_J_ARGS="-j ${j_flag}" -O ${obj_dir} -T ${obj_dir}/tooldir -Uu -m ${arch} -j ${j_flag} release
201:
202: Test will later run anita and run regression test suite from it
203: #!/bin/sh
204:
205: dirname=$1
206: arch=$(echo $dirname | cut -f 2 -d\-);
207:
208: buildslave_dir="/usr/devel/buildslave/"
209: obj_dir="${buildslave_dir}/obj/${dirname}"
210: anita_dir="${buildslave_dir}/anita/${dirname}"
211: release_dir="${obj_dir}/releasedir/${arch}/"
212:
213: echo "Running tests on ${arch}"
214: anita --workdir ${anita_dir} test ${release_dir}
215: ## Buildslave setup
216:
217: Buildslave is actual buildbot cluster worker which does worked scheduled to him by buildbot. Command used to created buildslave contains buildir [buildbot hostname:port] buildslave name password. Name and password must match those in master.cfg file.
218:
219: As buildbot user:
220: buildslave create-slave /usr/devel/buildslave localhost:9989 buildslave buildslavepass
221:
222: # Anita
223:
224: ## Anita integration with buildbot
CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb
![[BACK]](/icons/back.gif){kind=icon}
Return to continuous_building_and_testing_netbsd_with_buildbot.mdwn CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [NetBSD Developer Wiki] / wikisrc / tutorials