Annotation of wikisrc/tutorials/pkgsrc/pbulk.mdwn, revision 1.2

1.1       imil        1: ## Using pbulk to create a pkgsrc binary repository
                      2: 
                      3: *pkgtools/pbulk* package consists in a set of tools designed to ease
                      4: mass-building of *pkgsrc* packages, and the creation your own *pkgsrc* binaries
                      5: repository.
                      6: 
                      7: Its setup needs a bit of work, here is how to prepare and run your bulk-build
                      8: box. In this article we will only consider a one-node machine.
                      9: 
                     10: This documentation is based on the [pbulk-HOWTO](http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/doc/HOWTO-pbulk?rev=1.7&content-type=text/x-cvsweb-markup).
                     11: 
                     12: ### Prerequisites
                     13: 
                     14: These are the prerequisites needed by *pbulk*:
                     15: 
                     16: * A *pkgsrc* source tree
                     17: * A *src* source tree
                     18: * Possibly (not mandatory) a *xsrc* source tree
                     19: * Possibly (not mandatory) a tool like *misc/screen* as the full build process can takes a **very** long time.
                     20: 
                     21: For example, let's say you'd want to regularly build *pkgsrc-2011Q3* packages for *NetBSD 5.1* on your build box, you would prepare the following:
                     22: 
                     23:        # cd /usr
                     24:        # cvs -d anoncvs@anoncvs.netbsd.org:/cvsroot co -rnetbsd-5-1
                     25:        # cvs -d anoncvs@anoncvs.netbsd.org:/cvsroot co -rpkgsrc-2011Q3
                     26: 
                     27: And get sure those are always up-to-date by adding the following to a *crontab*:
                     28: 
                     29:        30 1 * * * cd /usr/pkgsrc && /usr/bin/cvs up -Pd >/dev/null 2>&1
                     30:        00 2 * * * cd /usr/src && /usr/bin/cvs up -Pd >/dev/null 2>&1
                     31: 
                     32: ### Prepare a chroot
                     33: 
                     34: In order to isolate the bulk build, it is advised you run all the operations
                     35: within a *chroot*. Running *pbulk* on your real environment would wipe all
                     36: of your installed packages, and would modify your base system with lots of
                     37: directories, users and groups you don't need.
                     38: 
                     39: Fortunately, a tool called *mksandbox* will simplify this process. *mksandbox*
                     40: is located in *mk/bulk/mksandbox*, and it is called like this:
                     41: 
                     42:        # mk/bulk/mksandbox [optionnal flags] /path/to/sandbox
                     43: 
                     44: For example, to create a sandbox in */home/bulk* without the X11 system, run:
                     45: 
                     46:        # mk/bulk/mksandbox --without-x /home/bulk
                     47: 
                     48: This command will prepare and mount most of the needed directories, and will
                     49: place a shell script on top of the sandbox filesystem called *sandbox*. This
                     50: script is used to *mount*/*umount* your sandbox. It is a good idea to add
                     51: */var/spool* to the list of directories mounted as read/write in your sandbox
                     52: so the email report is actually sent. Simply add:
                     53: 
                     54:        /var/spool /var/spool rw \
                     55: 
                     56: to the list of directories in the *sandbox* script. *sandbox* script use is
                     57: really straightforward:
1.2     ! imil       58: 
1.1       imil       59:        # /path/to/your/sandbox/sandbox umount
1.2     ! imil       60: 
1.1       imil       61: Unmounts the sandbox
1.2     ! imil       62: 
1.1       imil       63:        # /path/to/your/sandbox/sandbox mount
1.2     ! imil       64: 
1.1       imil       65: Mounts the sandbox
                     66: 
                     67: ### Prepare the pbulk environment
                     68: 
                     69: Now that our sandbox is available and mounted, we will chroot to it:
                     70: 
                     71:        # chroot /home/bulk /bin/ksh
                     72: 
                     73: As the [pbulk-HOWTO](http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/doc/HOWTO-pbulk?rev=1.7&content-type=text/x-cvsweb-markup) suggests, we will create some helpers directories to keep things clean:
                     74: 
                     75:        # mkdir /bulklog # bulk meta data and log files of each package
                     76:        # mkdir /scratch # WRKOBJDIR
                     77:        # mkdir /distfiles # DISTDIR
                     78:        # mkdir /packages # PACKAGES
                     79: 
                     80: The */scratch* directory must be writable by a *bulk* user defined below in */usr/pkg_bulk/etc/pbulk.conf* as the build is done using DESTDIR.
                     81: 
                     82:        # useradd bulk
                     83:        # chown bulk /scratch
                     84: 
                     85: It is now time to bootstrap (yes, even on NetBSD) a *pbulk* environment so the latter is not wiped out when a new bulk build is started:
                     86: 
                     87:        # cd /usr/pkgsrc/bootstrap
                     88:        # ./bootstrap --prefix=/usr/pkg_bulk --pkgdbdir=/usr/pkg_bulk/.pkgdb
                     89:        # rm -rf work
                     90: 
                     91: Then install *pbulk* itself on this newly created separate prefix:
                     92: 
                     93:        # cd /usr/pkgsrc/pkgtools/pbulk
                     94:        # env PATH=/usr/pkg_bulk/bin:/usr/pkg_bulk/sbin:${PATH} bmake install
                     95: 
                     96: ### Configuring pbulk
                     97: 
                     98: *pbulk* configuration file is */usr/pkg_bulk/etc/pbulk.conf*, here are
                     99: some parameters you should modify to customize the run to your needs:
                    100: 
                    101:        base_url= # should point to the URL where the txt report will be
                    102:        master_mode=no # this will not be a distributed build
                    103:        # [...]
                    104:        report_recipients="your@email.com"
                    105:        # [...]
                    106:        #bootstrapkit= must be commented on NetBSD !
                    107:        unprivileged_user=bulk # used for unprivileged user-destdir builds
                    108:        # [...]
                    109:        bulklog=/bulklog
                    110:        packages=/packages
                    111:        prefix=/usr/pkg
                    112:        pkgsrc=/usr/pkgsrc
                    113:        pkgdb=/var/db/pkg
                    114:        varbase=/var
                    115:        # [...]
                    116:        make=/usr/bin/make # must be modified on NetBSD !
                    117: 
                    118: ### Configuring the build
                    119: 
                    120: Last but not least, you should configure your */etc/mk.conf* (in the *chroot* !).
                    121: Here's an example of what we use at [NetBSDfr](http://www.netbsdfr.org/):
                    122: 
                    123:        WRKOBJDIR=              /scratch
                    124:        PKGSRCDIR=              /usr/pkgsrc
                    125:        DISTDIR=                /distfiles
                    126:        PACKAGES=               /packages
                    127: 
                    128:        FAILOVER_FETCH=         yes
                    129: 
                    130:        X11_TYPE=               modular
                    131: 
                    132:        SKIP_LICENSE_CHECK=             yes
                    133:        ALLOW_VULNERABLE_PACKAGES=      yes
                    134: 
                    135:        PKG_DEVELOPER?=         yes
                    136: 
                    137:        PKG_OPTIONS.irssi=      perl inet6
                    138:        PKG_OPTIONS.mplayer=    oss
                    139: 
                    140:        DSPAM_STORAGE_DRIVER=   mysql
                    141:        PKG_OPTIONS.dspam+=     graphs
                    142:        PKG_OPTIONS.dovecot=    ssl ldap dovecot-sieve dovecot-managesieve
                    143:        PKG_OPTIONS.nagios-nrpe=ssl tcpwrappers
                    144: 
                    145: ### Running the build
                    146: 
                    147: Now that everything's in place, we can fire up the build from the *chroot* using
                    148: the following command:
                    149: 
                    150:        # /usr/pkg_bulk/bin/bulkbuild
                    151: 
                    152: It is recommended to run the build inside a tool like *misc/screen* or
                    153: *misc/tmux* as it will take a lot of time.
                    154: 
                    155: If the build is stopped, it is possible to restart it by invoking:
                    156: 
                    157:        # /usr/pkg_bulk/bin/bulkbuild-restart
                    158: 
                    159: ### Hints
                    160: 
                    161: If you'd like to restart the build to retry building some failing packages, **empty** */meta/error* (don't delete it):
                    162: 
                    163:        # cat > /meta/error << EOF
                    164:        EOF
                    165: 
                    166: If you'd like to rebuild a single package, use the *bulkbuild-rebuild* command followed by the package name.

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb