Annotation of wikisrc/tutorials/pkgsrc/pbulk.mdwn, revision 1.8

1.1       imil        1: ## Using pbulk to create a pkgsrc binary repository
                      2: 
                      3: *pkgtools/pbulk* package consists in a set of tools designed to ease
                      4: mass-building of *pkgsrc* packages, and the creation your own *pkgsrc* binaries
                      5: repository.
                      6: 
                      7: Its setup needs a bit of work, here is how to prepare and run your bulk-build
                      8: box. In this article we will only consider a one-node machine.
                      9: 
                     10: This documentation is based on the [pbulk-HOWTO](http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/doc/HOWTO-pbulk?rev=1.7&content-type=text/x-cvsweb-markup).
                     11: 
                     12: ### Prerequisites
                     13: 
                     14: These are the prerequisites needed by *pbulk*:
                     15: 
                     16: * A *pkgsrc* source tree
1.4       imil       17: * Possibly a *src* source tree, only some rare packages needs it. If you choose not to checkout *src*, simply create a */usr/src* directory so *mksandbox* (see below) doesn't complain about non-existing directory.
1.6       imil       18: * Possibly (not mandatory) a tool like *misc/screen* or *misc/tmux* as the full build process can take a **very** long time.
1.1       imil       19: 
1.8     ! imil       20: For example, to prepare a *pkgsrc-2011Q3* bulk build:
1.1       imil       21: 
                     22:        # cd /usr
1.4       imil       23:        # cvs -d anoncvs@anoncvs.netbsd.org:/cvsroot co -rpkgsrc-2011Q3 pkgsrc
1.1       imil       24: 
1.3       imil       25: Avoid automatic update of *pkgsrc* tree (cron or such), if you're in the middle of a build, it could lead to unfortunate results.
1.1       imil       26: 
                     27: ### Prepare a chroot
                     28: 
                     29: In order to isolate the bulk build, it is advised you run all the operations
                     30: within a *chroot*. Running *pbulk* on your real environment would wipe all
                     31: of your installed packages, and would modify your base system with lots of
                     32: directories, users and groups you don't need.
                     33: 
                     34: Fortunately, a tool called *mksandbox* will simplify this process. *mksandbox*
                     35: is located in *mk/bulk/mksandbox*, and it is called like this:
                     36: 
                     37:        # mk/bulk/mksandbox [optionnal flags] /path/to/sandbox
                     38: 
                     39: For example, to create a sandbox in */home/bulk* without the X11 system, run:
                     40: 
                     41:        # mk/bulk/mksandbox --without-x /home/bulk
                     42: 
                     43: This command will prepare and mount most of the needed directories, and will
                     44: place a shell script on top of the sandbox filesystem called *sandbox*. This
                     45: script is used to *mount*/*umount* your sandbox. It is a good idea to add
                     46: */var/spool* to the list of directories mounted as read/write in your sandbox
                     47: so the email report is actually sent. Simply add:
                     48: 
                     49:        /var/spool /var/spool rw \
                     50: 
                     51: to the list of directories in the *sandbox* script. *sandbox* script use is
                     52: really straightforward:
1.2       imil       53: 
1.1       imil       54:        # /path/to/your/sandbox/sandbox umount
1.2       imil       55: 
1.1       imil       56: Unmounts the sandbox
1.2       imil       57: 
1.1       imil       58:        # /path/to/your/sandbox/sandbox mount
1.2       imil       59: 
1.1       imil       60: Mounts the sandbox
                     61: 
                     62: ### Prepare the pbulk environment
                     63: 
                     64: Now that our sandbox is available and mounted, we will chroot to it:
                     65: 
                     66:        # chroot /home/bulk /bin/ksh
                     67: 
                     68: As the [pbulk-HOWTO](http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/doc/HOWTO-pbulk?rev=1.7&content-type=text/x-cvsweb-markup) suggests, we will create some helpers directories to keep things clean:
                     69: 
                     70:        # mkdir /bulklog # bulk meta data and log files of each package
                     71:        # mkdir /scratch # WRKOBJDIR
                     72:        # mkdir /distfiles # DISTDIR
                     73:        # mkdir /packages # PACKAGES
                     74: 
                     75: The */scratch* directory must be writable by a *bulk* user defined below in */usr/pkg_bulk/etc/pbulk.conf* as the build is done using DESTDIR.
                     76: 
                     77:        # useradd bulk
                     78:        # chown bulk /scratch
                     79: 
                     80: It is now time to bootstrap (yes, even on NetBSD) a *pbulk* environment so the latter is not wiped out when a new bulk build is started:
                     81: 
                     82:        # cd /usr/pkgsrc/bootstrap
                     83:        # ./bootstrap --prefix=/usr/pkg_bulk --pkgdbdir=/usr/pkg_bulk/.pkgdb
                     84:        # rm -rf work
                     85: 
1.3       imil       86: Add the line:
                     87: 
1.7       imil       88:        PACKAGES=       /scratch/pbulk_packages
1.3       imil       89: 
                     90: to *pbulk*'s prefix *mk.conf* (*/usr/pkg_bulk/etc/mk.conf*).
                     91: 
1.1       imil       92: Then install *pbulk* itself on this newly created separate prefix:
                     93: 
                     94:        # cd /usr/pkgsrc/pkgtools/pbulk
                     95:        # env PATH=/usr/pkg_bulk/bin:/usr/pkg_bulk/sbin:${PATH} bmake install
                     96: 
                     97: ### Configuring pbulk
                     98: 
                     99: *pbulk* configuration file is */usr/pkg_bulk/etc/pbulk.conf*, here are
                    100: some parameters you should modify to customize the run to your needs:
                    101: 
                    102:        base_url= # should point to the URL where the txt report will be
                    103:        master_mode=no # this will not be a distributed build
                    104:        # [...]
                    105:        report_recipients="your@email.com"
                    106:        # [...]
                    107:        #bootstrapkit= must be commented on NetBSD !
                    108:        unprivileged_user=bulk # used for unprivileged user-destdir builds
                    109:        # [...]
                    110:        bulklog=/bulklog
                    111:        packages=/packages
                    112:        prefix=/usr/pkg
                    113:        pkgsrc=/usr/pkgsrc
                    114:        pkgdb=/var/db/pkg
                    115:        varbase=/var
                    116:        # [...]
                    117:        make=/usr/bin/make # must be modified on NetBSD !
                    118: 
1.3       imil      119: Also, in order to avoid hangs, it might me a good idea to add the following to the top of *pbulk.conf*
                    120: 
                    121:        ulimit -t 1800 # set the limit on CPU time (in seconds)
                    122:        ulimit -v 2097152 # limits process address space
                    123: 
1.1       imil      124: ### Configuring the build
                    125: 
                    126: Last but not least, you should configure your */etc/mk.conf* (in the *chroot* !).
                    127: Here's an example of what we use at [NetBSDfr](http://www.netbsdfr.org/):
                    128: 
                    129:        WRKOBJDIR=              /scratch
                    130:        PKGSRCDIR=              /usr/pkgsrc
                    131:        DISTDIR=                /distfiles
                    132:        PACKAGES=               /packages
                    133: 
                    134:        FAILOVER_FETCH=         yes
                    135: 
                    136:        SKIP_LICENSE_CHECK=             yes
                    137:        ALLOW_VULNERABLE_PACKAGES=      yes
                    138: 
                    139:        PKG_DEVELOPER?=         yes
                    140: 
1.3       imil      141:        # site specific changes
                    142: 
1.1       imil      143:        PKG_OPTIONS.irssi=      perl inet6
                    144:        PKG_OPTIONS.mplayer=    oss
                    145: 
                    146:        DSPAM_STORAGE_DRIVER=   mysql
                    147:        PKG_OPTIONS.dspam+=     graphs
                    148:        PKG_OPTIONS.dovecot=    ssl ldap dovecot-sieve dovecot-managesieve
                    149:        PKG_OPTIONS.nagios-nrpe=ssl tcpwrappers
                    150: 
1.3       imil      151:        X11_TYPE=               modular
                    152: 
1.1       imil      153: ### Running the build
                    154: 
                    155: Now that everything's in place, we can fire up the build from the *chroot* using
                    156: the following command:
                    157: 
                    158:        # /usr/pkg_bulk/bin/bulkbuild
                    159: 
                    160: It is recommended to run the build inside a tool like *misc/screen* or
                    161: *misc/tmux* as it will take a lot of time.
                    162: 
                    163: If the build is stopped, it is possible to restart it by invoking:
                    164: 
                    165:        # /usr/pkg_bulk/bin/bulkbuild-restart
                    166: 
                    167: ### Hints
                    168: 
                    169: If you'd like to restart the build to retry building some failing packages, **empty** */meta/error* (don't delete it):
                    170: 
1.3       imil      171:        # printf "" > /meta/error
1.1       imil      172: 
                    173: If you'd like to rebuild a single package, use the *bulkbuild-rebuild* command followed by the package name.

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb