Annotation of wikisrc/tutorials/pkgsrc/pbulk.mdwn, revision 1.1

1.1     ! imil        1: ## Using pbulk to create a pkgsrc binary repository
        !             2: 
        !             3: *pkgtools/pbulk* package consists in a set of tools designed to ease
        !             4: mass-building of *pkgsrc* packages, and the creation your own *pkgsrc* binaries
        !             5: repository.
        !             6: 
        !             7: Its setup needs a bit of work, here is how to prepare and run your bulk-build
        !             8: box. In this article we will only consider a one-node machine.
        !             9: 
        !            10: This documentation is based on the [pbulk-HOWTO](http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/doc/HOWTO-pbulk?rev=1.7&content-type=text/x-cvsweb-markup).
        !            11: 
        !            12: ### Prerequisites
        !            13: 
        !            14: These are the prerequisites needed by *pbulk*:
        !            15: 
        !            16: * A *pkgsrc* source tree
        !            17: * A *src* source tree
        !            18: * Possibly (not mandatory) a *xsrc* source tree
        !            19: * Possibly (not mandatory) a tool like *misc/screen* as the full build process can takes a **very** long time.
        !            20: 
        !            21: For example, let's say you'd want to regularly build *pkgsrc-2011Q3* packages for *NetBSD 5.1* on your build box, you would prepare the following:
        !            22: 
        !            23:        # cd /usr
        !            24:        # cvs -d anoncvs@anoncvs.netbsd.org:/cvsroot co -rnetbsd-5-1
        !            25:        # cvs -d anoncvs@anoncvs.netbsd.org:/cvsroot co -rpkgsrc-2011Q3
        !            26: 
        !            27: And get sure those are always up-to-date by adding the following to a *crontab*:
        !            28: 
        !            29:        30 1 * * * cd /usr/pkgsrc && /usr/bin/cvs up -Pd >/dev/null 2>&1
        !            30:        00 2 * * * cd /usr/src && /usr/bin/cvs up -Pd >/dev/null 2>&1
        !            31: 
        !            32: ### Prepare a chroot
        !            33: 
        !            34: In order to isolate the bulk build, it is advised you run all the operations
        !            35: within a *chroot*. Running *pbulk* on your real environment would wipe all
        !            36: of your installed packages, and would modify your base system with lots of
        !            37: directories, users and groups you don't need.
        !            38: 
        !            39: Fortunately, a tool called *mksandbox* will simplify this process. *mksandbox*
        !            40: is located in *mk/bulk/mksandbox*, and it is called like this:
        !            41: 
        !            42:        # mk/bulk/mksandbox [optionnal flags] /path/to/sandbox
        !            43: 
        !            44: For example, to create a sandbox in */home/bulk* without the X11 system, run:
        !            45: 
        !            46:        # mk/bulk/mksandbox --without-x /home/bulk
        !            47: 
        !            48: This command will prepare and mount most of the needed directories, and will
        !            49: place a shell script on top of the sandbox filesystem called *sandbox*. This
        !            50: script is used to *mount*/*umount* your sandbox. It is a good idea to add
        !            51: */var/spool* to the list of directories mounted as read/write in your sandbox
        !            52: so the email report is actually sent. Simply add:
        !            53: 
        !            54:        /var/spool /var/spool rw \
        !            55: 
        !            56: to the list of directories in the *sandbox* script. *sandbox* script use is
        !            57: really straightforward:
        !            58:        # /path/to/your/sandbox/sandbox umount
        !            59: Unmounts the sandbox
        !            60:        # /path/to/your/sandbox/sandbox mount
        !            61: Mounts the sandbox
        !            62: 
        !            63: ### Prepare the pbulk environment
        !            64: 
        !            65: Now that our sandbox is available and mounted, we will chroot to it:
        !            66: 
        !            67:        # chroot /home/bulk /bin/ksh
        !            68: 
        !            69: As the [pbulk-HOWTO](http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/doc/HOWTO-pbulk?rev=1.7&content-type=text/x-cvsweb-markup) suggests, we will create some helpers directories to keep things clean:
        !            70: 
        !            71:        # mkdir /bulklog # bulk meta data and log files of each package
        !            72:        # mkdir /scratch # WRKOBJDIR
        !            73:        # mkdir /distfiles # DISTDIR
        !            74:        # mkdir /packages # PACKAGES
        !            75: 
        !            76: The */scratch* directory must be writable by a *bulk* user defined below in */usr/pkg_bulk/etc/pbulk.conf* as the build is done using DESTDIR.
        !            77: 
        !            78:        # useradd bulk
        !            79:        # chown bulk /scratch
        !            80: 
        !            81: It is now time to bootstrap (yes, even on NetBSD) a *pbulk* environment so the latter is not wiped out when a new bulk build is started:
        !            82: 
        !            83:        # cd /usr/pkgsrc/bootstrap
        !            84:        # ./bootstrap --prefix=/usr/pkg_bulk --pkgdbdir=/usr/pkg_bulk/.pkgdb
        !            85:        # rm -rf work
        !            86: 
        !            87: Then install *pbulk* itself on this newly created separate prefix:
        !            88: 
        !            89:        # cd /usr/pkgsrc/pkgtools/pbulk
        !            90:        # env PATH=/usr/pkg_bulk/bin:/usr/pkg_bulk/sbin:${PATH} bmake install
        !            91: 
        !            92: ### Configuring pbulk
        !            93: 
        !            94: *pbulk* configuration file is */usr/pkg_bulk/etc/pbulk.conf*, here are
        !            95: some parameters you should modify to customize the run to your needs:
        !            96: 
        !            97:        base_url= # should point to the URL where the txt report will be
        !            98:        master_mode=no # this will not be a distributed build
        !            99:        # [...]
        !           100:        report_recipients="your@email.com"
        !           101:        # [...]
        !           102:        #bootstrapkit= must be commented on NetBSD !
        !           103:        unprivileged_user=bulk # used for unprivileged user-destdir builds
        !           104:        # [...]
        !           105:        bulklog=/bulklog
        !           106:        packages=/packages
        !           107:        prefix=/usr/pkg
        !           108:        pkgsrc=/usr/pkgsrc
        !           109:        pkgdb=/var/db/pkg
        !           110:        varbase=/var
        !           111:        # [...]
        !           112:        make=/usr/bin/make # must be modified on NetBSD !
        !           113: 
        !           114: ### Configuring the build
        !           115: 
        !           116: Last but not least, you should configure your */etc/mk.conf* (in the *chroot* !).
        !           117: Here's an example of what we use at [NetBSDfr](http://www.netbsdfr.org/):
        !           118: 
        !           119:        WRKOBJDIR=              /scratch
        !           120:        PKGSRCDIR=              /usr/pkgsrc
        !           121:        DISTDIR=                /distfiles
        !           122:        PACKAGES=               /packages
        !           123: 
        !           124:        FAILOVER_FETCH=         yes
        !           125: 
        !           126:        X11_TYPE=               modular
        !           127: 
        !           128:        SKIP_LICENSE_CHECK=             yes
        !           129:        ALLOW_VULNERABLE_PACKAGES=      yes
        !           130: 
        !           131:        PKG_DEVELOPER?=         yes
        !           132: 
        !           133:        PKG_OPTIONS.irssi=      perl inet6
        !           134:        PKG_OPTIONS.mplayer=    oss
        !           135: 
        !           136:        DSPAM_STORAGE_DRIVER=   mysql
        !           137:        PKG_OPTIONS.dspam+=     graphs
        !           138:        PKG_OPTIONS.dovecot=    ssl ldap dovecot-sieve dovecot-managesieve
        !           139:        PKG_OPTIONS.nagios-nrpe=ssl tcpwrappers
        !           140: 
        !           141: ### Running the build
        !           142: 
        !           143: Now that everything's in place, we can fire up the build from the *chroot* using
        !           144: the following command:
        !           145: 
        !           146:        # /usr/pkg_bulk/bin/bulkbuild
        !           147: 
        !           148: It is recommended to run the build inside a tool like *misc/screen* or
        !           149: *misc/tmux* as it will take a lot of time.
        !           150: 
        !           151: If the build is stopped, it is possible to restart it by invoking:
        !           152: 
        !           153:        # /usr/pkg_bulk/bin/bulkbuild-restart
        !           154: 
        !           155: ### Hints
        !           156: 
        !           157: If you'd like to restart the build to retry building some failing packages, **empty** */meta/error* (don't delete it):
        !           158: 
        !           159:        # cat > /meta/error << EOF
        !           160:        EOF
        !           161: 
        !           162: If you'd like to rebuild a single package, use the *bulkbuild-rebuild* command followed by the package name.

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb