Annotation of wikisrc/pkgsrc/how_to_upgrade_packages.mdwn, revision 1.8

1.1       mspo        1: There are various techniques for upgrading packages either by using pre-built binary package tarballs or by building new packages via pkgsrc build system. This wiki page hopefully will summarize all of the different ways this can be done, with examples and pointing you to further information.
                      2: 
                      3: **Contents**
                      4: 
                      5: [[!toc levels=2]]
                      6: 
1.2       wiki        7: # Methods using only binary packages
1.1       mspo        8: 
1.4       wiz         9: 
                     10: ## pkgin
                     11: 
                     12: The recommended way to manage your system with binary packages is by using [pkgtools/pkgin](http://pkgsrc.se/pkgtools/pkgin).
                     13: 
                     14:     pkg_add pkgin
                     15: 
                     16: Then configure your binary repository from which you want to install packages in /usr/pkg/etc/pkgin/repositories.conf.
                     17: Run 'pkgin update' to get the list of available packages. You can then install packages using 'pkgin install firefox'.
                     18: 
                     19: To update all installed packages, just run
                     20: 
                     21:        pkgin update
1.8     ! gutterid   22:        pkgin upgrade
1.4       wiz        23: 
                     24: 
1.2       wiki       25: ## pkg_add -uu
1.1       mspo       26: 
                     27: pkg_add's -u option is used to update a package. Basically: it saves the package's current list of packages that depend on it (+REQUIRED_BY), installs the new package, and replaces that list of dependencies.
                     28: 
                     29: By using the -uu (option used twice), it will attempt to update prerequisite packages also.
                     30: 
1.3       wiz        31: See the manual page, [pkg_add(1)](http://netbsd.gw.com/cgi-bin/man-cgi?pkg_add+1+NetBSD-current), for details.
1.1       mspo       32: 
                     33: 
1.2       wiki       34: ## pkg_chk -b
1.1       mspo       35: 
1.6       cnst       36: Use "-b -P URL" where URL is where the binary packages are (e.g. <ftp://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/5.1/All/>).
1.1       mspo       37: 
                     38: For example, to update any missing packages by using binary packages:
                     39: 
                     40:     pkg_chk -b -P URL -u
                     41: 
                     42: Or to automatically add any missing packages using just binary packages:
                     43: 
                     44:     pkg_chk -b -P URL -a -C pkg_chk.conf
                     45: 
                     46: If both -b and -P are given, no pkgsrc tree is used. If packages are on the local machine, they are scanned directly, otherwise the pkg_summary database is fetched. (Using pkg_summary for local packages is on the TODO list.)
                     47: 
1.3       wiz        48: (pkg_chk is also covered below.)
1.1       mspo       49: 
                     50: 
1.2       wiki       51: # Methods that build packages from source
1.1       mspo       52: 
                     53: **Note: Be sure you have an updated "pkgsrc" directory. It is recommended that you don't update just parts of the pkgsrc directory.**
                     54: 
1.2       wiki       55: ## make update
1.1       mspo       56: 
                     57: 'make update', invoked in a pkgsrc package directory, will remove the package and all packages that depend on it, keeping a list of such packages. It will then attempt to rebuild and install the package and all the packages that were removed.
                     58: 
                     59: It is possible, and in the case of updating a package with hundreds of dependencies, arguably even likely that the process will fail at some point. One can fix problems and resume the update by typing make update in the original directory, but the system can have unusuable packages for a prolonged period of time. Thus, many people find 'make update' too dangerous, particularly for something like glib on a system using gnome.
                     60: 
                     61: To use binary packages if available with "make update", use "UPDATE_TARGET=bin-install". If package tarball is not available in ${PACKAGES} locally or at URLs (defined with BINPKG_SITES), it will build a package from source.
                     62: 
1.3       wiz        63: To enable manual rollback one can keep binary packages. One method is to always use 'make package', and to have "DEPENDS_TARGET=package" in /etc/mk.conf. Another is to use pkg_tarup to save packages before starting.
1.1       mspo       64: 
                     65: 
1.2       wiki       66: ## make replace
1.1       mspo       67: 
                     68: The make replace target should only be used by those who understand that there may be ABI issues and can deal with fixing the resulting problems. It is possible that a replaced package will have a different binary interface and thus packages that depend on the replaced packages may not work correctly. This can be because of a shlib (shared library) version bump, where depending package binaries will no longer run, or something more subtle. In these cases, the correct fix is to 'make replace' the problematic depending package. The careful reader will note that this process can in theory require all packages that depend (recursively) on a replaced package to be replaced. See the pkg_rolling-replace section for a way to automate this process.
                     69: 
                     70: The "make replace" target preserves the existing +REQUIRED_BY file, uninstalls the currently installed package, installs the newly built package, reinstalls the +REQUIRED_BY file, and changes depending packages to reference the new package instead. It also marks such depending packages with the "unsafe_depends" build variable, set to YES.
                     71: 
                     72: It also uses the pkg_tarup tool to create a tarball package for the the currently installed package first, just in case there is a problem.
                     73: 
                     74: make replace should preserve the "automatic" build variable, but does not.
                     75: 
                     76: If you are an expert (and don't plan to share your packages publically), you can also use in your mk.conf:
                     77: 
                     78:     USE_ABI_DEPENDS?=no
                     79: 
1.3       wiz        80: This is for ignoring the ABI dependency recommendations and just use the required DEPENDS.
1.1       mspo       81: 
                     82: 
1.2       wiki       83: ### Problems with make replace
1.1       mspo       84: 
                     85: Besides ABI changes (for which pkg_rolling-replace is a good solution), make replace can fail if packages are named or split. A particularly tricky case is when package foo is installed, but in pkgsrc has been split into foo and foo-libs. In this case, make replace will try to build the new foo (while the old monolithic foo is installed). The foo package depends on foo-libs, and so pkgsrc will go to build and install foo-libs. This will fail because foo-libs will conflict with the old foo. There are three approaches:
                     86: 
1.2       wiki       87:   * manually save the foo +REQUIRED_BY file, pkg_delete foo, and then make package of the new foo. Put back the +REQUIRED_BY, and pkg_admin set unsafe_depends=YES all packages in the +REQUIRED_BY.
                     88:   * pkg_delete -r foo, and make package on everything you still want. Or do make update. Note that this could delete a lot.
                     89:   * Automating the first option would be a useful contribution to pkgsrc.
1.1       mspo       90: 
1.3       wiz        91: In addition, any problem that can occur with building a package can occur with make replace. Usually, the solution is not make replace specific
1.1       mspo       92: 
                     93: 
1.2       wiki       94: ## pkg_chk
1.1       mspo       95: 
1.2       wiki       96: See all packages which need upgrading:
1.1       mspo       97: 
                     98:     pkg_chk -u -q
                     99: 
1.2       wiki      100: Update packages from sources:
1.1       mspo      101: 
                    102:     pkg_chk -u -s
                    103: 
1.2       wiki      104: You can set UPDATE_TARGET=package in /etc/mk.conf and specify the -b flag, so that the results of compilation work are saved for later use, and binary packages are used if they are not outdated or dependent on outdated packages.
1.1       mspo      105: 
1.3       wiz       106: The main problem with pkg_chk, is that it deinstalls all to-be-upgraded candidates before reinstalling then. However a failure is not fatal, because the current state of packages is saved in a pkg_chk* file at the root of the pkgsrc directory.
1.1       mspo      107: 
                    108: 
1.2       wiki      109: ## pkg_rolling-replace
1.1       mspo      110: 
                    111: [pkgtools/pkg_rolling-replace](http://pkgsrc.se/pkgtools/pkg_rolling-replace) is a shell script available via pkgsrc. It makes a list of all packages that need updating, and sorts them in dependency order. Then, it invokes "make replace" on the first one, and repeats. A package needs updating if it is marked unsafe_depends or if it is marked rebuild (=YES). If pkg_rolling-replace is invoked with -u, a package needs updating if [pkgtools/pkg_chk](http://pkgsrc.se/pkgtools/pkg_chk) reports that the installed version differs from the source version. On error, pkg_rolling-replace exits. The user should remove all working directories and fix the reported problem. This can be tricky, but the same process that is appropriate for a make replace should be followed.
                    112: 
                    113: Because pkg_rolling-replace just invokes make replace, the problems of ABI changes with make replace apply to pkg_rolling-replace, and the system will be in a state which might be inconsistent while pkg_rolling-replace is executing. But, by the time pkg_rolling-replace has successfully finished, the system will be consistent because every package that has a depending package 'make replaced' out from under it will be marked unsafe_depends, and then replaced itself. This replace "rolls" up the dependency tree because pkg_rolling-replace sorts the packages by dependency and replaces the earliest needing-rebuild package first.
                    114: 
                    115: See the pkg_rolling-replace man page (installed by the pkg) for further details.
                    116: 
                    117: 
1.2       wiki      118: ### Example
1.1       mspo      119: 
1.2       wiki      120: Find essentials package, that we would rather update manually:
1.1       mspo      121: 
                    122:     cd /var/db/pkg
                    123:     find . -name "+PRESERVE" | awk -F/ '{print $2}'
                    124: 
1.2       wiki      125: Update everything except the packages above:
1.1       mspo      126: 
                    127:     pkg_rolling-replace -rsuvX bmake,bootstrap-mk-files,pax,pkg_install
                    128: 
                    129: 
1.2       wiki      130: ### Real-world experience with pkg_rolling-replace
1.1       mspo      131: 
                    132: Even if a lot of packages need to be updated, make replace usually works very well if the interval from the last 'pkg_rolling-replace -u' run is not that long (a month or so). With a longer interval, like a year or two, the odds of package renaming/splitting are higher. Still, for those who can resolve the issues, this is a fairly low-pain and reliable way to update.
                    133: 
                    134: 
1.2       wiki      135: ## Delete everything
1.1       mspo      136: 
                    137: If you don't have a production environment or don't care if your packages will be missing for a while, you can just delete everything and reinstall.
                    138: 
                    139: This method is the easiest:
                    140: 
                    141:     # pkg_delete -Rr '*-*'
                    142: 
1.3       wiz       143: -or-
1.1       mspo      144: 
                    145:     # pkg_delete -ff '*-*'
                    146: 
                    147: This expands to all packages, and deletes all packages without caring about dependencies. The second version of the command should be faster, as it does not perform any dependency recursion. (The quotes around the wildcards are so it doesn't get expanded by the shell first.)
                    148: 
                    149: Here is one idea (from posting on pkgsrc-users):
                    150: 
                    151: Get a list of packages installed:
                    152: 
                    153:     # pkg_info -Q PKGPATH -a > pkgs_i_want_to_have
                    154: 
                    155: Remove all the packages:
                    156: 
                    157:     # pkg_info -a | sed 's/ .*//' | tail -r | while read p ; do pkg_delete $p ; done
                    158: 
                    159: (There are many ways to do this.)
                    160: 
                    161: Then edit your "pkgs_i_want_to_have" (created above) as needed. And reinstall just those from it:
                    162: 
                    163:     # cat pkgs_i_want_to_have | (while read pp ; do cd /usr/pkgsrc/$pp ; make && make install ; done)
                    164: 
1.3       wiz       165: An alternative way to choose the packages you want installed is to create your own custom meta-package. A meta-package doesn't install any files itself, but just depends on other packages (usually within a similar topic or need). Have a look at pkgsrc/meta-pkgs category for various examples. If your new meta-package is generic enough and useful for others, please be sure to share it.
1.1       mspo      166: 
                    167: 
1.2       wiki      168: ## chroot environment
1.1       mspo      169: 
                    170: Nothing special here. This is really basically the same as just having a another physical system to build packages.
                    171: 
                    172: Manually setup a directory containing your base operating system (including compilers, libraries, shells, etc). Put a copy of your pkgsrc tree and distfiles into there or use mount to mount shared directories containing these. Then use the "chroot" command to chroot into that new directory. You can even switch users from root to a regular user in the new environment.
                    173: 
                    174: Then build and remove packages as you wish with out affecting your real production system. Be sure to create packages for everything.
                    175: 
                    176: Then use other technique from this list to install from these packages (built in chroot).
                    177: 
1.3       wiz       178: Or instead of using this manual method, use pkg_comp's chroot.
1.1       mspo      179: 
1.2       wiki      180: ## pkg_comp's chroot
1.1       mspo      181: 
                    182: Apart from the examples in the man page, it's necessary supply a list of packages you want to build. The command 'pkg_info -u -Q PKGPATH' will produce a list of packages you explicitly requested be installed; in some strong sense, it's what you want to rebuild.
                    183: 
                    184: After you've built new new packages, you need the list of files to reinstall. Assume that you saved the output of 'pkg_info -u -Q PKGPATH' in /etc/pkgs. The following script will produce the name of the binary packages:
                    185: 
                    186:     while read x
                    187:     do
1.2       wiki      188:         cd /usr/pkgsrc/$x && make show-var VARNAME=PKGNAME
                    189:     done < /etc/pkgs
1.1       mspo      190: 
                    191: If you use PKGBASE instead of PKGNAME, you get the basename of the file.
                    192: 
1.7       leot      193: See [this BSDFreak Article](http://web.archive.org/web/20081002065622/http://www.bsdfreak.org/modules/news/article.php?storyid=1) for a nice tutorial on how to set up and use pkg_comp.
1.1       mspo      194: 
1.2       wiki      195: ## bulk build framework
1.1       mspo      196: 
1.5       cnst      197: Use the scripts in pkgsrc/mk/bulk/, e.g. as pointed out in <http://www.netbsd.org/Documentation/pkgsrc/binary.html#bulkbuild>.
1.1       mspo      198: 
1.5       cnst      199: To go easy on the existing pkgsrc installation, creating a sandbox (automated chroot environment) is highly recommended here: <http://www.netbsd.org/Documentation/pkgsrc/binary.html#setting-up-a-sandbox>.
1.1       mspo      200: 
                    201: You can later mount the pkgsrc/packages/ via NFS wherever you want and install them like:
                    202: 
                    203:     set PKG_PATH /mnt/packages/All && pkg_add <pkg>
                    204: 
1.5       cnst      205: Or upload them on a www-site and pkg_add <http://www.site/packages/All/><pkg>
1.1       mspo      206: 
1.2       wiki      207: ## wip/distbb - distributed bulk builds
1.1       mspo      208: 
                    209: Using wip/distbb you may build packages in parallel using several machines and/or chroots. Read PREFIX/share/doc/distbb/README file for the instructions.
                    210: 
1.2       wiki      211: ## pkgdepgraph
1.1       mspo      212: 
                    213: Look at the EXAMPLES in the man page.
                    214: 
                    215: TODO: how to do this using binary packages only? Put in section above.
                    216: 
1.2       wiki      217: ## Alternative LOCALBASE and PKG_DBDIR
1.1       mspo      218: 
1.2       wiki      219: Use an alternative LOCALBASE setting to install the packages under a new location and an alternate PKG_DBDIR for your alternate database of installed packages.
1.1       mspo      220: 
                    221: You can choose the PKG_DBDIR via shell environment variable or using the -K switch with any of the standard pkg_* tools.
                    222: 
                    223: And set the LOCALBASE and your PKG_DBDIR in your mk.conf file.
                    224: 
                    225: You could also simply just have a symlink from /usr/pkg to your new LOCALBASE (and /var/db/pkg to your new PKG_DBDIR) and change this when ever you are ready.

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb