Annotation of wikisrc/tracking_current.mdwn, revision 1.3

1.1       jdf         1: # Tracking NetBSD-current
                      2: 
                      3: **Contents**
                      4: 
                      5: [[!toc levels=2]]
                      6: 
                      7: ## Why track NetBSD-current?
                      8: 
                      9: The developers of NetBSD have made the current development sources
                     10: available to the public for several reasons. Overall, providing
                     11: NetBSD-current helps us to create a more stable, accessible system.
                     12: 
                     13: It makes it easier for people to become involved in the development of
                     14: NetBSD. Distributing the current development sources allows a greater
                     15: number of people to see where the system is going, and to become
                     16: involved with new features as they are implemented.
                     17: 
                     18: It also makes changes from users easier to integrate. If users make
                     19: changes against the current development sources, then virtually no
                     20: integration is needed to get them into the master source tree.
                     21: 
                     22: It also allows wider testing of the software as it is developed. Users
                     23: of NetBSD-current are encouraged to send in
                     24: [bug reports](http://netbsd.org/support/send-pr.html) about the current sources,
                     25: and that helps find and fix bugs. Because people are testing the software soon
                     26: after it's written, more bugs can be found and eliminated.
                     27: 
                     28: ## Things you need to remember
                     29: 
                     30:  * People using NetBSD-current are strongly encouraged to subscribe to
                     31:    the [current-users](http://netbsd.org/mailinglists/#current-users) mailing
                     32:    list. The [source-changes](http://netbsd.org/mailinglists/#source-changes)
                     33:    mailing list is also of interest.
                     34: 
                     35:  * When upgrading to a more recent version of -current you should
1.2       jdf        36:    *always* install and boot a new kernel before installing any new libs (unless 
                     37:    you are certain there have been no new system calls added, but do it anyway; 
                     38:    it's safer).In general the best approach is to try the new kernel before 
                     39:    anything else, and if you hit any problems see the entry in the
                     40:    [Kernel FAQ](http://netbsd.org/docs/kernel/#problems_compiling_a_current_kernel).
1.1       jdf        41: 
                     42:  * When compiling a -current kernel, always remember to include the
1.3     ! sevan      43:    `COMPAT_<lastrelease>` option (e.g., `COMPAT_80`). As current diverges from
        !            44:    the last stable release, compatibility code will be added, but it will only
1.1       jdf        45:    be enabled if this option is present. At a bare minimum, you will need this
                     46:    compatibility code for the time between booting the new kernel and finishing
                     47:    your build via `build.sh`
                     48: 
                     49: ## Updating an existing system from a current snapshot
                     50: 
                     51: *Please remember to check
                     52: [src/UPDATING](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/UPDATING) for
                     53: quirks around certain specific changes.*
                     54: 
                     55: To quickly begin using current, start with a snapshot generated by release
                     56: engineering. The current status of each platform can be seen at
                     57: [NetBSD Autobuild](http://releng.NetBSD.org/cgi-bin/builds.cgi) and the
                     58: corresponding releases found in by date and platform.
                     59: 
                     60:  1. Hunt down to the desired `binary/sets` directory, and `mget *.tgz` files
                     61:     into your favorite local administrative directory (for example,
                     62:     `$HOME/current`); when limited by disk space and/or time, only
                     63:        `kern-GENERIC`, `etc`, `base`, and `comp` (if you want a compiler) are
                     64:        essential.
                     65:  2. Extract the desired kernel (usually `GENERIC`), copy it into (root) directory.
                     66: 
                     67:         # cd /root
                     68:         # tar -zxpf ~/kern-GENERIC.tgz
                     69:         # ln -fh /netbsd /netbsd.old
                     70:         # cp netbsd /netbsd.new
                     71:         # ln -fh /netbsd.new /netbsd
                     72: 
                     73:     > **Warning**: Don't extract any userland binary sets before rebooting your
                     74:     > machine with the new kernel. Newer binaries might use new system calls an
                     75:     > old running kernel doesn't support.
                     76: 
                     77:  3. Check if there are any other files which might also be required by a new
                     78:     kernel. Again,
                     79:     [src/UPDATING](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/UPDATING)
                     80:     might mention possible quirks on daily changes.
                     81:        
                     82:        The following items are typical files that possibly need to be updated:
                     83: 
                     84:      1. bootloader
                     85:         
                     86:         Usually a machine specific bootloader passes several parameters to a 
                     87:         loaded kernel. If some new parameters have been added or some existing 
                     88:         APIs between bootloader and kernel are changed you might also have to 
                     89:         install new bootloader files for a new kernel to handle new features. A 
                     90:         method to update bootloader files is quite machine dependent, so check 
                     91:         boot(8) and installboot(8) man pages for details.
                     92:         
                     93:         
                     94:         On i386 and amd64, if you are using FFSv1 for root file system on `wd0a` 
                     95:         (i.e. first ATA drive), typical commands to update bootloaders are:
                     96: 
                     97:             # tar -C /tmp -zxf ~/base.tgz ./usr/mdec
                     98:             # cp /tmp/usr/mdec/boot /
                     99:             # installboot -v /dev/rwd0a /tmp/usr/mdec/bootxx_ffsv1
                    100: 
                    101:         If you are using FFSv2 for root file system use the following commands 
                    102:         instead:
                    103: 
                    104:             # tar -C /tmp -zxf ~/base.tgz ./usr/mdec
                    105:             # cp /tmp/usr/mdec/boot /
                    106:             # installboot -v /dev/rwd0a /tmp/usr/mdec/bootxx_ffsv2
                    107: 
                    108:         Note `/usr/mdec/bootxx_ffsv1` and `/usr/mdec/bootxx_ffsv2` are primary 
                    109:         bootloaders which are file system dependent. `/usr/mdec/boot` is 
                    110:         secondary loader and it's file system independenet.
                    111: 
                    112:         If you forget your root file system type (FFSv1 or FFSv2), you can check 
                    113:         it by dumpfs(8) command:
                    114: 
                    115:             # dumpfs /dev/rwd0a | head -3
                    116:             file system: /dev/rwd0a
                    117:             format  FFSv2
                    118:             endian  little-endian
                    119: 
                    120:      2. kernel modules
                    121: 
1.3     ! sevan     122:         A new framework *kernel modules* was been introduced after netbsd-5 was 
        !           123:         branched.
        !           124:         The kernel module files 
1.1       jdf       125:         will be loaded dynamically by the kernel to support various kernel 
                    126:         options (including file systems) on demand, rather than linking all 
                    127:         necessary (but possibly unused) object files into the kernel binary. 
                    128:         This means if you are trying to boot a new `GENERIC` kernel, you also 
                    129:         have to prepare new kernel module files for the new kernel. 
                    130: 
                    131:         To prepare new kernel module files, you can simply use a new `modules` 
                    132:         set file which has been prepared since September 2009:
                    133: 
                    134:             # cd /
                    135:             # tar -zxpf ~/modules.tgz
                    136:                 
                    137:         Note i386 port also provides `MONOLITHIC` kernel binary in 
                    138:         `kern-MONOLITHIC.tgz` set file since October 2009. The `MONOLITHIC` 
                    139:         kernel includes all necessary options in its kernel as well as 5.0 and 
                    140:         prior `GENERIC` kernels and it doesn't depend on kernel module files at 
                    141:         all. If you would just like to test new features of a new kernel without 
                    142:         updating kernel modules, using `MONOLITHIC` kernel is easier way for the 
                    143:         first and quick trial. 
                    144: 
                    145:         It's also a good idea to put an old `MONOLITHIC` kernel into `/` (root) 
                    146:         directory for emergency and recovery because if newer modules have some 
                    147:         fatal issue there is no easy way to specify an alternative path of old 
                    148:         module files to a modular'ized kernel (and you can't rename directories 
                    149:         without a working kernel).
                    150: 
                    151:  4. Reboot machine with the new kernel:
                    152: 
                    153:         # shutdown -r now
                    154: 
                    155:  5. Make sure the new kernel boots and works properly. If your new kernel has
                    156:     any trouble, you can recover it by loading the renamed old one. If you are 
1.3     ! sevan     157:     using a modularised GENERIC kernel as mentioned above, you might also have to 
1.1       jdf       158:     restore old kernel module files.
                    159: 
                    160:  6. Extract the matching base, and any other desirable feature sets **except 
                    161:     etc**:
                    162: 
                    163:         # cd /
                    164:         # tar -zxpf ~/base.tgz
                    165:         # tar -zxpf ~/comp.tgz
                    166:         # ...
                    167:             
                    168: 
                    169:     Don't forget to specify "p" option (preserve permissions) on tar(1) command 
                    170:     otherwise setuid'ed commands (like su(1)) won't work.
                    171: 
                    172:     **Warning**: Extracting `etc.tgz` on the installed system will overwrite your 
                    173:     local settings.
                    174: 
                    175:  7. [Update](http://netbsd.org/docs/current/index.html#etcupdate) `/etc` as the 
                    176:     last step: postinstall(8) will first check and fix most things that can be 
                    177:     automated, and etcupdate(8) in the second step will ask on what to merge:
                    178: 
                    179:         # /usr/sbin/postinstall -s ~/etc.tgz check
                    180:         # /usr/sbin/postinstall -s ~/etc.tgz fix
                    181:         # /usr/sbin/etcupdate -s ~/etc.tgz
                    182:         # shutdown -r now
                    183:             
                    184:     If you have the X sets installed (xbase, ...), you can repeat the 
                    185:        postinstall and etcupdate steps with xetc.tgz as argument before rebooting.
                    186: 
                    187: At this point, you are relatively current and ready to build your own current 
                    188: source.
                    189: 
                    190: ## Downloading current source
                    191: 
                    192: See the [Obtaining the sources](http://netbsd.org/docs/guide/en/chap-fetch.html) 
                    193: section in the [[NetBSD Guide|guide/index]].
                    194: 
                    195: ## Building a release from source
                    196: 
                    197: See the [Crosscompiling NetBSD](http://netbsd.org/docs/guide/en/chap-build.html) 
                    198: section in the [[NetBSD Guide|guide/index]].
                    199: 
                    200: ## Updating an existing system from source
                    201: 
                    202: See the [[Updating an existing system from sources|guide/updating]] section in 
                    203: the [[NetBSD Guide|guide/index]].
                    204: 
                    205: ## Updating the configuration and startup files
                    206: 
1.2       jdf       207: See the
                    208: [[More details about the updating of configuration and startup files|guide/updating]]
                    209: section in the [[NetBSD Guide|guide/index]].
1.1       jdf       210: 
                    211: ## What if I get an error?
                    212: 
                    213: If you try to build -current, either from a snapshot or an earlier -current, and 
                    214: it doesn't work, don't panic. Try these steps:
                    215: 
                    216:  1. Read the 
                    217:     [src/UPDATING](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/UPDATING) 
                    218:     file from the release you're trying to build.
                    219: 
                    220:  2. Read the [current-users archive](http://mail-index.netbsd.org/current-users/)
                    221:     for hints.
                    222: 
                    223:  3. Update again. You may have caught the repository in the middle of a commit 
                    224:     to several related files, or the problem might have already been fixed.
                    225: 
                    226:  4. If all else fails, send email to current-users explaining the problem. 
                    227:     Include the date, time, and method you used to get your -current sources, as 
                    228:     well as any local changes you've made. Then put in a **short** script that 
                    229:     includes the error messages you're getting. Somebody will probably fix the 
                    230:     problem momentarily.
                    231: 
                    232: ## Tracking NetBSD-current with anoncvs
                    233: 
                    234: See the
                    235: [Fetching by CVS](http://netbsd.org/docs/guide/en/chap-fetch.html#chap-fetch-cvs)
                    236: section in the [[NetBSD Guide|guide/index]].
                    237: 
                    238: ### To check out the sources from a certain date
                    239: 
                    240:     $ cvs checkout -D 20020501-UTC src
                    241: 
                    242: ### To check out the sources from a certain branch
                    243: 
1.3     ! sevan     244:     $ cvs checkout -rnetbsd-8-0 src
1.1       jdf       245: 
                    246: See 
                    247: [src/doc/BRANCHES](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/doc/BRANCHES) 
                    248: for a description of the branches in the CVS repository.
                    249: 
                    250: ### Useful hints
                    251: 
                    252:  * Do not use the cvs `-z` flag. The data stream gets out of sync,
                    253:    leading to corruption on the client, or causing the client to hang
                    254:    completely. The additional load is also hard on the cvs server.
                    255:  * If you want to check out a certain branch of the tree, you may want
                    256:    to take caution not to overwrite any existing directories by
                    257:    creating a new directory for this branch:
                    258: 
                    259:        $ cd /parent/dir/to/checkout/into
                    260:        $ mkdir NewName-temp
                    261:        $ cd NewName-temp
                    262:        $ cvs checkout ... src
                    263:        $ mv src ../NewName
                    264:        $ cd ..
                    265:        $ rmdir NewName-temp
                    266:         
                    267:  * You will have to use objdirs in order for cvs updates to work correctly. If 
                    268:    you happen to get errors from cvs saying things like:
                    269: 
                    270:        cvs [update aborted]: could not chdir to gnu/usr.bin/gdb/gdb: Not a directory
                    271: 
                    272:    You should do a make cleandir and try again. Make sure to run make obj after 
                    273:    the cvs update.
                    274: 
                    275:  * You can put switches for specific commands in a `.cvsrc` in your home 
                    276:    directory, and they will be automatically used. A sample `.cvsrc` would be:
                    277: 
1.3     ! sevan     278:     cvs -q
        !           279:     checkout -P
        !           280:     update -dP
        !           281:     diff -upN
        !           282:     rdiff -u
        !           283:     release -d
1.1       jdf       284: 
                    285: ## Importing and merging sources.
                    286: 
                    287: Sources are imported as follows:
                    288: 
                    289:     $ cvs -d /misc/cvsrep import -I ! -I CVS netbsd netbsd current-date
                    290: 
                    291: `date` is replaced by the date of the SUP for tracking purposes. The
                    292: `-I ! -I CVS` options ensure that no file in the source tree is ignored
                    293: except 'CVS' directories. This is because some NetBSD source files have
                    294: extensions which are normally ignored by CVS. If there are any conflicts
                    295: with local patches the import command will report them and will describe
                    296: a command to merge the conflicts something like:
                    297: 
                    298:     $ cvs checkout -jnetbsd:yesterday -jnetbsd netbsd
                    299: 
                    300: This merge command will correctly merge the imported NetBSD sources but
                    301: it will not handle the removal of files locally which have already been
                    302: removed by the SUP process. To do this the merge command would be:
                    303: 
                    304:     $ cvs update -jprevious_import_tag -j current-date
                    305: 
                    306: `previous_import_tag` should be replaced with the name of the tag used
                    307: for the previous cvs import. `date` should be replaced with the current
                    308: date to yield the same tag used on the current import that has just been
                    309: merged.
                    310: 
                    311: The conflicts reported by the import command are potential conflicts.
                    312: These are usually merged by the update command but in some cases a real
                    313: conflict occurs. In these cases a manual merge of the conflicting lines
                    314: will be required. A real conflict will be reported in the cvs update
                    315: output as a `C` followed by a filename.
                    316: 
                    317: Merging conflicts manually is not a simple process but in most cases it
                    318: should be resolved by removing the local changes and making the file
                    319: like the original NetBSD source code.
                    320: 
                    321: CVS marks conflicts as follows:
                    322: 
                    323:     <<<<<<
                    324:       code from local file
                    325:     ======
                    326:       code from imported file
                    327:     >>>>>> local revision number of newly imported revision
                    328: 
                    329: If the import reports no conflicts the checked out copy of the tree
                    330: should be updated in exactly the same way as for the conflicts case.
                    331: 
                    332: All update and checkout commands should be done in the directory where
                    333: the sources have been checked out. On my system this is
                    334: `/usr/src/netbsd`.
                    335: 
                    336: If this is the first import then there will be no sources checked out.
                    337: Assuming you wish to create the source tree in `/usr/src/netbsd` The
                    338: following commands will check out the source and no merge step is
                    339: required.
                    340: 
                    341:     $ cd /usr/src
                    342:     $ cvs -d /misc/cvsrep checkout netbsd
                    343: 
                    344: ## Tagging a successful build
                    345: 
                    346: If the build completes successfully, and produces a working set of binaries, it 
                    347: can be useful to tag the working sources. This allows rewinding to a working 
                    348: build tree with a single CVS command in the event that the current tree becomes 
                    349: unbuildable for any reason. This can be performed by issuing the following 
                    350: command:
                    351: 
                    352:     $ cvs tag successful-build-BUILD date
                    353: 
                    354: ### Notes
1.2       jdf       355:  * If the NetBSD customised version of CVS, which recognises *\$NetBSD\$*
1.1       jdf       356:    markers in files, is not used, the NetBSD revision number of the file is 
                    357:    available for reference purposes when build problems occur.
                    358:  * The sup/import/merge sequence described above is quite easily
                    359:    automatable. The following Perl script automates this process.
                    360: 
                    361:        #!/usr/pkg/bin/perl
                    362:        #
                    363:        # Script to SUP NetBSD-current, import it into CVS and merge it with
                    364:        # any local changes.
                    365:        #
                    366:        # NOTES:
                    367:        # This script does no error handling so is not really suitable for
                    368:        # non-interactive use.
                    369:        #
                    370:        # This script has only been test with cvs-1.10.1 and cvs-1.9.18.
                    371:        #
                    372:        $SRCROOT="/usr/src/netbsd";
                    373:        $IMPORTROOT="/misc/import";
                    374:        $CVSROOT="/misc/cvsrep";
                    375:        #run the sup into a perl stream
                    376:        system "/usr/sbin/sup -zsv" ; # This may need to change for none
                    377:                                      # current systems
                    378:        # now import the new files into CVS
                    379:        
                    380:        chdir $IMPORTROOT or die "Could not cd to $IMPORTROOT\n";
                    381: 
                    382:        ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime;
                    383:        $date = localtime;
                    384:        $shortdate = sprintf "%02d%02d%04d",$mday,$mon+1,1900+$year;
                    385:        system "/usr/local/bin/cvs -d$CVSROOT import -I ! -m\"SUP Import $date\" netbsd netbsd current-$shortdate ";
                    386: 
                    387:        # make the working directory the local NetBSD Tree
                    388:        chdir $SRCROOT or die "Could not change to $SRCROOT directory\n";
                    389: 
                    390:        # Now do the import.
                    391:        $lastimport = `cat /usr/src/netbsd/.tag`; # `s are backquotes
                    392:        $lastimport =~ s/\n//; # strip off any trailing newline in the string
                    393:        system "/usr/local/bin/cvs update -j $lastimport  -j
                    394:        current-$shortdate ";
                    395:        # Now write the current file into tag save file
                    396:        open TAG,">$SRCROOT/.tag" or die "Could not open new tag file";
                    397:         print TAG "current-$shortdate";
                    398:        close TAG;
                    399: 
                    400:    This script was written in Perl since it is the scripting tool which the 
                    401:    author has the most experience with. It should be fairly straightforward to 
                    402:    write a shell script to perform the same task.
                    403: 
                    404:  * Techniques for tracking current with CVS have been discussed several times on 
                    405:    the NetBSD current-users mailing list. For alternative techniques try 
                    406:    searching the NetBSD mailing lists.
                    407: 
                    408: If you have any comments or suggestions please send them to
                    409: [Mike Pumford](mailto:mpumford@black-star.demon.co.uk) (who maintains this 
                    410: entry) or <www@NetBSD.org>.
                    411: 
                    412: ## Getting the whole repository
                    413: 
                    414: All the procedures described above allow you keeping your own changes in your 
                    415: repository, which has its advantages if you develop your own software based on 
                    416: NetBSD. If you don't want to maintain your own CVS repository, but just want to 
                    417: mirror NetBSD's CVS repository, there are three ways to do so.
                    418: 
                    419: Each of the methods described briefly below will get you a copy of the NetBSD 
                    420: CVS repository (i.e. the RCS ,v files, not the checked out files!). You can then 
                    421: setup your own anoncvs server or check out to a local harddisk. It's also useful 
                    422: for fast access to the history information stored in the repository.
                    423: 
                    424: The methods to retrieve the whole repository are:
                    425: 
                    426:  * **sup**
                    427: 
                    428:    If you use sup already to mirror other parts of the NetBSD source,
                    429:    you will want to add the following lines to your sup config file:
                    430: 
                    431:        anoncvs release=all  host=sup.NetBSD.org hostbase=/ftp/pub \
                    432:        base=/usr prefix=/usr backup use-rel-suffix compress
                    433:         
                    434:    After that, run `sup /path/to/supfile anoncvs` to retrieve the files.
                    435: 
1.3     ! sevan     436:    Some example sup files are available in `/usr/share/examples/supfiles`.
1.1       jdf       437: 
                    438:  * **rsync**
                    439: 
                    440:    Note that rsync puts quite a heavy load on our rsync server, and as such the 
                    441:    number of concurrent rsync users is restricted. If you still want to try 
                    442:    rsync, the command to retrieve the repository is:
                    443: 
                    444:        $ rsync -v -a --delete --exclude '#cvs.lock' rsync://anoncvs.NetBSD.org/cvsroot/src .
                    445: 
                    446:    Please see our [list of rsync mirrors](http://netbsd.org/mirrors/#rsync)!

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb