Annotation of wikisrc/tracking_current.mdwn, revision 1.1

1.1     ! jdf         1: # Tracking NetBSD-current
        !             2: 
        !             3: **Contents**
        !             4: 
        !             5: [[!toc levels=2]]
        !             6: 
        !             7: ## Why track NetBSD-current?
        !             8: 
        !             9: The developers of NetBSD have made the current development sources
        !            10: available to the public for several reasons. Overall, providing
        !            11: NetBSD-current helps us to create a more stable, accessible system.
        !            12: 
        !            13: It makes it easier for people to become involved in the development of
        !            14: NetBSD. Distributing the current development sources allows a greater
        !            15: number of people to see where the system is going, and to become
        !            16: involved with new features as they are implemented.
        !            17: 
        !            18: It also makes changes from users easier to integrate. If users make
        !            19: changes against the current development sources, then virtually no
        !            20: integration is needed to get them into the master source tree.
        !            21: 
        !            22: It also allows wider testing of the software as it is developed. Users
        !            23: of NetBSD-current are encouraged to send in
        !            24: [bug reports](http://netbsd.org/support/send-pr.html) about the current sources,
        !            25: and that helps find and fix bugs. Because people are testing the software soon
        !            26: after it's written, more bugs can be found and eliminated.
        !            27: 
        !            28: ## Things you need to remember
        !            29: 
        !            30:  * People using NetBSD-current are strongly encouraged to subscribe to
        !            31:    the [current-users](http://netbsd.org/mailinglists/#current-users) mailing
        !            32:    list. The [source-changes](http://netbsd.org/mailinglists/#source-changes)
        !            33:    mailing list is also of interest.
        !            34: 
        !            35: # XXX
        !            36:  * When upgrading to a more recent version of -current you should
        !            37:     *always* install and boot a new kernel before installing any new
        !            38:     libs ([[http://netbsd.org/docs/current/index.html#star|*]]). In general the best approach is to try the new
        !            39:     kernel before anything else, and if you hit any problems see the
        !            40:     entry in the [[http://netbsd.org/docs/kernel/#problems_compiling_a_current_kernel|Kernel FAQ]].
        !            41: 
        !            42:  * When compiling a -current kernel, always remember to include the
        !            43:    `COMPAT_<lastrelease>` option (e.g., `COMPAT_16`). As current diverges from
        !            44:    the last s"table release, compatibility code will be added, but it will only
        !            45:    be enabled if this option is present. At a bare minimum, you will need this
        !            46:    compatibility code for the time between booting the new kernel and finishing
        !            47:    your build via `build.sh`
        !            48: 
        !            49: # XXX
        !            50: \*: Unless you are certain there have been no new system calls added,
        !            51: but do it anyway; it's safer.
        !            52: 
        !            53: ## Updating an existing system from a current snapshot
        !            54: 
        !            55: *Please remember to check
        !            56: [src/UPDATING](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/UPDATING) for
        !            57: quirks around certain specific changes.*
        !            58: 
        !            59: To quickly begin using current, start with a snapshot generated by release
        !            60: engineering. The current status of each platform can be seen at
        !            61: [NetBSD Autobuild](http://releng.NetBSD.org/cgi-bin/builds.cgi) and the
        !            62: corresponding releases found in by date and platform.
        !            63: 
        !            64:  1. Hunt down to the desired `binary/sets` directory, and `mget *.tgz` files
        !            65:     into your favorite local administrative directory (for example,
        !            66:     `$HOME/current`); when limited by disk space and/or time, only
        !            67:        `kern-GENERIC`, `etc`, `base`, and `comp` (if you want a compiler) are
        !            68:        essential.
        !            69:  2. Extract the desired kernel (usually `GENERIC`), copy it into (root) directory.
        !            70: 
        !            71:         # cd /root
        !            72:         # tar -zxpf ~/kern-GENERIC.tgz
        !            73:         # ln -fh /netbsd /netbsd.old
        !            74:         # cp netbsd /netbsd.new
        !            75:         # ln -fh /netbsd.new /netbsd
        !            76: 
        !            77:     > **Warning**: Don't extract any userland binary sets before rebooting your
        !            78:     > machine with the new kernel. Newer binaries might use new system calls an
        !            79:     > old running kernel doesn't support.
        !            80: 
        !            81:  3. Check if there are any other files which might also be required by a new
        !            82:     kernel. Again,
        !            83:     [src/UPDATING](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/UPDATING)
        !            84:     might mention possible quirks on daily changes.
        !            85:        
        !            86:        The following items are typical files that possibly need to be updated:
        !            87: 
        !            88:      1. bootloader
        !            89:         
        !            90:         Usually a machine specific bootloader passes several parameters to a 
        !            91:         loaded kernel. If some new parameters have been added or some existing 
        !            92:         APIs between bootloader and kernel are changed you might also have to 
        !            93:         install new bootloader files for a new kernel to handle new features. A 
        !            94:         method to update bootloader files is quite machine dependent, so check 
        !            95:         boot(8) and installboot(8) man pages for details.
        !            96:         
        !            97:         
        !            98:         On i386 and amd64, if you are using FFSv1 for root file system on `wd0a` 
        !            99:         (i.e. first ATA drive), typical commands to update bootloaders are:
        !           100: 
        !           101:             # tar -C /tmp -zxf ~/base.tgz ./usr/mdec
        !           102:             # cp /tmp/usr/mdec/boot /
        !           103:             # installboot -v /dev/rwd0a /tmp/usr/mdec/bootxx_ffsv1
        !           104: 
        !           105:         If you are using FFSv2 for root file system use the following commands 
        !           106:         instead:
        !           107: 
        !           108:             # tar -C /tmp -zxf ~/base.tgz ./usr/mdec
        !           109:             # cp /tmp/usr/mdec/boot /
        !           110:             # installboot -v /dev/rwd0a /tmp/usr/mdec/bootxx_ffsv2
        !           111: 
        !           112:         Note `/usr/mdec/bootxx_ffsv1` and `/usr/mdec/bootxx_ffsv2` are primary 
        !           113:         bootloaders which are file system dependent. `/usr/mdec/boot` is 
        !           114:         secondary loader and it's file system independenet.
        !           115: 
        !           116:         If you forget your root file system type (FFSv1 or FFSv2), you can check 
        !           117:         it by dumpfs(8) command:
        !           118: 
        !           119:             # dumpfs /dev/rwd0a | head -3
        !           120:             file system: /dev/rwd0a
        !           121:             format  FFSv2
        !           122:             endian  little-endian
        !           123: 
        !           124:      2. kernel modules
        !           125: 
        !           126:         A new framework *kernel modules* has been introduced after netbsd-5 was 
        !           127:         branched, and `GENERIC` kernel on i386 port has been switched to using 
        !           128:         the kernel module files since November 2008. The kernel module files 
        !           129:         will be loaded dynamically by the kernel to support various kernel 
        !           130:         options (including file systems) on demand, rather than linking all 
        !           131:         necessary (but possibly unused) object files into the kernel binary. 
        !           132:         This means if you are trying to boot a new `GENERIC` kernel, you also 
        !           133:         have to prepare new kernel module files for the new kernel. 
        !           134: 
        !           135:         To prepare new kernel module files, you can simply use a new `modules` 
        !           136:         set file which has been prepared since September 2009:
        !           137: 
        !           138:             # cd /
        !           139:             # tar -zxpf ~/modules.tgz
        !           140:                 
        !           141:         Note i386 port also provides `MONOLITHIC` kernel binary in 
        !           142:         `kern-MONOLITHIC.tgz` set file since October 2009. The `MONOLITHIC` 
        !           143:         kernel includes all necessary options in its kernel as well as 5.0 and 
        !           144:         prior `GENERIC` kernels and it doesn't depend on kernel module files at 
        !           145:         all. If you would just like to test new features of a new kernel without 
        !           146:         updating kernel modules, using `MONOLITHIC` kernel is easier way for the 
        !           147:         first and quick trial. 
        !           148: 
        !           149:         It's also a good idea to put an old `MONOLITHIC` kernel into `/` (root) 
        !           150:         directory for emergency and recovery because if newer modules have some 
        !           151:         fatal issue there is no easy way to specify an alternative path of old 
        !           152:         module files to a modular'ized kernel (and you can't rename directories 
        !           153:         without a working kernel).
        !           154: 
        !           155:         **Warning**: The infrastructure of kernel module files mentioned here is 
        !           156:         still under discussion on -current development. It could be changed at 
        !           157:         some point before the next 6.0 release and in that case the description 
        !           158:         in this section will be obsolete. Again, check 
        !           159:         [src/UPDATING](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/UPDATING) 
        !           160:         and
        !           161:         [currenet-users mailing list](http://mail-index.netbsd.org/current-users/)
        !           162:         for updated information.
        !           163: 
        !           164:         There is
        !           165:         [a possible alternative structure for kernel modules](http://mail-index.netbsd.org/current-users/2009/05/10/msg009372.html)
        !           166:         which was proposed on May 2009, but we have not got any conclusion yet. 
        !           167:         This would be because most -current users build their own custom kernels 
        !           168:         from sources, but kernel modules might be rather useful for users who 
        !           169:         don't want to bother to compile their own kernels from sources to just 
        !           170:         try to use optinal functions. Anyway, any feedback about this brandnew 
        !           171:         kernel modules is quite appreciated.
        !           172: 
        !           173:  4. Reboot machine with the new kernel:
        !           174: 
        !           175:         # shutdown -r now
        !           176: 
        !           177:  5. Make sure the new kernel boots and works properly. If your new kernel has
        !           178:     any trouble, you can recover it by loading the renamed old one. If you are 
        !           179:     using modular'ized GENERIC kernel mentioned above, you might also have to 
        !           180:     restore old kernel module files.
        !           181: 
        !           182:  6. Extract the matching base, and any other desirable feature sets **except 
        !           183:     etc**:
        !           184: 
        !           185:         # cd /
        !           186:         # tar -zxpf ~/base.tgz
        !           187:         # tar -zxpf ~/comp.tgz
        !           188:         # ...
        !           189:             
        !           190: 
        !           191:     Don't forget to specify "p" option (preserve permissions) on tar(1) command 
        !           192:     otherwise setuid'ed commands (like su(1)) won't work.
        !           193: 
        !           194:     **Warning**: Extracting `etc.tgz` on the installed system will overwrite your 
        !           195:     local settings.
        !           196: 
        !           197:  7. [Update](http://netbsd.org/docs/current/index.html#etcupdate) `/etc` as the 
        !           198:     last step: postinstall(8) will first check and fix most things that can be 
        !           199:     automated, and etcupdate(8) in the second step will ask on what to merge:
        !           200: 
        !           201:         # /usr/sbin/postinstall -s ~/etc.tgz check
        !           202:         # /usr/sbin/postinstall -s ~/etc.tgz fix
        !           203:         # /usr/sbin/etcupdate -s ~/etc.tgz
        !           204:         # shutdown -r now
        !           205:             
        !           206:     If you have the X sets installed (xbase, ...), you can repeat the 
        !           207:        postinstall and etcupdate steps with xetc.tgz as argument before rebooting.
        !           208: 
        !           209: At this point, you are relatively current and ready to build your own current 
        !           210: source.
        !           211: 
        !           212: ## Downloading current source
        !           213: 
        !           214: See the [Obtaining the sources](http://netbsd.org/docs/guide/en/chap-fetch.html) 
        !           215: section in the [[NetBSD Guide|guide/index]].
        !           216: 
        !           217: ## Building a release from source
        !           218: 
        !           219: See the [Crosscompiling NetBSD](http://netbsd.org/docs/guide/en/chap-build.html) 
        !           220: section in the [[NetBSD Guide|guide/index]].
        !           221: 
        !           222: ## Updating an existing system from source
        !           223: 
        !           224: See the [[Updating an existing system from sources|guide/updating]] section in 
        !           225: the [[NetBSD Guide|guide/index]].
        !           226: 
        !           227: ## Updating the configuration and startup files
        !           228: 
        !           229: # XXX
        !           230: See the [[http://netbsd.org/docs/guide/en/chap-updating.html#updating-etcupdate|More details about the updating of configuration and startup files]] section in the [[http://netbsd.org/docs/guide/en/|NetBSD Guide]].
        !           231: 
        !           232: ## What if I get an error?
        !           233: 
        !           234: If you try to build -current, either from a snapshot or an earlier -current, and 
        !           235: it doesn't work, don't panic. Try these steps:
        !           236: 
        !           237:  1. Read the 
        !           238:     [src/UPDATING](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/UPDATING) 
        !           239:     file from the release you're trying to build.
        !           240: 
        !           241:  2. Read the [current-users archive](http://mail-index.netbsd.org/current-users/)
        !           242:     for hints.
        !           243: 
        !           244:  3. Update again. You may have caught the repository in the middle of a commit 
        !           245:     to several related files, or the problem might have already been fixed.
        !           246: 
        !           247:  4. If all else fails, send email to current-users explaining the problem. 
        !           248:     Include the date, time, and method you used to get your -current sources, as 
        !           249:     well as any local changes you've made. Then put in a **short** script that 
        !           250:     includes the error messages you're getting. Somebody will probably fix the 
        !           251:     problem momentarily.
        !           252: 
        !           253: ## Tracking NetBSD-current with anoncvs
        !           254: 
        !           255: See the
        !           256: [Fetching by CVS](http://netbsd.org/docs/guide/en/chap-fetch.html#chap-fetch-cvs)
        !           257: section in the [[NetBSD Guide|guide/index]].
        !           258: 
        !           259: ### To check out the sources from a certain date
        !           260: 
        !           261:     $ cvs checkout -D 20020501-UTC src
        !           262: 
        !           263: ### To check out the sources from a certain branch
        !           264: 
        !           265:     $ cvs checkout -rnetbsd-5-0 src
        !           266: 
        !           267: See 
        !           268: [src/doc/BRANCHES](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/doc/BRANCHES) 
        !           269: for a description of the branches in the CVS repository.
        !           270: 
        !           271: ### Useful hints
        !           272: 
        !           273:  * Do not use the cvs `-z` flag. The data stream gets out of sync,
        !           274:    leading to corruption on the client, or causing the client to hang
        !           275:    completely. The additional load is also hard on the cvs server.
        !           276:  * If you want to check out a certain branch of the tree, you may want
        !           277:    to take caution not to overwrite any existing directories by
        !           278:    creating a new directory for this branch:
        !           279: 
        !           280:        $ cd /parent/dir/to/checkout/into
        !           281:        $ mkdir NewName-temp
        !           282:        $ cd NewName-temp
        !           283:        $ cvs checkout ... src
        !           284:        $ mv src ../NewName
        !           285:        $ cd ..
        !           286:        $ rmdir NewName-temp
        !           287:         
        !           288:  * You will have to use objdirs in order for cvs updates to work correctly. If 
        !           289:    you happen to get errors from cvs saying things like:
        !           290: 
        !           291:        cvs [update aborted]: could not chdir to gnu/usr.bin/gdb/gdb: Not a directory
        !           292: 
        !           293:    You should do a make cleandir and try again. Make sure to run make obj after 
        !           294:    the cvs update.
        !           295: 
        !           296:  * You can put switches for specific commands in a `.cvsrc` in your home 
        !           297:    directory, and they will be automatically used. A sample `.cvsrc` would be:
        !           298: 
        !           299:        update -dP
        !           300:          checkout -P
        !           301:          diff -u
        !           302: 
        !           303: ## Importing and merging sources.
        !           304: 
        !           305: Sources are imported as follows:
        !           306: 
        !           307:     $ cvs -d /misc/cvsrep import -I ! -I CVS netbsd netbsd current-date
        !           308: 
        !           309: `date` is replaced by the date of the SUP for tracking purposes. The
        !           310: `-I ! -I CVS` options ensure that no file in the source tree is ignored
        !           311: except 'CVS' directories. This is because some NetBSD source files have
        !           312: extensions which are normally ignored by CVS. If there are any conflicts
        !           313: with local patches the import command will report them and will describe
        !           314: a command to merge the conflicts something like:
        !           315: 
        !           316:     $ cvs checkout -jnetbsd:yesterday -jnetbsd netbsd
        !           317: 
        !           318: This merge command will correctly merge the imported NetBSD sources but
        !           319: it will not handle the removal of files locally which have already been
        !           320: removed by the SUP process. To do this the merge command would be:
        !           321: 
        !           322:     $ cvs update -jprevious_import_tag -j current-date
        !           323: 
        !           324: `previous_import_tag` should be replaced with the name of the tag used
        !           325: for the previous cvs import. `date` should be replaced with the current
        !           326: date to yield the same tag used on the current import that has just been
        !           327: merged.
        !           328: 
        !           329: The conflicts reported by the import command are potential conflicts.
        !           330: These are usually merged by the update command but in some cases a real
        !           331: conflict occurs. In these cases a manual merge of the conflicting lines
        !           332: will be required. A real conflict will be reported in the cvs update
        !           333: output as a `C` followed by a filename.
        !           334: 
        !           335: Merging conflicts manually is not a simple process but in most cases it
        !           336: should be resolved by removing the local changes and making the file
        !           337: like the original NetBSD source code.
        !           338: 
        !           339: CVS marks conflicts as follows:
        !           340: 
        !           341:     <<<<<<
        !           342:       code from local file
        !           343:     ======
        !           344:       code from imported file
        !           345:     >>>>>> local revision number of newly imported revision
        !           346: 
        !           347: If the import reports no conflicts the checked out copy of the tree
        !           348: should be updated in exactly the same way as for the conflicts case.
        !           349: 
        !           350: All update and checkout commands should be done in the directory where
        !           351: the sources have been checked out. On my system this is
        !           352: `/usr/src/netbsd`.
        !           353: 
        !           354: If this is the first import then there will be no sources checked out.
        !           355: Assuming you wish to create the source tree in `/usr/src/netbsd` The
        !           356: following commands will check out the source and no merge step is
        !           357: required.
        !           358: 
        !           359:     $ cd /usr/src
        !           360:     $ cvs -d /misc/cvsrep checkout netbsd
        !           361: 
        !           362: ## Tagging a successful build
        !           363: 
        !           364: If the build completes successfully, and produces a working set of binaries, it 
        !           365: can be useful to tag the working sources. This allows rewinding to a working 
        !           366: build tree with a single CVS command in the event that the current tree becomes 
        !           367: unbuildable for any reason. This can be performed by issuing the following 
        !           368: command:
        !           369: 
        !           370:     $ cvs tag successful-build-BUILD date
        !           371: 
        !           372: ### Notes
        !           373: 
        !           374:  * If the NetBSD customised version of CVS, which recognises `$NetBSD$`
        !           375:    markers in files, is not used, the NetBSD revision number of the file is 
        !           376:    available for reference purposes when build problems occur.
        !           377:  * The sup/import/merge sequence described above is quite easily
        !           378:    automatable. The following Perl script automates this process.
        !           379: 
        !           380:        #!/usr/pkg/bin/perl
        !           381:        #
        !           382:        # Script to SUP NetBSD-current, import it into CVS and merge it with
        !           383:        # any local changes.
        !           384:        #
        !           385:        # NOTES:
        !           386:        # This script does no error handling so is not really suitable for
        !           387:        # non-interactive use.
        !           388:        #
        !           389:        # This script has only been test with cvs-1.10.1 and cvs-1.9.18.
        !           390:        #
        !           391:        $SRCROOT="/usr/src/netbsd";
        !           392:        $IMPORTROOT="/misc/import";
        !           393:        $CVSROOT="/misc/cvsrep";
        !           394:        #run the sup into a perl stream
        !           395:        system "/usr/sbin/sup -zsv" ; # This may need to change for none
        !           396:                                      # current systems
        !           397:        # now import the new files into CVS
        !           398:        
        !           399:        chdir $IMPORTROOT or die "Could not cd to $IMPORTROOT\n";
        !           400: 
        !           401:        ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime;
        !           402:        $date = localtime;
        !           403:        $shortdate = sprintf "%02d%02d%04d",$mday,$mon+1,1900+$year;
        !           404:        system "/usr/local/bin/cvs -d$CVSROOT import -I ! -m\"SUP Import $date\" netbsd netbsd current-$shortdate ";
        !           405: 
        !           406:        # make the working directory the local NetBSD Tree
        !           407:        chdir $SRCROOT or die "Could not change to $SRCROOT directory\n";
        !           408: 
        !           409:        # Now do the import.
        !           410:        $lastimport = `cat /usr/src/netbsd/.tag`; # `s are backquotes
        !           411:        $lastimport =~ s/\n//; # strip off any trailing newline in the string
        !           412:        system "/usr/local/bin/cvs update -j $lastimport  -j
        !           413:        current-$shortdate ";
        !           414:        # Now write the current file into tag save file
        !           415:        open TAG,">$SRCROOT/.tag" or die "Could not open new tag file";
        !           416:         print TAG "current-$shortdate";
        !           417:        close TAG;
        !           418: 
        !           419:    This script was written in Perl since it is the scripting tool which the 
        !           420:    author has the most experience with. It should be fairly straightforward to 
        !           421:    write a shell script to perform the same task.
        !           422: 
        !           423:  * Techniques for tracking current with CVS have been discussed several times on 
        !           424:    the NetBSD current-users mailing list. For alternative techniques try 
        !           425:    searching the NetBSD mailing lists.
        !           426: 
        !           427: If you have any comments or suggestions please send them to
        !           428: [Mike Pumford](mailto:mpumford@black-star.demon.co.uk) (who maintains this 
        !           429: entry) or <www@NetBSD.org>.
        !           430: 
        !           431: ## Getting the whole repository
        !           432: 
        !           433: All the procedures described above allow you keeping your own changes in your 
        !           434: repository, which has its advantages if you develop your own software based on 
        !           435: NetBSD. If you don't want to maintain your own CVS repository, but just want to 
        !           436: mirror NetBSD's CVS repository, there are three ways to do so.
        !           437: 
        !           438: Each of the methods described briefly below will get you a copy of the NetBSD 
        !           439: CVS repository (i.e. the RCS ,v files, not the checked out files!). You can then 
        !           440: setup your own anoncvs server or check out to a local harddisk. It's also useful 
        !           441: for fast access to the history information stored in the repository.
        !           442: 
        !           443: The methods to retrieve the whole repository are:
        !           444: 
        !           445:  * **sup**
        !           446: 
        !           447:    If you use sup already to mirror other parts of the NetBSD source,
        !           448:    you will want to add the following lines to your sup config file:
        !           449: 
        !           450:        anoncvs release=all  host=sup.NetBSD.org hostbase=/ftp/pub \
        !           451:        base=/usr prefix=/usr backup use-rel-suffix compress
        !           452:         
        !           453:    After that, run `sup /path/to/supfile anoncvs` to retrieve the files.
        !           454: 
        !           455:    Some example sup files are available in `/usr/share/examples/supfiles`. Also, 
        !           456:    check our
        !           457:    [list of SUP mirrors](http://netbsd.org/mirrors/#sup) to find the server 
        !           458:    closest to you!
        !           459: 
        !           460:  * **rsync**
        !           461: 
        !           462:    Note that rsync puts quite a heavy load on our rsync server, and as such the 
        !           463:    number of concurrent rsync users is restricted. If you still want to try 
        !           464:    rsync, the command to retrieve the repository is:
        !           465: 
        !           466:        $ rsync -v -a --delete --exclude '#cvs.lock' rsync://anoncvs.NetBSD.org/cvsroot/src .
        !           467: 
        !           468:    Please see our [list of rsync mirrors](http://netbsd.org/mirrors/#rsync)!
        !           469: 
        !           470:  * **cvsup**
        !           471: 
        !           472:  * CVSup is not currently available for all NetBSD architectures, since the M3 
        !           473:    compiler has not been ported. On i386, you can mirror the repository from 
        !           474:    cvsup.de.NetBSD.org with the `devel/cvsup` package and the following config 
        !           475:    file:
        !           476: 
        !           477:        *default host=cvsup.de.NetBSD.org
        !           478:        *default base=/usr
        !           479:        *default prefix=/local/NetBSD-cvs
        !           480:        *default release=cvs
        !           481:        *default delete use-rel-suffix
        !           482:        *default compress
        !           483: 
        !           484:        netbsd
        !           485: 
        !           486:    Please see our [list of CVSup mirrors](http://netbsd.org/mirrors/#cvsup)!

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb