File:  [NetBSD Developer Wiki] / wikisrc / Attic / tracking_current.mdwn
Revision 1.2: download - view: text, annotated - select for diffs
Sat May 11 09:15:13 2013 UTC (7 years, 6 months ago) by jdf
Branches: MAIN
CVS tags: HEAD
Cosmetics (XXXs and CVS tag removed).

    1: # Tracking NetBSD-current
    2: 
    3: **Contents**
    4: 
    5: [[!toc levels=2]]
    6: 
    7: ## Why track NetBSD-current?
    8: 
    9: The developers of NetBSD have made the current development sources
   10: available to the public for several reasons. Overall, providing
   11: NetBSD-current helps us to create a more stable, accessible system.
   12: 
   13: It makes it easier for people to become involved in the development of
   14: NetBSD. Distributing the current development sources allows a greater
   15: number of people to see where the system is going, and to become
   16: involved with new features as they are implemented.
   17: 
   18: It also makes changes from users easier to integrate. If users make
   19: changes against the current development sources, then virtually no
   20: integration is needed to get them into the master source tree.
   21: 
   22: It also allows wider testing of the software as it is developed. Users
   23: of NetBSD-current are encouraged to send in
   24: [bug reports](http://netbsd.org/support/send-pr.html) about the current sources,
   25: and that helps find and fix bugs. Because people are testing the software soon
   26: after it's written, more bugs can be found and eliminated.
   27: 
   28: ## Things you need to remember
   29: 
   30:  * People using NetBSD-current are strongly encouraged to subscribe to
   31:    the [current-users](http://netbsd.org/mailinglists/#current-users) mailing
   32:    list. The [source-changes](http://netbsd.org/mailinglists/#source-changes)
   33:    mailing list is also of interest.
   34: 
   35:  * When upgrading to a more recent version of -current you should
   36:    *always* install and boot a new kernel before installing any new libs (unless 
   37:    you are certain there have been no new system calls added, but do it anyway; 
   38:    it's safer).In general the best approach is to try the new kernel before 
   39:    anything else, and if you hit any problems see the entry in the
   40:    [Kernel FAQ](http://netbsd.org/docs/kernel/#problems_compiling_a_current_kernel).
   41: 
   42:  * When compiling a -current kernel, always remember to include the
   43:    `COMPAT_<lastrelease>` option (e.g., `COMPAT_16`). As current diverges from
   44:    the last s"table release, compatibility code will be added, but it will only
   45:    be enabled if this option is present. At a bare minimum, you will need this
   46:    compatibility code for the time between booting the new kernel and finishing
   47:    your build via `build.sh`
   48: 
   49: ## Updating an existing system from a current snapshot
   50: 
   51: *Please remember to check
   52: [src/UPDATING](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/UPDATING) for
   53: quirks around certain specific changes.*
   54: 
   55: To quickly begin using current, start with a snapshot generated by release
   56: engineering. The current status of each platform can be seen at
   57: [NetBSD Autobuild](http://releng.NetBSD.org/cgi-bin/builds.cgi) and the
   58: corresponding releases found in by date and platform.
   59: 
   60:  1. Hunt down to the desired `binary/sets` directory, and `mget *.tgz` files
   61:     into your favorite local administrative directory (for example,
   62:     `$HOME/current`); when limited by disk space and/or time, only
   63: 	`kern-GENERIC`, `etc`, `base`, and `comp` (if you want a compiler) are
   64: 	essential.
   65:  2. Extract the desired kernel (usually `GENERIC`), copy it into (root) directory.
   66: 
   67:         # cd /root
   68:         # tar -zxpf ~/kern-GENERIC.tgz
   69:         # ln -fh /netbsd /netbsd.old
   70:         # cp netbsd /netbsd.new
   71:         # ln -fh /netbsd.new /netbsd
   72: 
   73:     > **Warning**: Don't extract any userland binary sets before rebooting your
   74:     > machine with the new kernel. Newer binaries might use new system calls an
   75:     > old running kernel doesn't support.
   76: 
   77:  3. Check if there are any other files which might also be required by a new
   78:     kernel. Again,
   79:     [src/UPDATING](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/UPDATING)
   80:     might mention possible quirks on daily changes.
   81: 	
   82: 	The following items are typical files that possibly need to be updated:
   83: 
   84:      1. bootloader
   85:         
   86:         Usually a machine specific bootloader passes several parameters to a 
   87:         loaded kernel. If some new parameters have been added or some existing 
   88:         APIs between bootloader and kernel are changed you might also have to 
   89:         install new bootloader files for a new kernel to handle new features. A 
   90:         method to update bootloader files is quite machine dependent, so check 
   91:         boot(8) and installboot(8) man pages for details.
   92:         
   93:         
   94:         On i386 and amd64, if you are using FFSv1 for root file system on `wd0a` 
   95:         (i.e. first ATA drive), typical commands to update bootloaders are:
   96: 
   97:             # tar -C /tmp -zxf ~/base.tgz ./usr/mdec
   98:             # cp /tmp/usr/mdec/boot /
   99:             # installboot -v /dev/rwd0a /tmp/usr/mdec/bootxx_ffsv1
  100: 
  101:         If you are using FFSv2 for root file system use the following commands 
  102:         instead:
  103: 
  104:             # tar -C /tmp -zxf ~/base.tgz ./usr/mdec
  105:             # cp /tmp/usr/mdec/boot /
  106:             # installboot -v /dev/rwd0a /tmp/usr/mdec/bootxx_ffsv2
  107: 
  108:         Note `/usr/mdec/bootxx_ffsv1` and `/usr/mdec/bootxx_ffsv2` are primary 
  109:         bootloaders which are file system dependent. `/usr/mdec/boot` is 
  110:         secondary loader and it's file system independenet.
  111: 
  112:         If you forget your root file system type (FFSv1 or FFSv2), you can check 
  113:         it by dumpfs(8) command:
  114: 
  115:             # dumpfs /dev/rwd0a | head -3
  116:             file system: /dev/rwd0a
  117:             format  FFSv2
  118:             endian  little-endian
  119: 
  120:      2. kernel modules
  121: 
  122:         A new framework *kernel modules* has been introduced after netbsd-5 was 
  123:         branched, and `GENERIC` kernel on i386 port has been switched to using 
  124:         the kernel module files since November 2008. The kernel module files 
  125:         will be loaded dynamically by the kernel to support various kernel 
  126:         options (including file systems) on demand, rather than linking all 
  127:         necessary (but possibly unused) object files into the kernel binary. 
  128:         This means if you are trying to boot a new `GENERIC` kernel, you also 
  129:         have to prepare new kernel module files for the new kernel. 
  130: 
  131:         To prepare new kernel module files, you can simply use a new `modules` 
  132:         set file which has been prepared since September 2009:
  133: 
  134:             # cd /
  135:             # tar -zxpf ~/modules.tgz
  136:                 
  137:         Note i386 port also provides `MONOLITHIC` kernel binary in 
  138:         `kern-MONOLITHIC.tgz` set file since October 2009. The `MONOLITHIC` 
  139:         kernel includes all necessary options in its kernel as well as 5.0 and 
  140:         prior `GENERIC` kernels and it doesn't depend on kernel module files at 
  141:         all. If you would just like to test new features of a new kernel without 
  142:         updating kernel modules, using `MONOLITHIC` kernel is easier way for the 
  143:         first and quick trial. 
  144: 
  145:         It's also a good idea to put an old `MONOLITHIC` kernel into `/` (root) 
  146:         directory for emergency and recovery because if newer modules have some 
  147:         fatal issue there is no easy way to specify an alternative path of old 
  148:         module files to a modular'ized kernel (and you can't rename directories 
  149:         without a working kernel).
  150: 
  151:         **Warning**: The infrastructure of kernel module files mentioned here is 
  152:         still under discussion on -current development. It could be changed at 
  153:         some point before the next 6.0 release and in that case the description 
  154:         in this section will be obsolete. Again, check 
  155:         [src/UPDATING](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/UPDATING) 
  156:         and
  157:         [currenet-users mailing list](http://mail-index.netbsd.org/current-users/)
  158:         for updated information.
  159: 
  160:         There is
  161:         [a possible alternative structure for kernel modules](http://mail-index.netbsd.org/current-users/2009/05/10/msg009372.html)
  162:         which was proposed on May 2009, but we have not got any conclusion yet. 
  163:         This would be because most -current users build their own custom kernels 
  164:         from sources, but kernel modules might be rather useful for users who 
  165:         don't want to bother to compile their own kernels from sources to just 
  166:         try to use optinal functions. Anyway, any feedback about this brandnew 
  167:         kernel modules is quite appreciated.
  168: 
  169:  4. Reboot machine with the new kernel:
  170: 
  171:         # shutdown -r now
  172: 
  173:  5. Make sure the new kernel boots and works properly. If your new kernel has
  174:     any trouble, you can recover it by loading the renamed old one. If you are 
  175:     using modular'ized GENERIC kernel mentioned above, you might also have to 
  176:     restore old kernel module files.
  177: 
  178:  6. Extract the matching base, and any other desirable feature sets **except 
  179:     etc**:
  180: 
  181:         # cd /
  182:         # tar -zxpf ~/base.tgz
  183:         # tar -zxpf ~/comp.tgz
  184:         # ...
  185:             
  186: 
  187:     Don't forget to specify "p" option (preserve permissions) on tar(1) command 
  188:     otherwise setuid'ed commands (like su(1)) won't work.
  189: 
  190:     **Warning**: Extracting `etc.tgz` on the installed system will overwrite your 
  191:     local settings.
  192: 
  193:  7. [Update](http://netbsd.org/docs/current/index.html#etcupdate) `/etc` as the 
  194:     last step: postinstall(8) will first check and fix most things that can be 
  195:     automated, and etcupdate(8) in the second step will ask on what to merge:
  196: 
  197:         # /usr/sbin/postinstall -s ~/etc.tgz check
  198:         # /usr/sbin/postinstall -s ~/etc.tgz fix
  199:         # /usr/sbin/etcupdate -s ~/etc.tgz
  200:         # shutdown -r now
  201:             
  202:     If you have the X sets installed (xbase, ...), you can repeat the 
  203: 	postinstall and etcupdate steps with xetc.tgz as argument before rebooting.
  204: 
  205: At this point, you are relatively current and ready to build your own current 
  206: source.
  207: 
  208: ## Downloading current source
  209: 
  210: See the [Obtaining the sources](http://netbsd.org/docs/guide/en/chap-fetch.html) 
  211: section in the [[NetBSD Guide|guide/index]].
  212: 
  213: ## Building a release from source
  214: 
  215: See the [Crosscompiling NetBSD](http://netbsd.org/docs/guide/en/chap-build.html) 
  216: section in the [[NetBSD Guide|guide/index]].
  217: 
  218: ## Updating an existing system from source
  219: 
  220: See the [[Updating an existing system from sources|guide/updating]] section in 
  221: the [[NetBSD Guide|guide/index]].
  222: 
  223: ## Updating the configuration and startup files
  224: 
  225: See the
  226: [[More details about the updating of configuration and startup files|guide/updating]]
  227: section in the [[NetBSD Guide|guide/index]].
  228: 
  229: ## What if I get an error?
  230: 
  231: If you try to build -current, either from a snapshot or an earlier -current, and 
  232: it doesn't work, don't panic. Try these steps:
  233: 
  234:  1. Read the 
  235:     [src/UPDATING](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/UPDATING) 
  236:     file from the release you're trying to build.
  237: 
  238:  2. Read the [current-users archive](http://mail-index.netbsd.org/current-users/)
  239:     for hints.
  240: 
  241:  3. Update again. You may have caught the repository in the middle of a commit 
  242:     to several related files, or the problem might have already been fixed.
  243: 
  244:  4. If all else fails, send email to current-users explaining the problem. 
  245:     Include the date, time, and method you used to get your -current sources, as 
  246:     well as any local changes you've made. Then put in a **short** script that 
  247:     includes the error messages you're getting. Somebody will probably fix the 
  248:     problem momentarily.
  249: 
  250: ## Tracking NetBSD-current with anoncvs
  251: 
  252: See the
  253: [Fetching by CVS](http://netbsd.org/docs/guide/en/chap-fetch.html#chap-fetch-cvs)
  254: section in the [[NetBSD Guide|guide/index]].
  255: 
  256: ### To check out the sources from a certain date
  257: 
  258:     $ cvs checkout -D 20020501-UTC src
  259: 
  260: ### To check out the sources from a certain branch
  261: 
  262:     $ cvs checkout -rnetbsd-5-0 src
  263: 
  264: See 
  265: [src/doc/BRANCHES](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/doc/BRANCHES) 
  266: for a description of the branches in the CVS repository.
  267: 
  268: ### Useful hints
  269: 
  270:  * Do not use the cvs `-z` flag. The data stream gets out of sync,
  271:    leading to corruption on the client, or causing the client to hang
  272:    completely. The additional load is also hard on the cvs server.
  273:  * If you want to check out a certain branch of the tree, you may want
  274:    to take caution not to overwrite any existing directories by
  275:    creating a new directory for this branch:
  276: 
  277:        $ cd /parent/dir/to/checkout/into
  278:        $ mkdir NewName-temp
  279:        $ cd NewName-temp
  280:        $ cvs checkout ... src
  281:        $ mv src ../NewName
  282:        $ cd ..
  283:        $ rmdir NewName-temp
  284:         
  285:  * You will have to use objdirs in order for cvs updates to work correctly. If 
  286:    you happen to get errors from cvs saying things like:
  287: 
  288:        cvs [update aborted]: could not chdir to gnu/usr.bin/gdb/gdb: Not a directory
  289: 
  290:    You should do a make cleandir and try again. Make sure to run make obj after 
  291:    the cvs update.
  292: 
  293:  * You can put switches for specific commands in a `.cvsrc` in your home 
  294:    directory, and they will be automatically used. A sample `.cvsrc` would be:
  295: 
  296:        update -dP
  297:          checkout -P
  298:          diff -u
  299: 
  300: ## Importing and merging sources.
  301: 
  302: Sources are imported as follows:
  303: 
  304:     $ cvs -d /misc/cvsrep import -I ! -I CVS netbsd netbsd current-date
  305: 
  306: `date` is replaced by the date of the SUP for tracking purposes. The
  307: `-I ! -I CVS` options ensure that no file in the source tree is ignored
  308: except 'CVS' directories. This is because some NetBSD source files have
  309: extensions which are normally ignored by CVS. If there are any conflicts
  310: with local patches the import command will report them and will describe
  311: a command to merge the conflicts something like:
  312: 
  313:     $ cvs checkout -jnetbsd:yesterday -jnetbsd netbsd
  314: 
  315: This merge command will correctly merge the imported NetBSD sources but
  316: it will not handle the removal of files locally which have already been
  317: removed by the SUP process. To do this the merge command would be:
  318: 
  319:     $ cvs update -jprevious_import_tag -j current-date
  320: 
  321: `previous_import_tag` should be replaced with the name of the tag used
  322: for the previous cvs import. `date` should be replaced with the current
  323: date to yield the same tag used on the current import that has just been
  324: merged.
  325: 
  326: The conflicts reported by the import command are potential conflicts.
  327: These are usually merged by the update command but in some cases a real
  328: conflict occurs. In these cases a manual merge of the conflicting lines
  329: will be required. A real conflict will be reported in the cvs update
  330: output as a `C` followed by a filename.
  331: 
  332: Merging conflicts manually is not a simple process but in most cases it
  333: should be resolved by removing the local changes and making the file
  334: like the original NetBSD source code.
  335: 
  336: CVS marks conflicts as follows:
  337: 
  338:     <<<<<<
  339:       code from local file
  340:     ======
  341:       code from imported file
  342:     >>>>>> local revision number of newly imported revision
  343: 
  344: If the import reports no conflicts the checked out copy of the tree
  345: should be updated in exactly the same way as for the conflicts case.
  346: 
  347: All update and checkout commands should be done in the directory where
  348: the sources have been checked out. On my system this is
  349: `/usr/src/netbsd`.
  350: 
  351: If this is the first import then there will be no sources checked out.
  352: Assuming you wish to create the source tree in `/usr/src/netbsd` The
  353: following commands will check out the source and no merge step is
  354: required.
  355: 
  356:     $ cd /usr/src
  357:     $ cvs -d /misc/cvsrep checkout netbsd
  358: 
  359: ## Tagging a successful build
  360: 
  361: If the build completes successfully, and produces a working set of binaries, it 
  362: can be useful to tag the working sources. This allows rewinding to a working 
  363: build tree with a single CVS command in the event that the current tree becomes 
  364: unbuildable for any reason. This can be performed by issuing the following 
  365: command:
  366: 
  367:     $ cvs tag successful-build-BUILD date
  368: 
  369: ### Notes
  370:  * If the NetBSD customised version of CVS, which recognises *\$NetBSD\$*
  371:    markers in files, is not used, the NetBSD revision number of the file is 
  372:    available for reference purposes when build problems occur.
  373:  * The sup/import/merge sequence described above is quite easily
  374:    automatable. The following Perl script automates this process.
  375: 
  376:        #!/usr/pkg/bin/perl
  377:        #
  378:        # Script to SUP NetBSD-current, import it into CVS and merge it with
  379:        # any local changes.
  380:        #
  381:        # NOTES:
  382:        # This script does no error handling so is not really suitable for
  383:        # non-interactive use.
  384:        #
  385:        # This script has only been test with cvs-1.10.1 and cvs-1.9.18.
  386:        #
  387:        $SRCROOT="/usr/src/netbsd";
  388:        $IMPORTROOT="/misc/import";
  389:        $CVSROOT="/misc/cvsrep";
  390:        #run the sup into a perl stream
  391:        system "/usr/sbin/sup -zsv" ; # This may need to change for none
  392:                                      # current systems
  393:        # now import the new files into CVS
  394:        
  395:        chdir $IMPORTROOT or die "Could not cd to $IMPORTROOT\n";
  396: 
  397:        ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime;
  398:        $date = localtime;
  399:        $shortdate = sprintf "%02d%02d%04d",$mday,$mon+1,1900+$year;
  400:        system "/usr/local/bin/cvs -d$CVSROOT import -I ! -m\"SUP Import $date\" netbsd netbsd current-$shortdate ";
  401: 
  402:        # make the working directory the local NetBSD Tree
  403:        chdir $SRCROOT or die "Could not change to $SRCROOT directory\n";
  404: 
  405:        # Now do the import.
  406:        $lastimport = `cat /usr/src/netbsd/.tag`; # `s are backquotes
  407:        $lastimport =~ s/\n//; # strip off any trailing newline in the string
  408:        system "/usr/local/bin/cvs update -j $lastimport  -j
  409:        current-$shortdate ";
  410:        # Now write the current file into tag save file
  411:        open TAG,">$SRCROOT/.tag" or die "Could not open new tag file";
  412:         print TAG "current-$shortdate";
  413:        close TAG;
  414: 
  415:    This script was written in Perl since it is the scripting tool which the 
  416:    author has the most experience with. It should be fairly straightforward to 
  417:    write a shell script to perform the same task.
  418: 
  419:  * Techniques for tracking current with CVS have been discussed several times on 
  420:    the NetBSD current-users mailing list. For alternative techniques try 
  421:    searching the NetBSD mailing lists.
  422: 
  423: If you have any comments or suggestions please send them to
  424: [Mike Pumford](mailto:mpumford@black-star.demon.co.uk) (who maintains this 
  425: entry) or <www@NetBSD.org>.
  426: 
  427: ## Getting the whole repository
  428: 
  429: All the procedures described above allow you keeping your own changes in your 
  430: repository, which has its advantages if you develop your own software based on 
  431: NetBSD. If you don't want to maintain your own CVS repository, but just want to 
  432: mirror NetBSD's CVS repository, there are three ways to do so.
  433: 
  434: Each of the methods described briefly below will get you a copy of the NetBSD 
  435: CVS repository (i.e. the RCS ,v files, not the checked out files!). You can then 
  436: setup your own anoncvs server or check out to a local harddisk. It's also useful 
  437: for fast access to the history information stored in the repository.
  438: 
  439: The methods to retrieve the whole repository are:
  440: 
  441:  * **sup**
  442: 
  443:    If you use sup already to mirror other parts of the NetBSD source,
  444:    you will want to add the following lines to your sup config file:
  445: 
  446:        anoncvs release=all  host=sup.NetBSD.org hostbase=/ftp/pub \
  447:        base=/usr prefix=/usr backup use-rel-suffix compress
  448:         
  449:    After that, run `sup /path/to/supfile anoncvs` to retrieve the files.
  450: 
  451:    Some example sup files are available in `/usr/share/examples/supfiles`. Also, 
  452:    check our
  453:    [list of SUP mirrors](http://netbsd.org/mirrors/#sup) to find the server 
  454:    closest to you!
  455: 
  456:  * **rsync**
  457: 
  458:    Note that rsync puts quite a heavy load on our rsync server, and as such the 
  459:    number of concurrent rsync users is restricted. If you still want to try 
  460:    rsync, the command to retrieve the repository is:
  461: 
  462:        $ rsync -v -a --delete --exclude '#cvs.lock' rsync://anoncvs.NetBSD.org/cvsroot/src .
  463: 
  464:    Please see our [list of rsync mirrors](http://netbsd.org/mirrors/#rsync)!
  465: 
  466:  * **cvsup**
  467: 
  468:  * CVSup is not currently available for all NetBSD architectures, since the M3 
  469:    compiler has not been ported. On i386, you can mirror the repository from 
  470:    cvsup.de.NetBSD.org with the `devel/cvsup` package and the following config 
  471:    file:
  472: 
  473:        *default host=cvsup.de.NetBSD.org
  474:        *default base=/usr
  475:        *default prefix=/local/NetBSD-cvs
  476:        *default release=cvs
  477:        *default delete use-rel-suffix
  478:        *default compress
  479: 
  480:        netbsd
  481: 
  482:    Please see our [list of CVSup mirrors](http://netbsd.org/mirrors/#cvsup)!

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb