File:  [NetBSD Developer Wiki] / wikisrc / tracking_current.mdwn
Revision 1.1: download - view: text, annotated - select for diffs
Sat May 11 09:10:12 2013 UTC (11 months ago) by jdf
Branches: MAIN
CVS tags: HEAD
Convert the "Tracking current guide" (http://netbsd.org/docs/current/index.html,
htdocs/docs/current/index.xml resp.) from Docbook to Markdown.
This conversion was done by Georgi Tsanev in the scope of Google Code-In 2012.

# Tracking NetBSD-current

**Contents**

[[!toc levels=2]]

## Why track NetBSD-current?

The developers of NetBSD have made the current development sources
available to the public for several reasons. Overall, providing
NetBSD-current helps us to create a more stable, accessible system.

It makes it easier for people to become involved in the development of
NetBSD. Distributing the current development sources allows a greater
number of people to see where the system is going, and to become
involved with new features as they are implemented.

It also makes changes from users easier to integrate. If users make
changes against the current development sources, then virtually no
integration is needed to get them into the master source tree.

It also allows wider testing of the software as it is developed. Users
of NetBSD-current are encouraged to send in
[bug reports](http://netbsd.org/support/send-pr.html) about the current sources,
and that helps find and fix bugs. Because people are testing the software soon
after it's written, more bugs can be found and eliminated.

## Things you need to remember

 * People using NetBSD-current are strongly encouraged to subscribe to
   the [current-users](http://netbsd.org/mailinglists/#current-users) mailing
   list. The [source-changes](http://netbsd.org/mailinglists/#source-changes)
   mailing list is also of interest.

# XXX
 * When upgrading to a more recent version of -current you should
    *always* install and boot a new kernel before installing any new
    libs ([[http://netbsd.org/docs/current/index.html#star|*]]). In general the best approach is to try the new
    kernel before anything else, and if you hit any problems see the
    entry in the [[http://netbsd.org/docs/kernel/#problems_compiling_a_current_kernel|Kernel FAQ]].

 * When compiling a -current kernel, always remember to include the
   `COMPAT_<lastrelease>` option (e.g., `COMPAT_16`). As current diverges from
   the last s"table release, compatibility code will be added, but it will only
   be enabled if this option is present. At a bare minimum, you will need this
   compatibility code for the time between booting the new kernel and finishing
   your build via `build.sh`

# XXX
\*: Unless you are certain there have been no new system calls added,
but do it anyway; it's safer.

## Updating an existing system from a current snapshot

*Please remember to check
[src/UPDATING](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/UPDATING) for
quirks around certain specific changes.*

To quickly begin using current, start with a snapshot generated by release
engineering. The current status of each platform can be seen at
[NetBSD Autobuild](http://releng.NetBSD.org/cgi-bin/builds.cgi) and the
corresponding releases found in by date and platform.

 1. Hunt down to the desired `binary/sets` directory, and `mget *.tgz` files
    into your favorite local administrative directory (for example,
    `$HOME/current`); when limited by disk space and/or time, only
	`kern-GENERIC`, `etc`, `base`, and `comp` (if you want a compiler) are
	essential.
 2. Extract the desired kernel (usually `GENERIC`), copy it into (root) directory.

        # cd /root
        # tar -zxpf ~/kern-GENERIC.tgz
        # ln -fh /netbsd /netbsd.old
        # cp netbsd /netbsd.new
        # ln -fh /netbsd.new /netbsd

    > **Warning**: Don't extract any userland binary sets before rebooting your
    > machine with the new kernel. Newer binaries might use new system calls an
    > old running kernel doesn't support.

 3. Check if there are any other files which might also be required by a new
    kernel. Again,
    [src/UPDATING](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/UPDATING)
    might mention possible quirks on daily changes.
	
	The following items are typical files that possibly need to be updated:

     1. bootloader
        
        Usually a machine specific bootloader passes several parameters to a 
        loaded kernel. If some new parameters have been added or some existing 
        APIs between bootloader and kernel are changed you might also have to 
        install new bootloader files for a new kernel to handle new features. A 
        method to update bootloader files is quite machine dependent, so check 
        boot(8) and installboot(8) man pages for details.
        
        
        On i386 and amd64, if you are using FFSv1 for root file system on `wd0a` 
        (i.e. first ATA drive), typical commands to update bootloaders are:

            # tar -C /tmp -zxf ~/base.tgz ./usr/mdec
            # cp /tmp/usr/mdec/boot /
            # installboot -v /dev/rwd0a /tmp/usr/mdec/bootxx_ffsv1

        If you are using FFSv2 for root file system use the following commands 
        instead:

            # tar -C /tmp -zxf ~/base.tgz ./usr/mdec
            # cp /tmp/usr/mdec/boot /
            # installboot -v /dev/rwd0a /tmp/usr/mdec/bootxx_ffsv2

        Note `/usr/mdec/bootxx_ffsv1` and `/usr/mdec/bootxx_ffsv2` are primary 
        bootloaders which are file system dependent. `/usr/mdec/boot` is 
        secondary loader and it's file system independenet.

        If you forget your root file system type (FFSv1 or FFSv2), you can check 
        it by dumpfs(8) command:

            # dumpfs /dev/rwd0a | head -3
            file system: /dev/rwd0a
            format  FFSv2
            endian  little-endian

     2. kernel modules

        A new framework *kernel modules* has been introduced after netbsd-5 was 
        branched, and `GENERIC` kernel on i386 port has been switched to using 
        the kernel module files since November 2008. The kernel module files 
        will be loaded dynamically by the kernel to support various kernel 
        options (including file systems) on demand, rather than linking all 
        necessary (but possibly unused) object files into the kernel binary. 
        This means if you are trying to boot a new `GENERIC` kernel, you also 
        have to prepare new kernel module files for the new kernel. 

        To prepare new kernel module files, you can simply use a new `modules` 
        set file which has been prepared since September 2009:

            # cd /
            # tar -zxpf ~/modules.tgz
                
        Note i386 port also provides `MONOLITHIC` kernel binary in 
        `kern-MONOLITHIC.tgz` set file since October 2009. The `MONOLITHIC` 
        kernel includes all necessary options in its kernel as well as 5.0 and 
        prior `GENERIC` kernels and it doesn't depend on kernel module files at 
        all. If you would just like to test new features of a new kernel without 
        updating kernel modules, using `MONOLITHIC` kernel is easier way for the 
        first and quick trial. 

        It's also a good idea to put an old `MONOLITHIC` kernel into `/` (root) 
        directory for emergency and recovery because if newer modules have some 
        fatal issue there is no easy way to specify an alternative path of old 
        module files to a modular'ized kernel (and you can't rename directories 
        without a working kernel).

        **Warning**: The infrastructure of kernel module files mentioned here is 
        still under discussion on -current development. It could be changed at 
        some point before the next 6.0 release and in that case the description 
        in this section will be obsolete. Again, check 
        [src/UPDATING](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/UPDATING) 
        and
        [currenet-users mailing list](http://mail-index.netbsd.org/current-users/)
        for updated information.

        There is
        [a possible alternative structure for kernel modules](http://mail-index.netbsd.org/current-users/2009/05/10/msg009372.html)
        which was proposed on May 2009, but we have not got any conclusion yet. 
        This would be because most -current users build their own custom kernels 
        from sources, but kernel modules might be rather useful for users who 
        don't want to bother to compile their own kernels from sources to just 
        try to use optinal functions. Anyway, any feedback about this brandnew 
        kernel modules is quite appreciated.

 4. Reboot machine with the new kernel:

        # shutdown -r now

 5. Make sure the new kernel boots and works properly. If your new kernel has
    any trouble, you can recover it by loading the renamed old one. If you are 
    using modular'ized GENERIC kernel mentioned above, you might also have to 
    restore old kernel module files.

 6. Extract the matching base, and any other desirable feature sets **except 
    etc**:

        # cd /
        # tar -zxpf ~/base.tgz
        # tar -zxpf ~/comp.tgz
        # ...
            

    Don't forget to specify "p" option (preserve permissions) on tar(1) command 
    otherwise setuid'ed commands (like su(1)) won't work.

    **Warning**: Extracting `etc.tgz` on the installed system will overwrite your 
    local settings.

 7. [Update](http://netbsd.org/docs/current/index.html#etcupdate) `/etc` as the 
    last step: postinstall(8) will first check and fix most things that can be 
    automated, and etcupdate(8) in the second step will ask on what to merge:

        # /usr/sbin/postinstall -s ~/etc.tgz check
        # /usr/sbin/postinstall -s ~/etc.tgz fix
        # /usr/sbin/etcupdate -s ~/etc.tgz
        # shutdown -r now
            
    If you have the X sets installed (xbase, ...), you can repeat the 
	postinstall and etcupdate steps with xetc.tgz as argument before rebooting.

At this point, you are relatively current and ready to build your own current 
source.

## Downloading current source

See the [Obtaining the sources](http://netbsd.org/docs/guide/en/chap-fetch.html) 
section in the [[NetBSD Guide|guide/index]].

## Building a release from source

See the [Crosscompiling NetBSD](http://netbsd.org/docs/guide/en/chap-build.html) 
section in the [[NetBSD Guide|guide/index]].

## Updating an existing system from source

See the [[Updating an existing system from sources|guide/updating]] section in 
the [[NetBSD Guide|guide/index]].

## Updating the configuration and startup files

# XXX
See the [[http://netbsd.org/docs/guide/en/chap-updating.html#updating-etcupdate|More details about the updating of configuration and startup files]] section in the [[http://netbsd.org/docs/guide/en/|NetBSD Guide]].

## What if I get an error?

If you try to build -current, either from a snapshot or an earlier -current, and 
it doesn't work, don't panic. Try these steps:

 1. Read the 
    [src/UPDATING](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/UPDATING) 
    file from the release you're trying to build.

 2. Read the [current-users archive](http://mail-index.netbsd.org/current-users/)
    for hints.

 3. Update again. You may have caught the repository in the middle of a commit 
    to several related files, or the problem might have already been fixed.

 4. If all else fails, send email to current-users explaining the problem. 
    Include the date, time, and method you used to get your -current sources, as 
    well as any local changes you've made. Then put in a **short** script that 
    includes the error messages you're getting. Somebody will probably fix the 
    problem momentarily.

## Tracking NetBSD-current with anoncvs

See the
[Fetching by CVS](http://netbsd.org/docs/guide/en/chap-fetch.html#chap-fetch-cvs)
section in the [[NetBSD Guide|guide/index]].

### To check out the sources from a certain date

    $ cvs checkout -D 20020501-UTC src

### To check out the sources from a certain branch

    $ cvs checkout -rnetbsd-5-0 src

See 
[src/doc/BRANCHES](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/doc/BRANCHES) 
for a description of the branches in the CVS repository.

### Useful hints

 * Do not use the cvs `-z` flag. The data stream gets out of sync,
   leading to corruption on the client, or causing the client to hang
   completely. The additional load is also hard on the cvs server.
 * If you want to check out a certain branch of the tree, you may want
   to take caution not to overwrite any existing directories by
   creating a new directory for this branch:

       $ cd /parent/dir/to/checkout/into
       $ mkdir NewName-temp
       $ cd NewName-temp
       $ cvs checkout ... src
       $ mv src ../NewName
       $ cd ..
       $ rmdir NewName-temp
        
 * You will have to use objdirs in order for cvs updates to work correctly. If 
   you happen to get errors from cvs saying things like:

       cvs [update aborted]: could not chdir to gnu/usr.bin/gdb/gdb: Not a directory

   You should do a make cleandir and try again. Make sure to run make obj after 
   the cvs update.

 * You can put switches for specific commands in a `.cvsrc` in your home 
   directory, and they will be automatically used. A sample `.cvsrc` would be:

       update -dP
         checkout -P
         diff -u

## Importing and merging sources.

Sources are imported as follows:

    $ cvs -d /misc/cvsrep import -I ! -I CVS netbsd netbsd current-date

`date` is replaced by the date of the SUP for tracking purposes. The
`-I ! -I CVS` options ensure that no file in the source tree is ignored
except 'CVS' directories. This is because some NetBSD source files have
extensions which are normally ignored by CVS. If there are any conflicts
with local patches the import command will report them and will describe
a command to merge the conflicts something like:

    $ cvs checkout -jnetbsd:yesterday -jnetbsd netbsd

This merge command will correctly merge the imported NetBSD sources but
it will not handle the removal of files locally which have already been
removed by the SUP process. To do this the merge command would be:

    $ cvs update -jprevious_import_tag -j current-date

`previous_import_tag` should be replaced with the name of the tag used
for the previous cvs import. `date` should be replaced with the current
date to yield the same tag used on the current import that has just been
merged.

The conflicts reported by the import command are potential conflicts.
These are usually merged by the update command but in some cases a real
conflict occurs. In these cases a manual merge of the conflicting lines
will be required. A real conflict will be reported in the cvs update
output as a `C` followed by a filename.

Merging conflicts manually is not a simple process but in most cases it
should be resolved by removing the local changes and making the file
like the original NetBSD source code.

CVS marks conflicts as follows:

    <<<<<<
      code from local file
    ======
      code from imported file
    >>>>>> local revision number of newly imported revision

If the import reports no conflicts the checked out copy of the tree
should be updated in exactly the same way as for the conflicts case.

All update and checkout commands should be done in the directory where
the sources have been checked out. On my system this is
`/usr/src/netbsd`.

If this is the first import then there will be no sources checked out.
Assuming you wish to create the source tree in `/usr/src/netbsd` The
following commands will check out the source and no merge step is
required.

    $ cd /usr/src
    $ cvs -d /misc/cvsrep checkout netbsd

## Tagging a successful build

If the build completes successfully, and produces a working set of binaries, it 
can be useful to tag the working sources. This allows rewinding to a working 
build tree with a single CVS command in the event that the current tree becomes 
unbuildable for any reason. This can be performed by issuing the following 
command:

    $ cvs tag successful-build-BUILD date

### Notes

 * If the NetBSD customised version of CVS, which recognises `$NetBSD: tracking_current.mdwn,v 1.1 2013/05/11 09:10:12 jdf Exp $`
   markers in files, is not used, the NetBSD revision number of the file is 
   available for reference purposes when build problems occur.
 * The sup/import/merge sequence described above is quite easily
   automatable. The following Perl script automates this process.

       #!/usr/pkg/bin/perl
       #
       # Script to SUP NetBSD-current, import it into CVS and merge it with
       # any local changes.
       #
       # NOTES:
       # This script does no error handling so is not really suitable for
       # non-interactive use.
       #
       # This script has only been test with cvs-1.10.1 and cvs-1.9.18.
       #
       $SRCROOT="/usr/src/netbsd";
       $IMPORTROOT="/misc/import";
       $CVSROOT="/misc/cvsrep";
       #run the sup into a perl stream
       system "/usr/sbin/sup -zsv" ; # This may need to change for none
                                     # current systems
       # now import the new files into CVS
       
       chdir $IMPORTROOT or die "Could not cd to $IMPORTROOT\n";

       ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime;
       $date = localtime;
       $shortdate = sprintf "%02d%02d%04d",$mday,$mon+1,1900+$year;
       system "/usr/local/bin/cvs -d$CVSROOT import -I ! -m\"SUP Import $date\" netbsd netbsd current-$shortdate ";

       # make the working directory the local NetBSD Tree
       chdir $SRCROOT or die "Could not change to $SRCROOT directory\n";

       # Now do the import.
       $lastimport = `cat /usr/src/netbsd/.tag`; # `s are backquotes
       $lastimport =~ s/\n//; # strip off any trailing newline in the string
       system "/usr/local/bin/cvs update -j $lastimport  -j
       current-$shortdate ";
       # Now write the current file into tag save file
       open TAG,">$SRCROOT/.tag" or die "Could not open new tag file";
        print TAG "current-$shortdate";
       close TAG;

   This script was written in Perl since it is the scripting tool which the 
   author has the most experience with. It should be fairly straightforward to 
   write a shell script to perform the same task.

 * Techniques for tracking current with CVS have been discussed several times on 
   the NetBSD current-users mailing list. For alternative techniques try 
   searching the NetBSD mailing lists.

If you have any comments or suggestions please send them to
[Mike Pumford](mailto:mpumford@black-star.demon.co.uk) (who maintains this 
entry) or <www@NetBSD.org>.

## Getting the whole repository

All the procedures described above allow you keeping your own changes in your 
repository, which has its advantages if you develop your own software based on 
NetBSD. If you don't want to maintain your own CVS repository, but just want to 
mirror NetBSD's CVS repository, there are three ways to do so.

Each of the methods described briefly below will get you a copy of the NetBSD 
CVS repository (i.e. the RCS ,v files, not the checked out files!). You can then 
setup your own anoncvs server or check out to a local harddisk. It's also useful 
for fast access to the history information stored in the repository.

The methods to retrieve the whole repository are:

 * **sup**

   If you use sup already to mirror other parts of the NetBSD source,
   you will want to add the following lines to your sup config file:

       anoncvs release=all  host=sup.NetBSD.org hostbase=/ftp/pub \
       base=/usr prefix=/usr backup use-rel-suffix compress
        
   After that, run `sup /path/to/supfile anoncvs` to retrieve the files.

   Some example sup files are available in `/usr/share/examples/supfiles`. Also, 
   check our
   [list of SUP mirrors](http://netbsd.org/mirrors/#sup) to find the server 
   closest to you!

 * **rsync**

   Note that rsync puts quite a heavy load on our rsync server, and as such the 
   number of concurrent rsync users is restricted. If you still want to try 
   rsync, the command to retrieve the repository is:

       $ rsync -v -a --delete --exclude '#cvs.lock' rsync://anoncvs.NetBSD.org/cvsroot/src .

   Please see our [list of rsync mirrors](http://netbsd.org/mirrors/#rsync)!

 * **cvsup**

 * CVSup is not currently available for all NetBSD architectures, since the M3 
   compiler has not been ported. On i386, you can mirror the repository from 
   cvsup.de.NetBSD.org with the `devel/cvsup` package and the following config 
   file:

       *default host=cvsup.de.NetBSD.org
       *default base=/usr
       *default prefix=/local/NetBSD-cvs
       *default release=cvs
       *default delete use-rel-suffix
       *default compress

       netbsd

   Please see our [list of CVSup mirrors](http://netbsd.org/mirrors/#cvsup)!

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb