wikisrc/tracking_current.mdwn - view

File: [NetBSD Developer Wiki] / wikisrc / Attic / tracking_current.mdwn
Revision 1.1: download - view: text, annotated - select for diffs
Sat May 11 09:10:12 2013 UTC (10 years, 7 months ago) by jdf
Branches: MAIN
CVS tags: HEAD

Convert the "Tracking current guide" (http://netbsd.org/docs/current/index.html,
htdocs/docs/current/index.xml resp.) from Docbook to Markdown.
This conversion was done by Georgi Tsanev in the scope of Google Code-In 2012.

1: # Tracking NetBSD-current 2: 3: **Contents** 4: 5: [[!toc levels=2]] 6: 7: ## Why track NetBSD-current? 8: 9: The developers of NetBSD have made the current development sources 10: available to the public for several reasons. Overall, providing 11: NetBSD-current helps us to create a more stable, accessible system. 12: 13: It makes it easier for people to become involved in the development of 14: NetBSD. Distributing the current development sources allows a greater 15: number of people to see where the system is going, and to become 16: involved with new features as they are implemented. 17: 18: It also makes changes from users easier to integrate. If users make 19: changes against the current development sources, then virtually no 20: integration is needed to get them into the master source tree. 21: 22: It also allows wider testing of the software as it is developed. Users 23: of NetBSD-current are encouraged to send in 24: [bug reports](http://netbsd.org/support/send-pr.html) about the current sources, 25: and that helps find and fix bugs. Because people are testing the software soon 26: after it's written, more bugs can be found and eliminated. 27: 28: ## Things you need to remember 29: 30: * People using NetBSD-current are strongly encouraged to subscribe to 31: the [current-users](http://netbsd.org/mailinglists/#current-users) mailing 32: list. The [source-changes](http://netbsd.org/mailinglists/#source-changes) 33: mailing list is also of interest. 34: 35: # XXX 36: * When upgrading to a more recent version of -current you should 37: *always* install and boot a new kernel before installing any new 38: libs ([[http://netbsd.org/docs/current/index.html#star|*]]). In general the best approach is to try the new 39: kernel before anything else, and if you hit any problems see the 40: entry in the [[http://netbsd.org/docs/kernel/#problems_compiling_a_current_kernel|Kernel FAQ]]. 41: 42: * When compiling a -current kernel, always remember to include the 43: `COMPAT_<lastrelease>` option (e.g., `COMPAT_16`). As current diverges from 44: the last s"table release, compatibility code will be added, but it will only 45: be enabled if this option is present. At a bare minimum, you will need this 46: compatibility code for the time between booting the new kernel and finishing 47: your build via `build.sh` 48: 49: # XXX 50: \*: Unless you are certain there have been no new system calls added, 51: but do it anyway; it's safer. 52: 53: ## Updating an existing system from a current snapshot 54: 55: *Please remember to check 56: [src/UPDATING](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/UPDATING) for 57: quirks around certain specific changes.* 58: 59: To quickly begin using current, start with a snapshot generated by release 60: engineering. The current status of each platform can be seen at 61: [NetBSD Autobuild](http://releng.NetBSD.org/cgi-bin/builds.cgi) and the 62: corresponding releases found in by date and platform. 63: 64: 1. Hunt down to the desired `binary/sets` directory, and `mget *.tgz` files 65: into your favorite local administrative directory (for example, 66: `$HOME/current`); when limited by disk space and/or time, only 67: `kern-GENERIC`, `etc`, `base`, and `comp` (if you want a compiler) are 68: essential. 69: 2. Extract the desired kernel (usually `GENERIC`), copy it into (root) directory. 70: 71: # cd /root 72: # tar -zxpf ~/kern-GENERIC.tgz 73: # ln -fh /netbsd /netbsd.old 74: # cp netbsd /netbsd.new 75: # ln -fh /netbsd.new /netbsd 76: 77: > **Warning**: Don't extract any userland binary sets before rebooting your 78: > machine with the new kernel. Newer binaries might use new system calls an 79: > old running kernel doesn't support. 80: 81: 3. Check if there are any other files which might also be required by a new 82: kernel. Again, 83: [src/UPDATING](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/UPDATING) 84: might mention possible quirks on daily changes. 85: 86: The following items are typical files that possibly need to be updated: 87: 88: 1. bootloader 89: 90: Usually a machine specific bootloader passes several parameters to a 91: loaded kernel. If some new parameters have been added or some existing 92: APIs between bootloader and kernel are changed you might also have to 93: install new bootloader files for a new kernel to handle new features. A 94: method to update bootloader files is quite machine dependent, so check 95: boot(8) and installboot(8) man pages for details. 96: 97: 98: On i386 and amd64, if you are using FFSv1 for root file system on `wd0a` 99: (i.e. first ATA drive), typical commands to update bootloaders are: 100: 101: # tar -C /tmp -zxf ~/base.tgz ./usr/mdec 102: # cp /tmp/usr/mdec/boot / 103: # installboot -v /dev/rwd0a /tmp/usr/mdec/bootxx_ffsv1 104: 105: If you are using FFSv2 for root file system use the following commands 106: instead: 107: 108: # tar -C /tmp -zxf ~/base.tgz ./usr/mdec 109: # cp /tmp/usr/mdec/boot / 110: # installboot -v /dev/rwd0a /tmp/usr/mdec/bootxx_ffsv2 111: 112: Note `/usr/mdec/bootxx_ffsv1` and `/usr/mdec/bootxx_ffsv2` are primary 113: bootloaders which are file system dependent. `/usr/mdec/boot` is 114: secondary loader and it's file system independenet. 115: 116: If you forget your root file system type (FFSv1 or FFSv2), you can check 117: it by dumpfs(8) command: 118: 119: # dumpfs /dev/rwd0a | head -3 120: file system: /dev/rwd0a 121: format FFSv2 122: endian little-endian 123: 124: 2. kernel modules 125: 126: A new framework *kernel modules* has been introduced after netbsd-5 was 127: branched, and `GENERIC` kernel on i386 port has been switched to using 128: the kernel module files since November 2008. The kernel module files 129: will be loaded dynamically by the kernel to support various kernel 130: options (including file systems) on demand, rather than linking all 131: necessary (but possibly unused) object files into the kernel binary. 132: This means if you are trying to boot a new `GENERIC` kernel, you also 133: have to prepare new kernel module files for the new kernel. 134: 135: To prepare new kernel module files, you can simply use a new `modules` 136: set file which has been prepared since September 2009: 137: 138: # cd / 139: # tar -zxpf ~/modules.tgz 140: 141: Note i386 port also provides `MONOLITHIC` kernel binary in 142: `kern-MONOLITHIC.tgz` set file since October 2009. The `MONOLITHIC` 143: kernel includes all necessary options in its kernel as well as 5.0 and 144: prior `GENERIC` kernels and it doesn't depend on kernel module files at 145: all. If you would just like to test new features of a new kernel without 146: updating kernel modules, using `MONOLITHIC` kernel is easier way for the 147: first and quick trial. 148: 149: It's also a good idea to put an old `MONOLITHIC` kernel into `/` (root) 150: directory for emergency and recovery because if newer modules have some 151: fatal issue there is no easy way to specify an alternative path of old 152: module files to a modular'ized kernel (and you can't rename directories 153: without a working kernel). 154: 155: **Warning**: The infrastructure of kernel module files mentioned here is 156: still under discussion on -current development. It could be changed at 157: some point before the next 6.0 release and in that case the description 158: in this section will be obsolete. Again, check 159: [src/UPDATING](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/UPDATING) 160: and 161: [currenet-users mailing list](http://mail-index.netbsd.org/current-users/) 162: for updated information. 163: 164: There is 165: [a possible alternative structure for kernel modules](http://mail-index.netbsd.org/current-users/2009/05/10/msg009372.html) 166: which was proposed on May 2009, but we have not got any conclusion yet. 167: This would be because most -current users build their own custom kernels 168: from sources, but kernel modules might be rather useful for users who 169: don't want to bother to compile their own kernels from sources to just 170: try to use optinal functions. Anyway, any feedback about this brandnew 171: kernel modules is quite appreciated. 172: 173: 4. Reboot machine with the new kernel: 174: 175: # shutdown -r now 176: 177: 5. Make sure the new kernel boots and works properly. If your new kernel has 178: any trouble, you can recover it by loading the renamed old one. If you are 179: using modular'ized GENERIC kernel mentioned above, you might also have to 180: restore old kernel module files. 181: 182: 6. Extract the matching base, and any other desirable feature sets **except 183: etc**: 184: 185: # cd / 186: # tar -zxpf ~/base.tgz 187: # tar -zxpf ~/comp.tgz 188: # ... 189: 190: 191: Don't forget to specify "p" option (preserve permissions) on tar(1) command 192: otherwise setuid'ed commands (like su(1)) won't work. 193: 194: **Warning**: Extracting `etc.tgz` on the installed system will overwrite your 195: local settings. 196: 197: 7. [Update](http://netbsd.org/docs/current/index.html#etcupdate) `/etc` as the 198: last step: postinstall(8) will first check and fix most things that can be 199: automated, and etcupdate(8) in the second step will ask on what to merge: 200: 201: # /usr/sbin/postinstall -s ~/etc.tgz check 202: # /usr/sbin/postinstall -s ~/etc.tgz fix 203: # /usr/sbin/etcupdate -s ~/etc.tgz 204: # shutdown -r now 205: 206: If you have the X sets installed (xbase, ...), you can repeat the 207: postinstall and etcupdate steps with xetc.tgz as argument before rebooting. 208: 209: At this point, you are relatively current and ready to build your own current 210: source. 211: 212: ## Downloading current source 213: 214: See the [Obtaining the sources](http://netbsd.org/docs/guide/en/chap-fetch.html) 215: section in the [[NetBSD Guide|guide/index]]. 216: 217: ## Building a release from source 218: 219: See the [Crosscompiling NetBSD](http://netbsd.org/docs/guide/en/chap-build.html) 220: section in the [[NetBSD Guide|guide/index]]. 221: 222: ## Updating an existing system from source 223: 224: See the [[Updating an existing system from sources|guide/updating]] section in 225: the [[NetBSD Guide|guide/index]]. 226: 227: ## Updating the configuration and startup files 228: 229: # XXX 230: See the [[http://netbsd.org/docs/guide/en/chap-updating.html#updating-etcupdate|More details about the updating of configuration and startup files]] section in the [[http://netbsd.org/docs/guide/en/|NetBSD Guide]]. 231: 232: ## What if I get an error? 233: 234: If you try to build -current, either from a snapshot or an earlier -current, and 235: it doesn't work, don't panic. Try these steps: 236: 237: 1. Read the 238: [src/UPDATING](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/UPDATING) 239: file from the release you're trying to build. 240: 241: 2. Read the [current-users archive](http://mail-index.netbsd.org/current-users/) 242: for hints. 243: 244: 3. Update again. You may have caught the repository in the middle of a commit 245: to several related files, or the problem might have already been fixed. 246: 247: 4. If all else fails, send email to current-users explaining the problem. 248: Include the date, time, and method you used to get your -current sources, as 249: well as any local changes you've made. Then put in a **short** script that 250: includes the error messages you're getting. Somebody will probably fix the 251: problem momentarily. 252: 253: ## Tracking NetBSD-current with anoncvs 254: 255: See the 256: [Fetching by CVS](http://netbsd.org/docs/guide/en/chap-fetch.html#chap-fetch-cvs) 257: section in the [[NetBSD Guide|guide/index]]. 258: 259: ### To check out the sources from a certain date 260: 261: $ cvs checkout -D 20020501-UTC src 262: 263: ### To check out the sources from a certain branch 264: 265: $ cvs checkout -rnetbsd-5-0 src 266: 267: See 268: [src/doc/BRANCHES](http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/doc/BRANCHES) 269: for a description of the branches in the CVS repository. 270: 271: ### Useful hints 272: 273: * Do not use the cvs `-z` flag. The data stream gets out of sync, 274: leading to corruption on the client, or causing the client to hang 275: completely. The additional load is also hard on the cvs server. 276: * If you want to check out a certain branch of the tree, you may want 277: to take caution not to overwrite any existing directories by 278: creating a new directory for this branch: 279: 280: $ cd /parent/dir/to/checkout/into 281: $ mkdir NewName-temp 282: $ cd NewName-temp 283: $ cvs checkout ... src 284: $ mv src ../NewName 285: $ cd .. 286: $ rmdir NewName-temp 287: 288: * You will have to use objdirs in order for cvs updates to work correctly. If 289: you happen to get errors from cvs saying things like: 290: 291: cvs [update aborted]: could not chdir to gnu/usr.bin/gdb/gdb: Not a directory 292: 293: You should do a make cleandir and try again. Make sure to run make obj after 294: the cvs update. 295: 296: * You can put switches for specific commands in a `.cvsrc` in your home 297: directory, and they will be automatically used. A sample `.cvsrc` would be: 298: 299: update -dP 300: checkout -P 301: diff -u 302: 303: ## Importing and merging sources. 304: 305: Sources are imported as follows: 306: 307: $ cvs -d /misc/cvsrep import -I ! -I CVS netbsd netbsd current-date 308: 309: `date` is replaced by the date of the SUP for tracking purposes. The 310: `-I ! -I CVS` options ensure that no file in the source tree is ignored 311: except 'CVS' directories. This is because some NetBSD source files have 312: extensions which are normally ignored by CVS. If there are any conflicts 313: with local patches the import command will report them and will describe 314: a command to merge the conflicts something like: 315: 316: $ cvs checkout -jnetbsd:yesterday -jnetbsd netbsd 317: 318: This merge command will correctly merge the imported NetBSD sources but 319: it will not handle the removal of files locally which have already been 320: removed by the SUP process. To do this the merge command would be: 321: 322: $ cvs update -jprevious_import_tag -j current-date 323: 324: `previous_import_tag` should be replaced with the name of the tag used 325: for the previous cvs import. `date` should be replaced with the current 326: date to yield the same tag used on the current import that has just been 327: merged. 328: 329: The conflicts reported by the import command are potential conflicts. 330: These are usually merged by the update command but in some cases a real 331: conflict occurs. In these cases a manual merge of the conflicting lines 332: will be required. A real conflict will be reported in the cvs update 333: output as a `C` followed by a filename. 334: 335: Merging conflicts manually is not a simple process but in most cases it 336: should be resolved by removing the local changes and making the file 337: like the original NetBSD source code. 338: 339: CVS marks conflicts as follows: 340: 341: <<<<<< 342: code from local file 343: ====== 344: code from imported file 345: >>>>>> local revision number of newly imported revision 346: 347: If the import reports no conflicts the checked out copy of the tree 348: should be updated in exactly the same way as for the conflicts case. 349: 350: All update and checkout commands should be done in the directory where 351: the sources have been checked out. On my system this is 352: `/usr/src/netbsd`. 353: 354: If this is the first import then there will be no sources checked out. 355: Assuming you wish to create the source tree in `/usr/src/netbsd` The 356: following commands will check out the source and no merge step is 357: required. 358: 359: $ cd /usr/src 360: $ cvs -d /misc/cvsrep checkout netbsd 361: 362: ## Tagging a successful build 363: 364: If the build completes successfully, and produces a working set of binaries, it 365: can be useful to tag the working sources. This allows rewinding to a working 366: build tree with a single CVS command in the event that the current tree becomes 367: unbuildable for any reason. This can be performed by issuing the following 368: command: 369: 370: $ cvs tag successful-build-BUILD date 371: 372: ### Notes 373: 374: * If the NetBSD customised version of CVS, which recognises `$NetBSD: tracking_current.mdwn,v 1.1 2013/05/11 09:10:12 jdf Exp $` 375: markers in files, is not used, the NetBSD revision number of the file is 376: available for reference purposes when build problems occur. 377: * The sup/import/merge sequence described above is quite easily 378: automatable. The following Perl script automates this process. 379: 380: #!/usr/pkg/bin/perl 381: # 382: # Script to SUP NetBSD-current, import it into CVS and merge it with 383: # any local changes. 384: # 385: # NOTES: 386: # This script does no error handling so is not really suitable for 387: # non-interactive use. 388: # 389: # This script has only been test with cvs-1.10.1 and cvs-1.9.18. 390: # 391: $SRCROOT="/usr/src/netbsd"; 392: $IMPORTROOT="/misc/import"; 393: $CVSROOT="/misc/cvsrep"; 394: #run the sup into a perl stream 395: system "/usr/sbin/sup -zsv" ; # This may need to change for none 396: # current systems 397: # now import the new files into CVS 398: 399: chdir $IMPORTROOT or die "Could not cd to $IMPORTROOT\n"; 400: 401: ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime; 402: $date = localtime; 403: $shortdate = sprintf "%02d%02d%04d",$mday,$mon+1,1900+$year; 404: system "/usr/local/bin/cvs -d$CVSROOT import -I ! -m\"SUP Import $date\" netbsd netbsd current-$shortdate "; 405: 406: # make the working directory the local NetBSD Tree 407: chdir $SRCROOT or die "Could not change to $SRCROOT directory\n"; 408: 409: # Now do the import. 410: $lastimport = `cat /usr/src/netbsd/.tag`; # `s are backquotes 411: $lastimport =~ s/\n//; # strip off any trailing newline in the string 412: system "/usr/local/bin/cvs update -j $lastimport -j 413: current-$shortdate "; 414: # Now write the current file into tag save file 415: open TAG,">$SRCROOT/.tag" or die "Could not open new tag file"; 416: print TAG "current-$shortdate"; 417: close TAG; 418: 419: This script was written in Perl since it is the scripting tool which the 420: author has the most experience with. It should be fairly straightforward to 421: write a shell script to perform the same task. 422: 423: * Techniques for tracking current with CVS have been discussed several times on 424: the NetBSD current-users mailing list. For alternative techniques try 425: searching the NetBSD mailing lists. 426: 427: If you have any comments or suggestions please send them to 428: [Mike Pumford](mailto:mpumford@black-star.demon.co.uk) (who maintains this 429: entry) or <www@NetBSD.org>. 430: 431: ## Getting the whole repository 432: 433: All the procedures described above allow you keeping your own changes in your 434: repository, which has its advantages if you develop your own software based on 435: NetBSD. If you don't want to maintain your own CVS repository, but just want to 436: mirror NetBSD's CVS repository, there are three ways to do so. 437: 438: Each of the methods described briefly below will get you a copy of the NetBSD 439: CVS repository (i.e. the RCS ,v files, not the checked out files!). You can then 440: setup your own anoncvs server or check out to a local harddisk. It's also useful 441: for fast access to the history information stored in the repository. 442: 443: The methods to retrieve the whole repository are: 444: 445: * **sup** 446: 447: If you use sup already to mirror other parts of the NetBSD source, 448: you will want to add the following lines to your sup config file: 449: 450: anoncvs release=all host=sup.NetBSD.org hostbase=/ftp/pub \ 451: base=/usr prefix=/usr backup use-rel-suffix compress 452: 453: After that, run `sup /path/to/supfile anoncvs` to retrieve the files. 454: 455: Some example sup files are available in `/usr/share/examples/supfiles`. Also, 456: check our 457: [list of SUP mirrors](http://netbsd.org/mirrors/#sup) to find the server 458: closest to you! 459: 460: * **rsync** 461: 462: Note that rsync puts quite a heavy load on our rsync server, and as such the 463: number of concurrent rsync users is restricted. If you still want to try 464: rsync, the command to retrieve the repository is: 465: 466: $ rsync -v -a --delete --exclude '#cvs.lock' rsync://anoncvs.NetBSD.org/cvsroot/src . 467: 468: Please see our [list of rsync mirrors](http://netbsd.org/mirrors/#rsync)! 469: 470: * **cvsup** 471: 472: * CVSup is not currently available for all NetBSD architectures, since the M3 473: compiler has not been ported. On i386, you can mirror the repository from 474: cvsup.de.NetBSD.org with the `devel/cvsup` package and the following config 475: file: 476: 477: *default host=cvsup.de.NetBSD.org 478: *default base=/usr 479: *default prefix=/local/NetBSD-cvs 480: *default release=cvs 481: *default delete use-rel-suffix 482: *default compress 483: 484: netbsd 485: 486: Please see our [list of CVSup mirrors](http://netbsd.org/mirrors/#cvsup)!