Contents

  1. Crosscompiling NetBSD with build.sh
    1. Building the crosscompiler
    2. Configuring the kernel manually
    3. Crosscompiling the kernel manually
    4. Crosscompiling the kernel with build.sh
    5. Crosscompiling the userland
    6. Crosscompiling the X Window System
    7. Changing build behaviour
      1. Changing the Destination Directory
      2. Static Builds
      3. Using build.sh options
      4. make(1) variables used during build

Crosscompiling NetBSD with build.sh

When targeting a product for an embedded platform, it's not feasible to have all the development tools available on that same platform. Instead, some method of crosscompiling is usually used today. NetBSD 1.6 and forward comes with a framework to build both the operating system's kernel and the whole userland for either the same platform that the compiler runs on, or for a different platform, using crosscompiling. Crosscompiling requires assembler, linker, compiler etc. to be available and built for the target platform. The new build scheme will take care of creating these tools for a given platform, and make them available ready to use to do development work.

In this chapter, we will show how to use build.sh to first create a crosscompiling toolchain, including cross-compiler, cross-assembler, cross-linker and so on. While native kernel builds are covered in Compiling the kernel, these tools are then used to manually configure and crosscompile a kernel for a different platform, and then show how to use build.sh as a convenient alternative. After that works, the whole NetBSD userland will be compiled and packed up in the format of a NetBSD release. In the examples, we will use the Sun UltraSPARC (sparc64) 64-bit platform as target platform, any other platform supported by NetBSD can be targetted as well specifying its name (see /usr/src/sys/arch).

Before starting, take note that it is assumed that the NetBSD sources from the netbsd-4-0 branch are available in /usr/src as described in Obtaining the sources.

A more detailed description of the build.sh framework can be found in Luke Mewburn and Matthew Green's paper and their presentation from BSDCon 2003 as well as in /usr/src/BUILDING.

Building the crosscompiler

The first step to do cross-development is to get all the necessary tools available. In NetBSD terminology, this is called the "toolchain", and it includes BSD-compatible make(1), C/C++ compilers, linker, assembler, config(8), as well as a fair number of tools that are only required when crosscompiling a full NetBSD release, which we won't cover here.

The command to create the crosscompiler is quite simple, using NetBSD's new src/build.sh script. Please note that all the commands here can be run as normal (non-root) user:

$ cd /usr/src
$ ./build.sh -m sparc64 tools

Make sure that the directory /usr/obj does exist, or add a -O option to the build.sh call, redirecting the object directory someplace else.

If the tools have been built previously and they only need updated, then the update option -u can be used to only rebuild tools that have changed:

$ ./build.sh -u -m sparc64 tools

When the tools are built, information about them and several environment variables is printed out:

...
===> build.sh started: Thu Dec  2 22:18:11 CET 2007
===> build.sh ended:   Thu Dec  2 22:28:22 CET 2007
===> Summary of results:
         build.sh command: ./build.sh -m sparc64 tools
         build.sh started: Thu Dec  2 22:18:11 CET 2007
         No nonexistent/bin/nbmake, needs building.
         Bootstrapping nbmake
         MACHINE:          sparc64
         MACHINE_ARCH:     sparc64
         TOOLDIR path:     /usr/src/tooldir.NetBSD-4.0-i386
         DESTDIR path:     /usr/src/destdir.sparc64
         RELEASEDIR path:  /usr/src/releasedir
         Created /usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake
         makewrapper:      /usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake-sparc64
         Updated /usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake-sparc64
         Tools built to /usr/src/tooldir.NetBSD-4.0-i386
         build.sh started: Thu Dec  2 22:18:11 CET 2007
         build.sh ended:   Thu Dec  2 22:28:22 CET 2007
===> .

During the build, object directories are used consistently, i.e. special directories are kept that keep the platform-specific object files and compile results. In our example, they will be kept in directories named obj.sparc64 as we build for UltraSPARC as target platform.

The toolchain itself is part of this, but as it's hosted and compiled for a i386 system, it will get placed in its own directory indicating where to cross-build from. Here's where our crosscompiler tools are located:

$ pwd
/usr/src
$ ls -d tooldir.*
tooldir.NetBSD-4.0-i386

So the general rule of thumb is for a given host and target system combination, the crosscompiler will be placed in the src/tooldir.host directory by default. A full list of all tools created for crosscompiling the whole NetBSD operating system includes:

$ ls tooldir.NetBSD-4.0-i386/bin/
nbasn1_compile          nbmakefs                nbzic
nbcap_mkdb              nbmakeinfo              sparc64--netbsd-addr2li
nbcat                   nbmakewhatis            sparc64--netbsd-ar
nbcksum                 nbmenuc                 sparc64--netbsd-as
nbcompile_et            nbmkcsmapper            sparc64--netbsd-c++
nbconfig                nbmkdep                 sparc64--netbsd-c++filt
nbcrunchgen             nbmkesdb                sparc64--netbsd-cpp
nbctags                 nbmklocale              sparc64--netbsd-dbsym
nbdb                    nbmknod                 sparc64--netbsd-g++
nbeqn                   nbmktemp                sparc64--netbsd-g77
nbfgen                  nbmsgc                  sparc64--netbsd-gcc
nbfile                  nbmtree                 sparc64--netbsd-gcc-3.3
nbgencat                nbnroff                 sparc64--netbsd-gccbug
nbgroff                 nbpax                   sparc64--netbsd-gcov
nbhexdump               nbpic                   sparc64--netbsd-ld
nbhost-mkdep            nbpwd_mkdb              sparc64--netbsd-lint
nbindxbib               nbrefer                 sparc64--netbsd-mdsetim
nbinfo                  nbrpcgen                sparc64--netbsd-nm
nbinfokey               nbsoelim                sparc64--netbsd-objcopy
nbinstall               nbstat                  sparc64--netbsd-objdump
nbinstall-info          nbsunlabel              sparc64--netbsd-ranlib
nbinstallboot           nbtbl                   sparc64--netbsd-readelf
nblex                   nbtexi2dvi              sparc64--netbsd-size
nblorder                nbtexindex              sparc64--netbsd-strings
nbm4                    nbtsort                 sparc64--netbsd-strip
nbmake                  nbuudecode
nbmake-sparc64          nbyacc

As you can see, most of the tools that are available native on NetBSD are present with some program prefix to identify the target platform for tools that are specific to a certain target platform.

One important tool that should be pointed out here is nbmake-sparc64. This is a shell wrapper for a BSD compatible make(1) command that's setup to use all the right commands from the crosscompiler toolchain. Using this wrapper instead of /usr/bin/make allows crosscompiling programs that were written using the NetBSD Makefile infrastructure (see src/share/mk). We will use this make(1) wrapper in a second to cross compile the kernel!

Configuring the kernel manually

Now that we have a working crosscompiler available, the "usual" steps for building a kernel are needed - create a kernel config file, run config(1), then build. As the config(1) program used to create header files and Makefile for a kernel build is platform specific, we need to use the nbconfig program that's part of our new toolchain. That aside, the procedure is just as like compiling a "native" NetBSD kernel. Commands involved here are:

$ cd /usr/src/sys/arch/sparc64/conf
$ cp GENERIC MYKERNEL
$ vi MYKERNEL
$ /usr/src/tooldir.NetBSD-4.0-i386/bin/nbconfig MYKERNEL

That's all. This command has created a directory ../compile/MYKERNEL with a number of header files defining information about devices to compile into the kernel, a Makefile that is setup to build all the needed files for the kernel, and link them together.

Crosscompiling the kernel manually

We have all the files and tools available to crosscompile our UltraSPARC-based kernel from our Intel-based host system, so let's get to it! After changing in the directory created in the previous step, we need to use the crosscompiler toolchain's nbmake-sparc64 shell wrapper, which just calls make(1) with all the necessary settings for crosscompiling for a sparc64 platform:

$ cd ../compile/MYKERNEL/
$ /usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake-sparc64 depend
$ /usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake-sparc64

This will churn away a bit, then spit out a kernel:

...
   text    data     bss     dec     hex filename
5016899  163728  628752 5809379  58a4e3 netbsd
$ ls -l netbsd
-rwxr-xr-x  1 feyrer  666  5874663 Dec  2 23:17 netbsd
$ file netbsd
netbsd: ELF 64-bit MSB executable, SPARC V9, version 1 (SYSV), statically linked, not stripped

Now the kernel in the file netbsd can either be transferred to a UltraSPARC machine (via NFS, FTP, scp, etc.) and booted from a possible harddisk, or directly from our cross-development machine using NFS.

After configuring and crosscompiling the kernel, the next logical step is to crosscompile the whole system, and bring it into a distribution-ready format. Before doing so, an alternative approach to crosscompiling a kernel will be shown in the next section, using the build.sh script to do configuration and crosscompilation of the kernel in one step.

Crosscompiling the kernel with build.sh

A cross compiled kernel can be done manually as described in the previous sections, or by the easier method of using build.sh, which will be shown here.

Preparation of the kernel config file is the same as described above:

$ cd /usr/src/sys/arch/sparc64/conf
$ cp GENERIC MYKERNEL
$ vi MYKERNEL

Then edit MYKERNEL and once finished, all that needs to be done is to use build.sh to build the kernel (it will also configure it, running the steps shown above):

$ cd /usr/src
$ ./build.sh -u -m sparc64 kernel=MYKERNEL

Notice that update (-u) was specified, the tools are already built, there is no reason to rebuild all of the tools. Once the kernel is built, build.sh will print out the location of it along with other information:

...
===> Summary of results:
         build.sh command: ./build.sh -u -m sparc64 kernel=MYKERNEL
         build.sh started: Thu Dec  2 23:30:02 CET 2007
         No nonexistent/bin/nbmake, needs building.
         Bootstrapping nbmake
         MACHINE:          sparc64
         MACHINE_ARCH:     sparc64
         TOOLDIR path:     /usr/src/tooldir.NetBSD-4.0-i386
         DESTDIR path:     /usr/src/destdir.sparc64
         RELEASEDIR path:  /usr/src/releasedir
         Created /usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake
         makewrapper:      /usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake-sparc64
         Updated /usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake-sparc64
         Building kernel without building new tools
         Building kernel:  MYKERNEL
         Build directory:  /usr/src/sys/arch/sparc64/compile/obj.sparc64/GENERIC
         Kernels built from MYKERNEL:
          /usr/src/sys/arch/sparc64/compile/obj.sparc64/MYKERNEL/netbsd
         build.sh started: Thu Dec  2 23:30:02 CET 2007
         build.sh ended:   Thu Dec  2 23:38:22 CET 2007
===> .

The path to the kernel built is of interest here: /usr/src/sys/arch/sparc64/compile/obj.sparc64/MYKERNEL/netbsd, it can be used the same way as described above.

Crosscompiling the userland

By now it is probably becoming clear that the toolchain actually works in stages. First the crosscompiler is built, then a kernel. Since build.sh will attempt to rebuild the tools at every invocation, using update saves time. It is probably also clear that outside of a few options, the build.sh semantics are basically build.sh command. So, it stands to reason that building the whole userland and/or a release is a matter of using the right commands.

It should be no surprise that building and creating a release would look like the following:

$ ./build.sh -U -u -m sparc64 release

These commands will compile the full NetBSD userland and put it into a destination directory, and then build a release from it in a release directory. The -U switch is added here for an unprivileged build, i.e. one that's running as normal user and not as root. As no further switches to build.sh were given nor any environment variables were set, the defaults of DESTDIR=/usr/src/destdir.sparc64 and RELEASEDIR=/usr/src/releasedir are used, as shown in the build.sh-output above.

Crosscompiling the X Window System

The NetBSD project has its own copy of the X Window System's source which is currently based on XFree86 version 4, and which contains changes to make X going on as many of the platforms supported by NetBSD as possible. Due to this, it is desirable to use the X Window System version available from and for NetBSD, which can also be crosscompiled much like the kernel and base system. To do so, the xsrc sources must be checked out from CVS into /usr/xsrc just as src and pkgsrc were as described in Obtaining the sources.

After this, X can be crosscompiled for the target platform by adding the -x switch to build.sh, e.g. when creating a full release:

$ ./build.sh -U -x -u -m sparc64 release

The -U flag for doing unprivileged (non-root) builds and the -u flag for not removing old files before building as well as the -m arch option to define the target architecture have already been introduced, and the -x option to also (cross)compile xsrc is another option.

Changing build behaviour

Similar to the old, manual building method, the new toolchain has a lot of variables that can be used to direct things like where certain files go, what (if any) tools are used and so on. A look in src/BUILDING covers most of them. In this section some examples of changing default settings are given, each following its own ways.

Changing the Destination Directory

Many people like to track NetBSD-current and perform cross compiles of architectures that they use. The logic for this is simple, sometimes a new feature or device becomes available and someone may wish to use it. By keeping track of changes and building every now and again, one can be assured that these architectures can build their own release.

It is reasonable to assume that if one is tracking and building for more than one architecture, they might want to keep the builds in a different location than the default. There are two ways to go about this, either use a script to set the new DESTDIR, or simply do so interactively. In any case, it can be set the same way as any other variable (depending on your shell of course).

For bash, the Bourne or Korn shell, this is:

$ export DESTDIR=/usr/builds/sparc64

For tcsh and the C shell, the command is:

$ setenv DESTDIR /usr/builds/sparc64

Simple enough. When the build is run, the binaries and files will be sent to /usr/builds.

Static Builds

The NetBSD toolchain builds and links against shared libraries by default. Many users still prefer to be able to link statically. Sometimes a small system can be created without having shared libraries, which is a good example of doing a full static build. If a particular build machine will always need one environment variable set in a particular way, then it is easiest to simply add the changed setting to /etc/mk.conf.

To make sure a build box always builds statically, simply add the following line to /etc/mk.conf:

LDSTATIC=-static

Using build.sh options

Besides variables in environment and /etc/mk.conf, the build process can be influenced by a number of switches to the build.sh script itself, as we have already seen when forcing unprivileged (non-root) builds, selecting the target architecture or preventing deletion of old files before the build. All these options can be listed by running build.sh -h:

$ cd /usr/src
$ build.sh -h
Usage: build.sh [-EnorUux] [-a arch] [-B buildid] [-D dest] [-j njob]
        [-M obj] [-m mach] [-N noisy] [-O obj] [-R release] [-T tools]
        [-V var=[value]] [-w wrapper] [-X x11src] [-Z var]
        operation [...]

 Build operations (all imply "obj" and "tools"):
    build               Run "make build".
    distribution        Run "make distribution" (includes DESTDIR/etc/ files).
    release             Run "make release" (includes kernels and distrib media).

 Other operations:
    help                Show this message and exit.
    makewrapper         Create nbmake-${MACHINE} wrapper and nbmake.
                        Always performed.
    obj                 Run "make obj".  [Default unless -o is used]
    tools               Build and install tools.
    install=idir        Run "make installworld" to `idir' to install all sets
            except `etc'.  Useful after "distribution" or "release"
    kernel=conf         Build kernel with config file `conf'
    releasekernel=conf  Install kernel built by kernel=conf to RELEASEDIR.
    sets                Create binary sets in RELEASEDIR/MACHINE/binary/sets.
            DESTDIR should be populated beforehand.
    sourcesets          Create source sets in RELEASEDIR/source/sets.
    params              Display various make(1) parameters.

 Options:
    -a arch     Set MACHINE_ARCH to arch.  [Default: deduced from MACHINE]
    -B buildId  Set BUILDID to buildId.
    -D dest     Set DESTDIR to dest.  [Default: destdir.MACHINE]
    -E          Set "expert" mode; disables various safety checks.
                Should not be used without expert knowledge of the build system.
    -j njob     Run up to njob jobs in parallel; see make(1) -j.
    -M obj      Set obj root directory to obj; sets MAKEOBJDIRPREFIX.
                Unsets MAKEOBJDIR.
    -m mach     Set MACHINE to mach; not required if NetBSD native.
    -N noisy    Set the noisyness (MAKEVERBOSE) level of the build:
            0   Quiet
            1   Operations are described, commands are suppressed
            2   Full output
        [Default: 2]
    -n          Show commands that would be executed, but do not execute them.
    -O obj      Set obj root directory to obj; sets a MAKEOBJDIR pattern.
                Unsets MAKEOBJDIRPREFIX.
    -o          Set MKOBJDIRS=no; do not create objdirs at start of build.
    -R release  Set RELEASEDIR to release.  [Default: releasedir]
    -r          Remove contents of TOOLDIR and DESTDIR before building.
    -T tools    Set TOOLDIR to tools.  If unset, and TOOLDIR is not set in
                the environment, nbmake will be (re)built unconditionally.
    -U          Set MKUNPRIVED=yes; build without requiring root privileges,
            install from an UNPRIVED build with proper file permissions.
    -u          Set MKUPDATE=yes; do not run "make clean" first.
        Without this, everything is rebuilt, including the tools.
    -V v=[val]  Set variable `v' to `val'.
    -w wrapper  Create nbmake script as wrapper.
                [Default: ${TOOLDIR}/bin/nbmake-${MACHINE}]
    -X x11src   Set X11SRCDIR to x11src.  [Default: /usr/xsrc]
    -x          Set MKX11=yes; build X11R6 from X11SRCDIR
    -Z v        Unset ("zap") variable `v'.

As can be seen, a number of switches can be set to change the standard build behaviour. A number of them has already been introduced, others can be set as appropriate.

make(1) variables used during build

Several variables control the behaviour of NetBSD builds. Unless otherwise specified, these variables may be set in either the process environment or in the make(1) configuration file specified by MAKECONF. For a definitive list of these options, see BUILDING and share/mk/bsd.README files in the toplevel source directory.

The following variables only affect the top level Makefile and do not affect manually building subtrees of the NetBSD source code.

Add a comment