Using pbulk to create a pkgsrc binary repository

pkgtools/pbulk package consists in a set of tools designed to ease mass-building of pkgsrc packages, and the creation your own pkgsrc binaries repository.

Its setup needs a bit of work, here is how to prepare and run your bulk-build box. In this article we will only consider a one-node machine.

This documentation is based on the The pkgsrc Guide.

Prerequisites

These are the prerequisites needed by pbulk:

  • A pkgsrc source tree
  • Possibly a src source tree, only some rare packages needs it. If you choose not to checkout src, simply create a /usr/src directory so mksandbox (see below) doesn't complain about non-existing directory.
  • Possibly (not mandatory) a tool like misc/screen or misc/tmux as the full build process can take a very long time.

For example, to prepare a pkgsrc-2011Q3 bulk build:

# cd /usr
# cvs -d anoncvs@anoncvs.netbsd.org:/cvsroot co -rpkgsrc-2011Q3 pkgsrc

Avoid automatic update of pkgsrc tree (cron or such), if you're in the middle of a build, it could lead to unfortunate results.

Prepare a chroot

In order to isolate the bulk build, it is advised you run all the operations within a chroot. Running pbulk on your real environment would wipe all of your installed packages, and would modify your base system with lots of directories, users and groups you don't need.

Fortunately, a tool called mksandbox will simplify this process. mksandbox is located in the pkgtools/mksandbox package, and it is called like this:

# mksandbox [optional flags] /path/to/sandbox

For example, to create a sandbox in /home/bulk without the X11 system, run:

# mksandbox --without-x /home/bulk

This command will prepare and mount most of the needed directories, and will place a shell script on top of the sandbox filesystem called sandbox. This script is used to mount/umount your sandbox. It is a good idea to add /var/spool to the list of directories mounted as read/write in your sandbox so the email report is actually sent. Simply add:

/var/spool /var/spool rw \

to the list of directories in the sandbox script. sandbox script use is really straightforward:

# /path/to/your/sandbox/sandbox umount

Unmounts the sandbox

# /path/to/your/sandbox/sandbox mount

Mounts the sandbox

Prepare the pbulk environment

Now that our sandbox is available and mounted, we will chroot to it by executing the sandbox script without any parameter specified:

# /path/to/your/sandbox/sandbox

Create pbulk user (a Bourne style shell must be used)

# useradd -s /bin/sh pbulk

In a file, set the preferences to use when building packages (mk.conf fragment). Here is a sample mk.conf.frag file:

SKIP_LICENSE_CHECK=             yes
ALLOW_VULNERABLE_PACKAGES=      yes

PKG_DEVELOPER?=         yes

WRKDIR=                 /tmp/work

# site specific changes

PKG_OPTIONS.irssi=      perl inet6
PKG_OPTIONS.mplayer=    oss

DSPAM_STORAGE_DRIVER=   mysql
PKG_OPTIONS.dspam+=     graphs
PKG_OPTIONS.dovecot=    ssl ldap dovecot-sieve dovecot-managesieve
PKG_OPTIONS.nagios-nrpe=ssl tcpwrappers

X11_TYPE=               modular

Deploy and configure pbulk tools

# sh /usr/pkgsrc/mk/pbulk/pbulk.sh -n -c mk.conf.frag

pbulk configuration file is /usr/pbulk/etc/pbulk.conf. You may want to review and customize some parameters like "base_url" and "report_recipients".

Also, in order to avoid hangs, it might be a good idea to add the following to the top of pbulk.conf

ulimit -t 1800 # set the limit on CPU time (in seconds)
ulimit -v 2097152 # limits process address space

Running the build

Now that everything's in place, we can fire up the build from the chroot using the following command:

# /usr/pbulk/bin/bulkbuild

It is recommended to run the build inside a tool like misc/screen or misc/tmux as it will take a lot of time.

If the build is stopped, it is possible to restart it by invoking:

# /usr/pbulk/bin/bulkbuild-restart

Hints

If you'd like to rebuild a single package, use the bulkbuild-rebuild command followed by the package name.

Posted late Tuesday evening, November 1st, 2011

Project template description

The project template provides a consistent set of variables and tags to define a project proposal and/or specification.

The following parameters are supported:

  • title (required)
  • contact (required)
  • done_by (optional): set to the name of the person that completed the project. This adds a note to the project mentioning that it has been completed and removes it from the indexes. Do not move project pages or delete them; by setting this tag, the URL will remain valid.
  • mentors (optional)
  • category (required): one of "filesystems", "kernel", "misc", "networking", "pkgsrc", "ports" or "userland".
  • difficulty (required): one of "easy", "medium" or "hard".
  • funded (optional): set to the name of the organization or individual that is willing to fund this project.
  • duration (optional)
  • description (required)

The following tags should be set to classify the project into different indexes:

  • gsoc: Use this tag to denote a project suitable for the Google Summer of Code program. If you set this tag, the project must provide a set of mentors and its duration has to be 3 months.
Posted Saturday evening, November 5th, 2011

This page contains the data of all the available projects proposals as a single document to permit easy searching within the page.

Note that this page also contains an Atom/RSS feed. If you are interested in being notified about changes to the projects whenever these pages get updated, you should subscribe to this feed.

  • Contact: tech-kern
  • Duration estimate: ~1 month

Implement a fullfs, that is, a filesystem where writes fail with disk full. This is useful for testing applications and tools that often don't react well to this situation because it rarely occurs any more.

The basic fullfs would be just a layerfs layer that you can mount (like nullfs) to get a copy of an existing subtree or volume where writes are rejected with ENOSPC. This is the first thing to get running.

However, for testing it is good to have more than that, so the complete project includes the ability to control the behavior on the fly and a fullfsctl(8) binary that can be used to adjust it.

These are some things (feel free to brainstorm others) that it would be useful for fullfsctl to be able to do:

  • Turn on and off the fail state (so for example you can start up a program, let it run for a while, then have the disk appear to fill up under it)
  • Arm a "doom counter" that allows the next N writes to succeed and then switches to the fail state (to test what happens if the disk fills partway through a write or save operation)
  • Change what error it fails with (ENOSPC is the basic error, but at least EDQUOT and possibly other errors, such as NFS-related ones, are also interesting)

fullfs itself should be implented as a layerfs layer, not a whole filesystem.

fullfsctl should operate via one or more file-system-specific ioctls applied to the root directory of (or perhaps any file on) the fullfs volume.

There are many ways this could be extended further to provide for more general fault injection.

Posted Sunday night, January 21st, 2024 Tags:
  • Contact: tech-kern
  • Duration estimate: 1 month

Assess and, if appropriate, implement RFC 5927 countermeasures against IPv6 ICMP attacks on TCP. Write ATF tests for any countermeasures implemented, as well as ATF tests for the existing IPv4 countermeasures.

This project will close PR kern/35392.

The IPv4 countermeasures were previously implemented here: https://mail-index.NetBSD.org/source-changes/2005/07/19/msg166102.html

Posted at lunch time on Sunday, October 1st, 2023 Tags:

Implement automatic tests with ATF for all the PAM modules under src/lib/libpam/modules.

The modules, such as pam_krb5, are not currently automatically tested, despite being security-critical, which has led to severe regressions.

Posted mid-morning Friday, September 29th, 2023 Tags:

Buildlink is a framework in pkgsrc that controls what headers and libraries are seen by a package's configure and build processes. We'd like to expand on this isolation by:

  • Hiding -L${PREFIX}/{include,lib} from CFLAGS/LDFLAGS, so that (for instance) undeclared buildlink dependencies will give the needed "not found" earlier in the package-development cycle
  • Hiding ${PREFIX}/bin from PATH, so that (for instance) undeclared build- or run-time tools will give the needed "not found" earlier in the package-development cycle

Steps:

  1. Do bulk builds with the existing defaults on a few platforms (e.g., NetBSD, macOS, Illumos, Linux)
  2. Rerun the bulk builds from scratch, this time with the desired infrastructure changes
  3. Compare before and after to see which package builds break
  4. Fix them
  5. Enable the infrastructure changes by default for all users
Posted Monday afternoon, July 3rd, 2023 Tags:

bsd.pkg.mk says: # To sanitize the environment, set PKGSRC_SETENV=${SETENV} -i. We'd like to enable this isolation by default. Steps:

  1. Do bulk builds with the existing defaults on a few platforms (e.g., NetBSD, macOS, Illumos, Linux)
  2. Rerun the bulk builds from scratch, this time with the desired infrastructure change
  3. Compare before and after to see which package builds break
  4. Fix them
  5. Enable the infrastructure change by default for all users
Posted at lunch time on Friday, May 12th, 2023 Tags:

To create large sets of binary packages, pkgsrc uses the pbulk tool.

pbulk is designed to build packages in bulk in a chroot sandbox (which may be based on an older NetBSD release, a 32-bit release on a 64-bit machine, or similar).

However, managing multiple pbulk installations on a single machine can quickly become unwieldy.

It would be nice to have a web frontend for managing pbulk, restarting builds, creating new build sandboxes based on different NetBSD versions and package sets, etc.

For an example of a typical high-performance pbulk setup used on "large" hardware, see Nia's pbulk scripts. Notably, these include chroot management and can be used to build packages for multiple NetBSD (and pkgsrc) versions simultaneously by using different "base directories".

  • A management tool should support parallel pbulk across many chroots, but should also easily allow simultaneous builds for different NetBSD versions on the same machine.
  • It should be possible to schedule builds for later, somewhat like Jenkins.
  • There should be some kind of access management system.
  • pbulk creates status directories, it would be useful to present as much data from them as possible, as well as information on system resource usage (e.g. to find any stuck processes and monitor hardware health).
  • It should be possible for users to "kick" the builds (to quickly restart them, kill "stuck" processes, and so on).
  • It should be possible to schedule builds for upload after manual review of the reports, resulting in an rsync from a local staging directory to ftp.netbsd.org.
  • It should be usable "over a morning coffee".
  • For bonus points, it should be easily installable from pkgsrc, ideally with an RC script.
  • For bonus points, integrate well with the NetBSD base system tools (e.g. bozohttpd).
Posted late Wednesday evening, January 18th, 2023 Tags:

Currently NetBSD can be booted via UEFI firmware, but only offers the default boot loader setup so multi-boot environments are hard to create. This also causes cryptic displays in the firmware boot order menu or boot select menu, like "UEFI OS", instead of "NetBSD 10.0".

The UEFI spec offers support to configure load options, which include a path to the bootloader and a description of the operating system, see the UEFI spec. This project is to implement setting up proper load option variables at least on x86 machines booting via UEFI.

Part of the project is to find the best place to set this options up. Some integrations with sysinst might be needed, maybe sysinst is the right place to set this variables. If not, sysinst may simply be changed to use a different sub directory on the ESP for the NetBSD bootloader and the variables setup might happen elsewhere.

Currently the kernel interface to access the SetVariable() and other EFI runtime callbacks exists, but there is no userland tool to operate it.

It is not clear what the EFI path set in the variable should be, and mapping NetBSD disks/partitions to EFI path notation is not trivial.

Posted Thursday afternoon, January 12th, 2023 Tags:
  • Contact: tech-pkg
  • Mentors: unknown
  • Duration estimate: unknown

In recent years, packages whose builds download things on the fly have become an increasing problem. Such downloads violate both pkgsrc principles/rules and standard best practices: at best, they bypass integrity checks on the downloaded material, and at worst they may check out arbitrary recent changes from GitHub or other hosting sites, which in addition to being a supply-chain security risk also makes reliable repeatable builds impossible.

It has simultaneously grown more difficult to find and work around these issues by hand; these techniques have been growing increasingly popular among those who should perhaps know better, and meanwhile upstream build automation has been steadily growing more elaborate and more opaque. Consequently, we would like a way to automatically detect violations of the rules.

Currently, pbulk runs in two steps: first it scans the tree to find out what it needs to build, and once this is done it starts building. Builds progress in the usual pkgsrc way: the fetch phase comes first and downloads anything needed, then the rest of the build continues. This interleaves (expected) downloads and build operations, which on the one hand tends to give the best build throughput but on the other makes it difficult to apply network access restrictions.

The goal of this project is to set up infrastructure such that bulk builds can run without network access during the build phase. (Or, more specifically, in pkgsrc terms, everything other than the fetch phase.) Then attempts to download things on the fly will fail.

There are two ways to accomplish this and we will probably want both of them. One is to add a separate fetch step to pbulk, so it and the build step can run with different global network configuration. The other is to provide mechanisms so that expected downloads can proceed and unwanted ones cannot.

Separate fetch step

Since bulk builds are generally done on dedicated or mostly-dedicated systems (whether real or virtual) system-wide changes to the network configuration to prevent downloads during the build step will be in most setups acceptable, and also sufficient to accomplish the desired goals.

There are also two possible ways to implement a separate fetch step: as part of pbulk's own operational flow or as a separate external invocation of the pbulk components. The advantage of the former is that it requires less operator intervention, while the advantage of the latter is that it doesn't require teaching pbulk to manipulate the network state on its own. (In general it would need to turn external access off before the build step, and then restore it after in order to be able to upload the build results.) Since there are many possible ways one might manipulate the network state and the details vary between operating systems, and in some cases the operations might require privileges the pbulk process doesn't have, making pbulk do it adds considerable complication; on the other hand, setting up pbulk is already notoriously complicated and requiring additional user-written scripts to manipulate partial build states and adjust the network is a significant drawback.

Another consideration is that to avoid garbaging the failure reports, any download failures need to be recorded with pbulk during the download step, and the packages affeected marked as failed, so that those failures end up in the output results. Otherwise the build step will retry the fetch in an environment without a network, and that will then fail but nobody will be able to see why. For this reason it isn't sufficient to just, for example, run "make -k fetch" from the top level of pkgsrc before building.

Also note that rather than just iterating the package list and running "make fetch" from inside pbulk, it might be desirable to use "make fetch-list" or similar and then combine the results and feed them to a separate download tool. The simple approach can't readily adapt to available network bandwidth: depending on the build host's effective bandwidth from various download sites it might either flood the network or be horrendously slow, and no single fixed parallelism setting can avoid this. Trying to teach pbulk itself to do download load balancing is clearly the wrong idea. However, this is considerably more involved, if only because integrating failure results into the pbulk state is considerably more complicated.

Blocking unauthorized downloads

The other approach is to arrange things so that unwanted downloads will fail. There are a number of possible ways to arrange this on the basic assumption that the system has no network access to the outside world by default. (For example, it might have no default route, or it might have a firewall config that blocks outgoing connections.) Then some additional mechanism is introduced into the pkgsrc fetch stage so that authorized downloads can proceed. One mechanism is to set up a local HTTP proxy and only make the proxy config available to the fetch stage. Another, possibly preferable if the build is happening on a cluster of VMs or chroots, is to ssh to another local machine to download there; that allows mounting the distfiles directory read-only on the build hosts. There are probably others. Part of the goal of this project should be to select one or a small number of reasonable mechanisms and provide the necessary support in pbulk so each can be enabled in a relatively turn-key fashion. We want it to be easy to configure this restriction and ideally in the long term we'd like it to be able to default to "on".

Note that it will almost certainly be necessary to strengthen the pkgsrc infrastructure to support this. For example, conditionally passing around HTTP proxy config depending on the pkgsrc phase will require changes to pkgsrc itself. Also, while one can redirect fetch to a different tool via the FETCH_USING variable, as things stand this runs the risk of breaking packages that need to set it itself. There was at one point some talk about improving this but it apparently never went anywhere.

An additional possibility along these lines is to leverage OS-specific security frameworks to prohibit unwanted downloading. This has the advantage of not needing to disable the network in general, so it can be engaged even for ordinary builds and on non-dedicated hosts. However, because security frameworks and their state of useability vary, it isn't a general solution. In particular on NetBSD at the moment we do not have anything that can do this. (It might be possible to ues kauth for it, but if so the necessary policy modules do not currently exist.) Consequently this option is in some ways a separate project. Note if attempting it that simplistic solutions (e.g. blocking all attempts to create sockets) will probably not work adequately.

Other considerations

Note when considering this project:

  • Working on pbulk at all is not entirely trivial.
  • It's necessary to coordinate with Joerg and other stakeholders so that the changes can eventually be merged.
  • Any scheme will probably break at least a few innocent packages that will then probably need somebody to patch them.

Overall, one does hope that any package that attempts to download things and finds it can't will then fail rather than silently doing something different that we might or might not be able to detect. It is possible in the long run that we'll want to use a security framework that can log download attempts and provide an audit trail; however, first steps first.

Posted Sunday night, January 8th, 2023 Tags:

Java software like e.g. Eclipse is using SWT, the "Standard Widget Toolkit".

It would be good to have a NetBSD port. Since this is running on Linux already, it's probably mostly adapting that to NetBSD.

Posted at lunch time on Wednesday, September 28th, 2022 Tags:
  • Contact: nia
  • Mentors: nia
  • Duration estimate: 350h

NetBSD includes various simple, command-line audio tools by default, such as audioplay(1), audiorecord(1), mixerctl(1), aiomixer(1), audiocfg(1)...

These tools are useful because they provide almost everything a user needs to test basic functionality of their audio hardware. They are critically important for basic diagnostics.

It would be nice to have a tool to easily visualize audio input using a simple Curses interface. Some ideas for its possible functionality:

  • Display basic live-updating frequency graph using bars
  • Display channels separately
  • 'Echo' option (play back audio as it is input)
  • pad(4) support (NetBSD has support for 'virtual' audio devices. This is useful because you can record the output of an application by having it output to the audio device that opening /dev/pad creates. This can also 'echo' by outputting the data read from the pad device.)

You need NetBSD installed on physical hardware (older laptops work well and are cheaply available) and a microphone for this project. Applicants should be familiar with the C programming language.

Posted early Wednesday morning, May 11th, 2022 Tags:
  • Contact: nia
  • Mentors: nia
  • Duration estimate: 350h

pkgsrc is NetBSD's native package building system. It's also used on other platforms, such as illumos. It includes numerous graphical environments, including Xfce, MATE, and LXQt, but support for Enlightenment has since bitrotted and been largely removed. Support for its related fork Moksha is missing entirely.

Enlightenment is partiuclarly interesting for NetBSD because it's lightweight, BSD licensed, and suitable for mobile applications. We're not sure about the benefits of Moksha over Enlightenment proper, but it's worth investigating.

Since Enlightenment is written in C, the applicant should ideally have a basic understanding of C and Unix system APIs. In order for the port not to bit-rot in the future, it should be done well, with patches integrated upstream where possible. They should have a laptop with NetBSD installed (older laptops are likely more representative of typical NetBSD uses and can be picked up cheap from local auctions sites).

Integrating Enlightenment into pkgsrc will require a knowledge of build systems and make (pkgsrc in particuar is built on top of BSD make).

Milestones:

  • A basic port enables basic Enlightenment installation on NetBSD when installed from pkgsrc.
  • A more advanced and ideal port has tight integration with NetBSD system APIs, supporting features like native audio mixing and reading from sensors.
  • For extra brownie points, the pkgsrc package should work on illumos too.
Posted early Wednesday morning, May 11th, 2022 Tags:
  • Contact: nia
  • Mentors: nia
  • Duration estimate: 175h

A core component of NetBSD is the 'xsrc' repository, which contains a 'classic' distribution of X11, all related programs, and libraries, as found on Unix systems from times of yore.

xsrc uses the NetBSD build system and only BSD make to build, which means it builds extremely quickly, with minimal dependencies, and is easy to cross-compile. It currently includes an implementation of the OpenGL graphics API (Mesa), but not an implementation of the next-generation Vulkan graphics API, or OpenCL, the GPU-accelerated compute API, which can also be obtained from Mesa.

Most of modern X.Org is built with Meson and Python, so some level of translation is required to integrate new components.

This project involves making modifications to the Mesa Vulkan and OpenCL libraries in order to allow them to work on NetBSD (this part requires basic knowledge of the C programming language and Unix APIs), ideally submitting them upstream, until Vulkan and OpenCL support can be built on NetBSD, and then integrating the relevant components into the NetBSD build system using only BSD Make.

The candidate should ideally have some knowledge of the C programming language and build systems.

Posted early Wednesday morning, May 11th, 2022 Tags:
  • Contact: nia
  • Mentors: nia
  • Duration estimate: 350h

pkgsrc is NetBSD's native package building system It includes numerous graphical environments, including Xfce, MATE, and LXQt, but many have limited support for native NetBSD system APIs, e.g. support for reading battery levels, and audio volume.

We really would like better desktop environment integeration, and this requires some work on the upstream projects in C and in some cases C++.

An applicant should have basic familiarity with build systems, make, and C. They should be good at carefully reading documentation, as much of this stuff is documented in manual pages like audio(4) and envsys(4). They should have a laptop with NetBSD installed (older laptops are likely more representative of typical NetBSD uses and can be picked up cheap from local auctions sites).

They should be able to investigate the current level of support in various third-party projects and identify priority targets where native code for NetBSD can be written.

Nia is very experienced in writing native code for NetBSD audio and sensors and would be happy to answer questions.

As the project continues, we might even be able to start porting more applications and applets.

Posted early Wednesday morning, May 11th, 2022 Tags:

The NetBSD Wi-Fi stack is being reworked to support newer protocols, higher speeds, and fine-grained locking using code from FreeBSD. As part of this work, all existing NetBSD Wi-Fi drivers need to be reworked to the new Wi-Fi code base.

Successful completion of this project requires you to have access to hardware that is already supported by NetBSD but not yet converted. See the ?Driver state matrix for a list of devices to convert. Many older devices can be found cheap on sites like eBay.

When applying for this project, please note which driver(s) you want to convert.

Posted late Saturday afternoon, November 27th, 2021 Tags:

Currently resize_ffs(8) does not work with FFSv2 file systems.

This is a significant problem, since we currently rely on resize_ffs to provide live images for ARM, and FFSv1 is lacking in various ways (e.g. 2038-limited timestamps and maximum disk sizes of 1TB).

Posted late Thursday afternoon, November 18th, 2021 Tags:

We are currently not distributing official TNF binary packages with embedded signature. The pkgsrc infrastructure seems to be mostly there, but there are loose ends and this makes NetBSD fall back behind other pkgsrc users where everything needed comes with the bootstrap kit.

There have been various related experiments and discussions in the past, and the responsible persons are willing to change it now (that is: ideally have all binary pkgs for NetBSD 10 signed and verified already).

This project is about fixing the loose ends.

Intended user workflow

  • the user installs a new system
  • at the end of the sysinst installation the config page offers a binary pkgs setup
  • the user selects a repository (with a working default) and sysinst triggers all necessary configuration and installation actions (this may involve downloads and running fixup scripts, but may not require manual intervention)
  • after a reboot of the new machine, binary pkgs can be directly added and will be automatically verified (e.g.: "pkg_add firefox" or "pkg_add xfce4" will work)

Implementation details

The following drafts a possible pkgsrc/pkgbuilders/releng workflow and assumes x509 signing. This is just to make this project description clearer, the project does not require a x509 based solution.

Operational workflow for pkg creation

  • Thomas Klausner (wiz) is the keeper of the pkgsrc master CA key. He creates intermediate CA keys for every developer in charge of some pkgbuilding machines, signs them with the master CA key and distributes them.
  • The public part of the master CA certificate becomes part of the NetBSD release and is available as a trust anchor.
  • Every developer in charge of some pkgbuild machines creates a signing key (without passphrase) from their intermediate CA key and installs it on the individual pkg build machine

Main point of the whole process is that NetBSD and pkgsrc have different release cycles, and pkg building machines come and go. We do not want a fixed set of allowed machine signing keys distributed with a (long living) NetBSD release, but we do not want to just trust whatever the binary pkg repository offers, so there needs to be proper automatic validation of all keys used for a repository against some trust anchor provided with the base system. With the current size of the project it might be manageable to have all finally used signing keys signed directly by the pkgsrc master key, but a design that allows an interim step where individual signing keys could be created by the developers in charge of the machines would be preferable.

Deliverables for this project

  1. all required changes (if any) for the pkgtools and pkgsrc makefiles, or any new tools/scripts (either as a set of patches or commited).

  2. a description of the overall workflow, e.g. as a wiki page or as part of the web site.

  3. concrete instructions for the various parties involved in the deployment:

    • pkgsrc master key/cert handling (Thomas)
    • releng: how to make the trust anchor part of the release and what needs to be configured/done by sysinst
      • globally
      • post pkg repository selections
    • pkg build administrators: how to create signing keys and how to configure the pkgbuild machines

And of course all this needs to be tested upfront.

Bonus

If this project succeeds and does not use x509, propose removal of the bit rotted and not fully functional x509 support from pkg tools and the pkgsrc infrastructure.

Setup tried so far and how it fails

Thomas (wiz@) provided the certificate for the TNF CA, intended to be used to verify all signed binary pkgs. When everything works, this key should be part of the base system.

Thomas also created a cert+key for the test setup, signed by the TNF CA key, intended to be used to sign binary pkgs on a single pkg build setup.

The instructions for these two steps are in pkgsrc/pkgtools/pkg_install/files/x509/signing.txt - a script and a config file are in the same directory.

On the build machine, the setup is simple:

  • store the keys for example in /etc/pkg-certs. The names used below are 00.pem for the TNF CA cert and pkgkey_cert for the individual builder certificate and pkgkey_key.pem for the corresponding key (which needs to have no passphrase)
  • Add to /etc/mk.conf (or the equivalent in the bulk build tree)

     # signed binary pkgs, see
     # https://mail-index.netbsd.org/pkgsrc-users/2013/08/30/msg018511.html
     SIGN_PACKAGES=x509
     X509_KEY=/etc/pkg-certs/pkgkey_key.pem
     X509_CERTIFICATE=/etc/pkg-certs/pkgkey_cert.pem
    
  • Add to /etc/pkg_install.conf

     VERIFIED_INSTALLATIONS=always
     CERTIFICATE_ANCHOR_PKGS=/etc/pkg-certs/pkgkey_cert.pem
     CERTIFICATE_CHAIN=/etc/pkg-certs/00.pem
    

Then create a single pkg, like:

cd /usr/pkgsrc/pkgtools/digest
make package
make install

At the end of make package you should see successful signing of the binary pkg. But the make install will fail to verify the certificate.

Note: a key point of the whole setup is to avoid having to add the content of pkgkey_cert.pem to 00.pem (or similar). We want to avoid having to distribute many (changing) keys of build machines with the base system.

An alternative solution would make the key distribution part of the initial setup (e.g. download from a fixed relative path when selecting a pkg repository URL), but no off-the-shelf tools for that currently exist.

Posted late Wednesday morning, October 27th, 2021 Tags:

The current UI of pkgsrc MESSAGE as a couple of drawbacks:

  • When installing a lot of packages via pkg_add or pkgin it is often get lost
  • When updating packages via pkg_add or pkgin - also if the MESSAGE is not changed - it is printed anyway

For possible inspirations please look at OpenBSD ports' pkg-readmes and/or other package systems.

Posted mid-morning Monday, September 6th, 2021 Tags:

NetBSD has large amounts of documentation in XML DocBook: the Guide, the pkgsrc Guide, large parts of the website.

asciidoc is a much nicer, friendlier format than XML. It preserves the same advantages as DocBook: easy conversion to PDF and other formats for book distribution, and rich semantics.

There is a tool for converting DocBook to asciidoc, but it is likely not perfect, and output will require manual adjustment:

asciidoc itself can also output DocBook. We might leverage this so we can convert the website step-by-step, and keep the existing templates.

This project will require careful adjustment of build systems in the htdocs repository and those used by the pkgsrc guide.

A working proposal should demonstrate one page (of the website, pkgsrc guide, or NetBSD guide), converted to asciidoc and hooked up to the build system, integrated, with reasonable looking output.

Posted at lunch time on Sunday, May 2nd, 2021 Tags:
  • Contact: tech-kern
  • Funded by: coypu@sdf.org ($400 expires 1/July/2021)

The current xhci driver requires contiguous allocations, and with higher uptime, NetBSD's kernel memory becomes more and more fragmented.
Eventually, large contiguous allocations aren't possible, resulting in random USB driver failures.
This feature will improve user experience of NetBSD.

Posted late Thursday evening, March 25th, 2021 Tags:

When a system needs more memory but has free disk space it could auto create swap files and then delete them later.

The ideal solution would be configurable for:

  • thresholds for creation
  • min/max (don't fill up the disk entirely)
  • encryption settings

The "listener" for the file creation should avoid thrashing, have timeouts, and handle disk space usage sanely.

Posted Tuesday evening, October 13th, 2020 Tags:

OpenLDAP already has a SASL back-end for CYRUS-SASL.
In NetBSD, we have our own SASL-C library which has similar functionality and can be used in OpenLDAP instead of CYRUS.
Base postfix already does this.

There is a cyrus.c file where all the work is done.
We can make a saslc.c one that uses our library.
This will allow different authentication schemes to be used for the client programs (so we will be able to run ldapsearch against an Active Directory server using GSSAPI.

Posted at lunch time on Sunday, August 11th, 2019 Tags:

NetBSD has an extensive test suite that tests native kernel and userland code.

Mounting the root file system is one of the last steps the kernel does during boot before starting the first process (init(8)).

Root file system selection is not covered by the current test suite.

How to find the root file system is specfied in the kernel configuration file. E.g.:

config netbsd root on ? type ?  
config netbsd root on sd0a type ?  

The first is a wildcard specification which causes the kernel to look for the root file system on the device that the kernel was booted from. The second form specifies the device and partition that contains the root file system. Other forms are also possible.

The selection process is a complex interaction between various global variables that get initialized from the kernel configuration file and by machine specific code that processes the information passed by the bootloader about where the kernel was loaded from.

This selection process is performed mostly by a function named setroot in the file sys/kern/kern_subr.c.

The project could be completed in a number of steps:

  • Document the existing use cases and config ... syntax.
  • Document the processing steps and functions called by setroot.
  • Document how the various global variables interact.
  • Write unit tests using rumpservers for the ATF framework for the documented use cases.

The project would initially be focussed on x86 (amd64 and i386).

Posted Monday night, January 21st, 2019 Tags:

clang-format is a tool to format source code according to a set of rules and heuristics. Like most tools, it is not perfect nor covers every single case, but it is good enough to be helpful.

clang-format can be used for several purposes:

  • Quickly reformat a block of code to the NetBSD (KNF) style.
  • Spot style mistakes, typos and possible improvements in files.
  • Help to follow the coding style rules.

Milestones:

  • Create configuration file .clang-format that approximate the NetBSD coding style
  • Patch LibFormat to handle missing coding style rules.
  • Integrate .clang-format with the NetBSD distribution.
Posted late Friday afternoon, January 18th, 2019 Tags:

racoon(8) is the current IKEv1 implementation used in NetBSD. The racoon code is old and crufty and full of potential security issues. We would like to replace it. There are other implementations available, such as StrongSwan, openiked/isakmpd, racoon2.

This project has two stages:

  • Evaluate all 3 (or more) solutions, describe and document their pros and cons, and then settle into one of them.

  • Port it to NetBSD to replace racoon.

I have started working on that for racoon2 on https://github.com/zoulasc/racoon2/ (see the TODO file), and also have a build glue for NetBSD for it https://github.com/zoulasc/racoon2-glue/ and it works. I've also gotten openiked to compile (but not work).

Posted at teatime on Friday, January 18th, 2019 Tags:

the urtwn and rtwn have a lot of duplicate code.
Merging them will improve both.

This project is on hold due to the conversion project needing to be completed first.

Posted at teatime on Friday, January 18th, 2019 Tags:

NetBSD has the capability to run binaries compiled for Linux under compat_linux. This is a thin in-kernel translation layer that implements the same ABI as the Linux kernel, translating Linux system calls to NetBSD ones.

Not all Linux syscalls are implemented. This means some programs cannot run.

This project is about identifying critical missing syscalls and adding support for them.

In the course of this project, you should find at least one Linux binary that does not yet run on NetBSD using compat_linux to use as a test case (your mentor may have suggestions), trace the program to find the missing features it requires, make note of them, and begin implementing them in NetBSD's Linux compatibility layer.

Posted Friday afternoon, January 18th, 2019 Tags:

VMWare provides an emulator that could use graphical acceleration, but doesn't on NetBSD.
A DRM driver exists for linux which could be adapted, like other DRM drivers that were ported.

Posted Friday afternoon, January 18th, 2019 Tags:
  • Contact: port-arm
  • Duration estimate: 3-6 months

Android is an extremely popular platform, with good software support.

NetBSD has some COMPAT_LINUX support, and it might be possible to leverage this to run Android applications.
This is only done for GNU/Linux platforms right now (SUSE / Debian).

We need to start with Android x86, as COMPAT_LINUX for x86 already exists and is known to work.

As this is a difficult project, the project will need adjustments with time.

  • Create an anbox chroot on linux/x86, experiment with running it with NetBSD.
  • Experiment with running simplest Android program
  • Implement missing syscall emulation as needed
  • ??? (gap for difficulties we will find from this)
  • Package anbox-chroot in pkgsrc

Resources:

Posted Friday afternoon, January 18th, 2019 Tags:
  • Contact: tech-net
  • Duration estimate: 175h

Access to some hardware registers and other things can only be done by one CPU at a time.
An easy way to do this is to make the entire network stack runs with a single lock held, so operations only take place on one core.
This is inefficient, if you ever want to use more than one core, for faster performing cards.

Adapting old drivers to be able to run with the rest of the network stack not having this lock will improve NetBSD networking.
A large number of drivers must be adapted, and some of them can be emulated from virtual machines too, some examples:

  • Realtek RTL8139 Gigabit Ethernet re(4) (supported by QEMU)
  • AMD PCnet pcn(4) (supported by QEMU and VMware)
  • Novell NE1000 ne(4) (supported by QEMU)
  • Atheros/Killer Gigabit Ethernet alc(4)
  • Attansic Gigabit Ethernet ale(4)
  • Broadcom Gigabit Ethernet bge(4)
  • Broadcom NetXtreme bnx(4)

You may find others listed in pci(4). It is possible you have a computing device with a device for which the driver hasn't been converted yet.

The file src/doc/TODO.smpnet in the NetBSD source tree contains a list of fully converted drivers that you may use an an example, as well as some general guidelines.

When applying for this project, please note which driver you would like to work on.

Posted Friday afternoon, January 18th, 2019 Tags:

Raspberry Pi is a very popular ARM board.

It has a modern graphical driver, VC4.

NetBSD already supports several DRM drivers (from Linux 4.4), living in sys/external/bsd/drm2. Adapting this one will make Raspberry Pi work better out of the box.

While this project requires hardware, we can help with supplying a Raspberry Pi if needed.

Milestones for this project:

  • VC4 driver builds as part of netbsd source tree (no hardware access needed)
  • Adjust device tree configuration so VC4 driver is used
  • Iron out bugs that appear from running it
Posted Friday afternoon, January 18th, 2019 Tags:

The iscsictl(1) program manages the iSCSI instances on the local computer. It communicates with the iscsid(8) daemon to send queries using iSCSI protocol.

Possible enhancements:

  • Review of iscsictl(1) manpage. For instance, the command add_target has no description, [target-opts] could be refered to "Target Options".
  • Add a mode to iscsictl(1) program to log sessions at boot. It could be a batch command (the name could be discussed) that read a /etc/iscsi.conf file. Some parts of the iscsictl(1) from FreeBSD could be ported.
  • Implement the find_isns_servers.

The iscsi-target(8) server allows to setup iSCSI targets on a NetBSD host and to present block storage to the network. It can be used to test the iscsictl(1) implementation.

Posted late Monday evening, January 14th, 2019 Tags:

Improvements that can be done to NPF with reference to WIP code/descriptions:

  • Import thmap, needed for newer NPF
  • WIP dynamic NAT address and NETMAP
  • Use of "any"
    map $ext_if dynamic any -> $ext_v4 pass family inet4 from $int_net.
    needs a few syntactic fixes/wrappers (e.g. loading can just handle "any" here, since it's actually correct, just merely not supported by the syntax; you can replace it with 0.0.0.0, though)
  • traffic redirection, 1 2 I think it just needs IPv6 handling and testing
Posted Monday afternoon, December 10th, 2018 Tags:

The Citrus local code in NetBSD's libc stores the locale data and code in shared object files. Dynamic linking is used to load the code by name using dlopen(3). The static libc build (libc.a) can't use dlopen(3) to open locales and it only supports the C locale which is hard-coded.

This project is about adding make(1) rules to compile all the locales in libc and code to select each one by name. The same technique is used in libpam (libpam.a).

Posted late Sunday night, November 12th, 2018 Tags:

NetBSD has an extensive test suite that tests native kernel and userland code. NetBSD can run Linux binaries under emulation (notably on x86, but other platforms such as ARM have some support too). The Linux emulation is not covered by the test suite. It should be possible to run an appropriate subset of the tests when compiled as Linux binaries.

The project could be completed in a number of steps:

  • Determine tests that make sense to run under Linux emulation (e.g. syscalls)
  • Compile tests on Linux and then run on NetBSD
  • Add new/altered tests for Linux-specific APIs or features
  • Build cross-compilation environment to build Linux binaries on NetBSD, to make the test-suite self-hosting
  • Fix Linux emulation for tests that fail
  • Use tests to add Linux emulation for syscalls missing (e.g.timer_*)

It may also be instructive to look at the Linux Test Project.

The project would initially be focussed on x86 (amd64 and i386).

Posted at lunch time on Thursday, February 15th, 2018 Tags:

Debian's .deb packaging format is supported by mature and user-friendly packaging tools.
It's also the native packaging format in several systems, and would improve user experience on those systems.
It would be nice to generate pkgsrc packages to this format.

Prior work exists for generating packages in other formats.

Milestones

  • Lookup .deb format documentation and experiment with tooling
  • Investigate differences between pkgsrc's pkg and deb format
  • Import necessary tools to generate .deb packages
  • Build many packages and look for bugs
Posted Monday night, February 12th, 2018 Tags:
  • Adapt existing ed25519 and salsa20 implementations to netpgp, netpgpverify
  • Maintain compatibility and interoperability with gpg2's usage
  • Maintain compatibility with openssh's keys
  • Extend tests to cover new algorithms

Extended goals:

  • provide a standalone netpgp signing utility, to mirror the netpgpverify verification utility
Posted Monday evening, February 12th, 2018 Tags:

Integrate the LLVM Scudo with the basesystem framework. Build and execute base programs against Scudo.

Milestones:

  • Ensure completeness of the toolchain in the basesystem.
  • Add a new option for building the basesystem utilities with Scudo.
  • Finish the integration and report bugs.
  • Research Scudo for pkgsrc.
Posted Saturday afternoon, January 27th, 2018 Tags:

The NetBSD sourcecode is verified by a static analyzers like Coverity. Attempt to automate the process, report and if possible fix bugs.

Milestones:

  • Consult and research the available tools.
  • Integrate the tools for the purposes of the NetBSD project.
  • Scan the sources, report bugs, if possible fix the problems, or file bug reports
Posted Saturday afternoon, January 27th, 2018 Tags:

There is an initial functional support for syzkaller for NetBSD (as guest). Resume the porting effort, execute and report kernel bugs.

Milestones:

  • Ensure completeness of the current support.
  • Execute the fuzzer and gather reports, narrow down problems, translate to C reproducers.
  • Add missing features, fix bugs in the NetBSD support.
Posted Saturday afternoon, January 27th, 2018 Tags:

Integrate the LLVM libFuzzer with the basesystem framework. Build and execute base programs against libFuzzer.

Milestones:

  • Ensure completeness of the toolchain in the basesystem.
  • Add a new option for building the basesystem utilities with libFuzzer.
  • Finish the integration and report bugs.
Posted Saturday afternoon, January 27th, 2018 Tags:

Add support in the pkgsrc framework for building packages with sanitizers.

Expected sanitizer options:

  • Address (ASan),
  • Memory (MSan),
  • MemoryWithOrigin (MSan with tracking the origin)
  • Undefined (UBSan),
  • Thread (TSan),
  • Address;Undefined (ASan & UBSan)
  • "" (empty string) - the default option

Milestones:

  • Ensure the availability of the toolchain and prebuilt userland with the sanitizers.
  • Add new option in pkgsrc to build the packages with a each sanitizer.
  • Build the packages and report problems and bugs.
Posted Saturday afternoon, January 27th, 2018 Tags:

ALTQ (ALTernate Queueing) is an optional network packet scheduler for BSD systems. It provides various queueing disciplines and other quality of service (QoS) related components required to control resource usage.

It is currently integrated in pf(4) .

Unfortunately it was written a long time ago and it suffers from a lot of code duplication, dangerous code practices and can use improvements both in the API and implementation. After these problems have been addressed it should be integrated with npf(4) .

Posted late Thursday evening, September 14th, 2017 Tags:

Subtle changes in NetBSD may carry a significant performance penalty. Having a realistic performance test for various areas will allow us to find offending commits. Ideally, it would be possible to run the same tests on other operating systems so we can identify points for improvement.

It would be good to test for specific cases as well, such as:

  • Network operation with a packet filter in use
  • Typical disk workload
  • Performance of a multi-threaded program

Insert good performance testsuite examples here

It would be nice to easily test commits between particular dates. Automated runs could be implemented with the help of misc/py-anita.

Posted late Tuesday night, March 1st, 2017 Tags:

LFS is currently carrying around both support for the historical quota format ("quota1") and the recent newer 64-bit in-volume quota format ("quota2") from FFS, but neither is actually connected up or works. In fact, neither has ever worked -- there is no reason to worry about compatibility.

Define a new on-disk quota format that shares some of the properties of both the FFS formats:

  • it should be 64-bit (like quota2);

  • in LFS there is no reason to have a tree-within-a-file like quota2; just files indexed directly by uid/gid like quota1 is fine;

  • the quota files should be hidden away like in quota2;

  • there should be explicit support for deleting quota entries like quota2;

  • some of the additional bits in quota2, like default value handling, should be retained.

Implement the quota system by folding together the existing copies of the quota1 and quota2 code and writing as little new code as possible. Note that while the "ulfs" code should have more or less complete quota support and all the necessary hooks in place, you'll have to add quota hooks to the native lfs code.

You will also need to figure out (before you start coding) what if any special steps need to be taken to make sure that the quota files on disk end up consistent as segments are written out. This may require additional logic like the existing DIROP logic, which would likely be a pain.

Add an option to newfs_lfs to enable quotas. Then add code to fsck_lfs to crosscheck and correct the usage amounts. Also add logic in tunefs_lfs to turn quotas on and off. (It's ok to require a fsck afterwards when turning quotas on to compute the current usage amounts.)

(Note that as of this writing no tunefs_lfs program exists -- that will have to be corrected too.)

Posted in the wee hours of Friday night, August 27th, 2016 Tags:
  • Contact: tech-kern
  • Duration estimate: 1-2 months

The fallocate/posix_fallocate system call, and the file-system-level VOP_FALLOCATE operation backing it, allow preallocating underlying storage for files before writing into them.

This is useful for improving the physical layout of files that are written in arbitrary order, such as files downloaded by bittorrent; it also allows failing up front if disk space runs out while saving a file.

This functionality is not currently implemented for FFS; implement it. (For regular files.) There is not much to this besides running the existing block allocation logic in a loop. (Note though that as the loop might take a while for large allocations, it's important to check for signals periodically to allow the operation to be interrupted.)

Posted Friday evening, August 26th, 2016 Tags:
  • Contact: tech-kern
  • Duration estimate: 2 months

Apply TCP-like flow control to processes accessing to the filesystem, particularly writers. That is: put them to sleep when there is "congestion", to avoid generating enormous backlogs and provide more fair allocation of disk bandwidth.

This is a nontrivial undertaking as the I/O path wasn't intended to support this type of throttling. Also, the throttle should be underneath all caching (there is nothing to be gained by throttling cached accesses) but by that point attributing I/O actions to specific processes is awkward.

It might be worthwhile to develop a scheme to tag buffers with the upper-level I/O requests their pending changes came from, or something of the sort. That would likely help with readahead processing and other things as well as with flow control.

Posted early Friday morning, August 26th, 2016 Tags:

Heavily used file systems tend to spread the blocks all over the disk, especially if free space is scarce. The traditional fix for this is to use dump, newfs, and restore to rebuild a volume; however, this is not acceptable for current disk sizes.

The resize_ffs code already contains logic for relocating blocks, as it needs to be able to do that to shrink a volume; it is likely that it would serve nicely as a starting point for a defragmenting tool.

Note that safety (not scrambling the volume if the system crashes while defragging) is somewhat tricky and critically important.

Posted early Friday morning, August 26th, 2016 Tags:
  • Contact: tech-kern
  • Duration estimate: 1-2 months

NetBSD has a fixed, kernel-wide upper limit on transfer size, called MAXPHYS, which is currently set to 64k on most ports. This is too small to perform well on modern IDE and SCSI hardware; on the other hand some devices can't do more than 64k, and in some cases are even limited to less (the Xen virtual block device for example). Software RAID will also cause requests to be split in multiple smaller requests.

This limit should be per-IO-path rather than global and should be discovered/probed as those paths are created, based on driver capability and driver properties.

Much of the work has already been done and has been committed on the tls-maxphys branch. What's needed at this point is mostly testing and probably some debugging until it's ready to merge.

This project originaly suggested instead to make the buffer queue management logic (which currently only sorts the queue, aka disksort) capable of splitting too-large buffers or aggregating small contiguous buffers in order to conform to device-level requirements.

Once the MAXPHYS changes are finalized and committed, this project may be simply outdated. However, it may also be worthwhile to pursue this idea as well, particularly the aggregation part.

Posted early Friday morning, August 26th, 2016 Tags:

Right now /usr/games/gomoku is not very smart. This has two consequences: first, it's not that hard to beat it if one tries; and second, if you put it in a spot, it takes annoyingly long to play and chews up tons of RAM. On older hardware or a RAM-limited virtual machine, it'll run out of memory and bail.

A few years ago when I (dholland) looked into this, there was a small but apparently nonempty community of AI researchers working on gomoku brains. There was also some kind of interface standard for gomoku brains such that they could be conveniently run against one another.

This project is to:

  • track down the latest version of that standard and add external brain support to our gomoku(6);
  • package a good gomoku brain in pkgsrc if there's a suitable open-source one;
  • based on available research, improve the brain built into our gomoku (doesn't have to be competitive, just not terrible);
  • also (if applicable) improve the framework the built-in brain code uses so it can back out if it runs out of memory.

Some of these issues were originally raised in http://gnats.netbsd.org/3126.

Note to passersby: "gomoku" is not the same game as "go".

Posted in the wee hours of Sunday night, August 8th, 2016 Tags:

Currently BSD make emits tons and tons of stat(2) calls in the course of figuring out what to do, most notably (but not only) when matching suffix rules. This causes measurable amounts of overhead, especially when there are a lot of files like in e.g. libc. First step is to quantify this overhead so you can tell what you're accomplishing.

Fixing this convincingly requires a multi-step approach: first, give make an abstraction layer for directories it's working with. This alone is a nontrivial undertaking because make is such a mess inside. This step should also involve sorting out various issues where files in other directories (other than the current directory) sometimes don't get checked for changes properly.

Then, implement a cache under that abstraction layer so make can remember what it's learned instead of making the same stat calls over and over again. Also, teach make to scan a directory with readdir() instead of populating the cache by making thousands of scattershot stat() calls, and implement a simple heuristic to decide when to use the readdir method.

Unfortunately, in general after running a program the cache has to be flushed because we don't know what the program did. The final step in this project is to make use of kqueue or inotify or similar when running programs to synchronize make's directory cache so it doesn't have to be flushed.

As an additional step one might also have make warn when a recipe touches files it hasn't been declared to touch... but note that while this is desirable it is also somewhat problematic.

Posted terribly early Friday morning, May 27th, 2016 Tags:

When developing device drivers inside the kernel, mistakes will usually cause the whole kernel to crash unrecoverably and require a reboot. But device drivers need not run inside the kernel: with rump, device driver code can be run as a process inside userland.

However, userland code has only limited access to the hardware registers needed to control devices: currently, NetBSD supports only USB device drivers in userland, via ugen(4). NetBSD should additionally support developing PCI drivers in userland with rump -- at least one driver, iwm(4), was developed using rump, but on a Linux host!

There are several milestones to this project:

  1. Implement enough of the bus_space(9) and pci_mapreg() (pci(9)) APIs to map device registers from PCI BARs, using a pci(4) device (/dev/pciN). A first approximation can be done using pci(3) and simply mmapping from mem(4) (/dev/mem), but it would be better to cooperate with the kernel so that the kernel can limit the user to mapping ranges listed in PCI BARs without granting privileges to read all physical pages in the system. Cooperation with the kernel will also be necessary to implement port I/O instead of memory-mapped I/O, without raising the I/O privilege level of the userland process, on x86 systems.

  2. Expose PCI interrupts as events that can be read from a pci(4) (/dev/pciN) device instance, and use that to implement the pci_intr(9) API in userland. For many devices, this may require a small device-specific shim in the kernel to acknowledge interrupts while they are masked -- but that is a small price to pay for rapidly iterating driver development in userland.

  3. Devise a scheme for userland allocate and map memory for DMA in order to implement bus_dma(9). Again, a first approximation can be done by simply wiring pages with mlock(3) and then asking the kernel for a virtual-to-physical translation to program hardware DMA registers. However, this will not work on any machines with an IOMMU, which would help to prevent certain classes of catastrophic memory corruption in the case of a buggy driver. Cooperation with the kernel, and maintaining a kernel-side mirror of each bus_dmamem allocation and each bus_dmamap.

Inspiration may be found in the Linux uio framework. This project is not necessarily PCI-specific -- ideally, most of the code to manage bus_space(9), bus_dma(9), and interrupt event delivery should be generic. The focus is PCI because it is widely used and would be applicable to many drivers and devices for which someone has yet to write drivers.

Posted late Tuesday afternoon, February 23rd, 2016 Tags:

NetBSD configures a timer device to deliver a periodic timer interrupt, usually every 10 ms, in order to count time, wake threads that are sleeping, etc. This made sense when timer devices were expensive to program and CPUs ran at MHz. But today, CPUs run at GHz; timers on modern x86, arm, mips, etc. hardware are cheap to reprogram; programs expect greater than 10 ms resolution for sleep times; and mandatory periodic activity on idle machines wastes power.

There are four main milestones to this project:

  1. Choose a data structure for high-resolution timers, and a way to request high-resolution vs low-resolution sleeps, and adapt the various timeout functions (cv_timedwait, etc.) to use it. The current call wheel data structure for callouts provides good performance, but only for low-resolution sleeps. We need another data structure that provides good performance for high-resolution sleeps without hurting the performance of the existing call wheel for existing applications.

  2. Design a machine-independent high-resolution timer device API, implement it on a couple machines, and develop tests to confirm that it works. This might be done by adapting the struct timecounter interface to arm it for an interrupt, or might be done another way.

  3. Convert all the functions of the periodic 10 ms timer, hardclock, to schedule activity only when needed.

  4. Convert the various software subsystems that rely on periodic timer interrupts every tick, or every second, via callout(9), either to avoid periodic work altogether, or to batch it up only when the machine is about to go idle, in order to reduce the number of wakeups and thereby reduce power consumption.

Posted late Monday evening, February 22nd, 2016 Tags:
  • Contact: tech-kern
  • Duration estimate: 3 months

Programmatic interfaces in the system (at various levels) are currently specified in C: functions are declared with C prototypes, constants with C preprocessor macros, etc. While this is a reasonable choice for a project written in C, and good enough for most general programming, it falls down on some points, especially for published/supported interfaces. The system call interface is the most prominent such interface, and the one where these issues are most visible; however, other more internal interfaces are also candidates.

Declaring these interfaces in some other form, presumably some kind of interface description language intended for the purpose, and then generating the corresponding C declarations from that form, solves some of these problems. The goal of this project is to investigate the costs and benefits of this scheme, and various possible ways to pursue it sanely, and produce a proof-of-concept implementation for a selected interface that solves a real problem.

Note that as of this writing many candidate internal interfaces do not really exist in practice or are not adequately structured enough to support this treatment.

Problems that have been observed include:

  • Using system calls from languages that aren't C. While many compilers and interpreters have runtimes written in C, or C-oriented foreign function interfaces, many do not, and rely on munging system headers in (typically) ad hoc ways to extract necessary declarations. Others cut and paste declarations from system headers and then break silently if anything ever changes.

  • Generating test or validation code. For example, there is some rather bodgy code attached to the vnode interface that allows crosschecking the locking behavior observed at runtime against a specification. This code is currently generated from the existing vnode interface specification in vnode_if.src; however, the specification, the generator, and the output code all leave a good deal to be desired. Other interfaces where this kind of validation might be helpful are not handled at all.

  • Generating marshalling code. The code for (un)marshalling system call arguments within the kernel is all handwritten; this is laborious and error-prone, especially for the various compat modules. A code generator for this would save a lot of work. Note that there is already an interface specification of sorts in syscalls.master; this would need to be extended or converted to a better description language. Similarly, the code used by puffs and refuse to ship file system operations off to userland is largely or totally handwritten and in a better world would be automatically generated.

  • Generating trace or debug code. The code for tracing system calls and decoding the trace results (ktrace and kdump) is partly hand-written and partly generated with some fairly ugly shell scripts. This could be systematized; and also, given a complete specification to work from, a number of things that kdump doesn't capture very well could be improved. (For example, kdump doesn't decode the binary values of flags arguments to most system calls.)

  • Add your own here. (If proposing this project for GSOC, your proposal should be specific about what problem you're trying to solve and how the use of an interface definition language will help solve it. Please consult ahead of time to make sure the problem you're trying to solve is considered interesting/worthwhile.)

The project requires: (1) choose a target interface and problem; (2) evaluate existing IDLs (there are many) for suitability; (3) choose one, as roll your own should be only a last resort; (4) prepare a definition for the target interface in your selected IDL; (5) implement the code generator to produce the corresponding C declarations; (6) integrate the code generator and its output into the system (this should require minimal changes); (7) implement the code generator and other material to solve your target problem.

Note that while points 5, 6, and 7 are a reasonably simple matter of programming, parts 1-3 and possibly 4 constitute research -- these parts are as important as the code is, maybe more so, and doing them well is critical to success.

The IDL chosen should if at all possible be suitable for repeating steps 4-7 for additional choices of target interface and problem, as there are a lot of such choices and many of them are probably worth pursuing in the long run.

Some things to be aware of:

  • syscalls.master contains a partial specification of the functions in the system call interface. (But not the types or constants.)
  • vnode_if.src contains a partial specification of the vnode interface.
  • There is no specification, partial or otherwise, of the other parts of the VFS interface, other than the code and the (frequently outdated) section 9 man pages.
  • There are very few other interfaces within the kernel that are (as of this writing) structured enough or stable enough to permit using an IDL without regretting it. The bus interface might be one.
  • Another possible candidate is to pursue a specification language for handling all the conditional visibility and namespace control material in the userspace header files. This requires a good deal of familiarity with the kinds of requirements imposed by standards and the way these requirements are generally understood and is a both large and extremely detail-oriented.

Note: the purpose of an IDL is not to encode a parsing of what are otherwise C-language declarations. The purpose of an IDL is to encode some information about the semantics of an interface. Parsing the input is the easy part of the problem. For this reason, proposals that merely suggest replacing C declarations with e.g. equivalent XML or JSON blobs are unlikely to be accepted.

(Also note that XML-based IDLs are unlikely to be accepted by the community as XML is not in general considered suitable as a source language.)

Posted terribly early Wednesday morning, February 17th, 2016 Tags:
  • Contact: tech-kern
  • Duration estimate: 2-3 months

The fdiscard system call, and the file-system-level VOP_DISCARD operation backing it, allow dropping ranges of blocks from files to create sparse files with holes. This is a simple generalization of truncate.

Discard is not currently implemented for FFS; implement it. (For regular files.) It should be possible to do so by generalizing the existing truncate code; this should be not super-difficult, but somewhat messy.

Note that it's important (for both the WAPBL and non-WAPBL cases) to preserve the on-disk invariants needed for successful crash recovery.

Posted early Wednesday morning, November 25th, 2015 Tags:
  • Contact: tech-kern
  • Duration estimate: 1-2 months

Implement the block-discard operation (also known as "trim") for RAIDframe.

Higher-level code may call discard to indicate to the storage system that certain blocks no longer contain useful data. The contents of these blocks do not need to be preserved until they are later written again. This means, for example, that they need not be restored during a rebuild operation.

RAIDframe should also be made to call discard on the disk devices underlying it, so those devices can take similar advantage of the information. This is particularly important for SSDs, where discard ("trim") calls can increase both the performance and write lifetime of the device.

The complicating factor is that a block that has been discarded no longer has stable contents: it might afterwards read back as zeros, or it might not, or it might change to reading back zeros (or trash) at any arbitrary future point until written again.

The first step of this project is to figure out a suitable model for the operation of discard in RAIDframe. Does it make sense to discard single blocks, or matching blocks across stripes, or only whole stripe groups, or what? What metadata should be stored to keep track of what's going on, and where does it go? Etc.

Posted early Wednesday morning, November 25th, 2015 Tags:

Implement component 'scrubbing' in RAIDframe.

RAIDframe (raid(4)) provides various RAID levels to NetBSD, but currently has no facilities for 'scrubbing' (checking) the components for read errors.

Milestones:

  • implement a generic scrubbing routine that can be used by all RAID types
  • implement RAID-level-specific code for component scrubbing
  • add an option to raidctl (raidctl(8)) to allow the system to run the scrubbing as required
  • update raidctl (raidctl(8)) documentation to reflect the new scrubbing capabilities, and discuss what scrubbing can and cannot do (benefits and limitations)

Bonus:

  • Allow the user to select whether or not to attempt to 'repair' errors
  • Actually attempt to repair errors
Posted late Thursday afternoon, November 5th, 2015 Tags:

Abstract:

Xen now supports the ARM cpu architecture. We still don't have support in NetBSD for this architecture. A dom0 would be the first start in this direction.

Deliverables:

  • toolstack port to arm
  • Fully functioning dom0

This project would be much easier once pvops/pvh is ready, since the Xen ARM implementation only supports PVH.

See http://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions

Posted at midnight, August 19th, 2015 Tags:

Abstract: The blktap2 method provides a userland daemon to provide access to on disk files, arbitrated by the kernel. The NetBSD kernel lacks this support and currently disables blktap in xentools.

Deliverables:

  • blktap2 driver for /dev/blktapX
  • Tool support to be enabled in xentools
  • Enable access to various file formats (Qcow, VHD etc) , via the [ tap: ] disk specification.

Implementation:

The driver interacts with some areas of the kernel that are different from linux - eg: inter-domain page mapping, interdomain signalling. This may make it more challenging than a simple driver, especially with test/debug/performance risk.

Note: There's opportunity to compare notes with the FUSE/rump implementations.

See http://wiki.xenproject.org/wiki/Blktap and http://xenbits.xensource.com/hg/xen-unstable.hg/file/tip/tools/blktap2/README

Posted at midnight, August 19th, 2015 Tags:

Abstract:

The NetBSD boot path contains conditional compiled code for XEN. In the interests of moving towards unified pvops kernels, the code can be modified to be conditionally executed at runtime.

Deliverables:

  • Removal of #ifdef conditional code from all source files.
  • New minor internal API definitions, where required, to encapsulate platform specific conditional execution.
  • Performance comparison figures for before and after.
  • Clear up the API clutter to make way for pvops.

Note: Header files are exempt from #ifdef removal requirement.

Implementation:

The number of changes required are estimated as follows:

drone.sorcerer> pwd
/home/sorcerer/src/sys
drone.sorcerer> find . -type f -name '*.[cSs]' -exec grep -H "^#if.* *\\bXEN\\b" '{}' \; | wc -l
     222
drone.sorcerer>

The effort required in this task is likely to be highly variable, depending on the case. In some cases, minor API design may be required.

Most of the changes are likely to be mechanical.

Posted at midnight, August 19th, 2015 Tags:
  • Contact: port-xen
  • Duration estimate: 32 hours

Abstract:

NetBSD Dom0 kernels currently operate in fully PV mode. On amd64, this has the same performance drawbacks as for PV mode in domU. Instead, PVH mode provides a HVM container over pagetable management, while virtualising everything else. This mode is available on dom0, we attempt to support it.

Note: Xen/Linux is moving to this dom0 pvh support model.

Deliverables: PVH mode dom0 operation.

Implementation:

This project depends on the domU pvops/pvh project. The main work is in configuration and testing related (bringing in native device drivers as well as backend ones).

The bootstrap path may need to be tweaked for dom0 specific things.

Dependencies: This project depends on completion of the domU pvops/pvh project. This project can enable the NetBSD/Xen/ARM dom0 port.

See http://wiki.xenproject.org/wiki/Xen_Project_Hypervisor_Roadmap/4.6

Posted at midnight, August 19th, 2015 Tags:
  • Contact: port-xen
  • Duration estimate: 64 hours

Abstract: Dom0 is not SMP safe right now. The main culprits are the backend drivers for blk and netif (and possibly others, such as pciback)

Deliverables:

  • SMP capable dom0

Implementation:

This involves extensive stress testing of the dom0 kernel for concurrency and SMP workloads. Locking in various backend and other drivers need to be reviewed, reworked and tested.

Interrupt paths need to be reviewed, and the interrupt handling code needs to be reworked, potentially. The current event code doesn't multiplex well on vcpus. This needs reinvestigation.

This is a test/debug heavy task, since MP issues can crop up in various unrelated parts of the kernel.

Posted at midnight, August 19th, 2015 Tags:
  • Contact: port-xen
  • Duration estimate: 48 hours

Abstract: This is the first step towards PVH mode. This is relevant only for DomU. Speeds up amd64 - PV on amd64 has sub-optimal TLB and syscall overhead due to privilege level sharing between kernel and userland.

Deliverables:

  • operational PV drivers (netif.h, blkif.h) on HVM mode.

Implementation:

  • Xen needs to be detected at boottime.
  • The shared page and hypercall interface need to be setup.
  • xenbus(4) attachment during bootime configuration.

Scope (Timelines):

This project involves intrusive kernel configuration changes, since NetBSD currently has separate native (GENERIC) and xen (XEN3_DOMU) kernels. A "hybrid" build is required to bring in the PV drivers and xen infrastructure support such as xenbus(4).

Once the hypercall page is up however, progress should be faster.

See:

http://wiki.xenproject.org/wiki/XenParavirtOps and http://wiki.xenproject.org/wiki/PV_on_HVM

Posted at midnight, August 19th, 2015 Tags:
  • Contact: port-xen
  • Duration estimate: 48 hours

Abstract: This is the final step towards PVH mode. This is relevant only for DomU. Xen is moving to this eventual dom0 pvh support model. This project depends on completion of pv-on-hvm.

Deliverables:

  • operational interrupt (event) handling
  • PV only drivers.

Implementation:

Following on from the pv-on-hvm project, this project removes all remaining dependencies on native drivers. All drivers are PV only. Interrupt setup and handling is via the "event" mechanism.

Scope (Timelines):

This project has some uncertainty based on the change in the interrupt mechanism. Since the interrupt mechanism moves from apic + IDT based, to event based, there's room for debug/testing spillover.

Further, private APIs may need to be developed to parition the pv and native setup and management code for both mechanisms.

See: http://wiki.xenproject.org/wiki/Virtualization_Spectrum#Almost_fully_PV:_PVH_mode

Posted at midnight, August 19th, 2015 Tags:
  • Contact: port-xen
  • Duration estimate: 16 hours

Abstract:

Although Xen has ACPI support, the kernel needs to make explicit hypercalls for power management related functions.

sleep/resume are supported, and this support needs to be exported via the dom0 kernel.

Deliverables:

  • sleep/resume support for NetBSD/Xen dom0

Implementation:

There are two approaches to this. The first one involves using the kernel to export syscall nodes for ACPI.

The second one involves Xen aware power scripts, which can write directly to the hypercall kernfs node - /kern/privcmd. Device tree suspend could happen via drvctl(8), in theory.

Posted at midnight, August 19th, 2015 Tags:
  • Contact: port-xen
  • Duration estimate: 16 hours

Abstract: The pvfb video driver and the keyboard/point driver are useful for gui interaction with a domU console, including gui.

Deliverables:

A framebuffer device and console, which can be used to interact with the domU. The qemu process on the dom0 can then use this buffer to display to the regular X server using the SDL, as it does with HVM domUs. This is very useful for gui frontends to xen.

Implementation:

A simple framebuffer driver is implemented, along the lines of: http://xenbits.xensource.com/hg/linux-2.6.18-xen.hg/file/ca05cf1a9bdc/drivers/xen/fbfront/

Posted at midnight, August 19th, 2015 Tags:
  • Contact: port-xen
  • Duration estimate: 32 hours

Abstract:

Make the dom0 kernel use and provide drm(4) support. This enable gui use of dom0.

Deliverables:

Functioning X server using native kernel style drmkms support.

Implementation:

  • mtrr support for Xen
  • high memory RAM extent allocation needs special attention (See: kern/49330)
  • native driver integration and testing.
Posted at midnight, August 19th, 2015 Tags:
  • Contact: port-xen
  • Duration estimate: 64 hours

Abstract: This project enables NetBSD to have physical ram pages dynamically added via uvm_page_physload() and dynamically removed using a complimentary function.

Rationale: Makes NetBSD a viable datacentre OS, with hotspare RAM add/remove support.

Deliverables:

  • dynamic physical page add/remove support
  • Xen balloon driver usage of this support.

Implementation:

Further api and infrastructural changes may be required to accommodate these changes. The physical page management code will need to be enhanced to more than a fixed array, while taking into account memory attribute related tagging information.

Posted at midnight, August 19th, 2015 Tags:

Abstract: This project involves allowing SCSI devices on the dom0 to be passed through to the domU. Individual SCSI devices can be passed through dynamically. The domU needs to have a "frontend" driver that can communicate with the backend driver which runs on the dom0 and takes care of arbitrating and sometimes duplicating the requests.

Deliverables:

  • Functioning backend and frontend drivers for dom0 and domU respectively.
  • Make available high performance SCSI device to PV and PVHVM domains.

See http://wiki.xenproject.org/wiki/XenPVSCSI

Posted at midnight, August 19th, 2015 Tags:

Abstract: This project involves allowing USB devices on the dom0 to be passed through to the domU. Individual USB devices can be passed through dynamically. The domU needs to have a "frontend" driver that can communicate with the backend driver which runs on the dom0 and takes care of arbitrating and sometimes duplicating the requests.

Deliverables:

  • Functioning backend and frontend drivers for dom0 and domU respectively.
  • Make available USB device passthrough to PV and PVHVM domains.

Implementation:

Since the feature is under development in Xen, a frontend implementation is advisable to start to engage with the project. Once we have a functioning frontend on a non-NetBSD dom0, then we can engage with the community to develop the backend.

See http://wiki.xenproject.org/wiki/XenUSBPassthrough

Posted at midnight, August 19th, 2015 Tags:

Abstract: Current libvirt support in NetBSD doesn't include Xen support.

Rationale: Enables gui based domU managers. Enables the pvfb work to be easily accessible

Deliverables:

  • Fully functioning libvirt with Xen "driver" support.

See http://wiki.xenproject.org/wiki/Libxl_event_API_improvements and http://libvirt.org/drvxen.html

Posted at midnight, August 19th, 2015 Tags:

Abstract: NetBSD/Xen currently doesn't support __HAVE_DIRECT_MAP

Deliverables:

  • Direct Mapped functioning NetBSD/Xen, both dom0 and domU
  • Demonstrable performance numbers
  • Reduce TLB contention and clean up code for pvops/pvh

Implementation: This change involves testing for the Xen and CPU featureset, and tweaking the pmap code so that 1G superpages can be requested from the hypervisor and direct mapped the same way native code does.

Posted Tuesday night, August 18th, 2015 Tags:

Add a query optimizer to find(1).

Currently find builds a query plan for its search, and then executes it with little or no optimization. Add an optimizer pass on the plan that makes it run faster.

Things to concentrate on are transforms that allow skipping I/O: not calling stat(2) on files that will not be matched, for example, or not recursing into subdirectories whose contents cannot ever match. Combining successive string matches into a single match pattern might also be a win; so might precompiling match patterns into an executable match form (like with regcomp(3)).

To benefit from many of the possible optimizations it may be necessary to extend the fts(3) interface and/or extend the query plan schema or the plan execution logic. For example, currently it doesn't appear to be possible for an fts(3) client to take advantage of file type information returned by readdir(3) to avoid an otherwise unnecessary call to stat(2).

Step 1 of the project is to choose a number of candidate optimizations, and for each identify the internal changes needed and the expected benefits to be gained.

Step 2 is to implement a selected subset of these based on available time and cost/benefit analysis.

It is preferable to concentrate on opportunities that can be found in find invocations likely to actually be typed by users or issued by programs or infrastructure (e.g. in pkgsrc), vs. theoretical opportunities unlikely to appear in practice.

Posted late Wednesday evening, July 8th, 2015 Tags:

Test and debug the RAID 6 implementation in RAIDframe.

NetBSD uses RAIDframe (raid(4)) and stub code exists for RAID6 but is not very documented or well tested. Obviously, this code needs to be researched and vetted by an interested party. Other BSD projects should be consulted freely.

Milestones: * setup a working RAID 6 using RAIDFrame * document RAID6 in raid(4) manual page * port/develop a set of reliability and performance tests * fix bugs along the way * automate RAID testing in atf

Bonus: * Document how to add new RAID levels to RAIDframe * (you're awesome bonus) add RAID 1+0, etc

Posted Wednesday night, February 18th, 2015 Tags:
  • Contact: tech-kern
  • Duration estimate: 1-2 months

The mlock() system call locks pages in memory; however, it's meant for real-time applications that wish to avoid pageins.

There's a second class of applications that want to lock pages in memory: databases and anything else doing journaling, logging, or ordered writes of any sort want to lock pages in memory to avoid pageouts. That is, it should be possible to lock a (dirty) page in memory so that it does not get written out until after it's unlocked.

It is a bad idea to try to make mlock() serve this purpose as well as the purpose it already serves, so add a new call, perhaps mlockin(), and implement support in UVM.

Then for extra credit ram it through POSIX and make Linux implement it :-)

Posted terribly early Monday morning, February 16th, 2015 Tags:

Add publish-subscribe sockets to AF_UNIX (filesystem sockets).

The idea of a publish-subscribe socket (as opposed to a traditional socket) is that anything written to one end is read out by all processes reading from the other end.

This raises some issues that need attention; most notably, if a process doesn't read data sent to it, how long do we wait (or how much data do we allow to accumulate) before dropping the data or disconnecting that process?

It seems that one would want to be able to have both SOCK_STREAM and SOCK_DGRAM publish/subscribe channels, so it isn't entirely clear yet how best to graft this functionality into the socket API. Or the socket code. It might end up needing to be its own thing instead, or needing to extend the socket API, which would be unfortunate but perhaps unavoidable.

The goal for these sockets is to provide a principled and robust scheme for passing notices around. This will allow replacing multiple existing ad hoc callback schemes (such as for notices of device insertions) and also to allow various signaling schemes among user processes that have never been very workable in Unix. Those who remember AREXX will recognize the potential; but of course it will take time before anything like that can be orchestrated.

For this reason it's important that it be reasonably easy for one of the endpoints to be inside the kernel; but it's also important that peer credentials and permissions work.

Another goal is to be able to use publish/subscribe sockets to provide a compatibility library for Linux desktop software that wants to use dbus or any successor to dbus.

I'm marking this project hard as it's still only about half baked.

Posted terribly early Monday morning, February 16th, 2015 Tags:

Add support to UVM to allow processes to map pages directly from underlying objects that are already mapped into memory: for example, files in tmpfs; files in mfs; and also files on memory-mapped storage devices, like raw flash chips or experimental storage-class memory hardware.

This allows accesses (most notably, execution of user programs) to go directly to the backing object without copying the pages into main system memory. (Which in turn has obvious advantages.)

Note that uebayasi@ was working on this at one point but his changes did not get merged.

Posted terribly early Monday morning, February 16th, 2015 Tags:
  • Contact: tech-kern
  • Duration estimate: 2 months

Add per-user memory usage limits (also known as "quotas") to tmpfs, using the existing quota system infrastructure.

Posted terribly early Monday morning, February 16th, 2015 Tags:
  • Contact: tech-kern
  • Duration estimate: 2 months

While currently we have the cgd(4) driver for encrypting disks, setting it up is fairly involved. Furthermore, while it's fairly easy to use it just for /home, in an ideal world the entire disk should be encrypted; this leads to some nontrivial bootstrapping problems.

Develop a scheme for mounting root on cgd that does not require explicit manual setup, that passes cryptographic muster, and that protects everything on the root volume except for what absolutely must be exposed. Implement it.

The following is a non-exhaustive list of issues to consider:

  • How should we tell when root should be on cgd (perhaps in boot.cfg?)
  • When (and how) do we enter the passphrase needed to mount root (at mount-root time? in the bootloader? after mounting a fake root?)
  • Key management for the encryption passphrase
  • Where to keep the bootloader and/or kernels
  • Should the bootloader be able to read the cgd to get the boot kernel from it?
  • If the kernel is not on cgd, should it be signed to allow the bootloader to verify it?
  • Integration with sysinst so all you need to do to get FDE is to hit a checkbox
  • Perhaps, making it easy or at least reasonably possible to migrate an unencrypted root volume to cgd

Note that while init(8) currently has a scheme for mounting a temporary root and then chrooting to the real root afterwards, it doesn't work all that well. Improving it is somewhat difficult; also, ideally init(8) would be on the encrypted root volume. It would probably be better to support mounting the real root directly on cgd.

Another option is a pivot_root type of operation like Linux has, which allows mounting a fake root first and then shuffling the mount points to move something else into the / position. This has its drawbacks as well, and again ideally there would be no unencrypted fake root volume.

Posted terribly early Monday morning, February 16th, 2015 Tags:

The current asynchronous I/O (aio) implementation works by forking a background thread inside the kernel to do the I/O synchronously. This is a starting point, but one thread limits the amount of potential parallelism, and adding more threads falls down badly when applications want to have large numbers of requests outstanding at once.

Furthermore, the existing code doesn't even work in some cases; this is not well documented but there have been scattered reports of problems that nobody had time to look into in detail.

In order to make asynchronous I/O work well, the I/O path needs to be revised, particularly in the kernel's file system interface, so that all I/O operations are asynchronous requests by default. It is easy for high-level code to wait synchronously for lower-level asynchronous requests to complete; it is much more problematic for an asynchronous request to call into code that expects to be synchronous.

The ?flow control project, which also requires substantial revisions to the I/O path, naturally combines with this project.

Posted in the wee hours of Sunday night, February 16th, 2015 Tags:

NetBSD has preliminary DTrace support, so it supports SDT and FBT provider only.

riz@ has syscall provider patch.

Posted early Saturday morning, February 14th, 2015 Tags:

The Ext4 file system is standard file system for Linux (recent Ubuntu, and somewhat old Fedora). It is the successor of the Ext3 file system.

It has journaling, larger file system volumes, and improved timestamp etc features.

The NetBSD kernel support and accompanying userland code should be written.

It is not clear at the moment if this should be done by working on the existing ext2 code or not, and if not, whether it should be integrated with the ufs code or not.

Posted early Saturday morning, February 14th, 2015 Tags:

Linux provides an ISC-licensed Broadcom SoftMAC driver. Source code is included in Linux kernel tree,

Posted early Saturday morning, February 14th, 2015 Tags:

Xilinx MicroBlaze is RISC processort for Xilinx's FPGA chip. MicroBlaze can have MMU, and NetBSD can support it.

Posted early Saturday morning, February 14th, 2015 Tags:

Google Chrome is widely used nowadays, and it has some state-of-the-art functionalities.

Chromium web browser is open source edition of Google Chrome.

Currently Chromium is present in pkgsrc-wip as wip/chromium. Please see wip/chromium/TODO for a TODO list about it.

Posted terribly early Saturday morning, February 14th, 2015 Tags:

If one embarks on a set of package updates and it doesn't work out too well, it is nice to be able to roll back to a previous state. This entails two things: first, reverting the set of installed packages to what it was at some chosen prior time, and second, reverting changes to configuration, saved application state, and other such material that might be needed to restore to a fully working setup.

This project is about the first part, wihch is relatively tractable. The second part is Hard(TM), but it's pointless to even speculate about it until we can handle the first part, which we currently can't. Also, in many cases the first part is perfectly sufficient to recover from a problem.

Ultimately the goal would be to be able to do something like pkgin --revert yesterday but there are a number of intermediate steps to that to provide the infrastructure; after that adding the support to pkgin (or whatever else) should be straightforward.

The first thing we need is a machine-readable log somewhere of package installs and deinstalls. Any time a package is installed or removed, pkg_install adds a record to this log. It also has to be possible to trim the log so it doesn't grow without bound -- it might be interesting to keep the full history of all package manipulations over ten years, but for many people that's just a waste of disk space. Adding code to pkg_install to do this should be straightforward; the chief issues are

  • choosing the format
  • deciding where to keep the data

both of which require some measure of community consensus.

Preferably, the file should be text-based so it can be manipulated by hand if needed. If not, there ought to be some way to recover it if it gets corrupted. Either way the file format needs to be versioned and extensible to allow for future changes.

The file format should probably have a way to enter snapshots of the package state in addition to change records; otherwise computing the state at a particular point requires scanning the entire file. Note that this is very similar to deltas in version control systems and there's a fair amount of prior art.

Note that we can already almost do this using /var/backups/work/pkgs.current,v; but it only updates once a day and the format has assorted drawbacks for automated use.

The next thing needed is a tool (maybe part of pkg_info, maybe not) to read this log and both (a) report the installed package state as of a particular point in time, and (b) print the differences between then and now, or between then and some other point in time.

Given these two things it becomes possible to manually revert your installed packages to an older state by replacing newer packages with older packages. There are then two further things to do:

Arrange a mechanism to keep the .tgz files for old packages on file.

With packages one builds oneself, this can already be done by having them accumulate in /usr/pkgsrc/packages; however, that has certain disadvantages, most notably that old packages have to be pruned by hand. Also, for downloaded binary packages no comparable scheme exists yet.

Provide a way to compute the set of packages to alter, install, or remove to switch to a different state. This is somewhat different from, but very similar to, the update computations that tools like pkgin and pkg_rolling-replace do, so it probably makes sense to implement this in one or more of those tools rather than in pkg_install; but perhaps not.

There are some remaining issues, some of which aren't easily solved without strengthening other things in pkgsrc. The most notable one is: what about package options? If I rebuild a package with different options, it's still the "same" package (same name, same version) and even if I record the options in the log, there's no good way to distinguish the before and after binary packages.

Posted terribly early Saturday morning, August 30th, 2014 Tags:

Some of the games in base would be much more playable with even a simple graphic interface. The card games (e.g. canfield, cribbage) are particularly pointed examples; but quite a few others, e.g. atc, gomoku, hunt, and sail if anyone ever fixes its backend, would benefit as well.

There are two parts to this project: the first and basically mostly easy part is to pick a game and write an alternate user interface for it, using SDL or gtk2 or tk or whatever seems appropriate.

The hard part is to arrange the system-level infrastructure so that the new user interface appears as a plugin for the game that can be installed from pkgsrc but that gets found and run if appropriate when the game is invoked from /usr/games. The infrastructure should be sufficiently general that lots of programs in base can have plugins.

Some things this entails:

  • possibly setting up a support library in the base system for program plugins, if it appears warranted;

  • setting up infrastructure in the build system for programs with plugins, if needed;

  • preferably also setting up infrastructure in the build system for building plugins;

  • choosing a place to put the header files needed to build external plugins;

  • choosing a place to put plugin libraries, too, as there isn't a common model out there yet;

  • establishing a canonical way for programs in base to find things in pkgsrc, which is not technically difficult but will require a lot of wrangling to reach a community consensus;

  • setting up any pkgsrc infrastructure needed to build plugin packages for base (this should take little or no work).

It is possible that plugins warrant a toplevel directory under each prefix rather than being stuffed in lib/ directories; e.g. /usr/plugins, /usr/pkg/plugins, /usr/local/plugins, so the pkgsrc-installed plugins for e.g. rogue would go by default in /usr/pkg/plugins/rogue.

Note that while this project is disguised as a frivolous project on games, the infrastructure created will certainly be wanted in the medium to long term for other more meaty things. Doing it on games first is a way to get it done without depending on or conflicting with much of anything else.

Posted Wednesday afternoon, April 2nd, 2014 Tags:

In MacOS X, while logged in on the desktop, you can switch to another user (leaving yourself logged in) via a system menu.

The best approximation to this we have right now is to hit ctrl-alt-Fn, switch to a currently unused console, log in, and run startx with an alternate screen number. This has a number of shortcomings, both from the point of view of general polish (logging in on a console and running startx is very untidy, and you have to know how) and of technical underpinnings (starting multiple X servers uses buckets of memory, may cause driver or drmgem issues, acceleration may not work except in the first X server, etc.)

Ideally we'd have a better scheme for this. We don't necessarily need something as slick as OS X provides, and we probably don't care about Apple's compositing effects when switching, but it's useful to be able to switch users as a way of managing least privilege and it would be nice to make it easy to do.

The nature of X servers makes this difficult; for example, it isn't in any way safe to use the same X server for more than one user. It also isn't safe to connect a secure process to a user's X server to display things.

It seems that the way this would have to work is akin to job control: you have a switching supervisor process, which is akin to a shell, that runs as root in order to do authentication and switches (or starts) X servers for one or more users. The X servers would all be attached, I guess, to the same graphics backend device (wsfb, maybe? wsdrmfb?) and be switched in and out of the foreground in much the way console processes get switched in and out of the foreground on a tty.

You have to be able to get back to the switching supervisor from whatever user and X server you're currently running in. This is akin to ^Z to get back to your shell in job control. However, unlike in job control there are security concerns: the key combination has to be something that a malicious application, or even a malicious X server, can't intercept. This is the "secure attention key". Currently even the ctrl-alt-Fn sequences are handled by the X server; supporting this will take quite a bit of hacking.

Note that the secure attention key will also be wanted for other desktop things: any scheme that provides desktop-level access to system configuration needs to be able to authenticate a user as root. The only safe way to do this is to notify the switching supervisor and then have the user invoke the secure attention key; then the user authenticates to the switching supervisor, and that in turn provides some secured channel back to the application. This avoids a bunch of undesirable security plumbing as is currently found in (I think) GNOME; it also avoids the habit GNOME has of popping up unauthenticated security dialogs asking for the root password.

Note that while the switching supervisor could adequately run in text mode, making a nice-looking graphical one won't be difficult. Also that would allow the owner of the machine to configure the appearance (magic words, a custom image, etc.) to make it harder for an attacker to counterfeit the thing.

It is probably not even possible to think about starting this project until DRM/GEM/KMS stuff is more fully deployed, as the entire scheme presupposes being able to switch between X servers without the X servers' help.

Posted Wednesday afternoon, April 2nd, 2014 Tags:

Support for "real" desktops on NetBSD is somewhat limited and also scattershot. Part of the reason for this is that most of the desktops are not especially well engineered, and they contain poorly designed infrastructure components that interact incestuously with system components that are specific to and/or make sense only on Linux. Providing the interfaces these infrastructure components wish to talk to is a large job, and Linux churn being what it is, such interfaces are usually moving targets anyway. Reimplementing them for NetBSD is also a large job and often presupposes things that aren't true or would require that NetBSD adopt system-level design decisions that we don't want.

Rather than chase after Linux compatibility, we are better off designing the system layer and infrastructure components ourselves. It is easier to place poorly conceived interfaces on top of a solid design than to reimplement a poorly conceived design. Furthermore, if we can manage to do a few things right we should be able to get at least some uptake for them.

The purpose of this project, per se, is not to implement any particular piece of infrastructure. It is to identify pieces of infrastructure that the desktop software stacks need from the operating system, or provide themselves but ought to get from the operating system, figure out solid designs that are compatible with Unix and traditional Unix values, and generate project descriptions for them in order to get them written and deployed.

For example, GNOME and KDE rely heavily on dbus for sending notifications around. While dbus itself is a more or less terrible piece of software that completely fails to match the traditional Unix model (rather than doing one thing well, it does several things at once, pretty much all badly) it is used by GNOME and KDE because there are things a desktop needs to do that require sending messages around. We need a transport for such messages; not a reimplementation of dbus, but a solid and well-conceived way of sending notices around among running processes.

Note that one of the other things dbus does is start services up; we already have inetd for that, but it's possible that inetd could use some strengthening in order to be suitable for the purpose. It will probably also need to be taught about whatever message scheme we come up with. And we might want to e.g. set up a method for starting per-user inetd at login time.

(A third thing dbus does is serve as an RPC library. For this we already have XDR, but XDR sucks; it may be that in order to support existing applications we want a new RPC library that's more or less compatible with the RPC library parts of dbus. Or that may not be feasible.)

This is, however, just one of the more obvious items that arises.

Your goal working on this project is to find more. Please edit this page and make a list, and add some discussion/exposition of each issue and possible solutions. Ones that become fleshed out enough can be moved to their own project pages.

Posted Wednesday afternoon, April 2nd, 2014 Tags:

Port or implement NFSv4.

As of this writing the FreeBSD NFS code, including NFSv4, has been imported into the tree but no work has yet been done on it. The intended plan is to port it and eventually replace the existing NFS code, which nobody is particularly attached to.

It is unlikely that writing new NFSv4 code is a good idea, but circumstances might change.

Posted terribly early Thursday morning, February 27th, 2014 Tags:
  • Contact: tech-kern
  • Duration estimate: 3-6 months

Port Dragonfly's Hammer file system to NetBSD.

Note that because Dragonfly has made substantial changes (different from NetBSD's substantial changes) to the 4.4BSD vfs layer this port may be more difficult than it looks up front.

Posted terribly early Thursday morning, February 27th, 2014 Tags:

An important part of a binary-only environment is to support something similar to the existing options framework, but produce different combinations of pkgs.

There is an undefined amount of work required to accomplish this, and also a set of standards and naming conventions to be applied for the output of these pkgs.

This project should also save on having to rebuild "base" components of software to support different options: see amanda-base for an example.

OpenBSD supports this now and should be leveraged heavily for borrowing.

Preliminary work were done during Google Summer of Code 2016 as part of Split debug symbols for pkgsrc builds project. However, the support is far from being complete. In particular to complete the project the following tasks need to be accomplished:

  • Complete the SUBPACKAGES support similar to what was done during the GSoC (with SUBPACKAGES and !SUBPACKAGES case code duplication)
  • When the SUBPACKAGES support is complete we can switch to an implicit (and hidden) subpackage as suggested by ?joerg in order to get rid of code duplication and having a single control flow relative to SUBPACKAGES. In other words: every package will always have at least one subpackage.
  • Adapt mk/bsd.debugdata.mk to SUBPACKAGES and other possible other candidates like tex-*-doc, etc. After doing that look at less trivial possible SUBPACKAGES candidate like databases/postegresql*-*.
Posted Thursday evening, October 3rd, 2013 Tags:
  • Contact: tech-kern
  • Duration estimate: 3 months

Lua is obviously a great choice for an embedded scripting language in both kernel space and user space.

However, there exists a issue where Lua needs bindings to interface with NetBSD for it to be useful.

Someone needs to take on the grunt-work task of simply creating Lua interfaces into every possible NetBSD component.

That same someone needs to produce meaningful documentation.

As this enhances Lua's usefulness, more lua bindings will then be created, etc.

This project's goal is to set that inertia by attempting to define and create a solid baseline of Lua bindings.

Here is a list of things to possibly target

  • filesystem hooks
  • sysctl hooks
  • proplib
  • envsys
  • MI system bus code

WIP

Posted late Sunday afternoon, June 2nd, 2013 Tags:

Right now kqueue, the kernel event mechanism, only attaches to individual files. This works great for sockets and the like but doesn't help for a directory full of files.

The end result should be feature parity with linux's inotify (a single dir worth of notifications, not necessarily sub-dirs) and the design must be good enough to be accepted by other users of kqueue: osx, freebsd, etc

For an overview of inotify start on wikipedia.
Another example API is the windows event api.

I believe most of the work will be around genfs, namei, and vfs.

Posted at lunch time on Monday, April 15th, 2013 Tags:

pkgin is aimed at being an apt-/yum-like tool for managing pkgsrc binary packages. It relies on pkg_summary(5) for installation, removal and upgrade of packages and associated dependencies, using a remote repository.

While pkgin is now widely used and seems to handle packages installation, upgrade and removal correctly, there's room for improvement. In particular:

Main quest

  • Support for multi-repository
  • Speed-up installed packages database calculation
  • Speed-up local vs remote packages matching
  • Implement an automated-test system
  • Handle conflicting situations (MySQL conflicting with MySQL...)
  • Better logging for installed / removed / warnings / errors

To be confirmed / discussed:

  • Make pkgin independent from pkg_install binaries, use pkg_install libraries or abstract them

Bonus I

In order to ease pkgin's integration with third party tools, it would be useful to split it into a library (libpkgin) providing all methods needed to manipulate packages, i.e., all pkgin's runtime capabilities and flags.

Bonus II (operation "burn the troll")

It would be a very nice addition to abstract SQLite backend so pkgin could be plugged to any database system. A proof of concept using bdb or cdb would fulfill the task.

Useful steps:

  • Understand pkgsrc
  • Understand pkg_summary(5)'s logic
  • Understand SQLite integration
  • Understand pkgin's `impact.c' logic
Posted early Tuesday morning, April 9th, 2013 Tags:
  • Contact: tech-pkg
  • Duration estimate: 1 month

Mancoosi is a project to measure distribution quality by the number of conflicts in a software distribution environment.

The main work of this project would be to analyse possible incompatibilities of pkgsrc and mancoosi (i.e., incompatibilities between Debian and pkgsrc), to write a converter from pkg_summary(5) to CUDF (the format used by Mancoosi). You will need OCaml for doing so, as the libraries used by Mancoosi are written in OCaml.

When you are a French student in the third year of your Bachelor, or first year of your Master, you could also do this project as an internship for your studies.

Posted at lunch time on Wednesday, March 27th, 2013 Tags:

pygrub, which is part of the xentools package, allows booting a Linux domU without having to keep a copy of the kernel and ramdisk in the dom0 file system.

What it does is extract the grub.conf from the domU disk, processes it, and presents a grub like menu. It then extracts the appropriate kernel and ramdisk image from the domU disk, and stores temporary copies of them. Finally, it creates a file containing:

  • kernel=\
  • ramdisk=\
  • extras=\

This file gets processed by xend as if it were part of the domU config.

Most of the code required to do the NetBSD equivalent should be available as part of /boot or the libraries it depends on. The path to the source code for /boot is: sys/arch/i386/stand/boot.

Posted at midnight, December 27th, 2012 Tags:

recent rpmbuild for redhat creates a second rpm for later install containing debug symbols.

This is a very convenient system for having symbols available for debugging but not clogging up the system until crashes happen and debugging is required.

The end result of this project should be a mk.conf flag or build target (debug-build) which would result in a set of symbol files gdb could use, and which could be distributed with the pkg.

Most parts of this project were done during Google Summer of Code 2016 as part of Split debug symbols for pkgsrc builds project. However, in order to be properly integrated in pkgsrc it will need to support multipkg, some little adjustements and more testing in the wild.

Posted at lunch time on Sunday, December 2nd, 2012 Tags:
  • Contact: tech-pkg
  • Mentors: unknown
  • Duration estimate: unknown

Currently, bulk build results are sent to the pkgsrc-bulk mailing list. To figure out if a package is or has been building successfully, or when it broke, one must wade through the list archives and search the report e-mails by hand. Furthermore, to figure out what commits if any were correlated with a breakage, one must wade through the pkgsrc-changes archive and cross-reference manually.

The project is to produce a web/database application that can be run from the pkgsrc releng website on NetBSD.org that tracks bulk build successes and failures and provides search and crossreferencing facilities.

The application should subscribe to the pkgsrc-bulk and pkgsrc-changes mailing lists and ingest the data it finds into a SQL database. It should track commits to each package (and possibly infrastructure changes that might affect all packages) on both HEAD and the current stable branch, and also all successful and failed build reports on a per-platform (OS and/or machine type) basis.

The web part of the application should be able to retrieve summaries of currently broken packages, in general or for a specific platform and/or specific branch. It should also be able to generate a more detailed report about a single package, containing for example which platforms it has been built on recently and whether it succeeded or not; also, if it is broken, how long it has been broken, and the history of package commits and version bumps vs. build results. There will likely be other searches/reports wanted as well.

The application should also have an interface for people who do partial or individual-package check builds; that is, it should be able to generate a list of packages that have not been built since they were last committed, on a given platform or possibly on a per-user basis, and accept results from attempting to build these or subsets of these packages. It is not entirely clear what this interface should be (and e.g. whether it should be command-line-based, web-based, or what, and whether it should be limited to developers) and it's reasonable to expect that some refinements or rearrangements to it will be needed after the initial deployment.

The application should also be able to record cross-references to the bug database. To begin with at least it's reasonable for this to be handled manually.

This project should be a routine web/database application; there is nothing particularly unusual about it from that standpoint. The part that becomes somewhat less trivial is making all the data flows work: for example, it is probably necessary to coordinate an improvement in the way bulk build results are tagged by platform. It is also necessary to avoid importing the reports that appear occasionally on pkgsrc-bulk from misconfigured pbulk installs.

Note also that "OS" and "machine type" are not the only variables that can affect build outcome. There are also multiple compilers on some platforms, for which the results should be tracked separately, plus other factors such as non-default installation paths. Part of the planning phase for this project should be to identify all the variables of this type that should be tracked.

Also remember that what constitutes a "package" is somewhat slippery as well. The pkgsrc directory for a package is not a unique key; multiversion packages, such as Python and Ruby extensions, generate multiple results from a single package directory. There are also a few packages where for whatever reason the package name does not match the pkgsrc directory. The best key seems to be the pkgsrc directory paired with the package-name-without-version.

Some code already exists for this, written in Python using Postgres. Writing new code (rather than working on this existing code) is probably not a good plan.

Posted late Sunday night, April 9th, 2012 Tags:

Font handling in Unix has long been a disgrace. Every program that needs to deal with fonts has had to do its own thing, and there has never been any system-level support, not even to the point of having a standard place to install fonts in the system. (While there was/is a place to put X11 fonts, X is only one of the many programs doing their own thing.)

Font management should be a system service. There cannot be one type of font file -- it is far too late for that -- but there should be one standardized place for fonts, one unified naming scheme for fonts and styles, and one standard way to look up and open/retrieve/use fonts. (Note: "one standardized place" means relative to an installation prefix, e.g. share/fonts.)

Nowadays fontconfig is capable of providing much of this.

The project is:

  1. Figure out for certain if fontconfig is the right solution. Note that even if the code isn't, the model may be. If fontconfig is not the right solution, figure out what the right solution is. Also ascertain whether existing applications that use the fontconfig API can be supported easily/directly or if some kind of possibly messy wrapper layer is needed and doing things right requires changing applications. Convince the developer community that your conclusions are correct so that you can go on to step 2.

1a. Part of this is identifying exactly what is involved in "managing" and "handling" fonts and what applications require.

  1. Implement the infrastructure. If fontconfig is the right solution, this entails moving it from the X sources to the base sources. Also, some of the functionality/configurability of fontconfig is probably not needed in a canonicalized environment. All of this should be disabled if possible. If fontconfig is not the right solution, implement something else in base.

  2. Integrate the new solution in base. Things in base that use fonts should use fontconfig or the replacement for fontconfig. This includes console font handling, groff, mandoc, and X11. Also, the existing fonts that are currently available only to subsets of things in base should be made available to all software through fontconfig or its replacement.

3a. It would also be useful to kill off the old xfontsel and replace it with something that interoperates fully with the new system and also has a halfway decent user interface.

  1. Deploy support in pkgsrc. If the new system does not itself provide the fontconfig API, provide support for that via a wrapper package. Teach applications in pkgsrc that use fontconfig to recognize and use the new system. (This should be fairly straightforward and not require patching anything.) Make pkgsrc font installation interoperate with the new system. (This should be fairly straightforward too.) Take an inventory of applications that use fonts but don't use fontconfig. Patch one or two of these to use the new system to show that it can be done easily. If the new system is not fontconfig and has its own better API, take an inventory of applications that would benefit from being patched to use the new API. Patch one or two of these to demonstrate that it can be done easily. (If the answers from step 1 are correct, it should be fairly easy for most ordinary applications.)

  2. Persuade the rest of the world that we've done this right and try to get them to adopt the solution. This is particularly important if the solution is not fontconfig. Also, if the solution is not fontconfig, provide a (possibly limited) version of the implementation as a pkgsrc package that can be used on other platforms by packages and applications that support it.

Note that step 1 is the hard part. This project requires a good bit of experience and familiarity with Unix and operating system design to allow coming up with a good plan. If you think it's obvious that fontconfig is the right answer and/or you can't see why there might be any reason to change anything about fontconfig, then this may not be the right project for you.

Because of this, this project is really not suitable for GSoC.

Posted at midnight, March 26th, 2012 Tags:

Dependency handling in pkgsrc is rather complex task. There exist some cases (TeX packages, Perl packages) where it is hard to find build dependencies precisely and the whole thing is handled conservatively. E.g. the whole TeXLive meta-package is declared a build dependency even when rather small fraction of it is used actually. Another case is stale heavy dependency which is no longer required but still listed as prerequisite.

It would be nice to have a tool (or a set of them, if necessary) to detect which installed packages, libraries or tools were actually used to build new package. Ideally, the tool should report files used during configure, build, and test stages, and packages these files are provided by.

Milestones: * find or develop a good dependency graph algorithm * implement and demonstrate your new system in pkgsrc by adding a make target * expose this algorithm for use by websites such as pkgsrc.se

Posted late Saturday evening, March 17th, 2012 Tags:

There exist two ways of launching processes: one is forking them with fork or vfork and replacing the clone with exec-family function, another is spawning process directly with procedures like posix_spawn. Not all platforms implement fork model, and spawn model has its own merits.

pkgsrc relies heavily on launching subprocesses when building software. NetBSD posix_spawn support was implemented in GSoC 2011, it is included in NetBSD 6.0. Now that NetBSD supports both ways, it would be nice to compare the efficiency of both ways of launching subprocesses and measure its effect when using pkgsrc (both in user and developer mode). In order to accomplish that, the following tools should support posix_spawn:

  • devel/bmake
  • shells/pdksh
  • NetBSD base make
  • NetBSD sh
  • NetBSD base ksh
  • potentially some other tools (e.g. lang/nawk, shells/bash, lang/perl5)

Optionally, MinGW spawn support can be added as well.

Milestones:

  • support starting processes and subprocesses by posix_spawn in devel/bmake
  • support starting processes and subprocesses by posix_spawn in shells/pdksh,
  • measure its efficiency and compare it to traditional fork+exec.
Posted late Saturday evening, March 17th, 2012 Tags:

This project proposal is a subtask of smp networking.

The goal of this project is to implement lockless and atomic FIFO/LIFO queues in the kernel. The routines to be implemented allow for commonly typed items to be locklessly inserted at either the head or tail of a queue for either last-in, first-out (LIFO) or first-in, first-out (FIFO) behavior, respectively. However, a queue is not instrinsicly LIFO or FIFO. Its behavior is determined solely by which method each item was pushed onto the queue.

It is only possible for an item to removed from the head of queue. This removal is also performed in a lockless manner.

All items in the queue must share a atomic_queue_link_t member at the same offset from the beginning of item. This offset is passed to atomic_qinit.

The proposed interface looks like this:

  • void atomic_qinit(atomic_queue_t *q, size_t offset);

    Initializes the atomic_queue_t queue at q. offset is the offset to the atomic_queue_link_t inside the data structure where the pointer to the next item in this queue will be placed. It should be obtained using offsetof.

  • void *atomic_qpeek(atomic_queue_t *q);

    Returns a pointer to the item at the head of the supplied queue q. If there was no item because the queue was empty, NULL is returned. No item is removed from the queue. Given this is an unlocked operation, it should only be used as a hint as whether the queue is empty or not.

  • void *atomic_qpop(atomic_queue_t *q);

    Removes the item (if present) at the head of the supplied queue q and returns a pointer to it. If there was no item to remove because the queue was empty, NULL is returned. Because this routine uses atomic Compare-And-Store operations, the returned item should stay accessible for some indeterminate time so that other interrupted or concurrent callers to this function with this q can continue to deference it without trapping.

  • void atomic_qpush_fifo(atomic_queue_t *q, void *item);

    Places item at the tail of the atomic_queue_t queue at q.

  • void atomic_qpush_lifo(atomic_queue_t *q, void *item);

    Places item at the head of the atomic_queue_t queue at q.

Posted in the wee hours of Wednesday night, November 10th, 2011 Tags:

This project proposal is a subtask of smp networking.

The goal of this project is to implement lockless, atomic and generic Radix and Patricia trees. BSD systems have always used a radix tree for their routing tables. However, the radix tree implementation is showing its age. Its lack of flexibility (it is only suitable for use in a routing table) and overhead of use (requires memory allocation/deallocation for insertions and removals) make replacing it with something better tuned to today's processors a necessity.

Since a radix tree branches on bit differences, finding these bit differences efficiently is crucial to the speed of tree operations. This is most quickly done by XORing the key and the tree node's value together and then counting the number of leading zeroes in the result of the XOR. Many processors today (ARM, PowerPC) have instructions that can count the number of leading zeroes in a 32 bit word (and even a 64 bit word). Even those that do not can use a simple constant time routine to count them:

int
clz(unsigned int bits)
{
    int zeroes = 0;
    if (bits == 0)
        return 32;
    if (bits & 0xffff0000) bits &= 0xffff0000; else zeroes += 16;
    if (bits & 0xff00ff00) bits &= 0xff00ff00; else zeroes += 8;
    if (bits & 0xf0f0f0f0) bits &= 0xf0f0f0f0; else zeroes += 4;
    if (bits & 0xcccccccc) bits &= 0xcccccccc; else zeroes += 2;
    if (bits & 0xaaaaaaaa) bits &= 0xaaaaaaaa; else zeroes += 1;
    return zeroes;
}

The existing BSD radix tree implementation does not use this method but instead uses a far more expensive method of comparision. Adapting the existing implementation to do the above is actually more expensive than writing a new implementation.

The primary requirements for the new radix tree are:

  • Be self-contained. It cannot require additional memory other than what is used in its data structures.

  • Be generic. A radix tree has uses outside networking.

To make the radix tree flexible, all knowledge of how keys are represented has to be encapsulated into a pt_tree_ops_t structure with these functions:

  • bool ptto_matchnode(const void *foo, const void *bar, pt_bitoff_t max_bitoff, pt_bitoff_t *bitoffp, pt_slot_t *slotp);

    Returns true if both foo and bar objects have the identical string of bits starting at *bitoffp and ending before max_bitoff. In addition to returning true, *bitoffp should be set to the smaller of max_bitoff or the length, in bits, of the compared bit strings. Any bits before *bitoffp are to be ignored. If the string of bits are not identical, *bitoffp is set to the where the bit difference occured, *slotp is the value of that bit in foo, and false is returned. The foo and bar (if not NULL) arguments are pointers to a key member inside a tree object. If bar is NULL, then assume it points to a key consisting of entirely of zero bits.

  • bool ptto_matchkey(const void *key, const void *node_key, pt_bitoff_t bitoff, pt_bitlen_t bitlen);

    Returns true if both key and node_key objects have identical strings of bitlen bits starting at bitoff. The key argument is the same key argument supplied to ptree_find_filtered_node.

  • pt_slot_t ptto_testnode(const void *node_key, pt_bitoff_t bitoff, pt_bitlen_t bitlen);

    Returns bitlen bits starting at bitoff from node_key. The node_key argument is a pointer to the key members inside a tree object.

  • pt_slot_t ptto_testkey(const void *key, pt_bitoff_t bitoff, pt_bitlen_t bitlen);

    Returns bitlen bits starting at bitoff from key. The key argument is the same key argument supplied to ptree_find_filtered_node.

All bit offsets are relative to the most significant bit of the key,

The ptree programming interface should contains these routines:

  • void ptree_init(pt_tree_t *pt, const pt_tree_ops_t *ops, size_t ptnode_offset, size_t key_offset);

    Initializes a ptree. If pt points at an existing ptree, all knowledge of that ptree is lost. The pt argument is a pointer to the pt_tree_t to be initialized. The ops argument is a pointer to the pt_tree_ops_t used by the ptree. This has four members: The ptnode_offset argument contains the offset from the beginning of an item to its pt_node_t member. The key_offset argument contains the offset from the beginning of an item to its key data. This is used if 0 is used, a pointer to the beginning of the item will be generated.

  • void *ptree_find_filtered_node(pt_tree_t *pt, const void *key, pt_filter_t filter, void *filter_ctx);

    The filter argument is either NULL or a function bool (*)(const void *, void *, int);

  • bool ptree_insert_mask_node(pt_tree_t *pt, void *item, pt_bitlen_t masklen);

  • bool ptree_insert_node(pt_tree_t *pt, void *item);

  • void *ptree_iterate(pt_tree_t *pt, const void *node, pt_direction_t direction);

  • void ptree_remove_node(pt_tree_t *pt, const pt_tree_ops_t *ops, void *item);

Posted in the wee hours of Wednesday night, November 10th, 2011 Tags:

This project proposal is a subtask of smp networking.

The goal of this project is to enhance the networking protocols to process incoming packets more efficiently. The basic idea is the following: when a packet is received and it is destined for a socket, simply place the packet in the socket's receive PCQ (see atomic pcq) and wake the blocking socket. Then, the protocol is able to process the next packet.

The typical packet flow from ip_input is to {rip,tcp,udp}_input which:

  • Does the lookup to locate the socket which takes a reader lock on the appropriate pcbtable's hash bucket.
  • If found and in the proper state:
    • Do not lock the socket since that would might block and therefore stop packet demultiplexing.
    • pcq_put the packet to the pcb's pcq.
    • kcont_schedule the worker continuation with small delay (~100ms). See kernel continuations.
    • Lock the socket's cvmutex.
    • Release the pcbtable lock.
    • If TCP and in sequence, then if we need to send an immediate ACK:
      • Try to lock the socket.
      • If successful, send an ACK.
    • Set a flag to process the PCQ.
    • cv_signal the socket's cv.
    • Release the cvmutex.
  • If not found or not in the proper state:
    • Release the pcb hash table lock.
Posted in the wee hours of Wednesday night, November 10th, 2011 Tags:

This project proposal is a subtask of smp networking.

The goal of this project is to implement interrupt handling at the granularity of a networking interface. When a network device gets an interrupt, it could call <iftype>_defer(ifp) to schedule a kernel continuation (see kernel continuations) for that interface which could then invoke <iftype>_poll. Whether the interrupted source should be masked depends on if the device is a DMA device or a PIO device. This routine should then call (*ifp->if_poll)(ifp) to deal with the interrupt's servicing.

During servicing, any received packets should be passed up via (*ifp->if_input)(ifp, m) which would be responsible for ALTQ or any other optional processing as well as protocol dispatch. Protocol dispatch in <iftype>_input decodes the datalink headers, if needed, via a table lookup and call the matching protocol's pr_input to process the packet. As such, interrupt queues (e.g. ipintrq) would no longer be needed. Any transmitted packets can be processed as can MII events. Either true or false should be returned by if_poll depending on whether another invokation of <iftype>_poll for this interface should be immediately scheduled or not, respectively.

Memory allocation has to be prohibited in the interrupt routines. The device's if_poll routine should pre-allocate enough mbufs to do any required buffering. For devices doing DMA, the buffers are placed into receive descripors to be filled via DMA.

For devices doing PIO, pre-allocated mbufs are enqueued onto the softc of the device so when the interrupt routine needs one it simply dequeues one, fills in it in, and then enqueues it onto a completed queue, finally calls <iftype>_defer. If the number of pre-allocated mbufs drops below a threshold, the driver may decide to increase the number of mbufs that if_poll pre-allocates. If there are no mbufs left to receive the packet, the packets is dropped and the number of mbufs for if_poll to pre-allocate should be increased.

When interrupts are unmasked depends on a few things. If the device is interrupting "too often", it might make sense for the device's interrupts to remain masked and just schedule the device's continuation for the next clock tick. This assumes the system has a high enough value set for HZ.

Posted in the wee hours of Wednesday night, November 10th, 2011 Tags:

This project proposal is a subtask of smp networking.

The goal of this project is to implement continuations at the kernel level. Most of the pieces are already available in the kernel, so this can be reworded as: combine callouts, softints, and workqueues into a single framework. Continuations are meant to be cheap; very cheap.

These continuations are a dispatching system for making callbacks at scheduled times or in different thread/interrupt contexts. They aren't "continuations" in the usual sense such as you might find in Scheme code.

Please note that the main goal of this project is to simplify the implementation of SMP networking, so care must be taken in the design of the interface to support all the features required for this other project.

The proposed interface looks like the following. This interface is mostly derived from the callout(9) API and is a superset of the softint(9) API. The most significant change is that workqueue items are not tied to a specific kernel thread.

  • kcont_t *kcont_create(kcont_wq_t *wq, kmutex_t *lock, void (*func)(void *, kcont_t *), void *arg, int flags);

    A wq must be supplied. It may be one returned by kcont_workqueue_acquire or a predefined workqueue such as (sorted from highest priority to lowest):

    • wq_softserial, wq_softnet, wq_softbio, wq_softclock
    • wq_prihigh, wq_primedhigh, wq_primedlow, wq_prilow

    lock, if non-NULL, should be locked before calling func(arg) and released afterwards. However, if the lock is released and/or destroyed before the called function returns, then, before returning, kcont_set_mutex must be called with either a new mutex to be released or NULL. If acquiring lock would block, other pending kernel continuations which depend on other locks may be dispatched in the meantime. However, all continuations sharing the same set of { wq, lock, [ci] } need to be processed in the order they were scheduled.

    flags must be 0. This field is just provided for extensibility.

  • int kcont_schedule(kcont_t *kc, struct cpu_info *ci, int nticks);

    If the continuation is marked as INVOKING, an error of EBUSY should be returned. If nticks is 0, the continuation is marked as INVOKING while EXPIRED and PENDING are cleared, and the continuation is scheduled to be invoked without delay. Otherwise, the continuation is marked as PENDING while EXPIRED status is cleared, and the timer reset to nticks. Once the timer expires, the continuation is marked as EXPIRED and INVOKING, and the PENDING status is cleared. If ci is non-NULL, the continuation is invoked on the specified CPU if the continuations's workqueue has per-cpu queues. If that workqueue does not provide per-cpu queues, an error of ENOENT is returned. Otherwise when ci is NULL, the continuation is invoked on either the current CPU or the next available CPU depending on whether the continuation's workqueue has per-cpu queues or not, respectively.

  • void kcont_destroy(kcont_t *kc);

  • kmutex_t *kcont_getmutex(kcont_t *kc);

    Returns the lock currently associated with the continuation kc.

  • void kcont_setarg(kcont_t *kc, void *arg);

    Updates arg in the continuation kc. If no lock is associated with the continuation, then arg may be changed at any time; however, if the continuation is being invoked, it may not pick up the change. Otherwise, kcont_setarg must only be called when the associated lock is locked.

  • kmutex_t *kcont_setmutex(kcont_t *kc, kmutex_t *lock);

    Updates the lock associated with the continuation kc and returns the previous lock. If no lock is currently associated with the continuation, then calling this function with a lock other than NULL will trigger an assertion failure. Otherwise, kcont_setmutex must be called only when the existing lock (which will be replaced) is locked. If kcont_setmutex is called as a result of the invokation of func, then after kcont_setmutex has been called but before func returns, the replaced lock must have been released, and the replacement lock, if non-NULL, must be locked upon return.

  • void kcont_setfunc(kcont_t *kc, void (*func)(void *), void *arg);

    Updates func and arg in the continuation kc. If no lock is associated with the continuation, then only arg may be changed. Otherwise, kcont_setfunc must be called only when the associated lock is locked.

  • bool kcont_stop(kcont_t *kc);

    The kcont_stop function stops the timer associated the continuation handle kc. The PENDING and EXPIRED status for the continuation handle is cleared. It is safe to call kcont_stop on a continuation handle that is not pending, so long as it is initialized. kcont_stop will return a non-zero value if the continuation was EXPIRED.

  • bool kcont_pending(kcont_t *kc);

    The kcont_pending function tests the PENDING status of the continuation handle kc. A PENDING continuation is one who's timer has been started and has not expired. Note that it is possible for a continuation's timer to have expired without being invoked if the continuation's lock could not be acquired or there are higher priority threads preventing its invokation. Note that it is only safe to test PENDING status when holding the continuation's lock.

  • bool kcont_expired(kcont_t *kc);

    Tests to see if the continuation's function has been invoked since the last kcont_schedule.

  • bool kcont_active(kcont_t *kc);

  • bool kcont_invoking(kcont_t *kc);

    Tests the INVOKING status of the handle kc. This flag is set just before a continuation's function is being called. Since the scheduling of the worker threads may induce delays, other pending higher-priority code may run before the continuation function is allowed to run. This may create a race condition if this higher-priority code deallocates storage containing one or more continuation structures whose continuation functions are about to be run. In such cases, one technique to prevent references to deallocated storage would be to test whether any continuation functions are in the INVOKING state using kcont_invoking, and if so, to mark the data structure and defer storage deallocation until the continuation function is allowed to run. For this handshake protocol to work, the continuation function will have to use the kcont_ack function to clear this flag.

  • bool kcont_ack(kcont_t *kc);

    Clears the INVOKING state in the continuation handle kc. This is used in situations where it is necessary to protect against the race condition described under kcont_invoking.

  • kcont_wq_t *kcont_workqueue_acquire(pri_t pri, int flags);

    Returns a workqueue that matches the specified criteria. Thus if multiple requesters ask for the same criteria, they are all returned the same workqueue. pri specifies the priority at which the kernel thread which empties the workqueue should run.

    If flags is 0 then the standard operation is required. However, the following flag(s) may be bitwise ORed together:

    • WQ_PERCPU specifies that the workqueue should have a separate queue for each CPU, thus allowing continuations to be invoked on specific CPUs.
  • int kcont_workqueue_release(kcont_wq_t *wq);

    Releases an acquired workqueue. On the last release, the workqueue's resources are freed and the workqueue is destroyed.

Posted in the wee hours of Wednesday night, November 10th, 2011 Tags:

This project proposal is a subtask of smp networking.

The goal of this project is to improve the way the processing of incoming packets is handled.

Instead of having a set of active workqueue lwps waiting to service sockets, the kernel should use the lwp that is blocked on the socket to service the workitem. It is not productive being blocked and it has an interest in getting that workitem done, and maybe we can directly copy that data to user's address and avoid queuing in the socket at all.

Posted in the wee hours of Wednesday night, November 10th, 2011 Tags:

This project proposal is a subtask of smp networking.

The goal of this project is to remove the ARP, AARP, ISO SNPA, and IPv6 Neighbors from the routing table. Instead, the ifnet structure should have a set of nexthop caches (usually implemented using patricia trees), one per address family. Each nexthop entry should contain the datalink header needed to reach the neighbor.

This will remove cloneable routes from the routing table and remove the need to maintain protocol-specific code in the common Ethernet, FDDI, PPP, etc. code and put it back where it belongs, in the protocol itself.

Posted in the wee hours of Wednesday night, November 10th, 2011 Tags:

This project proposal is a subtask of smp networking.

The goal of this project is to make the SYN cache optional. For small systems, this is complete overkill and should be made optional.

Posted in the wee hours of Wednesday night, November 10th, 2011 Tags:

This project proposal is a subtask of smp networking and is elegible for funding independently.

The goal of this project is to implement full virtual network stacks. A virtual network stack collects all the global data for an instance of a network stack (excluding AF_LOCAL). This includes routing table, data for multiple domains and their protocols, and the mutexes needed for regulating access to it all. Instead, a brane is an instance of a networking stack.

An interface belongs to a brane, as do processes. This can be considered a chroot(2) for networking, e.g. chbrane(2).

Posted in the wee hours of Wednesday night, November 10th, 2011 Tags:

Design and program a scheme for extending the operating range of 802.11 networks by using techniques like frame combining and error-correcting codes to cope with low S/(N+I) ratio. Implement your scheme in one or two WLAN device drivers -- Atheros & Realtek, say.

This project is on hold due to the conversion project needing to be completed first.

Posted Sunday evening, November 6th, 2011 Tags:

Modern 802.11 NICs provide two or more transmit descriptor rings, one for each priority level or 802.11e access category. Add to NetBSD a generic facility for placing a packet onto a different hardware transmit queue according to its classification by pf or IP Filter. Demonstrate this facility on more than one 802.11 chipset.

This project is on hold due to the conversion project needing to be completed first.

Posted Sunday evening, November 6th, 2011 Tags:

BSD make (aka bmake) uses traditional suffix rules (.c.o: ...) instead of pattern rules like gmake's (%.c:%.o: ...) which are more general and flexible.

The suffix module should be re-written to work from a general match-and-transform function on targets, which is sufficient to implement not only traditional suffix rules and gmake pattern rules, but also whatever other more general transformation rule syntax comes down the pike next. Then the suffix rule syntax and pattern rule syntax can both be fed into this code.

Note that it is already possible to write rules where the source is computed explicitly based on the target, simply by using $(.TARGET) in the right hand side of the rule. Investigate whether this logic should be rewritten using the match-and-transform code, or if the match-and-transform code should use the logic that makes these rules possible instead.

Implementing pattern rules is widely desired in order to be able to read more makefiles written for gmake, even though gmake's pattern rules aren't well designed or particularly principled.

Posted Sunday evening, November 6th, 2011 Tags:
  • Contact: tech-kern
  • Duration estimate: 2-3 months

Currently the buffer handling logic only sorts the buffer queue (aka disksort). In an ideal world it would be able to coalesce adjacent small requests, as this can produce considerable speedups. It might also be worthwhile to split large requests into smaller chunks on the fly as needed by hardware or lower-level software.

Note that the latter interacts nontrivially with the ongoing dynamic MAXPHYS project and might not be worthwhile. Coalescing adjacent small requests (up to some potentially arbitrary MAXPHYS limit) is worthwhile regardless, though.

Posted Sunday evening, November 6th, 2011 Tags:

NetBSD version of compressed cache system (for low-memory devices): http://linuxcompressed.sourceforge.net/.

Posted Sunday evening, November 6th, 2011 Tags:

In a file system with symlinks, the file system can be seen as a graph rather than a tree. The meaning of .. potentially becomes complicated in this environment.

There is a fairly substantial group of people, some of them big famous names, who think that the usual behavior (where crossing a symlink is different from entering a subdirectory) is a bug, and have made various efforts from time to time to "fix" it. One such fix can be seen in the -L and -P options to ksh's pwd.

Rob Pike implemented a neat hack for this in Plan 9. It is described in http://cm.bell-labs.com/sys/doc/lexnames.html. This project is to implement that logic for NetBSD.

Note however that there's another fairly substantial group of people, some of them also big famous names, who think that all of this is a load of dingo's kidneys, the existing behavior is correct, and changing it would be a bug. So it needs to be possible to switch the implementation on and off as per-process state.

Posted Sunday evening, November 6th, 2011 Tags:
  • Contact: tech-kern
  • Duration estimate: 2-3 months

The ext2 file system is the lowest common denominator Unix-like file system in the Linux world, as ffs is in the BSD world. NetBSD has had kernel support for ext2 for quite some time.

However, the Linux world has moved on, with ext3 and now to some extent also ext4 superseding ext2 as the baseline. NetBSD has no support for ext3; the goal of this project is to implement that support.

Since ext3 is a backward-compatible extension that adds journaling to ext2, NetBSD can mount clean ext3 volumes as ext2 volumes. However, NetBSD cannot mount ext3 volumes with journaling and it cannot handle recovery for crashed volumes. As ext2 by itself provides no crash recovery guarantees whatsoever, this journaling support is highly desirable.

The ext3 support should be implemented by extending the existing ext2 support (which is in src/sys/ufs/ext2fs), not by rewriting the whole thing over from scratch. It is possible that some of the ostensibly filesystem-independent code that was added along with the ffs WAPBL journaling extensions might be also useable as part of an ext3 implementation; but it also might not be.

The full requirements for this project include complete support for ext3 in both the kernel and the userland tools. It is possible that a reduced version of this project with a clearly defined subset of these requirements could still be a viable GSOC project; if this appeals to you please coordinate with a prospective mentor. Be advised, however, that several past ext3-related GSOC projects have failed; it is a harder undertaking than you might think.

An additional useful add-on goal would be to audit the locking in the existing ext2 code; the ext2 code is not tagged MPSAFE, meaning it uses a biglock on multiprocessor machines, but it is likely that either it is in fact already safe and just needs to be tagged, or can be easily fixed. (Note that while this is not itself directly related to implementing ext3, auditing the existing ext2 code is a good way to become familiar with it.)

Posted Sunday evening, November 6th, 2011 Tags:

Implement a flash translation layer.

A flash translation layer does block remapping, translating from visible block addresses used by a file system to physical cells on one or more flash chips. This provides wear leveling, which is essential for effective use of flash, and also typically some amount of read caching and write buffering. (And it takes care of excluding cells that have gone bad.)

This allows FFS, LFS, msdosfs, or whatever other conventional file system to be used on raw flash chips. (Note that SSDs and USB flash drives and so forth contain their own FTLs.)

FTLs involve quite a bit of voodoo and there is a lot of prior art and research; do not just sit down and start coding.

There are also some research FTLs that we might be able to get the code for; it is probably worth looking into this.

Note that NAND flash and NOR flash are different and need different handling, and the various cell types and other variations also warrant different policy choices.

The degree of overprovisioning (that is, the ratio of the raw capacity of the flash chips to the advertised size of the resulting device) should be configurable as this is a critical factor for performance.

Making the device recoverable rather than munching itself in system crashes or power failures is a nice extra, although apparently the market considers this an optional feature for consumer devices.

The flash translation layer should probably be packaged a driver that attaches to one or more flash chips and provides a disk-type block/character device pair.

Posted Sunday evening, November 6th, 2011 Tags:

Use puffs or refuse to write an imapfs that you can mount on /var/mail, either by writing a new one or porting the old existing Plan 9 code that does this.

Note: there might be existing solutions, please check upfront and let us know.

Posted Sunday evening, November 6th, 2011 Tags:
  • Contact: tech-kern
  • Duration estimate: 4 months and up

There are many caches in the kernel. Most of these have knobs and adjustments, some exposed and some not, for sizing and writeback rate and flush behavior and assorted other voodoo, and most of the ones that aren't adjustable probably should be.

Currently all or nearly all of these caches operate on autopilot independent of the others, which does not necessarily produce good results, especially if the system is operating in a performance regime different from when the behavior was tuned by the implementors.

It would be nice if all these caches were instead coordinated, so that they don't end up fighting with one another. Integrated control of sizing, for example, would allow explicitly maintaining a sensible balance between different memory uses based on current conditions; right now you might get that, depending on whether the available voodoo happens to work adequately under the workload you have, or you might not. Also, it is probably possible to define some simple rules about eviction, like not evicting vnodes that have UVM pages still to be written out, that can help avoid unnecessary thrashing and other adverse dynamic behavior. And similarly, it is probably possible to prefetch some caches based on activity in others. It might even be possible to come up with one glorious unified cache management algorithm.

Also note that cache eviction and prefetching is fundamentally a form of scheduling, so all of this material should also be integrated with the process scheduler to allow it to make more informed decisions.

This is a nontrivial undertaking.

Step 1 is to just find all the things in the kernel that ought to participate in a coordinated caching and scheduling scheme. This should not take all that long. Some examples include:

  • UVM pages
  • file system metadata buffers
  • VFS name cache
  • vnode cache
  • size of the mbuf pool

Step 2 is to restructure and connect things up so that it is readily possible to get the necessary information from all the random places in the kernel that these things occupy, without making a horrible mess and without trashing system performance in the process or deadlocking out the wazoo. This is not going to be particularly easy or fast.

Step 3 is to take some simple steps, like suggested above, to do something useful with the coordinated information, and hopefully to show via benchmarks that it has some benefit.

Step 4 is to look into more elaborate algorithms for unified control of everything. The previous version of this project cited IBM's ARC ("Adaptive Replacement Cache") as one thing to look at. (But note that ARC may be encumbered -- someone please check on that and update this page.) Another possibility is to deploy machine learning algorithms to look for and exploit patterns.

Note: this is a serious research project. Step 3 will yield a publishable minor paper; step 4 will yield a publishable major paper if you manage to come up with something that works, and it quite possibly contains enough material for a PhD thesis.

Posted Sunday evening, November 6th, 2011 Tags:

Add support for Apple's extensions to ISO9660 to makefs, especially the ability to label files with Type & Creator IDs. See http://developer.apple.com/technotes/fl/fl_36.html.

Posted Sunday evening, November 6th, 2011 Tags:
  • Contact: tech-kern
  • Duration estimate: 1 year for port; 3 years for rewrite by one developer

Implement a BSD licensed JFS. A GPL licensed implementation of JFS is available at http://jfs.sourceforge.net/.

Alternatively, or additionally, it might be worthwhile to do a port of the GPL code and allow it to run as a kernel module.

Posted Sunday evening, November 6th, 2011 Tags:

Today a number of OS provide some form of kernel-level virtualization that offer better isolation mechanisms that the traditional (yet more portable) &chroot(2). Currently, NetBSD lacks functionality in this field; there have been multiple attempts (gaols, mult) to implement a jails-like system, but none so far has been integrated in base.

The purpose of this project is to study the various implementations found elsewhere (FreeBSD Jails, Solaris Zones, Linux Containers/VServers, ...), and eventually see their plus/minus points. An additional step would be to see how this can be implemented the various architectural improvements NetBSD gained, especially rump(3) and kauth(9).

Caution: this is a research project.

Posted Sunday evening, November 6th, 2011 Tags:

kernfs is a virtual file system that reports information about the running system, and in some cases allows adjusting this information. procfs is a virtual file system that provides information about currently running processes. Both of these file systems work by exposing virtual files containing textual data.

The current implementations of these file systems are redundant and both are non-extensible. For example, kernfs is a hardcoded table that always exposes the same set of files; there is no way to add or remove entries on the fly, and even adding new static entries is a nuisance. procfs is similarly limited; there is no way to add additional per-process data on the fly. Furthermore, the current code is not modular, not well designed, and has been a source of security bugs in the past.

We would like to have a new implementation for both of these file systems that rectifies these problems and others, as outlined below:

  • kernfs and procfs should share most of their code, and in particular they should share all the code for managing lists of virtual files. They should remain separate entities, however, at least from the user perspective: community consensus is that mixing system and per-process data, as Linux always has, is ugly.

  • It should be possible to add and remove entries on the fly, e.g. as modules are loaded and unloaded.

  • Because userlevel programs can become dependent on the format of the virtual files (Linux has historically had compatibility problems because of this) they should if possible not have complex formats at all, and if they do the format should be clearly specifiable in some way that isn't procedural code. (This makes it easier to reason about, and harder for it to get changed by accident.)

  • There is an additional interface in the kernel for retrieving and adjusting arbitrary kernel information: sysctl. Currently the sysctl code is a third completely separate mechanism, on many points redundant with kernfs and/or procfs. It is somewhat less primitive, but the current implementation is cumbersome and not especially liked. Integrating kernfs and procfs with sysctl (perhaps somewhat like the Linux sysfs) is not automatically the right design choice, but it is likely to be a good idea. At a minimum we would like to be able to have one way to handle reportable/adjustable data within the kernel, so that kernfs, procfs, and/or sysctl can be attached to any particular data element as desired.

  • While most of the implementations of things like procfs and sysctl found in the wild (including the ones we currently have) work by attaching callbacks, and then writing code all over the kernel to implement the callback API, it is possible to design instead to attach data, that is, pointers to variables within the kernel, so that the kernfs/procfs or sysctl code itself takes responsibility for fetching that data. Please consider such a design strongly and pursue it if feasible, as it is much tidier. (Note that attaching data also in general requires specifying a locking model and, for writeable data, possibly a condition variable to signal on when the value changes and/or a mechanism for checking new values for validity.)

It is possible that using tmpfs as a backend for kernfs and procfs, or sharing some code with tmpfs, would simplify the implementation. It also might not. Consider this possibility, and assess the tradeoffs; do not treat it as a requirement.

Alternatively, investigate FreeBSD's pseudofs and see if this could be a useful platform for this project and base for all the file systems mentioned above.

When working on this project, it is very important to write a complete regression test suite for procfs and kernfs beforehand to ensure that the rewrites do not create incompatibilities.

Posted Sunday evening, November 6th, 2011 Tags:

Improve on the Kismet design and implementation in a Kismet replacement for BSD.

Posted Sunday evening, November 6th, 2011 Tags:

Libvirt is a project that aims at bringing yet-another-level of abstraction to the management of different virtualization technologies; it supports a wide range of virtualization technologies, like Xen, VMWare, KVM and containers.

A package for libvirt was added to pkgsrc under sysutils/libvirt, however it requires more testing before all platforms supported by pkgsrc can also seamlessly support libvirt.

The purpose of this project is to investigate what is missing in libvirt (in terms of patches or system integration) so it can work out-of-the-box for platforms that can benefit from it. GNU/Linux, NetBSD, FreeBSD and Solaris are the main targets.

Posted Sunday evening, November 6th, 2011 Tags:

While NetBSD has had LiveCDs for a while, there has not yet been a LiveCD that allows users to install NetBSD after test-driving it. A LiveCD that contains a GUI based installer and reliably detects the platforms features would be very useful.

Posted Sunday evening, November 6th, 2011 Tags:

Apply statistical AI techniques to the problem of monitoring the logs of a busy system. Can one identify events of interest to a sysadmin, or events that merit closer inspection? Failing that, can one at least identify some events as routine and provide a filtered log that excludes them? Also, can one group a collection of related messages together into a single event?

Posted Sunday evening, November 6th, 2011 Tags:

NetBSD currently requires a system with an MMU. This obviously limits the portability. We'd be interested in an implementation/port of NetBSD on/to an MMU-less system.

Posted Sunday evening, November 6th, 2011 Tags:

The policy code in the kernel that controls file caching and readahead behavior is necessarily one-size-fits-all, and the knobs available to applications to tune it, like madvise() and posix_fadvise(), are fairly blunt hammers. Furthermore, it has been shown that the overhead from user<->kernel domain crossings makes syscall-driven fine-grained policy control ineffective. (Though, that was shown in the past when processors were much slower relative to disks and it may not be true any more.)

Is it possible to use BPF, or create a different BPF-like tool (that is, a small code generator with very simple and very clear safety properties) to allow safe in-kernel fine-grained policy control?

Caution: this is a research project.

Posted Sunday evening, November 6th, 2011 Tags:
  • Contact: tech-net
  • Duration estimate: 3 months

Implement the ability to route based on properties like QoS label, source address, etc.

Posted Sunday evening, November 6th, 2011 Tags:
  • Contact: tech-net
  • Duration estimate: 2 months

Write tests for the routing code and re-factor. Use more and better-named variables.

PCBs and other structures are sprinkled with route caches (struct route). Sometimes they do not get updated when they should. It's necessary to modify rtalloc(), at least. Fix that. Note XXX in rtalloc(); this may mark a potential memory leak!

Posted Sunday evening, November 6th, 2011 Tags:

Create/modify a 802.11 link-adaptation module that works at least as well as SampleRate, but is generic enough to be re-used by device drivers for ADMtek, Atheros, Intel, Ralink, and Realtek 802.11 chips. Make the module export a link-quality metric (such as ETT) suitable for use in linkstate routing. The SampleRate module in the Atheros device driver sources may be a suitable starting point.

Posted Sunday evening, November 6th, 2011 Tags:

Currently booting a sgimips machine requires different boot commands depending on the architecture. It is not possible to use the firmware menu to boot from CD.

An improved primary bootstrap should ask the firmware for architecture detail, and automatically boot the correct kernel for the current architecture by default.

A secondary objective of this project would be to rearrange the generation of a bootably CD image so that it could just be loaded from the firmware menu without going through the command monitor.

Posted Sunday evening, November 6th, 2011 Tags:

NetBSD/sgimips currently runs on a number of SGI hardware, but support for IP27 (Origin) and IP30 (Octane) is not yet available.

See also NetBSD/sgimips.

Posted Sunday evening, November 6th, 2011 Tags:

NetBSD/sgimips currently runs on O2s with R10k (or similar) CPUs, but for example speculative loads are not handled correctly. It is unclear if this is pure kernel work or the toolchain needs to be changed too.

Currently softfloat is used, and bugs seem to exist in the hardware float support. Resolving these bugs and switching to hardware float would improve performance.

See also NetBSD/sgimips.

Posted Sunday evening, November 6th, 2011 Tags:

Certain real time chips, and other related power hardware, have a facility within them to allow the kernel to set a specific time and date at which time the machine will power itself on. One such chip is the DS1685 RTC. A kernel API should be developed to allow such devices to have a power-on-time set from userland. Additionally, the API should be made available through a userland program, or added to an existing utility, such as halt(8).

It may also be useful to make this a more generic interface, to allow for configuring other devices, such as Wake-On-Lan ethernet chips, to turn them on/off, set them up, etc.

Posted Sunday evening, November 6th, 2011 Tags:
  • Contact: tech-kern
  • Duration estimate: 8-12 months

Remove the residual geometry code and datastructures from FFS (keep some kind of allocation groups but without most of what cylinder groups now have) and replace blocks and fragments with extents, yielding a much simpler filesystem well suited for modern disks.

Note that this results in a different on-disk format and will need to be a different file system type.

The result would be potentially useful to other operating systems beyond just NetBSD, since UFS/FFS is used in so many different kernels.

Posted Sunday evening, November 6th, 2011 Tags:

It would be nice to support these newer highly SMP processors from Sun. A Linux port already exists, and Sun has contributed code to the FOSS community.

(Some work has already been done and committed - see https://wiki.netbsd.org/ports/sparc64/sparc64sun4v/

Posted Sunday evening, November 6th, 2011 Tags:

syspkgs is the concept of using pkgsrc's pkg_* tools to maintain the base system. That is, allow users to register and components of the base system with more ease.

There has been a lot of work in this area already, but it has not yet been finalized. Which is a diplomatic way of saying that this project has been attempted repeatedly and failed every time.

Posted Sunday evening, November 6th, 2011 Tags:

Add memory-efficient snapshots to tmpfs. A snapshot is a view of the filesystem, frozen at a particular point in time. The snapshotted filesystem is not frozen, only the view is. That is, you can continue to read/write/create/delete files in the snapshotted filesystem.

The interface to snapshots may resemble the interface to null mounts, e.g., 'mount -t snapshot /var/db /db-snapshot' makes a snapshot of /var/db/ at /db-snapshot/.

You should exploit features of the virtual memory system like copy-on-write memory pages to lazily make copies of files that appear both in a live tmpfs and a snapshot. This will help conserve memory.

Posted Sunday evening, November 6th, 2011 Tags:

While we now have mandoc for handling man pages, we currently still need groff in the tree to handle miscellaneous docs that are not man pages.

This is itself an inadequate solution as the groff we have does not support PDF output (which in this day and age is highly desirable) ... and while newer groff does support PDF output it does so via a Perl script. Also, importing a newer groff is problematic for assorted other reasons.

We need a way to typeset miscellaneous articles that we can import into base and that ideally is BSD licensed. (And that can produce PDFs.) Currently it looks like there are three decent ways forward:

  • Design a new roff macro package that's comparable to mdoc (e.g. supports semantic markup) but is for miscellaneous articles rather than man pages, then teach mandoc to handle it.

  • Design a new set of markup tags comparable to mdoc (e.g. supports semantic markup) but for miscellaneous articles, and a different less ratty syntax for it, then teach mandoc to handle this.

  • Design a new set of markup tags comparable to mdoc (e.g. supports semantic markup) but for miscellaneous articles, and a different less ratty syntax for it, and write a new program akin to mandoc to handle it.

These are all difficult and a lot of work, and in the case of new syntax are bound to cause a lot of shouting and stamping. Also, many of the miscellaneous documents use various roff preprocessors and it isn't clear how much of this mandoc can handle.

None of these options is particularly appealing.

There are also some less decent ways forward:

  • Pick one of the existing roff macro packages for miscellaneous articles (ms, me, ...) and teach mandoc to handle it. Unfortunately all of these macro packages are pretty ratty, they're underpowered compared to mdoc, and none of them support semantic markup.

  • Track down one of the other older roff implementations, that are now probably more or less free (e.g. ditroff), then stick to the existing roff macro packages as above. In addition to the drawbacks cited above, any of these programs are likely to be old nasty code that needs a lot of work.

  • Teach the groff we have how to emit PDFs, then stick to the existing roff macro packages as above. In addition to the drawbacks cited above, this will likely be pretty nasty work and it's still got the wrong license.

  • Rewrite groff as BSD-licensed code and provide support for generating PDFs, then stick to the existing roff macro packages as above. In addition to the drawbacks cited above, this is a horrific amount of work.

  • Try to make something else do what we want. Unfortunately, TeX is a nonstarter and the only other halfway realistic candidate is lout... which is GPLv3 and at least at casual inspection looks like a horrible mess of its own.

These options are even less appealing.

Maybe someone can think of a better idea. There are lots of choices if we give up on typeset output, but that doesn't seem like a good plan either.

Posted Sunday evening, November 6th, 2011 Tags:

Due to the multitude of supported machine architectures NetBSD has to deal with many different partitioning schemes. To deal with them in a uniform way (without imposing artificial restrictions that are not enforced by the underlying firmware or bootloader partitioning scheme) wedges have been designed.

While the kernel part of wedges is mostly done (and missing parts are easy to add), a userland tool to edit wedges and to synthesize defaults from (machine/arch dependent) on-disk content is needed.

Posted Sunday evening, November 6th, 2011 Tags:

Create an easy to use wifi setup widget for NetBSD: browse and select networks in the vicinity by SSID, BSSID, channel, etc.

The guts should probably be done as a library so that user interfaces of increasing slickness can be built on top of it as desired. (That is: there ought to be some form of this in base; but a nice looking gtk interface version would be good to have as well.)

Posted Sunday evening, November 6th, 2011 Tags:

Add socket options to NetBSD for controlling WLAN transmit parameters like transmit power, fragmentation threshold, RTS/CTS threshold, bitrate, 802.11e access category, on a per-socket and per-packet basis. To set transmit parameters, pass radiotap headers using sendmsg(2) and setsockopt(2).

This project is on hold due to the conversion project needing to be completed first.

Posted Sunday evening, November 6th, 2011 Tags:

The NetBSD website building infrastructure is rather complex and requires significant resources. We need to make it easier for anybody to contribute without having to install a large number of complex applications from pkgsrc or without having to learn the intricacies of the build process.

A more detailed description of the problem is described in this and this email and the following discussion on the netbsd-docs.

This work requires knowledge of XML, XSLT and make. This is not a request for visual redesign of the website.

Posted Sunday evening, November 6th, 2011 Tags:

With the push of virtualization, the x86 world started recently to gain a more widespread attention towards supporting IOMMUs; similar to MMUs that translate virtual addresses into physical ones, an IOMMU translates device/bus addresses into physical addresses. The purpose of this project is to add AMD and Intel IOMMU support in NetBSD's machine-independent bus abstraction layers bus.space(9) and bus.dma(9).

Posted Sunday evening, November 6th, 2011 Tags:

Latest Xen versions come with a number of features that are currently not supported by NetBSD: USB/VGA passthrough, RAS (Reliability, Availability and Serviceability) options - CPU and memory hotpluging - , Fault tolerancy with Remus, and debugging with gdbx (lightweight debugger included with Xen).

The purpose of this project is to add the missing parts inside NetBSD. Most of the work is composed of smaller components that can be worked on independently from others.

Posted Sunday evening, November 6th, 2011 Tags:
  • Contact: tech-kern
  • Duration estimate: 1 year for port; 3 years for rewrite by one developer

Implement a BSD licensed XFS. A GPL licensed implementation of XFS is available at http://oss.sgi.com/projects/xfs/.

Alternatively, or additionally, it might be worthwhile to do a port of the GPL code and allow it to run as a kernel module.

See also FreeBSD's port.

Posted Sunday evening, November 6th, 2011 Tags:

Enhance zeroconfd, the Multicast DNS daemon, that was begun in NetBSD's Google Summer of Code 2005 (see work in progress: http://netbsd-soc.sourceforge.net/projects/zeroconf/). Develop a client library that lets a process publish mDNS records and receive asynchronous notification of new mDNS records. Adapt zeroconfd to use event(3) and queue(3). Demonstrate comparable functionality to the GPL or APSL alternatives (Avahi, Howl, ...), but in a smaller storage footprint, with no dependencies outside of the NetBSD base system.

Posted Sunday evening, November 6th, 2011 Tags:

Traditionally, the NetBSD kernel code had been protected by a single, global lock. This lock ensured that, on a multiprocessor system, two different threads of execution did not access the kernel concurrently and thus simplified the internal design of the kernel. However, such design does not scale to multiprocessor machines because, effectively, the kernel is restricted to run on a single processor at any given time.

The NetBSD kernel has been modified to use fine grained locks in many of its different subsystems, achieving good performance on today's multiprocessor machines. Unfotunately, these changes have not yet been applied to the networking code, which remains protected by the single lock. In other words: NetBSD networking has evolved to work in a uniprocessor envionment; switching it to use fine-grained locked is a hard and complex problem.

This project is currently claimed

Funding

At this time, The NetBSD Foundation is accepting project specifications to remove the single networking lock. If you want to apply for this project, please send your proposal to the contact addresses listed above.

Due to the size of this project, your proposal does not need to cover everything to qualify for funding. We have attempted to split the work into smaller units, and you can submit funding applications for these smaller subtasks independently as long as the work you deliver fits in the grand order of this project. For example, you could send an application to make the network interfaces alone MP-friendly (see the work plan below).

What follows is a particular design proposal, extracted from an original text written by Matt Thomas. You may choose to work on this particular proposal or come up with your own.

Tentative specification

The future of NetBSD network infrastructure has to efficiently embrace two major design criteria: Symmetric Multi-Processing (SMP) and modularity. Other design considerations include not only supporting but taking advantage of the capability of newer network devices to do packet classification, payload splitting, and even full connection offload.

You can divide the network infrastructure into 5 major components:

  • Interfaces (both real devices and pseudo-devices)
  • Socket code
  • Protocols
  • Routing code
  • mbuf code.

Part of the complexity is that, due to the monolithic nature of the kernel, each layer currently feels free to call any other layer. This makes designing a lock hierarchy difficult and likely to fail.

Part of the problem are asynchonous upcalls, among which include:

  • ifa->ifa_rtrequest for route changes.
  • pr_ctlinput for interface events.

Another source of complexity is the large number of global variables scattered throughout the source files. This makes putting locks around them difficult.

Subtasks

The proposed solution presented here include the following tasks (in no particular order) to achieve the desired goals of SMP support and modularity:

Work plan

Aside from the list of tasks above, the work to be done for this project can be achieved by following these steps:

  1. Move ARP out of the routing table. See the nexthop cache project.

  2. Make the network interfaces MP, which are one of the few users of the big kernel lock left. This needs to support multiple receive and transmit queues to help reduce locking contention. This also includes changing more of the common interfaces to do what the tsec driver does (basically do everything with softints). This also needs to change the *_input routines to use a table to do dispatch instead of the current switch code so domain can be dynamically loaded.

  3. Collect global variables in the IP/UDP/TCP protocols into structures. This helps the following items.

  4. Make IPV4/ICMP/IGMP/REASS MP-friendly.

  5. Make IPV6/ICMP/IGMP/ND MP-friendly.

  6. Make TCP MP-friendly.

  7. Make UDP MP-friendly.

Radical thoughts

You should also consider the following ideas:

LWPs in user space do not need a kernel stack

Those pages are only being used in case the an exception happens. Interrupts are probably going to their own dedicated stack. One could just keep a set of kernel stacks around. Each CPU has one, when a user exception happens, that stack is assigned to the current LWP and removed as the active CPU one. When that CPU next returns to user space, the kernel stack it was using is saved to be used for the next user exception. The idle lwp would just use the current kernel stack.

LWPs waiting for kernel condition shouldn't need a kernel stack

If an LWP is waiting on a kernel condition variable, it is expecting to be inactive for some time, possibly a long time. During this inactivity, it does not really need a kernel stack.

When the exception handler get an usermode exeception, it sets LWP restartable flag that indicates that the exception is restartable, and then services the exception as normal. As routines are called, they can clear the LWP restartable flag as needed. When an LWP needs to block for a long time, instead of calling cv_wait, it could call cv_restart. If cv_restart returned false, the LWPs restartable flag was clear so cv_restart acted just like cv_wait. Otherwise, the LWP and CV would have been tied together (big hand wave), the lock had been released and the routine should have returned ERESTART. cv_restart could also wait for a small amount of time like .5 second, and only if the timeout expires.

As the stack unwinds, eventually, it would return to the last the exception handler. The exception would see the LWP has a bound CV, save the LWP's user state into the PCB, set the LWP to sleeping, mark the lwp's stack as idle, and call the scheduler to find more work. When called, cpu_switchto would notice the stack is marked idle, and detach it from the LWP.

When the condition times out or is signalled, the first LWP attached to the condition variable is marked runnable and detached from the CV. When the cpu_switchto routine is called, the it would notice the lack of a stack so it would grab one, restore the trapframe, and reinvoke the exception handler.

Posted late Sunday afternoon, November 6th, 2011 Tags:

Refactor utilities in the base system, such as netstat, top, and vmstat, that format & display tables of statistics.

One possible refactoring divides each program into three:

  • one program reads the statistics from the kernel and writes them in machine-readable form to its standard output
  • a second program reads machine-readable tables from its standard input and writes them in a human-readable format to the standard output
  • and a third program supervises a pipeline of the first two programs, browses and refreshes a table.

Several utilities will share the second and third program.

Posted late Saturday night, November 6th, 2011 Tags:

Currently, the puffs(3) interface between the kernel and userspace uses various system structures for passing information. Examples are struct stat and struct uucred. If these change in layout (such as with the time_t size change in NetBSD 6.0), old puffs servers must be recompiled.

The project milestones are:

  • define a binary-independent protocol
  • implement support
  • measure the performance difference with direct kernel struct passing
  • if there is a huge difference, investigate the possibility for having both an internal and external protocol. The actual decision to include support will be made on the relative complexity of the code for dual support.

While this project will be partially implemented in the kernel, it is fairly well-contained and prior kernel experience is not necessary.

If there is time and interest, a suggested subproject is making sure that p2k(3) does not suffer from similar issues. This is a required subproject if dual support as mentioned above is not necessary.

Posted late Saturday night, November 6th, 2011 Tags:

Write a script that aids refactoring C programs by extracting subroutines from fragments of C code.

Do not reinvent the wheel: wherever possible, use existing technology for the parsing and comprehension of C code. Look at projects such as sparse and Coccinelle.

Your script should work as a filter that puts the free variables at the top, encloses the rest in curly braces, does something helpful with break, continue, and return statements.

That's just a start.

This project is tagged "hard" because it's not clearly specified. The first step (like with many projects) is to work out the specification in more detail.

Posted late Saturday night, November 6th, 2011 Tags:
  • Contact: tech-pkg
  • Duration estimate: 3 months

The goal of this project is to generate a package or packages that will set up a cross-compiling environment for one (slow) NetBSD architecture on another (fast) NetBSD architecture, starting with (and using) the NetBSD toolchain in src.

The package will require a checked out NetBSD src tree (or as a refinement, parts thereof) and is supposed to generate the necessary cross-building tools using src/build.sh with appropriate flags; for the necessary /usr/include and libraries of the slow architecture, build these to fit or use the slow architecture's base.tgz and comp.tgz (or parts thereof). As an end result, you should e.g. be able to install the binary cross/pkgsrc-NetBSD-amd64-to-atari package on your fast amd64 and start building packages for your slow atari system.

Use available packages, like eg pkgtools/pkg_comp to build the cross-compiling environment where feasible.

As test target for the cross-compiling environment, use pkgtools/pkg_install, which is readily cross-compilable by itself.

If time permits, test and fix cross-compiling pkgsrc X.org, which was made cross-compilable in an earlier GSoC project, but may have suffered from feature rot since actually cross-compiling it has been too cumbersome for it to see sufficient use.

Posted late Saturday night, November 6th, 2011 Tags:

pkgsrc duplicates NetBSD efforts in maintaining X.org packages. pkgsrc X11, being able to cross-build, could replace base X11 distribution.

The latter decoupling should simplify maintainance of software; updating X11 and associated software becomes easier because of pkgsrc's shorter release cycle and more volatility.

Cross-buildable and cross-installable pkgsrc tools could also simplify maintainance of slower systems by utilising power of faster ones.

The goal of this project is to make it possible to bootstrap pkgsrc using available cross-build tools (e.g. NetBSD's).

This project requires good understanding of cross-development, some knowledge of NetBSD build process or ability to create cross-development toolchain, and familiarity with pkgsrc bootstrapping.

Note: basic infrastructure for this exists as part of various previous GSoC projects. General testing is lacking.

Posted late Saturday night, November 6th, 2011 Tags:
  • Contact: tech-kern
  • Duration estimate: 3 months

Make NetBSD behave gracefully when a "live" USB/FireWire disk drive is accidentally detached and re-attached by, for example, creating a virtual block device that receives block-read/write commands on behalf of the underlying disk driver.

This device will delegate reads and writes to the disk driver, but it will keep a list of commands that are "outstanding," that is, reads that the disk driver has not completed, and writes that have not "hit the platter," so to speak.

Milestones:

  • Provide a character device for userland to read indications that a disk in use was abruptly detached.
  • Following disk re-attachment, the virtual block device replays its list of outstanding commands. A correct solution will not replay commands to the wrong disk if the removable was replaced instead of re-attached.

Open questions: Prior art? Isn't this how the Amiga worked? How will this interact with mount/unmount—is there a use-count on devices? Can you leverage "wedges" in your solution? Does any/most/all removable storage indicate reliably when a block written has actually reached the medium?

Posted late Saturday night, November 6th, 2011 Tags:

Write a program that can read an email, infer quoting conventions, discern bottom-posted emails from top-posted emails, and rewrite the email using the conventions that the reader prefers. Then, take it a step further: write a program that can distill an entire email discussion thread into one document where every writer's contribution is properly attributed and appears in its proper place with respect to interleaved quotations.

Posted late Saturday night, November 6th, 2011 Tags:

Design and implement a mechanism that allows for fast user level access to kernel time data structures for NetBSD. For certain types of small data structures the system call overhead is significant. This is especially true for frequently invoked system calls like clock_gettime(2) and gettimeofday(2). With the availability of user level readable high frequency counters it is possible to create fast implementations for precision time reading. Optimizing clock_gettime(2) and alike will reduce the strain from applications frequently calling these system calls and improves timing information quality for applications like NTP. The implementation would be based on a to-be-modified version of the timecounters implementation in NetBSD.

Milestones:

  • Produce optimizations for clock_gettime
  • Produce optimizations for gettimeofday
  • Show benchmarks before and after
  • start evolving timecounters in NetBSD, demonstrating your improvements

See also the Paper on Timecounters by Poul-Henning Kamp.

Posted late Saturday night, November 6th, 2011 Tags:

The existing puffs protocol gives a way to forward kernel-level file system actions to a userspace process. This project generalizes that protocol to allow forwarding file system actions arbitrarily across a network. This will make it possible to mount any kernel file system type from any location on the network, given a suitable arrangement of components.

The file system components to be used are puffs and rump. puffs is used to forward local file system requests from the kernel to userspace and rump is used to facilitate running the kernel file system in userspace as a service daemon.

The milestones are the following:

  • Write the necessary code to be able to forward requests from one source to another. This involves most likely reworking a bit of the libpuffs option parsing code and creating a puffs client (say, mount_puffs) to be able to forward requests from one location to another. The puffs protocol should be extended to include the necessary new features or a new protocol invented.

    Proof-of-concept code for this has already been written. (Where is it?)

  • Currently the puffs protocol used for communication between the kernel and userland is machine dependent. To facilitate forwarding the protocol to remote hosts, a machine independent version must be specified.

  • To be able to handle multiple clients, the file systems must be converted to daemons instead of being utilities. This will also, in the case of kernel file system servers, include adding locking to the communication protocol.

The end result will look something like this:

# start serving ffs from /dev/wd0a on port 12675
onehost> ffs_serv -p 12675 /dev/wd0a
# start serving cd9660 from /dev/cd0a on port 12676
onehost> cd9660_serv -p 12675 /dev/cd0a

# meanwhile in outer space, mount anotherhost from port 12675
anotherhost> mount_puffs -t tcp onehost:12675 /mnt
anotherhost> mount
...
anotherhost:12675 on /mnt type <negotiated>
...
anotherhost> cd /mnt
anotherhost> ls
  ... etc

The implementor should have some familiarity with file systems and network services.

Any proposal should include answers to at least the following questions:

  • How is this different from NFS?

  • How is the protocol different from 9p?

  • How is this scheme going to handle the hard things that NFS doesn't do very well, such as distributed cache consistency?

  • Given industry trends, why is this project proposing a new protocol instead of conforming to the SOA model?

Posted late Saturday night, November 6th, 2011 Tags:

inetd is a classic method for launching network programs on-the-fly and some of its ideas are coming back into vogue. Enhancing this daemon should include investigations into other similar systems in other operating systems.

Primary milestones:

  • Prefork: Support pre-forking multiple children and keeping them alive for multiple invocations.
  • Per service configuration file: Add a per-service configuration file similar to xinetd.
  • Make the rate-limiting feature configurable on a per-service basis.
  • Improve the logging and make logging modes configurable on a per-service basis.

Nice to have:

  • Add include directives to the configuration language to allow service definitions to be installed in /usr/share or /usr/pkg/share.
  • Add a separate way to turn services on and off, so they can be defined statically (such as in /usr/share) and turned on and off from /etc.
  • Allow non-privileged users to add/remove/change their own services using a separate utility.
  • Integrate with the new blocklist daemon.
  • Configuration compatibility for systemd socket activations
Posted late Saturday night, November 6th, 2011 Tags:

Produce lessons and a list of affordable parts and free software that NetBSD hobbyists can use to teach themselves JTAG. Write your lessons for a low-cost embedded system or expansion board that NetBSD already supports.

Posted late Saturday night, November 6th, 2011 Tags:

Launchd is a MacOS/X utility that is used to start and control daemons similar to init(8), but much more powerful. There was an effort to port launchd to FreeBSD, but it seems to be abandoned. We should first investigate what happened to the FreeBSD effort to avoid duplicating work. The port is not trivial because launchd uses a lot of mach features.

Milestones:

  • report of FreeBSD efforts (past and present)
  • launchd port replacing: init
  • launchd port replacing: rc
  • launchd port compatible with: rc.d scripts
  • launchd port replacing: watchdogd

Nice to have:

  • launchd port replacing/integrating: inetd
  • launchd port replacing: atd
  • launchd port replacing: crond
  • launchd port replacing: (the rest)
Posted late Saturday night, November 6th, 2011 Tags:

Design and implement a general API for control of LED and LCD type devices on NetBSD. The API would allow devices to register individual LED and LCD devices, along with a set of capabilities for each one. Devices that wish to display status via an LED/LCD would also register themselves as an event provider. A userland program would control the wiring of each event provider, to each output indicator. The API would need to encompass all types of LCD displays, such as 3 segment LCDs, 7 segment alphanumerics, and simple on/off state LED's. A simple example is a keyboard LED, which is an output indicator, and the caps-lock key being pressed, which is an event provider.

There is prior art in OpenBSD; it should be checked for suitability, and any resulting API should not differ from theirs without reason.

Milestones:

  • a port of OpenBSD's LED tools
  • a userland tool to control LED
  • demonstration of functionality
  • integration into NetBSD
Posted late Saturday night, November 6th, 2011 Tags:

Write a library of routines that estimate the expected transmission duration of a queue of 802.3 or 802.11 frames based on the current bit rate, applicable overhead, and the backoff level. The kernel will use this library to help it decide when packet queues have grown too long in order to mitigate "buffer bloat."

Posted late Saturday night, November 6th, 2011 Tags:

The current lpr/lpd system in NetBSD is ancient, and doesn't support modern printer systems very well. Interested parties would do a from scratch rewrite of a new, modular lpr/lpd system that would support both the old lpd protocol and newer, more modern protocols like IPP, and would be able to handle modern printers easily.

This project is not intrinsically difficult, but will involve a rather large chunk of work to complete.

Note that the goal of this exercise is not to reimplement cups -- cups already exists and one like it was enough.

Some notes:

It seems that a useful way to do this would be to divide the printing system in two: a client-side system, which is user-facing and allows submitting print jobs to arbitrary print servers, and a server-side system, which implements queues and knows how to talk to actual printer devices. In the common case where you don't have a local printer but use printers that are out on the network somewhere, the server-side system wouldn't be needed at all. When you do have a local printer, the client-side system would submit jobs to the local server-side system using the lpr protocol (or IPP or something else) over a local socket but otherwise treat it no differently from any other print server.

The other important thing moving forward: lpr needs to learn about MIME types and accept an argument to tell it the MIME types of its input files. The current family of legacy options lpr accepts for file types are so old as to be almost completely useless; meanwhile the standard scheme of guessing file types inside the print system is just a bad design overall. (MIME types aren't great but they're what we have.)

Posted late Saturday night, November 6th, 2011 Tags:

swcrypto could use a variety of enhanacements

Milestones/deliverables:

  • use multiple cores efficiently (that already works reasonably well for multiple request streams)
  • use faster versions of complex transforms (CBC, counter modes) from our in-tree OpenSSL or elsewhere (eg libtomcrypt)
  • add support for asymmetric operations (public key)

Extra credit:

  • Tie public-key operations into veriexec somehow for extra credit (probably a very good start towards an undergrad thesis project).
Posted late Saturday night, November 6th, 2011 Tags:

Change infrastructure so that dependency information (currently in buildlink3.mk files) is installed with a binary package and is used from there

This is not an easy project.

pkgsrc currently handles dependencies by including buildlink3.mk files spread over pkgsrc. The problem is that these files describe the current state of pkgsrc, not the current state of the installed packages.

For this reason and because most of the information in the files is templated, the buildlink3.mk files should be replaced by the relevant information in an easily parsable manner (not necessarily make syntax) that can be installed with the package. Here the task is to come up with a definitive list of information necessary to replace all the stuff that's currently done in buildlink3.mk files (including: dependency information, lists of headers to pass through or replace by buildlink magic, conditional dependencies, ...)

The next step is to come up with a proposal how to store this information with installed packages and how to make pkgsrc use it.

Then the coding starts to adapt the pkgsrc infrastructure to do it and show with a number of trivial and non-trivial packages that this proposal works.

It would be good to provide scripts that convert from the current state to the new one, and test it with a bulk build.

Of course it's not expected that all packages be converted to the new framework in the course of this project, but the further steps should be made clear.

goals/milestones:

  • invent a replacement for buildlink3.mk files, keeping current features
  • demonstrate your new tool as a buildlink3.mk replacement including new features
  • execute a bulk build with as many packages as possible using the new buildlink
Posted late Saturday night, November 6th, 2011 Tags:

Put config files (etc/) installed by pkgsrc into some version control system to help keeping track of changes and updating them.

The basic setup might look like this:

  • There is a repository containing the config files installed by pkgsrc, starting out empty.
  • During package installation, pkgsrc imports the package's config files into the repository onto a branch tagged with the name and version of the package (if available, on a vendor branch). (e.g.: digest-20080510)
  • After installation, there are two cases:

    1. the package was not installed before: the package's config files get installed into the live configuration directory and committed to the head of the config repository
    2. the package was installed before: a configuration update tool should display changes between the new and the previous original version as well as changes between the previous original and installed config file, for each config file the package uses, and support merging the changes made necessary by the package update into the installed config file. Commit the changes to head when the merge is done.
  • Regular automated check-ins of the entire live pkgsrc configuration should be easy to set up, but also manual check-ins of singular files so the local admin can use meaningful commit messages when they change their config, even if they are not experienced users of version control systems

The actual commands to the version control system should be hidden behind an abstraction layer, and the vcs operations should be kept simple, so that other compatibility layers can be written, and eventually the user can pick their vcs of choice (and also a vcs location of choice, in case e.g. the enterprise configuration repository is on a central subversion server).

milestones/goals:

  • choose a VCS system (BSD licensed is a nice-to-have)
  • write wrappers around it, or embed its functionality
  • demonstrate usage in upgrades

bonus:

  • extend functionality into additional VCS systems

This project was done during Google Summer of Code 2018 by Keivan Motavalli Configuration files versioning in pkgsrc project. At the moment the code need to be reviewed and imported in pkgsrc.

Posted late Saturday night, November 6th, 2011 Tags:

Using kernel virtualization offered by rump it is possible to start a virtually unlimited amount of TCP/IP stacks on one host and interlink the networking stacks in an arbitrary fashion. The goal of this project is to create a visual GUI tool for creating and managing the networks of rump kernel instances. The main goal is to support testing and development activities.

The implementation should be split into a GUI frontend and a command-line network-control backend. The former can be used to design and edit the network setup, while the latter can be used to start and monitor the virtual network from e.g. automated tests.

The GUI frontend can be implemented in any language that is convenient. The backend should be implemented in a language and fashion which makes it acceptable to import into the NetBSD base system.

The suggested milestones for the project are:

  1. get familiar and comfortable with starting and stopping rump kernels and configuring virtual network topologies
  2. come up with an initial backend command language. It can be something as simple as bourne shell functions.
  3. come with an initial specification and implementation of the GUI tool.
  4. make the GUI tool be able to save and load network specifications.
  5. create a number of automated tests to evaluate the usefulness of the tool in test creation.

In case of a capable student, the project goals can be set on a more demanding level. Examples of additional features include:

  • defining network characteristics such as jitter and packet loss for links
  • real-time monitoring of network parameters during a test run
  • UI gimmicks such as zooming and node grouping
Posted late Saturday night, November 6th, 2011 Tags:

All architectures suffer from code injection issues because the only writable segment is the PLT/GOT. RELRO (RELocation Read Only) is a mitigation technique that is used during dynamic linking to prevent access to the PLT/GOT. There is partial RELRO which protects that GOT but leaves the PLT writable, and full RELRO that protects both at the expense of performing a full symbol resolution at startup time. The project is about making the necessary modifications to the dynamic loader (ld_elf.so) to make RELRO work.

If that is completed, then we can also add the following improvement: Currently kernels with options PAX_MPROTECT can not execute dynamically linked binaries on most RISC architectures, because the PLT format defined by the ABI of these architectures uses self-modifying code. New binutils versions have introduced a different PLT format (enabled with --secureplt) for alpha and powerpc.

Milestones:

  • For all architectures we can improve security by implementing relro2.
  • Once this is done, we can improve security for the RISC architectures by adding support for the new PLT formats introduced in binutils 2.17 and gcc4.1 This will require changes to the dynamic loader (ld.elf_so), various assembly headers, and library files.
  • Support for both the old and new formats in the same invocation will be required.

Status: * Added support to the dynamic loader (ld.elf_so) to handle protecting the GNU relro section. * Enabled partial RELRO by default on x86.

Posted late Saturday night, November 6th, 2011 Tags:

pkgsrc is a very flexible package management system. It provides a comprehensible framework to build, test, deploy, and maintain software in its original form (with porter/packager modifications where applicable) as well as with site local modifications and customizations. All this makes pkgsrc suitable to use in diverse environments ranging from small companies up to large enterprises.

While pkgsrc already contains most elements needed to build an authentication server (or an authentication server failover pair), in order to install one, considerable knowledge about the neccessary elements is needed, plus the correct configuration, while in most cases pretty much identical, is tedious and not without pitfalls.

The goal of this project is to create a meta-package that will deploy and pre-configure an authentication server suitable for a single sign-on infrastructure.

Necessary tasks: provide missing packages, provide packages for initial configuration, package or create corresponding tools to manage user accounts, document.

The following topics should be covered:

  • PAM integration with OpenLDAP and DBMS;
  • Samba with PAM, DBMS and directory integration;
  • Kerberos setup;
  • OpenLDAP replication;
  • DBMS (PostgreSQL is a must, MySQL optional, if time permits), replication (master-master, if possible);
  • DNS server with a sane basic dynamic DNS update config using directory and database backend;
  • user account management tools (web interface, command line interface, see user(8) manual page, perhaps some scripting interface);
  • configuration examples for integration of services (web services, mail, instant messaging, PAM is a must, DBMS and directory optional).

All covered services should be documented, in particular documentation should include:

  • initial deployment instructions;
  • sample configuration for reasonably simple case;
  • instructions how to test if the software works;
  • references to full documentation and troubleshooting instructions (if any), both should be provided on-site (i.e. it should be possible to have everything available given pkgsrc snapshot and corresponding distfiles and/or packages on physical media).

In a nutshell, the goal of the project is to make it possible for a qualified system administrator to deploy basic service (without accounts) with a single pkg_add.

Posted late Saturday night, November 6th, 2011 Tags:

The goal of this project is to provide an alternative version of the NetBSD system installer with a simple, X based graphical user interface.

The installer currently uses a "homegrown" (similar to CUA) text based interface, thus being easy to use over serial console as well as on a framebuffer console.

The current installer code is partly written in plain C, but in big parts uses C fragments embedded into its own definition language, preprocessed by the "menuc" tool into plain C code and linked against libterminfo.

During this project, the "menuc" tool is modified to optionally generate a different version of the C code, which then is linked against standard X libraries. The C stub fragments sprinkled throughout the menu definitions need to be modified to be reusable for both (text and X) versions. Where needed the fragments can just call C functions, which have different implementations (selected via a new ifdef).

Since the end result should be able to run off an enhanced install CD, the selection of widgets used for the GUI is limited. Only base X libraries are available. A look & feel similar to current xdm would be a good start.

Developement can be done on an existing system, testing does not require actuall installation on real hardware.

An optional extension of the project is to modify the creation of one or more port's install CD to make use of the new xsysinst.

Milestones include: * modify the "menuc" tool to support X * keep text/serial console installing * demonstrate a GUI install * demonstrate fallback to the text installer

The candidate must have:

  • familiarity with the system installer. You should have used sysinst to install the system.
  • familiarity with C and X programming.

The following would also be useful:

  • familiarity with NetBSD.
  • familiarity with user interface programming using in-tree X widgets.

References:

Posted late Saturday night, November 6th, 2011 Tags:

Port valgrind to NetBSD for pkgsrc, then use it to do an audit of any memory leakage.

See also http://valgrind.org and http://vg4nbsd.berlios.de for work in progress.

Posted late Saturday night, November 6th, 2011 Tags:

NetBSD supports a number of platforms where both 32bit and 64bit execution is possible. The more well known example is the i386/AMD64 pair and the other important one is SPARC/SPARC64. On this platforms it is highly desirable to allow running all 32bit applications with a 64bit kernel. This is the purpose of the netbsd32 compatibility layer.

At the moment, the netbsd32 layer consists of a number of system call stubs and structure definitions written and maintained by hand. It is hard to ensure that the stubs and definitions are up-to-date and correct. One complication is the difference in alignment rules. On i386 uint64_t has a 32bit alignment, but on AMD64 it uses natural (64bit) alignment. This and the resulting padding introduced by the compiler can create hard to find bugs.

goals/milestones:

  • replace the manual labour with an automatic tool

    This tool should allow both verification / generation of structure definitions for use in netbsd32 code allow generation of system call stubs and conversion functions. Generated stubs should also ensure that no kernel stack data is leaked in hidden padding without having to resort to unnecessary large memset calls.

For this purpose, the Clang C parser or the libclang frontend can be used to analyse the C code.

Posted late Saturday night, November 6th, 2011 Tags:
Posted late Saturday night, November 6th, 2011

This page contains the list of all available projects, broken by topic and difficulty. The topics are as follows:

File system / storage projects

Easy

Medium

Hard

Networking projects

Easy

Medium

Hard

Port-related projects

Easy

Medium

Hard

Other kernel-level projects

Easy

Medium

Hard

Userland projects

Easy

Medium

Hard

Desktop projects

Easy

Medium

Hard

Code Quality Improvement projects

Easy

Medium

Hard

pkgsrc projects

Easy

Medium

Hard

Miscellaneous projects

Easy

Medium

Hard

Unclassified projects

This section contains the list of projects that have not been classified: i.e. projects that lack a tag defining their category and/or their difficulty.

Theoretically, this section should be empty. In practice, however, it is all too easy to forget to tag a project appropriately when defining it, and this section is intended to help in spotting such misclassified projects. Please note that misclassified projects may not appear in other indexes, so it is important to spot them!

Projects without a category definition

Projects without a difficulty definition

Notes

This page is (mostly) generated automatically from the project pages themselves. To add a new project, just add a new project page and tag it with the proper tags. Also note that the last modification date appearing below is the last modification date of the list template, not of any of the projects themselves; nor is it the date when the last new project was added.

Posted late Saturday night, November 6th, 2011

NetBSD supports a number of platforms where both 32bit and 64bit execution is possible. The more well known example is the i386/AMD64 pair and the other important one is SPARC/SPARC64. On this platforms it is highly desirable to allow running all 32bit applications with a 64bit kernel. This is the purpose of the netbsd32 compatibility layer.

At the moment, the netbsd32 layer consists of a number of system call stubs and structure definitions written and maintained by hand. It is hard to ensure that the stubs and definitions are up-to-date and correct. One complication is the difference in alignment rules. On i386 uint64_t has a 32bit alignment, but on AMD64 it uses natural (64bit) alignment. This and the resulting padding introduced by the compiler can create hard to find bugs.

goals/milestones:

  • replace the manual labour with an automatic tool

    This tool should allow both verification / generation of structure definitions for use in netbsd32 code allow generation of system call stubs and conversion functions. Generated stubs should also ensure that no kernel stack data is leaked in hidden padding without having to resort to unnecessary large memset calls.

For this purpose, the Clang C parser or the libclang frontend can be used to analyse the C code.

Posted late Saturday night, November 6th, 2011 Tags:

Port valgrind to NetBSD for pkgsrc, then use it to do an audit of any memory leakage.

See also http://valgrind.org and http://vg4nbsd.berlios.de for work in progress.

Posted late Saturday night, November 6th, 2011 Tags:

The goal of this project is to provide an alternative version of the NetBSD system installer with a simple, X based graphical user interface.

The installer currently uses a "homegrown" (similar to CUA) text based interface, thus being easy to use over serial console as well as on a framebuffer console.

The current installer code is partly written in plain C, but in big parts uses C fragments embedded into its own definition language, preprocessed by the "menuc" tool into plain C code and linked against libterminfo.

During this project, the "menuc" tool is modified to optionally generate a different version of the C code, which then is linked against standard X libraries. The C stub fragments sprinkled throughout the menu definitions need to be modified to be reusable for both (text and X) versions. Where needed the fragments can just call C functions, which have different implementations (selected via a new ifdef).

Since the end result should be able to run off an enhanced install CD, the selection of widgets used for the GUI is limited. Only base X libraries are available. A look & feel similar to current xdm would be a good start.

Developement can be done on an existing system, testing does not require actuall installation on real hardware.

An optional extension of the project is to modify the creation of one or more port's install CD to make use of the new xsysinst.

Milestones include: * modify the "menuc" tool to support X * keep text/serial console installing * demonstrate a GUI install * demonstrate fallback to the text installer

The candidate must have:

  • familiarity with the system installer. You should have used sysinst to install the system.
  • familiarity with C and X programming.

The following would also be useful:

  • familiarity with NetBSD.
  • familiarity with user interface programming using in-tree X widgets.

References:

Posted late Saturday night, November 6th, 2011 Tags:

IMPORTANT: This project was completed by Eugene Lozovoy. You may still contact the people above for details, but please do not submit an application for this project.

The goal of this project is to enhance the NetBSD system installer (sysinst) to provide additional support for (in order):

  • partition disks using GPT
  • prepare multiple disks
  • combine multiple partitions to raid/lvm volumes
  • encrypt partitions via cgd
  • other enhancements

The installer currently supports installing the system to any available single disk. It is possible to select which parts (distribution sets) of the system to install, and also to customise the disk partition layout. Sysinst can also use vnode pseudo disks, so can be tested without the need to re-install the host system.

The first goal is to allow partitioning disks using the GUID partition table (GPT). The current partitioning code is tied heavily to the BSD disklabel. Fixing it is straight forward, but both methods have to be offered and only some architectures can boot from GPT disks.

The second goal is to allow preparing several disks. This part would also be usefull (see "other enhancements" below) as a stand-alone tool. Various disks may be partitioned using different shemes, for example when the boot disk can not use GPT, but secondary (large) disks should use it. This part also is a direct prerequisite for the following one.

The third goal is to (optionally) create logical volumes from multiple partitions or disks, either using raidframe or LVM. This includes making the volumes bootable, if possible.

The fourth goal is to add support for creating and installing on to cgd (encrypted) partitions. The initial support will not be for the boot partition, but other partitions should be supported.

The other enhancements that might be possible are (not in priority order):

user interface

  • customise colours
  • add "back" and "forward" menu options
  • run parts of the installer independently (e.g. disk partitioning, set installation)

cgd enhancements

  • add the ability to encrypt the whole disk and to enter the decryption key at boot time

automated test setup

  • add the ability to install Anita for automated testing

The candidate must have:

  • familiarity with the system installer. You should have used sysinst to install the system.
  • familiarity with C programming. The system installer program consists of C code.
  • a test system, preferably with a 2nd bootable device.

The following would also be useful:

  • familiarity with NetBSD.
  • familiarity with user interface programming using curses.

References:

Posted late Saturday night, November 6th, 2011 Tags:

IMPORTANT: This project was completed by various NetBSD developers. You may still contact the people above for details, but please do not submit an application for this project.

sysinst, the NetBSD installation tool, should be able to get NetBSD set up with most packages relevant to the end user during the installation step, to make it possible to get a usable system during the initial setup. The packages might be simple packages like screen or bigger ones like firefox. Configuration of the packages is not required to happen in sysinst.

A short overview of the milestones involved:

  • Improve sysinst so it can list a pkgsummary.gz file on the install media and offer to install a subset of them. There should be a chooser included in sysinst where the user can select the packages they want to install.

  • There should be some pre-defined sets included with sysinst that define the necessary packages for a Gnome or KDE desktop, so the user just chooses them (possibly just by adding appropriate meta packages to pkgsrc).

  • Support fetching a pkgsummary file from a remote host (usually ftp.NetBSD.org with useful default PATH for architecture/release, but overridable) and offer installations of packages from there as well.

For bonus points (just as last step when the rest works): * Come up with lists of packages for amd64 to fill a CD or DVD (including the NetBSD base system)

Posted late Saturday night, November 6th, 2011 Tags:

pkgsrc is a very flexible package management system. It provides a comprehensible framework to build, test, deploy, and maintain software in its original form (with porter/packager modifications where applicable) as well as with site local modifications and customizations. All this makes pkgsrc suitable to use in diverse environments ranging from small companies up to large enterprises.

While pkgsrc already contains most elements needed to build an authentication server (or an authentication server failover pair), in order to install one, considerable knowledge about the neccessary elements is needed, plus the correct configuration, while in most cases pretty much identical, is tedious and not without pitfalls.

The goal of this project is to create a meta-package that will deploy and pre-configure an authentication server suitable for a single sign-on infrastructure.

Necessary tasks: provide missing packages, provide packages for initial configuration, package or create corresponding tools to manage user accounts, document.

The following topics should be covered:

  • PAM integration with OpenLDAP and DBMS;
  • Samba with PAM, DBMS and directory integration;
  • Kerberos setup;
  • OpenLDAP replication;
  • DBMS (PostgreSQL is a must, MySQL optional, if time permits), replication (master-master, if possible);
  • DNS server with a sane basic dynamic DNS update config using directory and database backend;
  • user account management tools (web interface, command line interface, see user(8) manual page, perhaps some scripting interface);
  • configuration examples for integration of services (web services, mail, instant messaging, PAM is a must, DBMS and directory optional).

All covered services should be documented, in particular documentation should include:

  • initial deployment instructions;
  • sample configuration for reasonably simple case;
  • instructions how to test if the software works;
  • references to full documentation and troubleshooting instructions (if any), both should be provided on-site (i.e. it should be possible to have everything available given pkgsrc snapshot and corresponding distfiles and/or packages on physical media).

In a nutshell, the goal of the project is to make it possible for a qualified system administrator to deploy basic service (without accounts) with a single pkg_add.

Posted late Saturday night, November 6th, 2011 Tags:

All architectures suffer from code injection issues because the only writable segment is the PLT/GOT. RELRO (RELocation Read Only) is a mitigation technique that is used during dynamic linking to prevent access to the PLT/GOT. There is partial RELRO which protects that GOT but leaves the PLT writable, and full RELRO that protects both at the expense of performing a full symbol resolution at startup time. The project is about making the necessary modifications to the dynamic loader (ld_elf.so) to make RELRO work.

If that is completed, then we can also add the following improvement: Currently kernels with options PAX_MPROTECT can not execute dynamically linked binaries on most RISC architectures, because the PLT format defined by the ABI of these architectures uses self-modifying code. New binutils versions have introduced a different PLT format (enabled with --secureplt) for alpha and powerpc.

Milestones:

  • For all architectures we can improve security by implementing relro2.
  • Once this is done, we can improve security for the RISC architectures by adding support for the new PLT formats introduced in binutils 2.17 and gcc4.1 This will require changes to the dynamic loader (ld.elf_so), various assembly headers, and library files.
  • Support for both the old and new formats in the same invocation will be required.

Status: * Added support to the dynamic loader (ld.elf_so) to handle protecting the GNU relro section. * Enabled partial RELRO by default on x86.

Posted late Saturday night, November 6th, 2011 Tags:

Using kernel virtualization offered by rump it is possible to start a virtually unlimited amount of TCP/IP stacks on one host and interlink the networking stacks in an arbitrary fashion. The goal of this project is to create a visual GUI tool for creating and managing the networks of rump kernel instances. The main goal is to support testing and development activities.

The implementation should be split into a GUI frontend and a command-line network-control backend. The former can be used to design and edit the network setup, while the latter can be used to start and monitor the virtual network from e.g. automated tests.

The GUI frontend can be implemented in any language that is convenient. The backend should be implemented in a language and fashion which makes it acceptable to import into the NetBSD base system.

The suggested milestones for the project are:

  1. get familiar and comfortable with starting and stopping rump kernels and configuring virtual network topologies
  2. come up with an initial backend command language. It can be something as simple as bourne shell functions.
  3. come with an initial specification and implementation of the GUI tool.
  4. make the GUI tool be able to save and load network specifications.
  5. create a number of automated tests to evaluate the usefulness of the tool in test creation.

In case of a capable student, the project goals can be set on a more demanding level. Examples of additional features include:

  • defining network characteristics such as jitter and packet loss for links
  • real-time monitoring of network parameters during a test run
  • UI gimmicks such as zooming and node grouping
Posted late Saturday night, November 6th, 2011 Tags:

IMPORTANT: This project was completed by Vlad Balan. You may still contact the people above for details, but please do not submit an application for this project.

When using connect(2) to connect the client end of a socket, the system will choose the next number of the socket for you. Having an easily guessed port number can allow various attacks to take place. Choosing the next port number at random, whilst not perfect, gives more protection against these attacks. RFC 6056 gives an excellent overview of the algorithms in use when "randomising source ports", giving examples from FreeBSD, OpenBSD and Linux.

This project has a number of goals:

  • Evaluate and prioritise the algorithms in RFC 6056.
  • Implement the algorithms in RFC 6056, and make it possible to choose between them with sysctl.
Posted late Saturday night, November 6th, 2011 Tags:

IMPORTANT: This project was completed by Vyacheslav Matyushin. You may still contact the people above for details, but please do not submit an application for this project.

As is well-known, puffs(3) is the NetBSD userspace file system framework. It provides support for implementing file servers in userspace. A lesser known "cousin" of puffs is the Pass-to-Userspace Device, or pud(4) framework, which provides support for implementing character and block device servers in userspace. Both use putter(9) for transmitting requests to and from the kernel.

Currently, puffs includes a userspace support library: libpuffs. It provides two facets:

  • file system routines and callback interface
  • generic parts, including kernel request handling

On the other hand, pud is without a userspace support library and servers talk with kernel directly with read() and write().

The goal of the project is to modify libpuffs into a generic library which pud and puffs can share, and provide libpuffs and libpud built on this base. The submission should include a rough analysis of the source modules of libpuffs and what is going to happen to them during the project.

This project is fairly straightforward, but involves a reasonable amount of work. Plenty of documentation exists to make the learning curve manageable. This project is an excellent opportunity to practise getting your hands dirty with real world systems.

Posted late Saturday night, November 6th, 2011 Tags:

IMPORTANT: This project was completed by Anton Panev. You may still contact the people above for details, but please do not submit an application for this project.

In 2006 and 2007, the pkgsrc build system was abstracted to allow packaging for other package system packages. For details see pkgsrc/mk/flavor/README and the original commit message.

This means pkgsrc build infrastructure may be used to potentially create packages that may be installed using non-NetBSD packaging tools (i.e. not using NetBSD's pkg_add). Note: this is not about cross-builds; the packages may be built on an appropriate host system using the pkgsrc collection.

This project may include creating shell command wrappers to mimic pkgsrc build needs as documented in README. (The wrappers only needed for building packages and not for using packages.) Also the project may include implementing package-specific make targets as documented in README. Also see suggestions to do in the initial commit message.

The goals of this project include:

  • Add support for RPM, dpkg, SVR4, PC-BSD PBI, and/or the Solaris native package system(s).
  • Be able to build at least 100 packages and demonstrate that the packages can be installed and de-installed using the corresponding system's native package tools.
  • Document support and interaction with existing packages.
Posted late Saturday night, November 6th, 2011 Tags:

IMPORTANT: This project was completed by Johnny C. Lam. You may still contact the people above for details, but please do not submit an application for this project.

Instead of including install scripts from the infrastructure into every binary package, just include the necessary information and split the scripts off into a separate package that is installed first (right after bootstrap, as soon as the first package needs it). This affects user creation, installation of tex packages, ...

Milestones:

  • identify example packages installing users, groups, and documentation
  • demonstrate pkgsrc packages which add users, etc
  • Also add support for actions that happen once after a big upgrade session, instead of once per package (e.g. ls-lR rebuild for tex).
  • convert some existing packages to use this new framework
  • allow options framework to configure these resources per-package

An intermediate step would be to replace various remaining INSTALL scripts by declarative statements and install script snippets using them.

This was implemented via the new pkgtasks framework that can be enabled by setting in mk.conf:

_USE_NEW_PKGINSTALL=    yes
Posted late Saturday night, November 6th, 2011 Tags:

Put config files (etc/) installed by pkgsrc into some version control system to help keeping track of changes and updating them.

The basic setup might look like this:

  • There is a repository containing the config files installed by pkgsrc, starting out empty.
  • During package installation, pkgsrc imports the package's config files into the repository onto a branch tagged with the name and version of the package (if available, on a vendor branch). (e.g.: digest-20080510)
  • After installation, there are two cases:

    1. the package was not installed before: the package's config files get installed into the live configuration directory and committed to the head of the config repository
    2. the package was installed before: a configuration update tool should display changes between the new and the previous original version as well as changes between the previous original and installed config file, for each config file the package uses, and support merging the changes made necessary by the package update into the installed config file. Commit the changes to head when the merge is done.
  • Regular automated check-ins of the entire live pkgsrc configuration should be easy to set up, but also manual check-ins of singular files so the local admin can use meaningful commit messages when they change their config, even if they are not experienced users of version control systems

The actual commands to the version control system should be hidden behind an abstraction layer, and the vcs operations should be kept simple, so that other compatibility layers can be written, and eventually the user can pick their vcs of choice (and also a vcs location of choice, in case e.g. the enterprise configuration repository is on a central subversion server).

milestones/goals:

  • choose a VCS system (BSD licensed is a nice-to-have)
  • write wrappers around it, or embed its functionality
  • demonstrate usage in upgrades

bonus:

  • extend functionality into additional VCS systems

This project was done during Google Summer of Code 2018 by Keivan Motavalli Configuration files versioning in pkgsrc project. At the moment the code need to be reviewed and imported in pkgsrc.

Posted late Saturday night, November 6th, 2011 Tags:

Change infrastructure so that dependency information (currently in buildlink3.mk files) is installed with a binary package and is used from there

This is not an easy project.

pkgsrc currently handles dependencies by including buildlink3.mk files spread over pkgsrc. The problem is that these files describe the current state of pkgsrc, not the current state of the installed packages.

For this reason and because most of the information in the files is templated, the buildlink3.mk files should be replaced by the relevant information in an easily parsable manner (not necessarily make syntax) that can be installed with the package. Here the task is to come up with a definitive list of information necessary to replace all the stuff that's currently done in buildlink3.mk files (including: dependency information, lists of headers to pass through or replace by buildlink magic, conditional dependencies, ...)

The next step is to come up with a proposal how to store this information with installed packages and how to make pkgsrc use it.

Then the coding starts to adapt the pkgsrc infrastructure to do it and show with a number of trivial and non-trivial packages that this proposal works.

It would be good to provide scripts that convert from the current state to the new one, and test it with a bulk build.

Of course it's not expected that all packages be converted to the new framework in the course of this project, but the further steps should be made clear.

goals/milestones:

  • invent a replacement for buildlink3.mk files, keeping current features
  • demonstrate your new tool as a buildlink3.mk replacement including new features
  • execute a bulk build with as many packages as possible using the new buildlink
Posted late Saturday night, November 6th, 2011 Tags:

swcrypto could use a variety of enhanacements

Milestones/deliverables:

  • use multiple cores efficiently (that already works reasonably well for multiple request streams)
  • use faster versions of complex transforms (CBC, counter modes) from our in-tree OpenSSL or elsewhere (eg libtomcrypt)
  • add support for asymmetric operations (public key)

Extra credit:

  • Tie public-key operations into veriexec somehow for extra credit (probably a very good start towards an undergrad thesis project).
Posted late Saturday night, November 6th, 2011 Tags:

The current lpr/lpd system in NetBSD is ancient, and doesn't support modern printer systems very well. Interested parties would do a from scratch rewrite of a new, modular lpr/lpd system that would support both the old lpd protocol and newer, more modern protocols like IPP, and would be able to handle modern printers easily.

This project is not intrinsically difficult, but will involve a rather large chunk of work to complete.

Note that the goal of this exercise is not to reimplement cups -- cups already exists and one like it was enough.

Some notes:

It seems that a useful way to do this would be to divide the printing system in two: a client-side system, which is user-facing and allows submitting print jobs to arbitrary print servers, and a server-side system, which implements queues and knows how to talk to actual printer devices. In the common case where you don't have a local printer but use printers that are out on the network somewhere, the server-side system wouldn't be needed at all. When you do have a local printer, the client-side system would submit jobs to the local server-side system using the lpr protocol (or IPP or something else) over a local socket but otherwise treat it no differently from any other print server.

The other important thing moving forward: lpr needs to learn about MIME types and accept an argument to tell it the MIME types of its input files. The current family of legacy options lpr accepts for file types are so old as to be almost completely useless; meanwhile the standard scheme of guessing file types inside the print system is just a bad design overall. (MIME types aren't great but they're what we have.)

Posted late Saturday night, November 6th, 2011 Tags:

IMPORTANT: This project was completed by Brett Lymn. You may still contact the people above for details, but please do not submit an application for this project.

Updating an operating system image can be fraught with danger, an error could cause the system to be unbootable and require significant work to restore the system to operation. The aim of this project is to permit a system to be updated while it is running only requiring a reboot to activate the updated system and provide the facility to rollback to a "known good" system in the event that the update causes regressions.

Milestones for this project:

  • Make a copy of the currently running system
  • Either apply patches, install binary sets, run a source build with the copy as the install target
  • Update critical system files to reference the copy (things like fstab)
  • Update the boot menu to make the copy the default boot target, the current running system should be left as an option to boot to

The following link shows how live upgrade works on Solaris: http://docs.oracle.com/cd/E19455-01/806-7933/overview-3/index.html The aim is to implement something that is functionally similar to this which can not only be used for upgrading but for any risky operation where a reliable back out is required.

Posted late Saturday night, November 6th, 2011 Tags:

Write a library of routines that estimate the expected transmission duration of a queue of 802.3 or 802.11 frames based on the current bit rate, applicable overhead, and the backoff level. The kernel will use this library to help it decide when packet queues have grown too long in order to mitigate "buffer bloat."

Posted late Saturday night, November 6th, 2011 Tags:

IMPORTANT: This project was completed by William Morr. You may still contact the people above for details, but please do not submit an application for this project.

NetBSD provides a BSD licensed implementation of libintl. This implementation is based on the specifications from GNU gettext. It has not kept up with the development of gettext in the last few years though, lacking e.g. support for context specific translations. NetBSD currently also has to depend on GNU gettext to recreate the message catalogs.

Milestones for this project include:

  • Restore full API compatibility with current gettext. At the time of writing, this is gettext 0.18.1.1.
  • Implement support for the extensions in the message catalog format. libintl should be able to process all .mo files from current gettext and return the same results via the API.
  • Provide a clean implementation of msgfmt.
  • Other components of gettext like msgmerge and the gettext frontend should be evaluated case-by-case if they are useful for the base system and whether third party software in pkgsrc depends on them.
  • demonstrate the elimination of GNU gettext dependencies
Posted late Saturday night, November 6th, 2011 Tags:

IMPORTANT: This project was completed by Naman Jain. You may still contact the people above for details, but please do not submit an application for this project.

Update: This project was completed by Naman Jain during Google Summer of Code 2020 and merged into NetBSD. More tests are still needed for completion's sake.

The curses library is an important part of the NetBSD operating system, many applications rely on the correct functioning of the library. Performing modifications on the curses library can be difficult because the effects of the change may be subtle and can introduce bugs that are not detected for a long time.

The testing framework has been written to run under the atf framework but has not been committed to the tree yet.

The student undertaking this project will be provided with the testing framework and will use this to generate test cases for curses library calls. Most of the work will require analytical skills to verify the output of the test is actually correct before encapsulating that output into a validation file.

Milestones for this project:

  • produce a suite of high quality tests for the curses library
  • These tests should exercise every aspect of the library functionality.

This project will need a good understanding of the curses library and will provide the student with a much deeper understanding of the operation of curses.

Posted late Saturday night, November 6th, 2011 Tags:

Design and implement a general API for control of LED and LCD type devices on NetBSD. The API would allow devices to register individual LED and LCD devices, along with a set of capabilities for each one. Devices that wish to display status via an LED/LCD would also register themselves as an event provider. A userland program would control the wiring of each event provider, to each output indicator. The API would need to encompass all types of LCD displays, such as 3 segment LCDs, 7 segment alphanumerics, and simple on/off state LED's. A simple example is a keyboard LED, which is an output indicator, and the caps-lock key being pressed, which is an event provider.

There is prior art in OpenBSD; it should be checked for suitability, and any resulting API should not differ from theirs without reason.

Milestones:

  • a port of OpenBSD's LED tools
  • a userland tool to control LED
  • demonstration of functionality
  • integration into NetBSD
Posted late Saturday night, November 6th, 2011 Tags:

Launchd is a MacOS/X utility that is used to start and control daemons similar to init(8), but much more powerful. There was an effort to port launchd to FreeBSD, but it seems to be abandoned. We should first investigate what happened to the FreeBSD effort to avoid duplicating work. The port is not trivial because launchd uses a lot of mach features.

Milestones:

  • report of FreeBSD efforts (past and present)
  • launchd port replacing: init
  • launchd port replacing: rc
  • launchd port compatible with: rc.d scripts
  • launchd port replacing: watchdogd

Nice to have:

  • launchd port replacing/integrating: inetd
  • launchd port replacing: atd
  • launchd port replacing: crond
  • launchd port replacing: (the rest)
Posted late Saturday night, November 6th, 2011 Tags:

IMPORTANT: This project was completed by Vlad Balan. You may still contact the people above for details, but please do not submit an application for this project.

Many applications that use UDP unicast or multicast to receive data, need to store the data together with its reception time, or time the arrival time of packets as precisely as possible. This is required for example in order to be able to replay the data in simulated real-time to be able to do further performance analysis or quality control. Right now the only way to do this is to call recv(2)/recvmsg(2)/recvfrom(2) to grab the data, followed by gettimeofday(2)/clock_gettime(2). This is undesirable because:

  • It doubles the number of system calls limiting the effective maximum reception rate.
  • The timing of the packet is imprecise because there non-deterministic delay from the time that the packet hit the network adapter till the time userland received it.

Linux provides a socket control message to add timestamp ancillary data to the the socket, which can be controlled by 2 different socket options SO_TIMESTAMP and SO_TIMESTAMPNS, using setsockopt(2). Recently linux added more SO_TIMESTAMPING_* options to provide even more precise timestamps from NIC cards that support them. This project is about implementing those options on NetBSD. Their implementation should be done as close to the code that does reception as possible to minimize timestamp delay. The SO_TIMESTAMP option is already done in the kernel (perhaps only partially): kern, ip, raw, udp, ip6, sys. A good reading would also be timestamping.c

This project is to finish implementing SO_TIMESTAMP in the kernel, test it, add SO_TIMESTAMPNS (both very easy, should be a few hours of work), and then design and implement the SO_TIMESTAMPING_* options, get at least one driver working with them and test them.

Posted late Saturday night, November 6th, 2011 Tags:

Produce lessons and a list of affordable parts and free software that NetBSD hobbyists can use to teach themselves JTAG. Write your lessons for a low-cost embedded system or expansion board that NetBSD already supports.

Posted late Saturday night, November 6th, 2011 Tags:

inetd is a classic method for launching network programs on-the-fly and some of its ideas are coming back into vogue. Enhancing this daemon should include investigations into other similar systems in other operating systems.

Primary milestones:

  • Prefork: Support pre-forking multiple children and keeping them alive for multiple invocations.
  • Per service configuration file: Add a per-service configuration file similar to xinetd.
  • Make the rate-limiting feature configurable on a per-service basis.
  • Improve the logging and make logging modes configurable on a per-service basis.

Nice to have:

  • Add include directives to the configuration language to allow service definitions to be installed in /usr/share or /usr/pkg/share.
  • Add a separate way to turn services on and off, so they can be defined statically (such as in /usr/share) and turned on and off from /etc.
  • Allow non-privileged users to add/remove/change their own services using a separate utility.
  • Integrate with the new blocklist daemon.
  • Configuration compatibility for systemd socket activations
Posted late Saturday night, November 6th, 2011 Tags:

IMPORTANT: This project was completed by Dmitry Matveev. You may still contact the people above for details, but please do not submit an application for this project.

Monitoring of file system activity is a key piece of a modern desktop environment because it provides instant feedback to the user about any changes performed to the disk. In particular, file monitoring is an internal component of the Gnome infrastructure that allows the desktop to receive notifications when files or directories change. This way, if, say, you are viewing the Downloads folder in Nautilus and you start downloading a file from Epiphany into that folder, Nautilus will realize the new file and show it immediately without requiring a manual refresh.

How to monitor the file system depends on the operating system. There are basically two approaches: polling and asynchronous notifications. Polling is suboptimal because the notifications are usually delayed. Asynchronous notifications are tied to the operating system: Linux provides inotify, NetBSD provides kqueue and other systems provide their own APIs.

In the past, Gnome monitored the file system via a combination of FAM (a system-level service that provides an API to file system monitoring) and GNOME VFS (a high-level layer that hides the interaction with FAM). This approach was good in spirit (client/server separation) but didn't work well in NetBSD:

  • FAM is abandoned.
  • Does not support kqueue out of the box.
  • FAM runs as root.
  • FAM is too hard to set up: it requires rpcbind, an addition to etc/services, a sysctl tweak, and the configuration of a system-level daemon.

To solve some of these problems, a drop-in replacement for FAM was started. Gamin, as it is known, still does not fix everything:

  • Gamin is abandoned.
  • Supports kqueue out of the box, but does not work very well.
  • Actually, Gamin itself does not work. Running the tests provided by the distfile in a modern Linux system results in several test failures.

Did you notice the abandoned pattern above? This is important: in the new world order, Gnome does not use FAM any more.

The new structure to monitor files is: the low-level glib library provides the gio module, which has a new file system monitoring API. The GVFS module provides higher level abstractions to file system management, and relies on gio for file system monitoring. There is no GNOME VFS any more.

The problematic point is: gio uses inotify directly; no abstraction layers in between. FAM support is still there for platforms without inotify, but as it is not used in Linux any more, both the FAM package and the support for it rot.

The goal of this project is to fix this situation. There are two possible approaches. The first is to extend the gio library with a new module to support kqueue, the native interface in NetBSD for file monitoring; this new module can be submitted upstream and will benefit all other BSD systems, but it will not help other applications that use inotify. The second is to write a compatibility library that exposes an inotify interface on top of kqueue; this cannot be sent upstream but it helps any application using inotify.

The preferred method is to write a new gio plugin to support kqueue. Henceforth, the deliverables for the project include a new gio module to support kqueue and, if time permits, a standalone library that implements an inotify interface on top of kqueue.

Assuming you are not familiar with kqueue, you might want to follow these steps to get started:

  1. Read the kqueue(2) manual page to get a general idea of the API and this example for starters.
  2. Analyze the kqueue support for FAM which is in pkgsrc/sysutils/fam/files/IMonKQueue.c++. This might be helpful to see how to apply kqueue to monitor for the events used by GNOME.
  3. Read the modules/inotify* files in gnome-vfs and inspect how they are used in the "file method". The modules/Makefile.am contains a list of related sources in the libfile_la_SOURCES variable.
  4. Possibly the hardest part: write the required stuff (modules/kqueue*) to add kqueue support.
Posted late Saturday night, November 6th, 2011 Tags:

The existing puffs protocol gives a way to forward kernel-level file system actions to a userspace process. This project generalizes that protocol to allow forwarding file system actions arbitrarily across a network. This will make it possible to mount any kernel file system type from any location on the network, given a suitable arrangement of components.

The file system components to be used are puffs and rump. puffs is used to forward local file system requests from the kernel to userspace and rump is used to facilitate running the kernel file system in userspace as a service daemon.

The milestones are the following:

  • Write the necessary code to be able to forward requests from one source to another. This involves most likely reworking a bit of the libpuffs option parsing code and creating a puffs client (say, mount_puffs) to be able to forward requests from one location to another. The puffs protocol should be extended to include the necessary new features or a new protocol invented.

    Proof-of-concept code for this has already been written. (Where is it?)

  • Currently the puffs protocol used for communication between the kernel and userland is machine dependent. To facilitate forwarding the protocol to remote hosts, a machine independent version must be specified.

  • To be able to handle multiple clients, the file systems must be converted to daemons instead of being utilities. This will also, in the case of kernel file system servers, include adding locking to the communication protocol.

The end result will look something like this:

# start serving ffs from /dev/wd0a on port 12675
onehost> ffs_serv -p 12675 /dev/wd0a
# start serving cd9660 from /dev/cd0a on port 12676
onehost> cd9660_serv -p 12675 /dev/cd0a

# meanwhile in outer space, mount anotherhost from port 12675
anotherhost> mount_puffs -t tcp onehost:12675 /mnt
anotherhost> mount
...
anotherhost:12675 on /mnt type <negotiated>
...
anotherhost> cd /mnt
anotherhost> ls
  ... etc

The implementor should have some familiarity with file systems and network services.

Any proposal should include answers to at least the following questions:

  • How is this different from NFS?

  • How is the protocol different from 9p?

  • How is this scheme going to handle the hard things that NFS doesn't do very well, such as distributed cache consistency?

  • Given industry trends, why is this project proposing a new protocol instead of conforming to the SOA model?

Posted late Saturday night, November 6th, 2011 Tags:

IMPORTANT: This project was completed by Przemyslaw Sierocinski. You may still contact the people above for details, but please do not submit an application for this project.

This project requires the implementation of a new mount option, and a new system and user file system flag, which, when set, will write random data over file system blocks before they are to be deleted. In the NetBSD kernel, this will take place at the time of the last unlink of a file, and when ftruncate is called.

The project will involve the investigation of retrieving or generating random data within the kernel, along with research into ways of retrieving large amounts of low-quality random data, such as LSFR, Mersenne twisters, and PRNGs. As well as implementing the file system flags within the kernel, user-level programs and library functions which manipulate the flags will need to be modified. Documentation detailing the new functionality must also be provided.

Posted late Saturday night, November 6th, 2011 Tags:

Design and implement a mechanism that allows for fast user level access to kernel time data structures for NetBSD. For certain types of small data structures the system call overhead is significant. This is especially true for frequently invoked system calls like clock_gettime(2) and gettimeofday(2). With the availability of user level readable high frequency counters it is possible to create fast implementations for precision time reading. Optimizing clock_gettime(2) and alike will reduce the strain from applications frequently calling these system calls and improves timing information quality for applications like NTP. The implementation would be based on a to-be-modified version of the timecounters implementation in NetBSD.

Milestones:

  • Produce optimizations for clock_gettime
  • Produce optimizations for gettimeofday
  • Show benchmarks before and after
  • start evolving timecounters in NetBSD, demonstrating your improvements

See also the Paper on Timecounters by Poul-Henning Kamp.

Posted late Saturday night, November 6th, 2011 Tags:

Write a program that can read an email, infer quoting conventions, discern bottom-posted emails from top-posted emails, and rewrite the email using the conventions that the reader prefers. Then, take it a step further: write a program that can distill an entire email discussion thread into one document where every writer's contribution is properly attributed and appears in its proper place with respect to interleaved quotations.

Posted late Saturday night, November 6th, 2011 Tags:
  • Contact: tech-kern
  • Duration estimate: 3 months

Make NetBSD behave gracefully when a "live" USB/FireWire disk drive is accidentally detached and re-attached by, for example, creating a virtual block device that receives block-read/write commands on behalf of the underlying disk driver.

This device will delegate reads and writes to the disk driver, but it will keep a list of commands that are "outstanding," that is, reads that the disk driver has not completed, and writes that have not "hit the platter," so to speak.

Milestones:

  • Provide a character device for userland to read indications that a disk in use was abruptly detached.
  • Following disk re-attachment, the virtual block device replays its list of outstanding commands. A correct solution will not replay commands to the wrong disk if the removable was replaced instead of re-attached.

Open questions: Prior art? Isn't this how the Amiga worked? How will this interact with mount/unmount—is there a use-count on devices? Can you leverage "wedges" in your solution? Does any/most/all removable storage indicate reliably when a block written has actually reached the medium?

Posted late Saturday night, November 6th, 2011 Tags:

pkgsrc duplicates NetBSD efforts in maintaining X.org packages. pkgsrc X11, being able to cross-build, could replace base X11 distribution.

The latter decoupling should simplify maintainance of software; updating X11 and associated software becomes easier because of pkgsrc's shorter release cycle and more volatility.

Cross-buildable and cross-installable pkgsrc tools could also simplify maintainance of slower systems by utilising power of faster ones.

The goal of this project is to make it possible to bootstrap pkgsrc using available cross-build tools (e.g. NetBSD's).

This project requires good understanding of cross-development, some knowledge of NetBSD build process or ability to create cross-development toolchain, and familiarity with pkgsrc bootstrapping.

Note: basic infrastructure for this exists as part of various previous GSoC projects. General testing is lacking.

Posted late Saturday night, November 6th, 2011 Tags:
  • Contact: tech-pkg
  • Duration estimate: 3 months

The goal of this project is to generate a package or packages that will set up a cross-compiling environment for one (slow) NetBSD architecture on another (fast) NetBSD architecture, starting with (and using) the NetBSD toolchain in src.

The package will require a checked out NetBSD src tree (or as a refinement, parts thereof) and is supposed to generate the necessary cross-building tools using src/build.sh with appropriate flags; for the necessary /usr/include and libraries of the slow architecture, build these to fit or use the slow architecture's base.tgz and comp.tgz (or parts thereof). As an end result, you should e.g. be able to install the binary cross/pkgsrc-NetBSD-amd64-to-atari package on your fast amd64 and start building packages for your slow atari system.

Use available packages, like eg pkgtools/pkg_comp to build the cross-compiling environment where feasible.

As test target for the cross-compiling environment, use pkgtools/pkg_install, which is readily cross-compilable by itself.

If time permits, test and fix cross-compiling pkgsrc X.org, which was made cross-compilable in an earlier GSoC project, but may have suffered from feature rot since actually cross-compiling it has been too cumbersome for it to see sufficient use.

Posted late Saturday night, November 6th, 2011 Tags:

Write a script that aids refactoring C programs by extracting subroutines from fragments of C code.

Do not reinvent the wheel: wherever possible, use existing technology for the parsing and comprehension of C code. Look at projects such as sparse and Coccinelle.

Your script should work as a filter that puts the free variables at the top, encloses the rest in curly braces, does something helpful with break, continue, and return statements.

That's just a start.

This project is tagged "hard" because it's not clearly specified. The first step (like with many projects) is to work out the specification in more detail.

Posted late Saturday night, November 6th, 2011 Tags:

Currently, the puffs(3) interface between the kernel and userspace uses various system structures for passing information. Examples are struct stat and struct uucred. If these change in layout (such as with the time_t size change in NetBSD 6.0), old puffs servers must be recompiled.

The project milestones are:

  • define a binary-independent protocol
  • implement support
  • measure the performance difference with direct kernel struct passing
  • if there is a huge difference, investigate the possibility for having both an internal and external protocol. The actual decision to include support will be made on the relative complexity of the code for dual support.

While this project will be partially implemented in the kernel, it is fairly well-contained and prior kernel experience is not necessary.

If there is time and interest, a suggested subproject is making sure that p2k(3) does not suffer from similar issues. This is a required subproject if dual support as mentioned above is not necessary.

Posted late Saturday night, November 6th, 2011 Tags:

Refactor utilities in the base system, such as netstat, top, and vmstat, that format & display tables of statistics.

One possible refactoring divides each program into three:

  • one program reads the statistics from the kernel and writes them in machine-readable form to its standard output
  • a second program reads machine-readable tables from its standard input and writes them in a human-readable format to the standard output
  • and a third program supervises a pipeline of the first two programs, browses and refreshes a table.

Several utilities will share the second and third program.

Posted late Saturday night, November 6th, 2011 Tags:

IMPORTANT: This project was completed by riz. You may still contact the people above for details, but please do not submit an application for this project.

Amazon Web Services offers virtual hosts on demand via its Amazon Elastic Comput Cloud (or EC2). Users can run a number of operating systems via so-called "Amazon Machine Images" (AMIs), currently including various Linux flavors and OpenSolaris, but not NetBSD.

This project would entail developing a NetBSD "AMI" and providing an easy step-by-step guide to create custom NetBSD AMIs. The difficulty of this is estimated to not be very high, as prior work in this direction exists (instructions for creating a FreeBSD AMI, for example, are available). Depending on the progress, we may adjust the final goals -- possibilities include adding necessary (though optional) glue to the release process to create official AMIs for each release or to provide images with common applications pre-installed.

See also: this thread on the port-xen mailing list as well as this page.

Posted late Saturday night, November 6th, 2011 Tags:

IMPORTANT: This project was completed by Abhinav Upadhyay. You may still contact the people above for details, but please do not submit an application for this project.

NetBSD ships a lot of useful documentation in the form of manual pages. Finding the right manual page can be difficult though. If you look for a library function, it will sometimes fail, because it is part of a larger manual page and doesn't have a MLINKS entry. If you look for a program, but don't know the exact name, it can be hard to find as well.

Historically, the content of the NAME section of each manual page has been extracted and put into a special file. The apropos command has been used to search this file based on keywords. This brings severe limitations as it restricts the list of potential matches significantly and requires very good descriptions of the content of a manual page in typically one line.

The goal of this project is to provide a modern replacement based on the Full Text Search of SqLite. The most basic version of the new apropos builds an index from the text output of mandoc and queries it using appropriate SQL syntax. Some basic form of position indications should be provided as well (e.g. line number).

A more advanced version could use the mandoc parser directly too. This would easily allow relatively precise position marks for the HTML version of manual pages. It would also allow weighting the context of a word. Consider Google's preference of URLs that contain the keywords or documents containing them in the head lines as an example.

Another idea is to use the index for directly handling manual page aliases. This could replace the symbolic links currently used via the MLINKS mechanism. The aliases can be derived automatically from the .Nm macros in the manual page.

Posted late Saturday night, November 6th, 2011 Tags:

IMPORTANT: This project was completed by utkarsh009. You may still contact the people above for details, but please do not submit an application for this project.

Anita is a tool for automated testing of NetBSD. Anita automates the process of downloading a NetBSD distribution, installing it into a fresh virtual machine and running the ATF tests in the distribution in a fully-automated manner. Originally, the main focus of Anita was on testing the sysinst installation procedure and on finding regressions that cause the system to fail to install or boot, but Anita is now used as the canonical platform for running the ATF test suite distributed with NetBSD. (You can see the results of such tests in the Test Run Logs page.)

At the moment, Anita supports qemu and Xen as the system to create the virtual machine with. qemu gives us the ability to test several ports of NetBSD (because qemu emulates many different architectures), but qemu is very slow because it lacks hardware virtualization support in NetBSD. The goal of this project is to extend Anita to support other virtual machine systems.

There are many virtual machine systems out there, but this project focuses on adding support to at least VirtualBox, because it is the canonical free virtual machine system for workstation setups. Another possibility would be adding support for SIMH for running NetBSD/vax.

The obvious deliverable is a new version of Anita that can use any of the mentioned virtual machines at run time, without having to be rebuilt, and the updated pkgsrc packages to install such updated version.

The candidate is supposed to know Python and be familiar with at least one virtual machine system.

Posted late Saturday night, November 6th, 2011 Tags:

This page includes a brainstorming of ideas for project proposals. While most, if not all, of the projects in this page are good ideas to propose to potential applicants, the ideas themselves have not received enough attention to write up a project specification for them. Any project considered good enough to be thrown at applicants should get a page of its own within projects/project/* and get a well-defined write-up explaining what the project is about, why it is important for NetBSD, and what is expected from the applicant.

Kernel Projects

  1. related to (2) - rewrite sysctl to be less difficult to use

  2. Implement SYN cookies, stretch goal - extended syn cookies with timestamp usage

  3. difficult a QoS implementation for NetBSD in the year 2011

  4. Debug, and merge the iscsi initiator - stretch goal - get rid of the weird portal stuff which we don't need

  5. update linux emulation

    Update to what it was updated not long time ago to 2.6.18 kernel.
    Indeed, and RHEL 5 uses a modified 2.6.18 kernel. But RHEL 5 has been available since March 15, 2007, and is due to be "End of Production 1" in Q4 2011 -- agc

  6. difficult investigate ffs dir hashing, and fix related bugs (hard)

    I implemented a debugged generic dir hashing. It could be extended to also allow FFS, but FFS's directory stuff is all over the place and i fear to break it. Also FFS directory packing needs attention. -- Reinoud

  7. difficult single instance storage and de-duplicating (might be too long?)

    This might be possible to implement on top of device-mapper driver.
    I was thinking that Single Instance Storage and de-duplication really needs access to userspace, and is best implemented there -- agc

  8. difficult get some form of jails up and running

    Maybe restoring MULT to working state is an option here, another possibility is to get gaols working. Some fresh approach to jails might be implemented on RUMP because it basically allow us to create in process virtual machine and run them independently.

  9. add shred/wiping and memtest options to boot menu

  10. add ability to store entropy at shutdown time, and to boot and restore same entropy. at same time, add a monitor (chi2?) for randomness over small sample sizes to /dev/random. investigate other ways of generating pseudo random numbers - 2 mersenne twisters sampling at different speeds, and throwing away random interim values due to another re-keyed prng or better. i.e. beef up our random number generation.

  11. find a method to boot a boot.cfg entry once (and then fall back to the default method). This requires, especially, that the boot flip back to the default method if the once-off fails.

  12. difficult Write new DTrace provider for watching locks(lockdebug), userspace or any other from available ones http://wikis.sun.com/display/DTrace/Providers

  13. Change keyboard drivers to emit USB scancodes in event mode so for example ADB or Sun keyboards can coexist with things like USB keypads on the same mux and we don't need a separate InputDevice section in xorg.conf for each. This should be relatively easy.

  14. Port FreeBSD's DAHDI implementation to NetBSD, so that Asterisk on NetBSD will have hardware support. FreeBSD's DAHDI implementation is SMP, so this isn't a simple port. Also, just about every FreeBSD kernel API related to device drivers and modules are different from the NetBSD equivalent meaning that one needs to basically know both kernels.

  15. Merge all PC style floppy drivers to be similar to other "portable" drivers that use a bus frontend with a common chip backend (or possibly two: pseudo DMA and real DMA).

  16. Create a new HID framework so that we can merge the USB and Bluetooth HID drivers, also allowing creation of drivers to properly handle devices such as Apple Magic Mouse which use several [undeclared] reports.

  17. Improve tmpfs

  18. Update zfs pool to version 5000.

  19. Add MI interface for IEEE 802.3 Clause 45 MDIO access.

  20. IEEE 802.3az(Energy Efficiency Ethernet(EEE)) support.

  21. Add Support dot3 MIB.

Userspace Projects

  1. a Java implementation we can use across all netbsd platforms

  2. (controversial) a pcap-based port-knocking system, similar to pkgsrc/net/knock

  3. puffs-based cvsfs (there's one out there which relies on extattrs)

  4. fuse lowlevel

    Wasn't this already done ?
    ReFUSE does the upper layer of FUSE, and the excellent /dev/fuse does the device interface, but we are still lacking FUSE lowlevel functionality -- agc

  5. automatic setlist generation - see my post to tech-userlevel a few months ago for the code i used (and which was used to generate all the libisns entries, btw).

  6. Embedded World-Generator: a tool to build smaller distribution and using custom packages and software - aimed at the embedded market

  7. Device-mapper RUMP testing: Write device-mapper testing framework based on libdm and RUMP this should be able to allow us to develop new targets easily.

  8. RUMP ZFS testing: integrate and write ZFS testsuite for NetBSD.

  9. Update userspace build of ddb (/sbin/crash) add more architectures and more functions to live/post mortem kernel debugging.

  10. A new puffs(4) based auto-mount daemon which supports direct mounting (not via "/amd") and Solaris style auto-mounter maps.

  11. port valgrind to NetBSD - valgrind is a code instrumentation framework that can be used to find memory related bugs in programs as well as conduct performance analysis. Valgrind is written for linux, and recently there is has been a MacOS/X port. The code is very much linux specific, and we should talk to the maintainers before starting work on it to find out if they will accept a NetBSD port, and also provide some assistance with the design of a portability layer. We should also do some research to see if we can re-use the FreeBSD port. For previous attempt see http://vg4nbsd.berlios.de/

  12. Implement enough of libudev so that Gnome (KDE?) in pkgsrc can use it

  13. Stack Backtrace Library - Write a library that the kernel debugger and return_address(9) can use to walk down the call stack. Use the technique that David Laight recommends: TBD.

  14. Daily autotest system for ARM

pkgsrc projects

  1. Modular libtool: rewrite libtool replacement that is faster and more convenient to maintain, so that one can add compilers to it without rebuilding the whole package.

  2. Go back to previous pkgsrc install state: Write a tool to help with reverting the installation to some previous state. See pkg_tarup, pkg_chk.

  3. Add support for multi-packages (building multiple binary packages from single source).

    don't we do that already? (I guess this means: more info please :-) -- spz

    No, we don't. We don't produce more than one package during the single build.

  4. Running VirtualBox on NetBSD as a host.

Posted in the wee hours of Saturday night, November 6th, 2011

NetBSD participated successfully in the following Google's Summer of Code programs (see our results of 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023 )

This page contains a list of concrete suggestions for projects we would like to see applications for in the next Summer of Code. Note that they vary a lot in required skills and difficulty. We hope to get applications with a broad spectrum.

In addition, you may wish to discuss your proposal on IRC -- look for us on Libera.chat's #netbsd-code or for pkgsrc-related discussions, #pkgsrc. If you want to just meet the community, visit #netbsd.

We encourage you to come up with your own suggestions, if you cannot find a suitable project here. You can find more project ideas on the NetBSD projects page). These are not directly applicable to Summer-of-Code, but may serve as ideas for your own suggestions. You might find other ideas in src/doc/TODO and pkgsrc/doc/TODO.

Deadlines and directions for students' applications to the Google Summer-of-Code can be found on the Google pages.

Application process

To make the job of sorting out proposals and applications for NetBSD-related projects, e.g. in the Google Summer-of-Code, easier for us, there are a few questions that we would like to see answered.

If you are interested in working on any of the projects below, please contact the mailing list referenced on each item, and possibly answer as many questions from our project application guidelines as possible. The interested developers will be glad to respond to you there.

Please note that Google Summer-of-Code projects are a full (day-) time job.

A positive mid-term evaluation is only possible if usable code has been committed by that time. Make sure your schedule allows for this.

Kernel-level projects

Easy

Medium

Hard

Userland projects

Easy

Medium

Hard

Code Quality Improvement projects

Easy

Medium

Hard

pkgsrc projects

Easy

Medium

Hard

Other projects

Medium

Comments

We are trying to be fair; expect easy projects to require less knowledge and skill, but quite a bit of work.

Medium and hard projects are hard enough to qualify as practical part of a master's thesis (it'll qualify as thesis topic if you can add sufficient quality theoretical parts). We had the honor to mentor several in past GSoCs. Talk to your adviser(s) if and how you can claim academic credit for the project you do with us.

We have not yet failed a student who worked hard and actually talked (and listened) to their mentors and the community. If unexpected roadblocks make your project goals too hard to reach in the time given, the goals can be re-negotiated. They will not be for rampant slacking, though.

What we expect from contributors (both GSoC students and generally) is that they cooperate, that they are able to communicate (this will mean some English skills, sorry; we will try to support with mentors speaking the same contributor language if that can be a problem), and that they meet a minimum of good manners towards other people on our lists and other venues. Note that being a specific color, gender, nationality, religion, etc is not listed: If you are willing and able to contribute in a constructive manner, you are welcome.

Posted in the wee hours of Saturday night, November 6th, 2011

Traditionally, the NetBSD kernel code had been protected by a single, global lock. This lock ensured that, on a multiprocessor system, two different threads of execution did not access the kernel concurrently and thus simplified the internal design of the kernel. However, such design does not scale to multiprocessor machines because, effectively, the kernel is restricted to run on a single processor at any given time.

The NetBSD kernel has been modified to use fine grained locks in many of its different subsystems, achieving good performance on today's multiprocessor machines. Unfotunately, these changes have not yet been applied to the networking code, which remains protected by the single lock. In other words: NetBSD networking has evolved to work in a uniprocessor envionment; switching it to use fine-grained locked is a hard and complex problem.

This project is currently claimed

Funding

At this time, The NetBSD Foundation is accepting project specifications to remove the single networking lock. If you want to apply for this project, please send your proposal to the contact addresses listed above.

Due to the size of this project, your proposal does not need to cover everything to qualify for funding. We have attempted to split the work into smaller units, and you can submit funding applications for these smaller subtasks independently as long as the work you deliver fits in the grand order of this project. For example, you could send an application to make the network interfaces alone MP-friendly (see the work plan below).

What follows is a particular design proposal, extracted from an original text written by Matt Thomas. You may choose to work on this particular proposal or come up with your own.

Tentative specification

The future of NetBSD network infrastructure has to efficiently embrace two major design criteria: Symmetric Multi-Processing (SMP) and modularity. Other design considerations include not only supporting but taking advantage of the capability of newer network devices to do packet classification, payload splitting, and even full connection offload.

You can divide the network infrastructure into 5 major components:

  • Interfaces (both real devices and pseudo-devices)
  • Socket code
  • Protocols
  • Routing code
  • mbuf code.

Part of the complexity is that, due to the monolithic nature of the kernel, each layer currently feels free to call any other layer. This makes designing a lock hierarchy difficult and likely to fail.

Part of the problem are asynchonous upcalls, among which include:

  • ifa->ifa_rtrequest for route changes.
  • pr_ctlinput for interface events.

Another source of complexity is the large number of global variables scattered throughout the source files. This makes putting locks around them difficult.

Subtasks

The proposed solution presented here include the following tasks (in no particular order) to achieve the desired goals of SMP support and modularity:

Work plan

Aside from the list of tasks above, the work to be done for this project can be achieved by following these steps:

  1. Move ARP out of the routing table. See the nexthop cache project.

  2. Make the network interfaces MP, which are one of the few users of the big kernel lock left. This needs to support multiple receive and transmit queues to help reduce locking contention. This also includes changing more of the common interfaces to do what the tsec driver does (basically do everything with softints). This also needs to change the *_input routines to use a table to do dispatch instead of the current switch code so domain can be dynamically loaded.

  3. Collect global variables in the IP/UDP/TCP protocols into structures. This helps the following items.

  4. Make IPV4/ICMP/IGMP/REASS MP-friendly.

  5. Make IPV6/ICMP/IGMP/ND MP-friendly.

  6. Make TCP MP-friendly.

  7. Make UDP MP-friendly.

Radical thoughts

You should also consider the following ideas:

LWPs in user space do not need a kernel stack

Those pages are only being used in case the an exception happens. Interrupts are probably going to their own dedicated stack. One could just keep a set of kernel stacks around. Each CPU has one, when a user exception happens, that stack is assigned to the current LWP and removed as the active CPU one. When that CPU next returns to user space, the kernel stack it was using is saved to be used for the next user exception. The idle lwp would just use the current kernel stack.

LWPs waiting for kernel condition shouldn't need a kernel stack

If an LWP is waiting on a kernel condition variable, it is expecting to be inactive for some time, possibly a long time. During this inactivity, it does not really need a kernel stack.

When the exception handler get an usermode exeception, it sets LWP restartable flag that indicates that the exception is restartable, and then services the exception as normal. As routines are called, they can clear the LWP restartable flag as needed. When an LWP needs to block for a long time, instead of calling cv_wait, it could call cv_restart. If cv_restart returned false, the LWPs restartable flag was clear so cv_restart acted just like cv_wait. Otherwise, the LWP and CV would have been tied together (big hand wave), the lock had been released and the routine should have returned ERESTART. cv_restart could also wait for a small amount of time like .5 second, and only if the timeout expires.

As the stack unwinds, eventually, it would return to the last the exception handler. The exception would see the LWP has a bound CV, save the LWP's user state into the PCB, set the LWP to sleeping, mark the lwp's stack as idle, and call the scheduler to find more work. When called, cpu_switchto would notice the stack is marked idle, and detach it from the LWP.

When the condition times out or is signalled, the first LWP attached to the condition variable is marked runnable and detached from the CV. When the cpu_switchto routine is called, the it would notice the lack of a stack so it would grab one, restore the trapframe, and reinvoke the exception handler.

Posted late Sunday afternoon, November 6th, 2011 Tags:

The projects listed in this page have been pre-approved for funding. If you choose to work on any of these projects, contact first the corresponding person or group and discuss the exact details of your work, the schedule and the compensation you expect.

List of funded projects

Posted late Sunday afternoon, November 6th, 2011

Enhance zeroconfd, the Multicast DNS daemon, that was begun in NetBSD's Google Summer of Code 2005 (see work in progress: http://netbsd-soc.sourceforge.net/projects/zeroconf/). Develop a client library that lets a process publish mDNS records and receive asynchronous notification of new mDNS records. Adapt zeroconfd to use event(3) and queue(3). Demonstrate comparable functionality to the GPL or APSL alternatives (Avahi, Howl, ...), but in a smaller storage footprint, with no dependencies outside of the NetBSD base system.

Posted Sunday evening, November 6th, 2011 Tags:
  • Contact: tech-kern
  • Duration estimate: 1 year for port; 3 years for rewrite by one developer

Implement a BSD licensed XFS. A GPL licensed implementation of XFS is available at http://oss.sgi.com/projects/xfs/.

Alternatively, or additionally, it might be worthwhile to do a port of the GPL code and allow it to run as a kernel module.

See also FreeBSD's port.

Posted Sunday evening, November 6th, 2011 Tags:

Latest Xen versions come with a number of features that are currently not supported by NetBSD: USB/VGA passthrough, RAS (Reliability, Availability and Serviceability) options - CPU and memory hotpluging - , Fault tolerancy with Remus, and debugging with gdbx (lightweight debugger included with Xen).

The purpose of this project is to add the missing parts inside NetBSD. Most of the work is composed of smaller components that can be worked on independently from others.

Posted Sunday evening, November 6th, 2011 Tags:

With the push of virtualization, the x86 world started recently to gain a more widespread attention towards supporting IOMMUs; similar to MMUs that translate virtual addresses into physical ones, an IOMMU translates device/bus addresses into physical addresses. The purpose of this project is to add AMD and Intel IOMMU support in NetBSD's machine-independent bus abstraction layers bus.space(9) and bus.dma(9).

Posted Sunday evening, November 6th, 2011 Tags:

The NetBSD website building infrastructure is rather complex and requires significant resources. We need to make it easier for anybody to contribute without having to install a large number of complex applications from pkgsrc or without having to learn the intricacies of the build process.

A more detailed description of the problem is described in this and this email and the following discussion on the netbsd-docs.

This work requires knowledge of XML, XSLT and make. This is not a request for visual redesign of the website.

Posted Sunday evening, November 6th, 2011 Tags:

Add socket options to NetBSD for controlling WLAN transmit parameters like transmit power, fragmentation threshold, RTS/CTS threshold, bitrate, 802.11e access category, on a per-socket and per-packet basis. To set transmit parameters, pass radiotap headers using sendmsg(2) and setsockopt(2).

This project is on hold due to the conversion project needing to be completed first.

Posted Sunday evening, November 6th, 2011 Tags:

Create an easy to use wifi setup widget for NetBSD: browse and select networks in the vicinity by SSID, BSSID, channel, etc.

The guts should probably be done as a library so that user interfaces of increasing slickness can be built on top of it as desired. (That is: there ought to be some form of this in base; but a nice looking gtk interface version would be good to have as well.)

Posted Sunday evening, November 6th, 2011 Tags:

Due to the multitude of supported machine architectures NetBSD has to deal with many different partitioning schemes. To deal with them in a uniform way (without imposing artificial restrictions that are not enforced by the underlying firmware or bootloader partitioning scheme) wedges have been designed.

While the kernel part of wedges is mostly done (and missing parts are easy to add), a userland tool to edit wedges and to synthesize defaults from (machine/arch dependent) on-disk content is needed.

Posted Sunday evening, November 6th, 2011 Tags:

While we now have mandoc for handling man pages, we currently still need groff in the tree to handle miscellaneous docs that are not man pages.

This is itself an inadequate solution as the groff we have does not support PDF output (which in this day and age is highly desirable) ... and while newer groff does support PDF output it does so via a Perl script. Also, importing a newer groff is problematic for assorted other reasons.

We need a way to typeset miscellaneous articles that we can import into base and that ideally is BSD licensed. (And that can produce PDFs.) Currently it looks like there are three decent ways forward:

  • Design a new roff macro package that's comparable to mdoc (e.g. supports semantic markup) but is for miscellaneous articles rather than man pages, then teach mandoc to handle it.

  • Design a new set of markup tags comparable to mdoc (e.g. supports semantic markup) but for miscellaneous articles, and a different less ratty syntax for it, then teach mandoc to handle this.

  • Design a new set of markup tags comparable to mdoc (e.g. supports semantic markup) but for miscellaneous articles, and a different less ratty syntax for it, and write a new program akin to mandoc to handle it.

These are all difficult and a lot of work, and in the case of new syntax are bound to cause a lot of shouting and stamping. Also, many of the miscellaneous documents use various roff preprocessors and it isn't clear how much of this mandoc can handle.

None of these options is particularly appealing.

There are also some less decent ways forward:

  • Pick one of the existing roff macro packages for miscellaneous articles (ms, me, ...) and teach mandoc to handle it. Unfortunately all of these macro packages are pretty ratty, they're underpowered compared to mdoc, and none of them support semantic markup.

  • Track down one of the other older roff implementations, that are now probably more or less free (e.g. ditroff), then stick to the existing roff macro packages as above. In addition to the drawbacks cited above, any of these programs are likely to be old nasty code that needs a lot of work.

  • Teach the groff we have how to emit PDFs, then stick to the existing roff macro packages as above. In addition to the drawbacks cited above, this will likely be pretty nasty work and it's still got the wrong license.

  • Rewrite groff as BSD-licensed code and provide support for generating PDFs, then stick to the existing roff macro packages as above. In addition to the drawbacks cited above, this is a horrific amount of work.

  • Try to make something else do what we want. Unfortunately, TeX is a nonstarter and the only other halfway realistic candidate is lout... which is GPLv3 and at least at casual inspection looks like a horrible mess of its own.

These options are even less appealing.

Maybe someone can think of a better idea. There are lots of choices if we give up on typeset output, but that doesn't seem like a good plan either.

Posted Sunday evening, November 6th, 2011 Tags:

Add memory-efficient snapshots to tmpfs. A snapshot is a view of the filesystem, frozen at a particular point in time. The snapshotted filesystem is not frozen, only the view is. That is, you can continue to read/write/create/delete files in the snapshotted filesystem.

The interface to snapshots may resemble the interface to null mounts, e.g., 'mount -t snapshot /var/db /db-snapshot' makes a snapshot of /var/db/ at /db-snapshot/.

You should exploit features of the virtual memory system like copy-on-write memory pages to lazily make copies of files that appear both in a live tmpfs and a snapshot. This will help conserve memory.

Posted Sunday evening, November 6th, 2011 Tags:
Posted Sunday evening, November 6th, 2011 Tags:

syspkgs is the concept of using pkgsrc's pkg_* tools to maintain the base system. That is, allow users to register and components of the base system with more ease.

There has been a lot of work in this area already, but it has not yet been finalized. Which is a diplomatic way of saying that this project has been attempted repeatedly and failed every time.

Posted Sunday evening, November 6th, 2011 Tags:

It would be nice to support these newer highly SMP processors from Sun. A Linux port already exists, and Sun has contributed code to the FOSS community.

(Some work has already been done and committed - see https://wiki.netbsd.org/ports/sparc64/sparc64sun4v/

Posted Sunday evening, November 6th, 2011 Tags:
  • Contact: tech-kern
  • Duration estimate: 8-12 months

Remove the residual geometry code and datastructures from FFS (keep some kind of allocation groups but without most of what cylinder groups now have) and replace blocks and fragments with extents, yielding a much simpler filesystem well suited for modern disks.

Note that this results in a different on-disk format and will need to be a different file system type.

The result would be potentially useful to other operating systems beyond just NetBSD, since UFS/FFS is used in so many different kernels.

Posted Sunday evening, November 6th, 2011 Tags:

Certain real time chips, and other related power hardware, have a facility within them to allow the kernel to set a specific time and date at which time the machine will power itself on. One such chip is the DS1685 RTC. A kernel API should be developed to allow such devices to have a power-on-time set from userland. Additionally, the API should be made available through a userland program, or added to an existing utility, such as halt(8).

It may also be useful to make this a more generic interface, to allow for configuring other devices, such as Wake-On-Lan ethernet chips, to turn them on/off, set them up, etc.

Posted Sunday evening, November 6th, 2011 Tags:

NetBSD/sgimips currently runs on O2s with R10k (or similar) CPUs, but for example speculative loads are not handled correctly. It is unclear if this is pure kernel work or the toolchain needs to be changed too.

Currently softfloat is used, and bugs seem to exist in the hardware float support. Resolving these bugs and switching to hardware float would improve performance.

See also NetBSD/sgimips.

Posted Sunday evening, November 6th, 2011 Tags:

NetBSD/sgimips currently runs on a number of SGI hardware, but support for IP27 (Origin) and IP30 (Octane) is not yet available.

See also NetBSD/sgimips.

Posted Sunday evening, November 6th, 2011 Tags:

Currently booting a sgimips machine requires different boot commands depending on the architecture. It is not possible to use the firmware menu to boot from CD.

An improved primary bootstrap should ask the firmware for architecture detail, and automatically boot the correct kernel for the current architecture by default.

A secondary objective of this project would be to rearrange the generation of a bootably CD image so that it could just be loaded from the firmware menu without going through the command monitor.

Posted Sunday evening, November 6th, 2011 Tags:

Create/modify a 802.11 link-adaptation module that works at least as well as SampleRate, but is generic enough to be re-used by device drivers for ADMtek, Atheros, Intel, Ralink, and Realtek 802.11 chips. Make the module export a link-quality metric (such as ETT) suitable for use in linkstate routing. The SampleRate module in the Atheros device driver sources may be a suitable starting point.

Posted Sunday evening, November 6th, 2011 Tags:

IMPORTANT: This project was completed by Kritaps Dzonsons at openbsd. You may still contact the people above for details, but please do not submit an application for this project.

Create a BSD licensed drop-in replacement for rsync that can handle large numbers of files/directories and large files efficiently. The idea would be to not scan the filesystem for every client to detect changes that need transfer, but rather maintain some database that can be queried (and that also needs updating when files are changed on the server). See supservers(8) for some ideas of how this could work. Compatibility with existing rsync clients should be retained.

Posted Sunday evening, November 6th, 2011 Tags:
  • Contact: tech-net
  • Duration estimate: 2 months

Write tests for the routing code and re-factor. Use more and better-named variables.

PCBs and other structures are sprinkled with route caches (struct route). Sometimes they do not get updated when they should. It's necessary to modify rtalloc(), at least. Fix that. Note XXX in rtalloc(); this may mark a potential memory leak!

Posted Sunday evening, November 6th, 2011 Tags:
  • Contact: tech-net
  • Duration estimate: 3 months

Implement the ability to route based on properties like QoS label, source address, etc.

Posted Sunday evening, November 6th, 2011 Tags:

The policy code in the kernel that controls file caching and readahead behavior is necessarily one-size-fits-all, and the knobs available to applications to tune it, like madvise() and posix_fadvise(), are fairly blunt hammers. Furthermore, it has been shown that the overhead from user<->kernel domain crossings makes syscall-driven fine-grained policy control ineffective. (Though, that was shown in the past when processors were much slower relative to disks and it may not be true any more.)

Is it possible to use BPF, or create a different BPF-like tool (that is, a small code generator with very simple and very clear safety properties) to allow safe in-kernel fine-grained policy control?

Caution: this is a research project.

Posted Sunday evening, November 6th, 2011 Tags:

IMPORTANT: This project was completed by joerg. You may still contact the people above for details, but please do not submit an application for this project.

To create packages that are usable by anyone, pkgsrc currently requires that packages be built with superuser privileges. It is already possible to use pkgsrc in great parts without such privileges, but there haven't been many thoughts about how the resulting binary packages should be specified. For example, many packages don't care at all about the owner/group of their files, as long as they are not publicly overwritable. In the end, the binary packages should be as independent from the build environment as possible.

For more information about the current state, see the How to use pkgsrc as non-root section in the pkgsrc guide, Jörg's mail on DESTDIR support as well as pkgsrc/mk/unprivileged.mk.

Posted Sunday evening, November 6th, 2011 Tags:

NetBSD currently requires a system with an MMU. This obviously limits the portability. We'd be interested in an implementation/port of NetBSD on/to an MMU-less system.

Posted Sunday evening, November 6th, 2011 Tags:

Apply statistical AI techniques to the problem of monitoring the logs of a busy system. Can one identify events of interest to a sysadmin, or events that merit closer inspection? Failing that, can one at least identify some events as routine and provide a filtered log that excludes them? Also, can one group a collection of related messages together into a single event?

Posted Sunday evening, November 6th, 2011 Tags:

While NetBSD has had LiveCDs for a while, there has not yet been a LiveCD that allows users to install NetBSD after test-driving it. A LiveCD that contains a GUI based installer and reliably detects the platforms features would be very useful.

Posted Sunday evening, November 6th, 2011 Tags:

Libvirt is a project that aims at bringing yet-another-level of abstraction to the management of different virtualization technologies; it supports a wide range of virtualization technologies, like Xen, VMWare, KVM and containers.

A package for libvirt was added to pkgsrc under sysutils/libvirt, however it requires more testing before all platforms supported by pkgsrc can also seamlessly support libvirt.

The purpose of this project is to investigate what is missing in libvirt (in terms of patches or system integration) so it can work out-of-the-box for platforms that can benefit from it. GNU/Linux, NetBSD, FreeBSD and Solaris are the main targets.

Posted Sunday evening, November 6th, 2011 Tags:

Improve on the Kismet design and implementation in a Kismet replacement for BSD.

Posted Sunday evening, November 6th, 2011 Tags:

kernfs is a virtual file system that reports information about the running system, and in some cases allows adjusting this information. procfs is a virtual file system that provides information about currently running processes. Both of these file systems work by exposing virtual files containing textual data.

The current implementations of these file systems are redundant and both are non-extensible. For example, kernfs is a hardcoded table that always exposes the same set of files; there is no way to add or remove entries on the fly, and even adding new static entries is a nuisance. procfs is similarly limited; there is no way to add additional per-process data on the fly. Furthermore, the current code is not modular, not well designed, and has been a source of security bugs in the past.

We would like to have a new implementation for both of these file systems that rectifies these problems and others, as outlined below:

  • kernfs and procfs should share most of their code, and in particular they should share all the code for managing lists of virtual files. They should remain separate entities, however, at least from the user perspective: community consensus is that mixing system and per-process data, as Linux always has, is ugly.

  • It should be possible to add and remove entries on the fly, e.g. as modules are loaded and unloaded.

  • Because userlevel programs can become dependent on the format of the virtual files (Linux has historically had compatibility problems because of this) they should if possible not have complex formats at all, and if they do the format should be clearly specifiable in some way that isn't procedural code. (This makes it easier to reason about, and harder for it to get changed by accident.)

  • There is an additional interface in the kernel for retrieving and adjusting arbitrary kernel information: sysctl. Currently the sysctl code is a third completely separate mechanism, on many points redundant with kernfs and/or procfs. It is somewhat less primitive, but the current implementation is cumbersome and not especially liked. Integrating kernfs and procfs with sysctl (perhaps somewhat like the Linux sysfs) is not automatically the right design choice, but it is likely to be a good idea. At a minimum we would like to be able to have one way to handle reportable/adjustable data within the kernel, so that kernfs, procfs, and/or sysctl can be attached to any particular data element as desired.

  • While most of the implementations of things like procfs and sysctl found in the wild (including the ones we currently have) work by attaching callbacks, and then writing code all over the kernel to implement the callback API, it is possible to design instead to attach data, that is, pointers to variables within the kernel, so that the kernfs/procfs or sysctl code itself takes responsibility for fetching that data. Please consider such a design strongly and pursue it if feasible, as it is much tidier. (Note that attaching data also in general requires specifying a locking model and, for writeable data, possibly a condition variable to signal on when the value changes and/or a mechanism for checking new values for validity.)

It is possible that using tmpfs as a backend for kernfs and procfs, or sharing some code with tmpfs, would simplify the implementation. It also might not. Consider this possibility, and assess the tradeoffs; do not treat it as a requirement.

Alternatively, investigate FreeBSD's pseudofs and see if this could be a useful platform for this project and base for all the file systems mentioned above.

When working on this project, it is very important to write a complete regression test suite for procfs and kernfs beforehand to ensure that the rewrites do not create incompatibilities.

Posted Sunday evening, November 6th, 2011 Tags:

Today a number of OS provide some form of kernel-level virtualization that offer better isolation mechanisms that the traditional (yet more portable) &chroot(2). Currently, NetBSD lacks functionality in this field; there have been multiple attempts (gaols, mult) to implement a jails-like system, but none so far has been integrated in base.

The purpose of this project is to study the various implementations found elsewhere (FreeBSD Jails, Solaris Zones, Linux Containers/VServers, ...), and eventually see their plus/minus points. An additional step would be to see how this can be implemented the various architectural improvements NetBSD gained, especially rump(3) and kauth(9).

Caution: this is a research project.

Posted Sunday evening, November 6th, 2011 Tags:
  • Contact: tech-kern
  • Duration estimate: 1 year for port; 3 years for rewrite by one developer

Implement a BSD licensed JFS. A GPL licensed implementation of JFS is available at http://jfs.sourceforge.net/.

Alternatively, or additionally, it might be worthwhile to do a port of the GPL code and allow it to run as a kernel module.

Posted Sunday evening, November 6th, 2011 Tags:

Add support for Apple's extensions to ISO9660 to makefs, especially the ability to label files with Type & Creator IDs. See http://developer.apple.com/technotes/fl/fl_36.html.

Posted Sunday evening, November 6th, 2011 Tags:
  • Contact: tech-kern
  • Duration estimate: 4 months and up

There are many caches in the kernel. Most of these have knobs and adjustments, some exposed and some not, for sizing and writeback rate and flush behavior and assorted other voodoo, and most of the ones that aren't adjustable probably should be.

Currently all or nearly all of these caches operate on autopilot independent of the others, which does not necessarily produce good results, especially if the system is operating in a performance regime different from when the behavior was tuned by the implementors.

It would be nice if all these caches were instead coordinated, so that they don't end up fighting with one another. Integrated control of sizing, for example, would allow explicitly maintaining a sensible balance between different memory uses based on current conditions; right now you might get that, depending on whether the available voodoo happens to work adequately under the workload you have, or you might not. Also, it is probably possible to define some simple rules about eviction, like not evicting vnodes that have UVM pages still to be written out, that can help avoid unnecessary thrashing and other adverse dynamic behavior. And similarly, it is probably possible to prefetch some caches based on activity in others. It might even be possible to come up with one glorious unified cache management algorithm.

Also note that cache eviction and prefetching is fundamentally a form of scheduling, so all of this material should also be integrated with the process scheduler to allow it to make more informed decisions.

This is a nontrivial undertaking.

Step 1 is to just find all the things in the kernel that ought to participate in a coordinated caching and scheduling scheme. This should not take all that long. Some examples include:

  • UVM pages
  • file system metadata buffers
  • VFS name cache
  • vnode cache
  • size of the mbuf pool

Step 2 is to restructure and connect things up so that it is readily possible to get the necessary information from all the random places in the kernel that these things occupy, without making a horrible mess and without trashing system performance in the process or deadlocking out the wazoo. This is not going to be particularly easy or fast.

Step 3 is to take some simple steps, like suggested above, to do something useful with the coordinated information, and hopefully to show via benchmarks that it has some benefit.

Step 4 is to look into more elaborate algorithms for unified control of everything. The previous version of this project cited IBM's ARC ("Adaptive Replacement Cache") as one thing to look at. (But note that ARC may be encumbered -- someone please check on that and update this page.) Another possibility is to deploy machine learning algorithms to look for and exploit patterns.

Note: this is a serious research project. Step 3 will yield a publishable minor paper; step 4 will yield a publishable major paper if you manage to come up with something that works, and it quite possibly contains enough material for a PhD thesis.

Posted Sunday evening, November 6th, 2011 Tags:

Use puffs or refuse to write an imapfs that you can mount on /var/mail, either by writing a new one or porting the old existing Plan 9 code that does this.

Note: there might be existing solutions, please check upfront and let us know.

Posted Sunday evening, November 6th, 2011 Tags:

Implement a flash translation layer.

A flash translation layer does block remapping, translating from visible block addresses used by a file system to physical cells on one or more flash chips. This provides wear leveling, which is essential for effective use of flash, and also typically some amount of read caching and write buffering. (And it takes care of excluding cells that have gone bad.)

This allows FFS, LFS, msdosfs, or whatever other conventional file system to be used on raw flash chips. (Note that SSDs and USB flash drives and so forth contain their own FTLs.)

FTLs involve quite a bit of voodoo and there is a lot of prior art and research; do not just sit down and start coding.

There are also some research FTLs that we might be able to get the code for; it is probably worth looking into this.

Note that NAND flash and NOR flash are different and need different handling, and the various cell types and other variations also warrant different policy choices.

The degree of overprovisioning (that is, the ratio of the raw capacity of the flash chips to the advertised size of the resulting device) should be configurable as this is a critical factor for performance.

Making the device recoverable rather than munching itself in system crashes or power failures is a nice extra, although apparently the market considers this an optional feature for consumer devices.

The flash translation layer should probably be packaged a driver that attaches to one or more flash chips and provides a disk-type block/character device pair.

Posted Sunday evening, November 6th, 2011 Tags:
  • Contact: tech-kern
  • Duration estimate: 2-3 months

The ext2 file system is the lowest common denominator Unix-like file system in the Linux world, as ffs is in the BSD world. NetBSD has had kernel support for ext2 for quite some time.

However, the Linux world has moved on, with ext3 and now to some extent also ext4 superseding ext2 as the baseline. NetBSD has no support for ext3; the goal of this project is to implement that support.

Since ext3 is a backward-compatible extension that adds journaling to ext2, NetBSD can mount clean ext3 volumes as ext2 volumes. However, NetBSD cannot mount ext3 volumes with journaling and it cannot handle recovery for crashed volumes. As ext2 by itself provides no crash recovery guarantees whatsoever, this journaling support is highly desirable.

The ext3 support should be implemented by extending the existing ext2 support (which is in src/sys/ufs/ext2fs), not by rewriting the whole thing over from scratch. It is possible that some of the ostensibly filesystem-independent code that was added along with the ffs WAPBL journaling extensions might be also useable as part of an ext3 implementation; but it also might not be.

The full requirements for this project include complete support for ext3 in both the kernel and the userland tools. It is possible that a reduced version of this project with a clearly defined subset of these requirements could still be a viable GSOC project; if this appeals to you please coordinate with a prospective mentor. Be advised, however, that several past ext3-related GSOC projects have failed; it is a harder undertaking than you might think.

An additional useful add-on goal would be to audit the locking in the existing ext2 code; the ext2 code is not tagged MPSAFE, meaning it uses a biglock on multiprocessor machines, but it is likely that either it is in fact already safe and just needs to be tagged, or can be easily fixed. (Note that while this is not itself directly related to implementing ext3, auditing the existing ext2 code is a good way to become familiar with it.)

Posted Sunday evening, November 6th, 2011 Tags:

In a file system with symlinks, the file system can be seen as a graph rather than a tree. The meaning of .. potentially becomes complicated in this environment.

There is a fairly substantial group of people, some of them big famous names, who think that the usual behavior (where crossing a symlink is different from entering a subdirectory) is a bug, and have made various efforts from time to time to "fix" it. One such fix can be seen in the -L and -P options to ksh's pwd.

Rob Pike implemented a neat hack for this in Plan 9. It is described in http://cm.bell-labs.com/sys/doc/lexnames.html. This project is to implement that logic for NetBSD.

Note however that there's another fairly substantial group of people, some of them also big famous names, who think that all of this is a load of dingo's kidneys, the existing behavior is correct, and changing it would be a bug. So it needs to be possible to switch the implementation on and off as per-process state.

Posted Sunday evening, November 6th, 2011 Tags:

NetBSD version of compressed cache system (for low-memory devices): http://linuxcompressed.sourceforge.net/.

Posted Sunday evening, November 6th, 2011 Tags:
  • Contact: tech-kern
  • Duration estimate: 2-3 months

Currently the buffer handling logic only sorts the buffer queue (aka disksort). In an ideal world it would be able to coalesce adjacent small requests, as this can produce considerable speedups. It might also be worthwhile to split large requests into smaller chunks on the fly as needed by hardware or lower-level software.

Note that the latter interacts nontrivially with the ongoing dynamic MAXPHYS project and might not be worthwhile. Coalescing adjacent small requests (up to some potentially arbitrary MAXPHYS limit) is worthwhile regardless, though.

Posted Sunday evening, November 6th, 2011 Tags:

BSD make (aka bmake) uses traditional suffix rules (.c.o: ...) instead of pattern rules like gmake's (%.c:%.o: ...) which are more general and flexible.

The suffix module should be re-written to work from a general match-and-transform function on targets, which is sufficient to implement not only traditional suffix rules and gmake pattern rules, but also whatever other more general transformation rule syntax comes down the pike next. Then the suffix rule syntax and pattern rule syntax can both be fed into this code.

Note that it is already possible to write rules where the source is computed explicitly based on the target, simply by using $(.TARGET) in the right hand side of the rule. Investigate whether this logic should be rewritten using the match-and-transform code, or if the match-and-transform code should use the logic that makes these rules possible instead.

Implementing pattern rules is widely desired in order to be able to read more makefiles written for gmake, even though gmake's pattern rules aren't well designed or particularly principled.

Posted Sunday evening, November 6th, 2011 Tags:

Modern 802.11 NICs provide two or more transmit descriptor rings, one for each priority level or 802.11e access category. Add to NetBSD a generic facility for placing a packet onto a different hardware transmit queue according to its classification by pf or IP Filter. Demonstrate this facility on more than one 802.11 chipset.

This project is on hold due to the conversion project needing to be completed first.

Posted Sunday evening, November 6th, 2011 Tags:

Design and program a scheme for extending the operating range of 802.11 networks by using techniques like frame combining and error-correcting codes to cope with low S/(N+I) ratio. Implement your scheme in one or two WLAN device drivers -- Atheros & Realtek, say.

This project is on hold due to the conversion project needing to be completed first.

Posted Sunday evening, November 6th, 2011 Tags:

If you plan to apply to work on any of the projects listed in this site, you should start by sending an email to the contact points listed at the top of every project description. The purpose of this email is to introduce yourself to the NetBSD community if you are new to it, and to explain in a great level of detail how you will work on the project (e.g. the overall plan, milestones, schedule, etc.) This is particularly important if the project is up for funding or if it is a Google Summer of Code project.

The sections below include a guideline on the kind of information we are interested in hearing in your initial communication. The level of detail involved will depend on how how familiar you are with the NetBSD Project and the developer community.

If you are a newcomer (e.g. you are a Google Summer of Code student that has just installed NetBSD for the first time), you are encouraged to answer as many questions as possible. If you are an old-timer, however, you can skip the most obvious questions and focus on preparing a detailed project specification instead.

About your project

  • What is the goal of the project? (Short overview)

  • What will be the deliverables of the project? (Code, documentation, ...)

  • Give an overview of how you intend to reach the project's goal in the form of milestones and a schedule.

  • Is similar software already available elsewhere, e.g. for Linux or any other BSD?

  • Is the project a port of software, or a rewrite? (remember: No GPL in the NetBSD kernel!)

About your project and NetBSD

  • If your working area is the core NetBSD operating system: have you installed NetBSD and made first experiences with hands-on configuration? Have you rebuilt the kernel and the userland, either in full or in parts? If you plan to work on pkgsrc, have you installed packages from source and binary? Have you created a package on your own?

  • Have you found the relevant places that your project is based on in the source code, and read through it?

  • How will your project integrate into NetBSD? (Userland tool, kernel subsystem, driver, patch set, pkgsrc, ...)

  • What interfaces in NetBSD will your project use? (Go into details here! What module/file names, functions, data structures etc. are of relevance for your project?)

  • To what degree are you familiar with those interfaces? (not/some/very, details?)

  • Is knowledge on other topics required for this project, e.g. on hardware, software other than NetBSD, APIs, protocols, etc.? If so, give details and references.

  • To what degree are you familiar with those? (not/some/very, details?)

  • If the project involves hardware (e.g. writing drivers, doing a port to new hardware, ...): do you own the hardware or have access to?

About you

  • Can you list some prior projects that you have worked on so far? Include details like programming language, duration, number of people involved, project goal, if you used CVS, SVN or similar, and whatever else we may find thrilling! If you have a CV/resume online, feel free to include a link.

  • Do you have any prior experience with programming NetBSD? In what area? If you did send some problem reports (PRs) or patches, please include references.

  • Have you previously discussed your project within NetBSD, either on a mailing list or with some specific developers? If so, please give us either the names/email addresses of those developers or point us towards the discussions on our list (via http://mail-index.NetBSD.org/).

  • How do we contact you for question, comments, suggestions, etc.?

  • Is there anything else you'd like us to know? Did we forget any important details or questions?

Posted late Sunday evening, November 6th, 2011

This is the archive of completed projects. Project proposals are preserved after they were completed for two reasons: first, to show that these project pages are useful and, second, to ensure that the URLs pointing to the projects remain valid.

Posted late Wednesday evening, November 9th, 2011

This project proposal is a subtask of smp networking and is elegible for funding independently.

The goal of this project is to implement full virtual network stacks. A virtual network stack collects all the global data for an instance of a network stack (excluding AF_LOCAL). This includes routing table, data for multiple domains and their protocols, and the mutexes needed for regulating access to it all. Instead, a brane is an instance of a networking stack.

An interface belongs to a brane, as do processes. This can be considered a chroot(2) for networking, e.g. chbrane(2).

Posted in the wee hours of Wednesday night, November 10th, 2011 Tags:

IMPORTANT: This project was completed by Tyler Retzlaff. You may still contact the people above for details, but please do not submit an application for this project.

The goal of this project is to split out obvious PR*_xxx that should have never been dispatched through the pr_usrreq/pr_ctloutput. Note that pr_ctloutput should be replaced by pr_getopt/pr_setopt:

  • PRU_CONTROL -> pr_ioctl
  • PRU_PURGEIF -> pr_purgeif
  • PRC0_GETOPT -> pr_getopt
  • PRC0_GETOPT -> pr_setopt

It's expected that pr_drain will just schedule a kernel continuation such as:

  • pr_init -> int pr_init(void *dsc);
  • int pr_fini(void *dsc)
Posted in the wee hours of Wednesday night, November 10th, 2011 Tags:

This project proposal is a subtask of smp networking.

The goal of this project is to make the SYN cache optional. For small systems, this is complete overkill and should be made optional.

Posted in the wee hours of Wednesday night, November 10th, 2011 Tags:

This project proposal is a subtask of smp networking.

The goal of this project is to remove the ARP, AARP, ISO SNPA, and IPv6 Neighbors from the routing table. Instead, the ifnet structure should have a set of nexthop caches (usually implemented using patricia trees), one per address family. Each nexthop entry should contain the datalink header needed to reach the neighbor.

This will remove cloneable routes from the routing table and remove the need to maintain protocol-specific code in the common Ethernet, FDDI, PPP, etc. code and put it back where it belongs, in the protocol itself.

Posted in the wee hours of Wednesday night, November 10th, 2011 Tags:

This project proposal is a subtask of smp networking.

The goal of this project is to improve the way the processing of incoming packets is handled.

Instead of having a set of active workqueue lwps waiting to service sockets, the kernel should use the lwp that is blocked on the socket to service the workitem. It is not productive being blocked and it has an interest in getting that workitem done, and maybe we can directly copy that data to user's address and avoid queuing in the socket at all.

Posted in the wee hours of Wednesday night, November 10th, 2011 Tags:

This project proposal is a subtask of smp networking.

The goal of this project is to implement continuations at the kernel level. Most of the pieces are already available in the kernel, so this can be reworded as: combine callouts, softints, and workqueues into a single framework. Continuations are meant to be cheap; very cheap.

These continuations are a dispatching system for making callbacks at scheduled times or in different thread/interrupt contexts. They aren't "continuations" in the usual sense such as you might find in Scheme code.

Please note that the main goal of this project is to simplify the implementation of SMP networking, so care must be taken in the design of the interface to support all the features required for this other project.

The proposed interface looks like the following. This interface is mostly derived from the callout(9) API and is a superset of the softint(9) API. The most significant change is that workqueue items are not tied to a specific kernel thread.

  • kcont_t *kcont_create(kcont_wq_t *wq, kmutex_t *lock, void (*func)(void *, kcont_t *), void *arg, int flags);

    A wq must be supplied. It may be one returned by kcont_workqueue_acquire or a predefined workqueue such as (sorted from highest priority to lowest):

    • wq_softserial, wq_softnet, wq_softbio, wq_softclock
    • wq_prihigh, wq_primedhigh, wq_primedlow, wq_prilow

    lock, if non-NULL, should be locked before calling func(arg) and released afterwards. However, if the lock is released and/or destroyed before the called function returns, then, before returning, kcont_set_mutex must be called with either a new mutex to be released or NULL. If acquiring lock would block, other pending kernel continuations which depend on other locks may be dispatched in the meantime. However, all continuations sharing the same set of { wq, lock, [ci] } need to be processed in the order they were scheduled.

    flags must be 0. This field is just provided for extensibility.

  • int kcont_schedule(kcont_t *kc, struct cpu_info *ci, int nticks);

    If the continuation is marked as INVOKING, an error of EBUSY should be returned. If nticks is 0, the continuation is marked as INVOKING while EXPIRED and PENDING are cleared, and the continuation is scheduled to be invoked without delay. Otherwise, the continuation is marked as PENDING while EXPIRED status is cleared, and the timer reset to nticks. Once the timer expires, the continuation is marked as EXPIRED and INVOKING, and the PENDING status is cleared. If ci is non-NULL, the continuation is invoked on the specified CPU if the continuations's workqueue has per-cpu queues. If that workqueue does not provide per-cpu queues, an error of ENOENT is returned. Otherwise when ci is NULL, the continuation is invoked on either the current CPU or the next available CPU depending on whether the continuation's workqueue has per-cpu queues or not, respectively.

  • void kcont_destroy(kcont_t *kc);

  • kmutex_t *kcont_getmutex(kcont_t *kc);

    Returns the lock currently associated with the continuation kc.

  • void kcont_setarg(kcont_t *kc, void *arg);

    Updates arg in the continuation kc. If no lock is associated with the continuation, then arg may be changed at any time; however, if the continuation is being invoked, it may not pick up the change. Otherwise, kcont_setarg must only be called when the associated lock is locked.

  • kmutex_t *kcont_setmutex(kcont_t *kc, kmutex_t *lock);

    Updates the lock associated with the continuation kc and returns the previous lock. If no lock is currently associated with the continuation, then calling this function with a lock other than NULL will trigger an assertion failure. Otherwise, kcont_setmutex must be called only when the existing lock (which will be replaced) is locked. If kcont_setmutex is called as a result of the invokation of func, then after kcont_setmutex has been called but before func returns, the replaced lock must have been released, and the replacement lock, if non-NULL, must be locked upon return.

  • void kcont_setfunc(kcont_t *kc, void (*func)(void *), void *arg);

    Updates func and arg in the continuation kc. If no lock is associated with the continuation, then only arg may be changed. Otherwise, kcont_setfunc must be called only when the associated lock is locked.

  • bool kcont_stop(kcont_t *kc);

    The kcont_stop function stops the timer associated the continuation handle kc. The PENDING and EXPIRED status for the continuation handle is cleared. It is safe to call kcont_stop on a continuation handle that is not pending, so long as it is initialized. kcont_stop will return a non-zero value if the continuation was EXPIRED.

  • bool kcont_pending(kcont_t *kc);

    The kcont_pending function tests the PENDING status of the continuation handle kc. A PENDING continuation is one who's timer has been started and has not expired. Note that it is possible for a continuation's timer to have expired without being invoked if the continuation's lock could not be acquired or there are higher priority threads preventing its invokation. Note that it is only safe to test PENDING status when holding the continuation's lock.

  • bool kcont_expired(kcont_t *kc);

    Tests to see if the continuation's function has been invoked since the last kcont_schedule.

  • bool kcont_active(kcont_t *kc);

  • bool kcont_invoking(kcont_t *kc);

    Tests the INVOKING status of the handle kc. This flag is set just before a continuation's function is being called. Since the scheduling of the worker threads may induce delays, other pending higher-priority code may run before the continuation function is allowed to run. This may create a race condition if this higher-priority code deallocates storage containing one or more continuation structures whose continuation functions are about to be run. In such cases, one technique to prevent references to deallocated storage would be to test whether any continuation functions are in the INVOKING state using kcont_invoking, and if so, to mark the data structure and defer storage deallocation until the continuation function is allowed to run. For this handshake protocol to work, the continuation function will have to use the kcont_ack function to clear this flag.

  • bool kcont_ack(kcont_t *kc);

    Clears the INVOKING state in the continuation handle kc. This is used in situations where it is necessary to protect against the race condition described under kcont_invoking.

  • kcont_wq_t *kcont_workqueue_acquire(pri_t pri, int flags);

    Returns a workqueue that matches the specified criteria. Thus if multiple requesters ask for the same criteria, they are all returned the same workqueue. pri specifies the priority at which the kernel thread which empties the workqueue should run.

    If flags is 0 then the standard operation is required. However, the following flag(s) may be bitwise ORed together:

    • WQ_PERCPU specifies that the workqueue should have a separate queue for each CPU, thus allowing continuations to be invoked on specific CPUs.
  • int kcont_workqueue_release(kcont_wq_t *wq);

    Releases an acquired workqueue. On the last release, the workqueue's resources are freed and the workqueue is destroyed.

Posted in the wee hours of Wednesday night, November 10th, 2011 Tags:

This project proposal is a subtask of smp networking.

The goal of this project is to implement interrupt handling at the granularity of a networking interface. When a network device gets an interrupt, it could call <iftype>_defer(ifp) to schedule a kernel continuation (see kernel continuations) for that interface which could then invoke <iftype>_poll. Whether the interrupted source should be masked depends on if the device is a DMA device or a PIO device. This routine should then call (*ifp->if_poll)(ifp) to deal with the interrupt's servicing.

During servicing, any received packets should be passed up via (*ifp->if_input)(ifp, m) which would be responsible for ALTQ or any other optional processing as well as protocol dispatch. Protocol dispatch in <iftype>_input decodes the datalink headers, if needed, via a table lookup and call the matching protocol's pr_input to process the packet. As such, interrupt queues (e.g. ipintrq) would no longer be needed. Any transmitted packets can be processed as can MII events. Either true or false should be returned by if_poll depending on whether another invokation of <iftype>_poll for this interface should be immediately scheduled or not, respectively.

Memory allocation has to be prohibited in the interrupt routines. The device's if_poll routine should pre-allocate enough mbufs to do any required buffering. For devices doing DMA, the buffers are placed into receive descripors to be filled via DMA.

For devices doing PIO, pre-allocated mbufs are enqueued onto the softc of the device so when the interrupt routine needs one it simply dequeues one, fills in it in, and then enqueues it onto a completed queue, finally calls <iftype>_defer. If the number of pre-allocated mbufs drops below a threshold, the driver may decide to increase the number of mbufs that if_poll pre-allocates. If there are no mbufs left to receive the packet, the packets is dropped and the number of mbufs for if_poll to pre-allocate should be increased.

When interrupts are unmasked depends on a few things. If the device is interrupting "too often", it might make sense for the device's interrupts to remain masked and just schedule the device's continuation for the next clock tick. This assumes the system has a high enough value set for HZ.

Posted in the wee hours of Wednesday night, November 10th, 2011 Tags:

This project proposal is a subtask of smp networking.

The goal of this project is to enhance the networking protocols to process incoming packets more efficiently. The basic idea is the following: when a packet is received and it is destined for a socket, simply place the packet in the socket's receive PCQ (see atomic pcq) and wake the blocking socket. Then, the protocol is able to process the next packet.

The typical packet flow from ip_input is to {rip,tcp,udp}_input which:

  • Does the lookup to locate the socket which takes a reader lock on the appropriate pcbtable's hash bucket.
  • If found and in the proper state:
    • Do not lock the socket since that would might block and therefore stop packet demultiplexing.
    • pcq_put the packet to the pcb's pcq.
    • kcont_schedule the worker continuation with small delay (~100ms). See kernel continuations.
    • Lock the socket's cvmutex.
    • Release the pcbtable lock.
    • If TCP and in sequence, then if we need to send an immediate ACK:
      • Try to lock the socket.
      • If successful, send an ACK.
    • Set a flag to process the PCQ.
    • cv_signal the socket's cv.
    • Release the cvmutex.
  • If not found or not in the proper state:
    • Release the pcb hash table lock.
Posted in the wee hours of Wednesday night, November 10th, 2011 Tags:

This project proposal is a subtask of smp networking.

The goal of this project is to implement lockless, atomic and generic Radix and Patricia trees. BSD systems have always used a radix tree for their routing tables. However, the radix tree implementation is showing its age. Its lack of flexibility (it is only suitable for use in a routing table) and overhead of use (requires memory allocation/deallocation for insertions and removals) make replacing it with something better tuned to today's processors a necessity.

Since a radix tree branches on bit differences, finding these bit differences efficiently is crucial to the speed of tree operations. This is most quickly done by XORing the key and the tree node's value together and then counting the number of leading zeroes in the result of the XOR. Many processors today (ARM, PowerPC) have instructions that can count the number of leading zeroes in a 32 bit word (and even a 64 bit word). Even those that do not can use a simple constant time routine to count them:

int
clz(unsigned int bits)
{
    int zeroes = 0;
    if (bits == 0)
        return 32;
    if (bits & 0xffff0000) bits &= 0xffff0000; else zeroes += 16;
    if (bits & 0xff00ff00) bits &= 0xff00ff00; else zeroes += 8;
    if (bits & 0xf0f0f0f0) bits &= 0xf0f0f0f0; else zeroes += 4;
    if (bits & 0xcccccccc) bits &= 0xcccccccc; else zeroes += 2;
    if (bits & 0xaaaaaaaa) bits &= 0xaaaaaaaa; else zeroes += 1;
    return zeroes;
}

The existing BSD radix tree implementation does not use this method but instead uses a far more expensive method of comparision. Adapting the existing implementation to do the above is actually more expensive than writing a new implementation.

The primary requirements for the new radix tree are:

  • Be self-contained. It cannot require additional memory other than what is used in its data structures.

  • Be generic. A radix tree has uses outside networking.

To make the radix tree flexible, all knowledge of how keys are represented has to be encapsulated into a pt_tree_ops_t structure with these functions:

  • bool ptto_matchnode(const void *foo, const void *bar, pt_bitoff_t max_bitoff, pt_bitoff_t *bitoffp, pt_slot_t *slotp);

    Returns true if both foo and bar objects have the identical string of bits starting at *bitoffp and ending before max_bitoff. In addition to returning true, *bitoffp should be set to the smaller of max_bitoff or the length, in bits, of the compared bit strings. Any bits before *bitoffp are to be ignored. If the string of bits are not identical, *bitoffp is set to the where the bit difference occured, *slotp is the value of that bit in foo, and false is returned. The foo and bar (if not NULL) arguments are pointers to a key member inside a tree object. If bar is NULL, then assume it points to a key consisting of entirely of zero bits.

  • bool ptto_matchkey(const void *key, const void *node_key, pt_bitoff_t bitoff, pt_bitlen_t bitlen);

    Returns true if both key and node_key objects have identical strings of bitlen bits starting at bitoff. The key argument is the same key argument supplied to ptree_find_filtered_node.

  • pt_slot_t ptto_testnode(const void *node_key, pt_bitoff_t bitoff, pt_bitlen_t bitlen);

    Returns bitlen bits starting at bitoff from node_key. The node_key argument is a pointer to the key members inside a tree object.

  • pt_slot_t ptto_testkey(const void *key, pt_bitoff_t bitoff, pt_bitlen_t bitlen);

    Returns bitlen bits starting at bitoff from key. The key argument is the same key argument supplied to ptree_find_filtered_node.

All bit offsets are relative to the most significant bit of the key,

The ptree programming interface should contains these routines:

  • void ptree_init(pt_tree_t *pt, const pt_tree_ops_t *ops, size_t ptnode_offset, size_t key_offset);

    Initializes a ptree. If pt points at an existing ptree, all knowledge of that ptree is lost. The pt argument is a pointer to the pt_tree_t to be initialized. The ops argument is a pointer to the pt_tree_ops_t used by the ptree. This has four members: The ptnode_offset argument contains the offset from the beginning of an item to its pt_node_t member. The key_offset argument contains the offset from the beginning of an item to its key data. This is used if 0 is used, a pointer to the beginning of the item will be generated.

  • void *ptree_find_filtered_node(pt_tree_t *pt, const void *key, pt_filter_t filter, void *filter_ctx);

    The filter argument is either NULL or a function bool (*)(const void *, void *, int);

  • bool ptree_insert_mask_node(pt_tree_t *pt, void *item, pt_bitlen_t masklen);

  • bool ptree_insert_node(pt_tree_t *pt, void *item);

  • void *ptree_iterate(pt_tree_t *pt, const void *node, pt_direction_t direction);

  • void ptree_remove_node(pt_tree_t *pt, const pt_tree_ops_t *ops, void *item);

Posted in the wee hours of Wednesday night, November 10th, 2011 Tags:

IMPORTANT: This project was completed by Matt Thomas. You may still contact the people above for details, but please do not submit an application for this project.

This project proposal is a subtask of smp networking and is elegible for funding independently.

The goal of this project is to implement lockess and atomic producer/consumer queues (PCQs) in the kernel. A PCQ allows multiple writers (producers) but only a single reader (consumer). Compare-And-Store operations are used to allow lockless updates. The consumer is expected to be protected by a mutex that covers the structure that the PCQ is embedded into (e.g. socket lock, ifnet hwlock). These queues operate in a First-In, First-Out (FIFO) manner. The act of inserting or removing an item from a PCQ does not modify the item in any way. A PCQ does not prevent an item being inserted multiple times into a single PCQ.

Since this structure is not specific to networking it has to be accessed via <sys/pcq.h> and the code has to live in kern/subr_pcq.c.

The proposed interface looks like this:

  • bool pcq_put(pcq_t *pcq, void *item);

    Places item at the end of the queue. If there is no room in the queue for the item, false is returned; otherwise true is returned. The item must not have the value NULL.

  • void *pcq_peek(pcq_t *pcq);

    Returns the next item to be consumed from the queue but does not remove it from the queue. If the queue is empty, NULL is returned.

  • void *pcq_get(pcq_t *pcq);

    Removes the next item to be consumed from the queue and returns it. If the queue is empty, NULL is returned.

  • size_t pcq_maxitems(pcq_t *pcq);

    Returns the maximum number of items that the queue can store at any one time.

  • pcq_t *pcq_create(size_t maxlen, km_flags_t kmflags);

  • void pcq_destroy(pcq_t *pcq);

Posted in the wee hours of Wednesday night, November 10th, 2011 Tags:

This project proposal is a subtask of smp networking.

The goal of this project is to implement lockless and atomic FIFO/LIFO queues in the kernel. The routines to be implemented allow for commonly typed items to be locklessly inserted at either the head or tail of a queue for either last-in, first-out (LIFO) or first-in, first-out (FIFO) behavior, respectively. However, a queue is not instrinsicly LIFO or FIFO. Its behavior is determined solely by which method each item was pushed onto the queue.

It is only possible for an item to removed from the head of queue. This removal is also performed in a lockless manner.

All items in the queue must share a atomic_queue_link_t member at the same offset from the beginning of item. This offset is passed to atomic_qinit.

The proposed interface looks like this:

  • void atomic_qinit(atomic_queue_t *q, size_t offset);

    Initializes the atomic_queue_t queue at q. offset is the offset to the atomic_queue_link_t inside the data structure where the pointer to the next item in this queue will be placed. It should be obtained using offsetof.

  • void *atomic_qpeek(atomic_queue_t *q);

    Returns a pointer to the item at the head of the supplied queue q. If there was no item because the queue was empty, NULL is returned. No item is removed from the queue. Given this is an unlocked operation, it should only be used as a hint as whether the queue is empty or not.

  • void *atomic_qpop(atomic_queue_t *q);

    Removes the item (if present) at the head of the supplied queue q and returns a pointer to it. If there was no item to remove because the queue was empty, NULL is returned. Because this routine uses atomic Compare-And-Store operations, the returned item should stay accessible for some indeterminate time so that other interrupted or concurrent callers to this function with this q can continue to deference it without trapping.

  • void atomic_qpush_fifo(atomic_queue_t *q, void *item);

    Places item at the tail of the atomic_queue_t queue at q.

  • void atomic_qpush_lifo(atomic_queue_t *q, void *item);

    Places item at the head of the atomic_queue_t queue at q.

Posted in the wee hours of Wednesday night, November 10th, 2011 Tags:

Setting up a secure SMTP server with AUTH and TLS enabled in Sendmail

While postfix is the basesystem's SMTP server, it is still possible to use the venerable Sendmail as your mail server of choice. Securing a sendmail SMTP gateway in order to use it from anywhere using your system's credentials is an easy task, here is how to achieve it.

Enabling Sendmail as the system's SMTP server

First thing is to disable postfix as the system's SMTP server. This action is controlled by the postfix parameter in /etc/rc.conf:

postfix=NO

We will then Install sendmail from pkgsrc with SASL for the authentication mechanism and TLS as the secure transport layer:

$ grep sendmail /etc/mk.conf
PKG_OPTIONS.sendmail=   tls sasl
ACCEPTABLE_LICENSES+=   sendmail-license

AUTH with SASL

Enabling SASL will build security/cyrus-sasl, but this package build failed with the following on my NetBSD 5.0.2 box:

db_ndbm.c:95: warning: passing argument 3 of 'utils->getcallback' from incompatible pointer type

So we will specify that cyrus-sasl should use berkeley as its database type:

$ grep SASL /home/bulk/etc/mk.conf
SASL_DBTYPE=            berkeley

We can now install sendmail with TLS and SASL support the classic way:

$ cd /usr/pkgsrc/mail/sendmail && sudo make install clean

cyrus-sasl package does now include any authentication plugin, it's up to us to pick one that will suit our needs. As we want to authenticate over system's login/password, we will use cy2-login:

$ cd /usr/pkgsrc/security/cy2-login && sudo make install

In order to use this method, we will have to install the saslauthd package. Saslauthd is in charge of plaintext authentications on behalf of the SASL library.

$ cd /usr/pkgsrc/security/cyrus-saslauthd && sudo make install clean

Of course, we want this daemon to start at every boot of this mail server:

# cp /usr/pkg/share/examples/rc.d/saslauthd /etc/rc.d
# echo "saslauthd=YES" >> /etc/rc.conf
# /etc/rc.d/saslauthd start

Now we have to inform the SASL library that it should use saslauthd whenever sendmail asks for an authentication:

# echo "pwcheck_method:saslauthd" > /usr/pkg/lib/sasl2/Sendmail.conf

Setting up the secure transport layer

As everything is in place for authentication, we will now prepare the TLS prerequisites. Instead of generating a self-signed certificate, I use to rely on CACert, "a community driven, Certificate Authority that issues certificates to the public at large for free." (from CACert.org).

In order to generate the certificate signing request (CSR), you can use the CSRGenerator script from CACert, which is really handy.

Once you have generated your server's private key with CSRGenerator and received your server certificate from CACert, simply copy them to /etc/mail/certs, along with CACert root certificate. Make sure your private key has strict permissions, sendmail will refuse to start if it is readable by everyone.

Configuring sendmail

It is now time to write our sendmail configuration. Create a mc file corresponding to your needs in /usr/pkg/share/sendmail/cf, for example:

# cat > /usr/pkg/share/sendmail/cf/korriban.mc << EOF
divert(0)dnl
VERSIONID(`Mustafar')
OSTYPE(bsd4.4)dnl
DOMAIN(generic)dnl

FEATURE(access_db, `hash -T<TMPF> /etc/mail/access')
FEATURE(blacklist_recipients)
FEATURE(mailertable, `hash -o /etc/mail/mailertable')
FEATURE(virtusertable, `hash -o /etc/mail/virtusertable')
FEATURE(genericstable, `hash -o /etc/mail/genericstable')
FEATURE(local_procmail)

dnl ### I use procmail as my MDA
define(`PROCMAIL_MAILER_PATH',`/usr/pkg/bin/procmail')
dnl ### and dspam as my antispam
define(`LOCAL_MAILER_PATH', `/usr/pkg/bin/dspam')
define(`LOCAL_MAILER_ARGS', `dspam -t -Y -a $h "--deliver=innocent" --user $u -d %u')

define(`confMAX_MESSAGE_SIZE', 5000000)

dnl ### here begins the secure SMTP gateway parameters
dnl ###
dnl ### enable SMTP AUTH with LOGIN mechanism
define(`confAUTH_MECHANISMS', `LOGIN')dnl
TRUST_AUTH_MECH(`LOGIN')dnl
dnl ### enable STARTTLS
define(`confCACERT_PATH',`/etc/mail/certs/')dnl
define(`confCACERT', `/etc/mail/certs/cacert.crt')
define(`confSERVER_CERT',`/etc/mail/certs/korriban_server.pem')dnl
define(`confSERVER_KEY',`/etc/mail/certs/korriban_privatekey.pem')dnl
dnl ### end of secure SMTP gateway parameters

MAILER(local)dnl
MAILER(smtp)dnl
MAILER(procmail)
EOF

Once your configuration is ready, build and install it using the following:

# make install-cf CF=korriban
rm -f korriban.cf
m4 ../m4/cf.m4 korriban.mc > korriban.cf || ( rm -f korriban.cf && exit 1 )
echo "### korriban.mc ###" >>korriban.cf
sed -e 's/^/# /' korriban.mc >>korriban.cf
chmod 444 korriban.cf
/usr/bin/install -c -o root -g wheel -m 0444 korriban.cf /etc/mail/sendmail.cf
/usr/bin/install -c -o root -g wheel -m 0444 korriban.cf /etc/mail/submit.cf

Now that sendmail is configured, fire it up by invoking:

# /etc/rc.d/sendmail start

And test that the features we've added are working:

# sendmail -d0.1 -bv root | grep SASL
    SASLv2 SCANF SOCKETMAP STARTTLS TCPWRAPPERS USERDB XDEBUG
$ telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 korriban.imil.net ESMTP Sendmail 8.14.5/8.14.5; Sat, 12 Nov 2011 16:43:40 +0100 (CET)
ehlo localhost
250-korriban.imil.net Hello localhost [127.0.0.1], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-EXPN
250-VERB
250-8BITMIME
250-SIZE 5000000
250-DSN
250-ETRN
250-AUTH LOGIN
250-STARTTLS
250-DELIVERBY
250 HELP

There you go! now configure your MUA so it always tries TLS for sending mail, using the LOGIN authentication method.

Posted Saturday afternoon, November 12th, 2011
Posted at teatime on Saturday, November 12th, 2011

Speeding up pkgsrc builds with ccache and distcc

Building an important amount of packages with pkgsrc can take a very long time. Two helper softwares can speed up operations significantly: ccache and distcc.

ccache

From package's DESCR:

ccache is a compiler cache. It acts as a caching pre-processor to C/C++ compilers, using the -E compiler switch and a hash to detect when a compilation can be satisfied from cache. This often results in a 5 to 10 times speedup in common compilations.

Using ccache in pkgsrc is very simple, just add the following line to your /etc/mk.conf:

PKGSRC_COMPILER=    ccache gcc

Declaring ccache as a compiler in mk.conf will make it a dependency for every package to be built.

distcc

From package's DESCR:

distcc is a program to distribute compilation of C or C++ code across several machines on a network. distcc should always generate the same results as a local compile, is simple to install and use, and is often two or more times faster than a local compile.

We will setup distcc with two hosts called hostA and hostB. First, install the software on both machines:

# cd /usr/pkgsrc/devel/distcc && make install clean
# cp /usr/pkg/share/examples/rc.d/distccd /etc/rc.d

Configure some parameters in order to allow hostA and hostB to use each other's distcc instances. hostA's IP address is 192.168.1.1, hostB's IP address is 192.168.1.2:

hostA$ grep distcc /etc/rc.conf
distccd=YES
distccd_flags="--allow 192.168.1.0/24 --allow 127.0.0.1 --listen 192.168.1.1 --log-file=/home/distcc/distccd.log"

hostB$ grep distcc /etc/rc.conf
distccd=YES
distccd_flags="--allow 192.168.1.0/24 --allow 127.0.0.1 --listen 192.168.1.2 --log-file=/home/distcc/distccd.log"

Instead of sending logs to syslog, we will use a custom logfile located in distcc's user home directory:

# mkdir /home/distcc && chown distcc /home/distcc

We can then fire up distcc on both hosts:

# /etc/rc.d/distccd start

In order to use hostnames instead of their IP addresses, add them to both /etc/hosts:

# tail -2 /etc/hosts
192.168.1.1 hostA
192.168.1.2 hostB

And finally tell pkgsrc to use distcc along with ccache by adding these lines to /etc/mk.conf on both machines:

PKGSRC_COMPILER=    ccache distcc gcc
DISTCC_HOSTS=       hostA hostB
MAKE_JOBS=      4

Here we define MAKE_JOBS to 4 because we are using two single-CPU hosts. The recommended value for MAKE_JOBS is number of CPUs*2 to avoid idle time.

Testing

To see distcc in action, simply watch the /home/distcc/distccd.log file while you are building a package:

$ tail -f /home/distcc/distccd.log
distccd[5218] (dcc_job_summary) client: 192.168.1.1:64865 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:175ms gcc lockfile.c
distccd[8292] (dcc_job_summary) client: 192.168.1.1:64864 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:222ms gcc counters.c
distccd[27779] (dcc_job_summary) client: 192.168.1.1:64881 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:3009ms gcc ccache.c
distccd[27779] (dcc_job_summary) client: 192.168.1.1:64863 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:152ms gcc compopt.c
Posted at lunch time on Sunday, November 13th, 2011

The file mk.conf is the central configuration file for everything that has to do with building software. It is used by the BSD-style Makefiles in /usr/share/mk and especially by pkgsrc. Usually, it is found in the /etc directory. If it doesn't exist there, feel free to create it.

Because all configuration takes place in a single file, there are some variables so the user can choose different configurations based on whether he is building the base system or packages from pkgsrc. These variables are:

  • BSD_PKG_MK: Defined when a pkgsrc package is built.
  • BUILDING_HTDOCS: Defined when the NetBSD web site is built.
  • None of the above: When the base system is built. The file /usr/share/mk/bsd.README is a good place to start in this case.

A typical mk.conf file would look like this:


# This is /etc/mk.conf
#

.if defined(BSD_PKG_MK) || defined(BUILDING_HTDOCS)
# The following lines apply to both pkgsrc and htdocs.

#...
LOCALBASE=          /usr/pkg
#...

.else
# The following lines apply to the base system.

WARNS= 4

.endif
Posted late Sunday evening, November 20th, 2011

Document status: DRAFT

In this article I will document how to transform a Solaris 10 x86 core installation into a pkgsrc-powered desktop system. The Solaris core installation does not include any X11, GNOME or GNU utilites. We will use modular X.org from pkgsrc. The guide assumes that the reader has some prior experience using Solaris and pkgsrc.

Contents

Installation

Begin by installing a Solaris core system. When done, mount the Solaris CD/DVD and install the following extra packages:

  • SUNWarc Lint Libraries (usr)
  • SUNWbtool CCS tools bundled with SunOS (Solaris 9)
  • SUNWbzip The bzip compression utility
  • SUNWdoc Documentation Tools
  • SUNWhea SunOS Header Files
  • SUNWlibm Math & Microtasking Library Headers & Lint Files (Usr)
  • SUNWlibmr Math Library Lint Files (Root) (Solaris 10)
  • SUNWman On-Line Manual Pages
  • SUNWscpr Source Compatibility, (Root)
  • SUNWscpu Source Compatibility, (Usr)
  • SUNWsprot Solaris Bundled tools
  • SUNWtoo SUNWtoo Programming Tools
  • SUNWxcu4 XCU4 Utilities

These packages are required if you intend to use modular-xorg-server from pkgsrc:

  • SUNWdfbh Dumb Frame Buffer Header Files
  • SUNWaudh Audio Header Files (don't ask why!)

# mount -F hsfs /dev/dsk/c1t1d0p0 /mnt
# cd /mnt/Solaris10/Product
# cp -r SUNW... /var/spool/pkg
# pkgadd

To see which SUNW packages are already installed, use the /usr/bin/pkginfo command.

Compiler setup

Now you need a compiler. You have a couple of options:

  • Use my prebuilt compiler kit, available from http://notyet
  • Install SUNWgcc from the Solaris DVD
  • Install Sun Studio 10
  • Install gcc from Sunfreeware.com
  • [advanced] Bootstrap your own gcc, using one of the above. If you get an error about a library not being found, remember to use crle -u -l libpath to add it to the link path. Make sure any /usr/pkg/* library paths are included early in this string so that pkgsrc binaries will only have dependencies on pkgsrc libraries as much as possible.

pkgsrc

Got a compiler? Good! Let's download and bootstrap pkgsrc.

Grab pkgsrc.tar.gz from ftp://ftp.NetBSD.org/pub/pkgsrc/current/ and untar to /usr, or get it from CVS.

# cd /usr/pkgsrc/bootstrap
# env CFLAGS=-O2 CC=/usr/local/gcc4/bin/gcc ./bootstrap

[coffee break]

Now we can tune /usr/pkg/etc/mk.conf. I use the following additional settings:


CFLAGS+=-O2
CXXFLAGS+=-O2
CC=/usr/local/gcc4/bin/gcc
CXX=/usr/local/gcc4/bin/g++
X11_TYPE=modular
PKG_DEVELOPER=yes
PKG_DEFAULT_OPTIONS+=freetype truetype mmx subpixel official-mozilla-branding

At this point you're free to install whatever packages you like.

On Solaris 9 (at least), Python 2.4 is the latest version that will build. You may want to set PYTHON_VERSION_DEFAULT= 24 in mk.conf in order to build python packages. (As of 13feb2010.)

Installing modular X.org from pkgsrc

It is mentioned above, but easy to miss... you really want to set X11_TYPE=modular in mk.conf, otherwise none of this will work. You will also want to set MOTIF_TYPE=openmotif or MOTIF_TYPE=lesstif to avoid having pkgsrc/mk/motif.buildlink3.mk choose /usr/dt/... which requires X11_TYPE=native instead of modular.

Packages needed for modular X.org are:

  • meta-pkgs/modular-xorg-fonts
    • xxx: fonts/mkfontdir needs a hack that I have not yet committed
  • meta-pkgs/modular-xorg-apps
  • x11/modular-xorg-server
    • xxx: needs some hacks that have not yet been committed (--disable-dri, libdrm, driproto KDSETMODE ioctl, vtname /dev/fb)
  • x11/xf86-input-keyboard
  • x11/xf86-input-mouse
  • x11/xf86-video-vesa (or pick a suitable driver for your card)
  • x11/xterm

Now run /usr/pkg/bin/Xorg -configure, which should work. Move the generated configuration file to /etc/X11/xorg.conf. Now you can attempt to start the server, by running Xorg with no arguments. If you get a picture, but the mouse isn't working, try to set your mouse device to "/dev/kdmouse" and the protocol to "PS/2" in xorg.conf.

TODO: write about installing firefox, desktop stuff, ...

Outstanding PR's with patches

There are some PR's with patches that solve Solaris build issues that are not yet committed. These may solve a problem you are having.

  • pkg/40153 - Synopsis: pkgsrc/devel/binutils fails to build on solaris 10 sparc
  • pkg/40201 - Synopsis: pkgsrc/sysutils/memconf update needed
  • pkg/39085 - Synopsis: firefox3 compile problem (just committed!)
  • pkg/40221 - Synopsis: pkgsrc/mail/p5-Mail-Mbox-MessageParser requires GNU grep (needed by grepmail)
  • pkg/40222 - Synopsis: pkgsrc/databases/sqsh fix build w/sybase libs on Solaris

Other hints

These may not be the 'right' think to do, but are still a help to get past some issues until the right thing can be figured out:

  • errors building libffi (required by python 2.5+) using Sun Studio compiler can be worked around by using gcc or setting:

    PYTHON_VERSION_DEFAULT=24

  • errors regarding a missing gtk-rebase can be worked around by installing the package textproc/gtk-doc

  • using a native jdk (on anything except SunOS-5.11-i386, which already works) can be done by adding these to /usr/pkg/etc/mk.conf:

    PKG_JVM_DEFAULT= sun-jdk6 _PKG_JVMS_ACCEPTED+= sun-jdk6

  • errors regarding wrong number of arguments to readdir_r() and some other xxx_r() functions can be fixed by adding to the package Makefile:

    CFLAGS.SunOS+= -D_POSIX_PTHREAD_SEMANTICS

If you encounter issues with missing libraries, refer to the pkgsrc guide first.

See also

  • ?The pkgsrc portability guide
  • ?How to use pkgsrc on Mac OS X

External links

Posted late Sunday evening, November 20th, 2011

I'm doing this guide on Tru64, but it should also apply to older versions of this fine OS, formerly branded Digital Unix and also as OSF1 (the system still identifies itself as OSF1).

Contents

Setting the environment

There is no bzip2 and cvs so we have to fetch the tarball by ftp. You can try to use ftp(1) or Netscape if you have DECWindows installed. You also can use the portable NetBSD ftp client included in the binary bootstrap if you decide to extract it first. I think tnftp is much nicer for downloading than anything else.

The system doesn't have gcc installed by default as some newer Unices have in the default install, but it has a nice compiler called ccc (Compaq C Compiler).

Extracting the files

You have to extract the downloaded sources:

# cd /usr
# gunzip -c /pkgsrc.tar.gz | tar xf -

Bootstrapping needs gcc:

# cd pkgsrc/bootstrap/
# env CC=/usr/local/gcc4/bin/gcc CFLAGS=-O2 ./bootstrap

Configuring pkgsrc

After the bootstrap is completed, you must decide which compiler to use. To keep using gcc, add the following to your mk.conf:

CC=/usr/local/gcc4/bin/gcc
CXX=/usr/local/gcc4/bin/g++

To use the native compiler, set PKGSRC_COMPILER=ccc in mk.conf. You will need at least Compaq C 6.4 (it supports VA_ARGS that tnftp(1) uses).

See also

http://pkgsrc-osf1.netbsd.se/

README.OSF1

Posted late Sunday evening, November 20th, 2011

This page should use cross references to avoid duplicate content. Please help us by cleaning it up. ?

Contents

See

README.Linux

pkgsrc guide

Software requirements

Before you can use pkgsrc, you may need a few packages installed on your Linux system upfront.

  • gcc (and libstdc++)
  • libncurses-devel
  • zlib and zlib-devel
  • openssl-devel (optional but required for some packages)
  • libudev-dev (optional but required for some packages)

The names may vary, depending on what Linux distribution you are using. Also be mindful of the platform you are using (eg. i686 vs. x86_64 - some have different pre-required packages). Also note that some very basic tools such as file, patch, sed, and others are required, as well.

Troubleshooting bootstrap

Shell's echo command is not BSD-compatible

If you see this error

ERROR: Your shell's echo command is not BSD-compatible.

This error is known to occur if /bin/sh is linked to /bin/dash (recent Ubuntu versions).

The fix is to issue the following before commencing bootstrap: export SH=/bin/bash

ARG_MAX undeclared

If bootstrap stops at

In file included from glob.c:12:                                                
__glob13.c: In function globextend:                                             
__glob13.c:836: error: ARG_MAX undeclared (first use in this function)

Then apply this patch 1

FORTIFY_SOURCE

If bootstrap stops at

/usr/pkgsrc/bootstrap/work/bmake/arch.c: In function 'Arch_Touch':
/usr/pkgsrc/bootstrap/work/bmake/arch.c:1038: warning: ignoring return
value of 'fwrite', declared with attribute warn_unused_result
*** Error code 1

This error occurs because Linux uses -DFORTIFY_SOURCES by default. Bootstrap can be achieved by:

CFLAGS="-U_FORTIFY_SOURCE" ./bootstrap 

libncurses not installed

If bootstrap stops at

ERROR: This package has set PKG_FAIL_REASON:
ERROR: No usable termcap library found on the system.

Then install the libncurses-dev package

On an RPM based system this might be via 'yum install ncurses-devel', and on a Debian/Ubuntu system 'apt-get install libncurses5-dev'

Posted late Sunday evening, November 20th, 2011

This page should use cross references to avoid duplicate content. Please help us by cleaning it up. ?

By default, IRIX is a quite hostile environment if one got used to systems where Bash, CVS and such are already installed. It also lacks many other tools (or at least sufficiently capable versions of them), so they all have to be built when bootstrapping pkgsrc.

Contents

Exploring the environment


$ echo $SHELL
/bin/ksh
$ cvs
ksh: cvs:  not found
$ zsh
ksh: zsh:  not found
$ bash
ksh: bash:  not found
$ bzip2
ksh: bzip2:  not found
$

So far, so bad. I will have to install all these tools via pkgsrc.

Getting pkgsrc

Since cvs is not available, I have to download the tarball from the FTP server.


$ ftp
ftp> open ftp.NetBSD.org
Connected to ftp.netbsd.org.
220 ftp.NetBSD.org FTP server (NetBSD-ftpd 20060923) ready.
Name (ftp.NetBSD.org:rillig): ftp
331 Guest login ok, type your name as password.
Password: 
...
ftp> cd /pub/pkgsrc/current
250 CWD command successful.
ftp> ls
227 Entering Passive Mode (204,152,184,36,251,197)
150 Opening ASCII mode data connection for '/bin/ls'.
total 141322
drwxrwxr-x  52 srcmastr  netbsd      1024 Jan  8 05:39 pkgsrc
-rw-rw-r--   1 srcmastr  netbsd  31658232 Jan  6 07:22 pkgsrc.tar.bz2
-rw-rw-r--   1 srcmastr  netbsd        56 Jan  6 07:22 pkgsrc.tar.bz2.MD5
-rw-rw-r--   1 srcmastr  netbsd        65 Jan  6 07:22 pkgsrc.tar.bz2.SHA1
-rw-rw-r--   1 srcmastr  netbsd  40628869 Jan  6 07:19 pkgsrc.tar.gz
-rw-rw-r--   1 srcmastr  netbsd        55 Jan  6 07:20 pkgsrc.tar.gz.MD5
-rw-rw-r--   1 srcmastr  netbsd        64 Jan  6 07:20 pkgsrc.tar.gz.SHA1
226 Transfer complete.
ftp> binary
200 Type set to I.
ftp> get pkgsrc.tar.gz
...
ftp> quit
221-
    Data traffic for this session was 0 bytes in 0 files.
    Total traffic for this session was 3445 bytes in 1 transfer.
221 Thank you for using the FTP service on ftp.NetBSD.org.
$

Extracting the files


$ mkdir proj
$ cd proj
$ gzcat ../pkgsrc.tar.gz | tar xf -
$ cd pkgsrc

$ CC=cc ./bootstrap/bootstrap --unprivileged --compiler=mipspro

... TODO: continue

Note: because nbsed cannot handle files with embedded '\0', and since GNU info files contain such characters, you should install textproc/gsed as soon as possible and then replace the TOOLS_PLATFORM.sed line in your mk.conf file.

See also

External links

Posted late Sunday evening, November 20th, 2011

This page should use cross references to avoid duplicate content. Please help us by cleaning it up. ?

Contents

Synopsis

HP-UX is a version of Unix for HP's PA-RISC and Integrity line of servers and workstations. HP-UX 11.x versions are pretty well supported by pkgsrc and it's also quite usable on 10.20.

Preparations

pkgsrc

Simply download the pkgsrc snapshot tarball as you would do on other architectures. You can also use CVS if its avalible.

XXX TODO: explain in pkgsrc page and link to there.

Patches

Read Readme.HPUX for the required patches and prerequisites.

Compiler

You will need a compiler which can be HP's ANSI C/C++ compiler or GCC, which is availible from HP or other third parties.

Bootstrap

Bootstrapping is done the usual way.

CC=path_to CXX=path_to ./bootstrap --abi 32 --compiler gcc

XXX TODO: explain in pkgsrc page and link to there.

Audio

Audio playback works pretty well on Series 700 workstations through HP Audio and esound.

You will need to install the following depot beforehand:

B6865AAA -- HP Audio Clients for Series 700 - English 

You can also use libao-esd with packages which support libao.

See also

Posted late Sunday evening, November 20th, 2011

This page should use cross references to avoid duplicate content. Please help us by cleaning it up.

Contents

Introduction

'pkgsrc' on AIX must be a fairly uncommon occurance considering the general state of the documentation around getting it working. This is probably partly due to the advent of the 'AIX toolkit' which allows RPMs to be installed with minimum hassle on an AIX box. On the plus side, this toolkit also makes what appears to have been a fernickity bootstrap procedure pretty simple.

Due to limited resources I've only attempted this on AIX 5.2 and AIX 5.3 but both appear to work smoothly. Notes, from previous versions regarding AIX 4.1 have been left.

Setup the base system

For the majority of systems the following will be a non-issue as, more often than not, the entire base system will have been installed on the box and thus the 'bos.adt' packages will be present. To verify your system has those tools run the following command

TODO: installp <check for 'bos.adt'>

you'll notice the ABC output. If it's not there then you need to add those packages by sticking in the relevant CD (first install disk, generally) and running your favourite admin tool or the following command.

TODO: installp <install the package>

Precisely which set of packages is the minimum required set of 'bos.adt' packages to bootstrap is left as an exercise for the reader.

Install a compiler

As you'll probably realise, the one thing that 'pkgsrc' can't do without is a complier. The simplest option is to use the 'gcc' complier which (I think) is available on most versions of AIX (perhaps someone else could clarify these options). There's no particular reason that you can't or shouldn't use another complier but you might want to note that many of the packages within 'pkgsrc' will be dependent on GNU extensions and may cause you problems. Mixing compliers will probably wreak havoc unless you are extremely careful and have an excellent understanding of the dependancies and technologies involved.

Option 1: Using 'gcc' from the AIX toolkit

I believe that 'gcc' is available in AIX but if you don't have it you should be able to download it from IBM like I did. However I'd send a word of warning about having the complete 'AIX linux toolkit' installed because you will inevitably get caught trying to avoid conflicts with the 'pkgsrc' versions of libraries. Consequently, I'd recommend you only get the complier you want and remove everything else via the IBM supplied 'rpm' command.

The first step is to ensure that you have the 'rpm' manager

installp <show rpm>

and install it (in a similar manner to above) if you find it's missing or has been stripped out by an enthusiastic security officer.

Thus, if you follow the above advice, running the command 'rpm -qa' should produce something similar to the following

# rpm -qa
libgcc
gcc
gcc-c++

if you don't, you'll have a much longer list but the above items should be included in it.

P.S: i'm using 'gcc4' from the AIX toolkit, however, it would probably be more prudent to use 'gcc3' which is also available as 'pkgsrc' does not appear to be particularly 'gcc4' friendly.

P.P.S: in achieving the above I removed the 'mkisofs' and 'cdrecord' packages from the system. This suited me fine, however, you may wish to verify whether that's going to impact some of the base system's backup option (i.e. backup to DVD) and/or whether re-instatement of those utilities via 'pkgsrc' will solve those issues.

Option 2: Use a-n-other compiler

I cannot recommend or caution against this option; I'm simply not able to afford the IBM compiler (or other, if there is one). Should anyone wish to give me a license I'll be happy to try it. However, apart from the problems you'll no doubt have with the general adherence to GCC and it's extensions within the 'pkgsrc' packages, it should work.

That's a heavily conditioned should, for those that didn't get it the first time.

Bootstrap away, bootstrap away, bootstrap away

Generally, I like to mirror the native system layout (it's one of the primary reasons I like 'pkgsrc') and so generally I put 'pkgsrc' into '/opt' and use the following bootstrap options

  • --prefix=/opt/pkg
  • --pkgdbdir=/var/opt/pkg

whilst leaving the others with their default 'pkgsrc' setting.

Where you put it is, of course, entirely up to yourself but remember to set the 'CC' environment variable before bootstrap or you'll get into a bit of a mess when the build wrappers fail to find it later. This

# CC=/opt/freeware/bin/gcc ; export CC

is probably what you want but, if you're using another complier, you'll need to change it correspondingly.

Go bootstrap.

# cd /opt/pkgsrc/bootstrap
# ./bootstrap --prefix=/opt/pkg --pkgdbdir=/var/opt/pkg
[ ... ]
Complete success and happiness has been achieved
[ ... ]
Fulfillment of other life ambitions imminient
[ ... ]
Bootstrap successful
#

Hopefully, that's the worst of it over.

Pack-up and wagons roll

Now you need to complete your local configuration

  • set library locations
  • set path
  • set build environment (i.e. mk.conf)
  • set other variables

And ensure that you double check 'README.AIX' for important changes.

The last decision you have to make (for now at least) is whether to use one of the 'pkgsrc' compilers instead of the AIX linux toolkit you just used. Personally, I see little reason, particularly as the latest compiler in 'pkgsrc' is 'gcc3' and the AIX toolkit give me a shiny gcc4.2 version. N.B: as noted above building with 'gcc4' may not be as resilient as 'pkgsrc' seems to be more settled on 'gcc3' at present.

The only thing left is for someone to re-generate the binary bootstrap package for the other AIX souls out there so that the above article is completely useless.

Known issues

The following section outlines the various problems encountered whilst trying to get the system working, note (to self) these should only really appear here if they are specific to AIX. Find somewhere friendlier for more generic notes.

Packages

This is a very limited selection of the packages problems that have been encountered if anyone does a complete build, power to them, perhaps they'll add some info here.

sysutils/fam

It will just not compile - The situation is a bit like for OSF1 as described in PR #31489 for AIX as well. After trying to fix the first errors I decided to add FAM_DEFAULT=gamin in my mk.conf. I've posted a PR for setting this permanently on AIX: #41815

devel/gettext

See bug (todo: look this up)

devel/pcre

Just does not compile (the C++ part cannot be linked) - a bug report in the PCRE project was opened.

lang/perl5

The Configure and Makefile.SH scripts delivered by perl are not properly working when compiling with gcc. They hand over -b arguments to gcc without prefixing them -Wl, which needs to be done because they are meant for ld. I've raised PR #41814 with a patch file included to fix this. The fix was tested on a Power2 System running AIX 5.1 with gcc 3.4 from pkgsrc. OliverLehmann

security/openssh

Another package that thinks it needs 'perl'; OK it does but we don't want to build it so lets just hack out the tools again.

security/openssl

One of those packages that depends on 'perl' simply hack the Makefile to remove 'perl' from the required build tools. This should allow it to build using the base system 'perl'.

Also had trouble with the linking of the shared libraries, not sure if this points to my use of 'gcc4' but manually hacked 'Makefile.shared' (think this is generated so you may need to hack this after the error) to include '--shared' with the linking command. You'll find this in the LINK_SO definition; look for the SHAREDLINK command. -- [ttw](file:///User:Ttw) 

P.S: remember to set the 'PERL5' variable above

security/gnupg2

This isn't actually a problem with the build of 'gnupg2' but it is a problem with the fact that it depends on a bizillion packages. I had issues with 'curl' and 'pinentry'. I should really log bugs for these, and I'll try but I need to get this @$%# completed and get work done too. Anyway, can't remember 'what the problem with 'curl' was but it must have been on of the standard ones here, probably my base perl. 'pinentry' posed more of a problem. Apparently there is a dependancy on 'getopt' missing from the build. I didn't actually fix this correctly, once I finally tracked it down i simply did the following

# ln -s /opt/pkg/include/getopt.h work/.buildlink/include/getopt.h
# for f in /opt/pkg/lib/libgetopt* ; do ln -s $f work/.buildlink/lib/`basename $f` ; done

Next we hit a block with the definition of FD_SETSIZE on AIX which is 65534 but the GNU portable threads that this package uses fixes a maximum of 1024 (although the change log appears to contradict this). Either way I hacked the 'Makefile' such that the static inclusion of the 'pth' stuff (under 'pre-configure' action) includes the '--with-fdsetsize=1024' option.

Current advice would be try 'security/gnupg' instead.

PS: Odd that the build still shows the '-L/opt/pkg/include' but cannot find it. Makes tracking down issues difficult. Need to investigate the 'pkgsrc' mechanics further to understand how/why this works. --ttw 23:45, 17 May 2009 (UTC)

lang/python25

This is a new one by me but the configure script for python explicitly overrides the 'CC' variable we defined when setting up pkgsrc. I've hacked this (once more, i know i should correct these things but i'm busy scratching other itches ... anyway) by adding a 'CONFIGURE_ARGS' to the 'Makefile' with the value '--with-gcc'. this conveniently avoids the evil configure code and allows the default 'CC' to be uses sanely.

NB: The above is only one issue. This doesn't work for me as is.

lang/python24

This is slightly less ugly than 'lang/python25' primarily because it doesn't appear dependent on GNU 'pth' but I didn't really track this down too hard either. Major grip is that IPv6 doesn't work due to a failure in the 'configure' script. That should be hackable but I've currently got it built with '--disable-ipv6', although unsatisfactory.

Further down the line there is a problem with the 'bsddb' package which has some threading issues that I've not investigated. I wasn't interested in it anyway so I simply disabled it (by hacking it out of the Makefile) and we have a build.

I'm not sure that counts as success but I'm ignoring it in favour of my own needs.

mail/dovecot

this is pretty nice all in all but I couldn't get this to build due to issues with 'rquota' in the 'quotas' plugins. the solution was to hack 'quota-fs.c' and change the local 'rquota.h' to the system version () and add the other necessaries (i.e. and ).

Unfortunately, there's a little more incompatability that needs hacking and that's more edits to 'quota-fs.c' where we hack the result.status' to the corresponding 'result.gqr_status' and the 'result.getquota_rslt_u.status' to 'result.gqr_rquota'.

I'm sure all this is "wrong" but it appears to build. Only thing required after that is to ensure you add the 'dovecot' user before doing the install. Happy days.

Operating system

As we all know, AIX isn't anymore singular than other systems and various problems arise on various versions, particularly as both systems are organic entities.

Feel free to add to that knowledge quotient here.

AIX 5L

'undef'd stdio functions within c++ library

the c++ standard libraries appear have a small problem wherein the 'cstdio' 'undef's the macros for 'fgetpos', 'fsetpos', 'fopen', 'freopen'. unfortunately, this is incorrect when using _LARGE_FILE extensions as these macros alias the 64 versions of the functions. if you alter your '/opt/freeware/lib/gcc/powerpc-ibm-aix5.3.0.0/4.2.0/include/c++/cstdio' file to correct this everything should start to flow smoothly again. the following is what i did

# cd /opt/freeware/lib/gcc/powerpc-ibm-aix5.3.0.0/4.2.0/include/c++
# ln cstdio cstdio,v1.1.1.1
# sed -e '/fgetpos/ d' -e '/fsetpos/ d' \
  -e '/fopen/ d' -e '/freopen/ d' \
  cstdio,v1.1.1.1 >cstdio,v1.2

then we add the following to the new file

#ifndef _LARGE_FILES
#undef fgetpos
#undef fsetpos
#undef fopen
#undef freopen
#endif /* _LARGE_FILES */

and replace the active one

# ln -f cstdio,v1.2 cstdio

we can now compile away happily again

'bos.net.nfs.adt' required for various builds

i would imagine that this issue will be transparent to most users and can probably be resolved by a quick search on google, however, i put it here for posterity and completeness and for those whom hacking at any level is alien. basically, you need 'rpcgen' for various builds and that's only included as part of the 'bos.net.nfs.adt' package from the base system. you'll probably have it already.

--ttw 15:18, 7 June 2009 (UTC): this may be incorrect, perhaps 'bos.net.nfs.server' package

AIX 4.1

AIX 4.1 is a pretty old system at this stage, however, AIX systems are renowned for their longevity, although I'd guess there are very few being used as development platforms these days.

Still if you can contribute to clarifying any of the following and/or find some issues of your own feel free to help shuffle the rest of the world round that blockade.

For older AIX releases like this, mirrors of the AIX Public Domain Software Library (aixpdslib) may prove useful to get started.

Conflicting type declarations

Following the instructions to bootstrap pkgsrc-2006Q3 did not work on my older 4.1 system. After several minutes the bootstrap process aborted with the following error:

sh makelist -bc ./vi.c ./emacs.c ./common.c > help.c
sh makelist -bh ./vi.c ./emacs.c ./common.c > help.h
gcc -g -O2 -I. -I./.. -I. -I.. -I./../libedit  -I./../libnetbsd -c chared.c
In file included from sys.h:142,
                 from chared.c:37:
/usr/include/regex.h:172: conflicting types for `regex'
/usr/include/libgen.h:31: previous declaration of `regex'
/usr/include/regex.h:173: conflicting types for `regcmp'
/usr/include/libgen.h:30: previous declaration of `regcmp'
*** Error code 1
Stop.
bmake: stopped in /usr/pkgsrc/bootstrap/work/tnftp/libedit
*** Error code 1
Stop.
bmake: stopped in /usr/pkgsrc/bootstrap/work/tnftp
===> exited with status 1
aborted.

I found an explanation of this error with help from Google:

  • Old Unix versions define these functions in libgen.h, newer in regex.h. It seems that AIX define in two places but with different prototypes.

Not having any skills in C programming, I was unable to resolve the issue by correcting the conflicting definitions and had to resort to sledgehammer tactics; I removed libgen.h but the bootstrap process then failed as it was not able to locate the file. I then overwrote the libgen.h file with a copy of regex.h. This sorted out the conflicting type declaration problem, but I am not sure if this will have any adverse effect on my system! The plan was just to get the bootstrap process to complete then re-instate the original libgen.h file.

You should never change your operating system's files just to make any third-party software run. The proper solution is to tell the author of tnftp (see pkgsrc/net/tnftp/Makefile.common) to have a look at it. --Rillig 13:11, 17 December 2006 (CET)

A workaround -- remove check for libgen.h from tnftp/configure. (Verified on AIX 4.3.2.0, pkgsrc-2007Q1.) Shattered 21:10, 9 July 2007 (CEST)

Undeclared variable

After restarting the bootstrap process, it failed again with the following error:

gcc -g -O2 -I. -I./.. -I. -I.. -I./../libedit  -I./../libnetbsd -c inet_ntop.c
gcc -g -O2 -I. -I./.. -I. -I.. -I./../libedit  -I./../libnetbsd -c inet_pton.c
inet_pton.c: In function `inet_pton4':
inet_pton.c:92: `uint32_t' undeclared (first use in this function)
inet_pton.c:92: (Each undeclared identifier is reported only once
inet_pton.c:92: for each function it appears in.)
inet_pton.c:92: parse error before `val'
inet_pton.c:108: `val' undeclared (first use in this function)
*** Error code 1
Stop.
bmake: stopped in /usr/pkgsrc/bootstrap/work/tnftp/libnetbsd
*** Error code 1
Stop.
bmake: stopped in /usr/pkgsrc/bootstrap/work/tnftp
===> exited with status 1
aborted.

This is as far as I have managed to get at the moment. I will update this page as and when I have a solution to this problem.

I think adding #include at line 25 of pkgsrc/net/tnftp/files/tnftp.h helps. --Rillig 14:03, 17 December 2006 (CET)

AIX 4.1.5 appears not to have inttypes.h. ? ChristTrekker 14:30, 22 April 2009 (UTC)

Which makes sense, since 4.1.5 predates C99. ? ChristTrekker 04:12, 29 April 2009 (UTC)

missing termcap library

Using AIX 4.1.5 and gcc 2.95.2, I get this far...

===> running: (cd /usr/pkgsrc/net/tnftp && /usr/pkgsrc/bootstrap/work/bin/bmake  -DPKG_PRESERVE MAKECONF=/usr/pkgsrc/bootstrap/work/mk.conf install)
ERROR: This package has set PKG_FAIL_REASON:
ERROR: No usable termcap library found on the system.
*** Error code 1

Stop.
bmake: stopped in /usr/pkgsrc/net/tnftp
===> exited with status 1
aborted.

Updates will be posted here as progress is made. ? ChristTrekker 18:57, 17 March 2009 (UTC)

It appears that AIX has a libtermcap.a but doesn't provide termcap.h. ? ChristTrekker 16:00, 18 March 2009 (UTC)

working bootstrap

I don't remember what I did now, but I managed to get bootstrapped. The problem now is that very few packages build, because when trying to make libtool-base I get configure: error: C compiler cannot create executables, which is clearly a bogus problem since I've obviously created executables. I've submitted some patches with PRs, so I really need to wipe out my pkgsrc installation and try bootstrapping from scratch again to see if the process is smoother. ? ChristTrekker 21:24, 9 June 2009 (UTC)

I think fixing this is just a matter of specifying CC on the command line. ? ChristTrekker 18:09, 21 July 2009 (UTC)

See also

README.AIX

External links

Open Source with AIX

Posted late Sunday evening, November 20th, 2011

This article is a stub. You can help by editing it.

This document specifies additional details helpful or necessary when /usr/pkgsrc is mounted via NFS. These are mostly related to /etc/mk.conf variables you can set.

Work files

The working files have to be placed in a writable directory, so if the mount is read-only, you need to add:

WRKOBJDIR=/usr/work

You also need to create that directory. Alternatively, if the mount is read/write, you just need to keep the work files from clobbering those from other platforms, so instead add:

OBJMACHINE=defined

You may want to set this in /usr/pkgsrc/mk/defaults/mk.conf if you intend on using it anywhere, so that it will be used everywhere and you don't shoot yourself in the foot accidentally.

Distribution files

The distribution files have to be placed in a writable directory, so if the mount is read-only, you need to add:

DISTDIR=/usr/distfiles

You also need to create that directory. Alternatively, you can mount the NFS partition read/write, which can be helpful if you build the same packages on multiple platforms.

Generated packages

If you generate binary packages, they have to be placed in a writable directory where they will not clobber those from other platforms, so add:

PACKAGES=/usr/pkg/packages

You may want to set this in /usr/pkgsrc/mk/defaults/mk.conf if you intend on using it anywhere, so that it will be used everywhere and you don't shoot yourself in the foot accidentally. (Is it possible to do something like /usr/pkg/packages/sys-ver/arch automatically?)

Posted late Sunday evening, November 20th, 2011

Contents

What is pkgsrc

Pkgsrc [spoken: package source] is the main package management framework for NetBSD. With pkgsrc you can easily add, remove and manage software on your system. Pkgsrc is basically a set of files, grouped by categories which contain information to install the software you have selected. All these files together are mostly referred to as the pkgsrc tree. This tree is maintained by the pkgsrc developers, who make changes to it every day. Therefore it is necessary to update the pkgsrc tree regularly.

Documentation

It is strongly advised to rely on information in The pkgsrc Guide This wiki is semi-official and is frequently outdated. Sometimes even misleading.

Preparing pkgsrc

Obtaining the current pkgsrc source tree

See The pkgsrc Guide

Creating WRKOBJDIR

To keep the tree clean and your work directories out of it, create a directory, e.g.

# mkdir /usr/work

and define WRKOBJDIR in /etc/mk.conf:

WRKOBJDIR=/usr/work

Creating DISTDIR

We also want our distfiles to be stored, outside of the pkgsrc directory. Therefore we add the DISTDIR variable to /etc/mk.conf

DISTDIR=/usr/distfiles

and create it with:

# mkdir /usr/distfiles

Installing packages

To install packages, we need to become root.

$ su 

then we change to the directory (category) and then to the package we want to install.

# cd /usr/pkgsrc/misc/figlet

to install we enter

# make install

afterwards we clean up and enter

# make clean

if this was a package with dependencies, we also enter

# make clean-depends

You can put them all in one line too.

# make install clean clean-depends

If you wish to clean the distfiles, the files that have been downloaded, you enter

# make distclean

List Packages

$ pkg_info

Removing Packages

# pkg_delete packagename

Updating Packages

You can update a single package using make update.

# make update

On-line help

Besides The pkgsrc Guide there is also a built-in on-line help system.

# make help

gives you the usage information. This requires you to already know the name of the target or variable you want more info on (just like man does).

Most targets and variable names are documented, but not all are. See also

Posted late Sunday evening, November 20th, 2011

Some folks don't realize that NetBSD provides an easy way to configure many packages. To see if a particular package has such options, while in the /usr/pkgsrc// directory type

make show-options

As an example, we'll use uim, an input method for Japanese.

cd /usr/pgksrc/inputmethod/uim
make show-options

I see the following


Any of the following general options may be selected:
       anthy    Use Anthy as Japanese conversion program.
       canna    Use Canna as Japanese conversion program.
       eb       Enable EB dictionary library support.
       gtk      Enable support for GTK.
       qt

These options are enabled by default: anthy canna gtk
These options are currently enabled: anthy canna gtk

If one only wants the default options, then a simple make install clean; make clean-depends will install them. However I don't want the defaults. I do want anthy and gtk however, I don't want canna and wish to add qt.

One can either do this at the command line or put the lines in /etc/mk.conf. Usually, the variable will be called PKG_OPTIONS.pkgname where pkgname is, oddly enough, the name of the package. Most packages that have options will have an options.mk file in their directory. A quick glance at that will show you the name of the variable. (If there is no options.mk file, the name of the variable can be found in the Makefile.) In this case the options.mk file has the line

PKG_OPTIONS_VAR=        PKG_OPTIONS.uim

So, I will type

make PKG_OPTIONS.uim="qt -canna" install clean; make clean-depends. 

This will install gtk, qt and anthy.

Most people will put these options in /etc/mk.conf so that they don't have to remember it each time. In this case, it's only a few options, but for something like /usr/pkgsrc/x11/xorg-server there are about 30 options, most of which you won't want. Typing make show-options in x11/xorg-server's directory gives me (shortened for the reader's sake)

These options are enabled by default: xorg-server-apm xorg-server-ark xorg-server-ati xorg-server-chips xorg-server-cirrus xorg-server-cyrix xorg-server-dummy xorg-server-glint xorg-server-i128 xorg-server-i740 xorg-server-i810 xorg-server-imstt xorg-server-mga xorg-server-neomagic xorg-server-newport xorg-server-nsc xorg-server-nv xorg-server-rendition xorg-server-s3 xorg-server-s3virge xorg-server-savage xorg-server-siliconmotion xorg-server-sis xorg-server-tdfx xorg-server-tga xorg-server-trident xorg-server-tseng xorg-server-vesa xorg-server-vga xorg-server-via xorg-server-vmware

I don't want to type make PKG_OPTIONS="-xorg-server-ati -xorg-server-cardb -xorg-server-cardc" and all the rest each time I reinstall it so I would definitely put it in /etc/mk.conf.

When adding PKG_OPTIONS.pkgname options to etc/mk.conf, don 't put quotes around them. In our example of uim, I would add this line to /etc/mk.conf

PKG_OPTIONS.uim=qt -canna

If I write

PKG_OPTIONS.uim="qt -canna" 

in /etc/mk.conf, I will get an error message when I try to build the package. (Or possibly when I try to build any package.)

So, to sum up

If you want default options, don't do anything. If you want available options that aren't enabled by default add them to the PKG_OPTIONS variable, either quoted on the command line, eg PKG_OPTIONS.uim="qt" or in /etc/mk.conf. If you put them in mk.conf don't use quotes.

If you don't want an option enabled by default, use a - in front of it, either quoted on the command line or without quotes in /etc/mk.conf

Posted late Sunday evening, November 20th, 2011

NetBSD consists of a core base system and any third-party software you install on top. The NetBSD base system can be upgraded in numerous ways, including booting a new installer image, using the sysupgrade tool, or building from source. For more information, see the NetBSD guide.

NetBSD uses pkgsrc for managing third-party software. The instructions for upgrading pkgsrc packages have moved to the pkgsrc guide.

Posted late Sunday evening, November 20th, 2011

This article describes by examples how a ?pkgsrc developer can update a package to a later version.

You may be looking for ?How to update a package with pkgsrc

Contents

Preparation

Please install pkgtools/pkgdiff. This tools helps creating patches for pkgsrc. It contains pkgvi, mkpatches and patchdiff.

Trivial: games/gogui from 0.9.1 to 0.9.3

Part 1: Get the new package to build

  • Change the version number in the DISTNAME variable to 0.9.3.
  • Run make mdi to download the new distfile and regenerate the checksums.
  • Run make.
  • Wait until the build has finished ...

Part 2: Install the new package

  • Run make deinstall to remove the old package.
  • Run make install PKG_DEVELOPER=yes, hoping that the list of installed files has not changed.
  • There is no document describing the changes. Only on the SourceForge project page, there are two comments about minor bugfixes. Keep them in mind for later.
  • The package has installed without any failures. That's good.
  • Run make package clean to finish the work.

Part 3: Committing the update

  • Run pkglint -Wall to see if the package looks good.
  • Run cvs ci to commit the update.

    • The text of the commit message is Updated gogui to 0.9.3, followed by an empty line, followed by a comment about the changes.
  • Run (cd ../../doc && cvs up)

  • Run make changes-entry
  • Run (cd ../../doc && cvs ci CHANGES-*).

    • The text of the commit message is Updated gogui to 0.9.3.

Simple: devel/rapidsvn from 0.9.3 to 0.9.4

Part 1: Get the new package to build

  • Change the version number in the DISTNAME variable to 0.9.4.
  • Run make mdi to download the new distfile and regenerate the checksums.

    ... fetch: Unable to fetch expected file rapidsvn-0.9.4.tar.gz

  • Look at the homepage (see HOMEPAGE) to see where the distfile is.

  • Change MASTER_SITES accordingly.
  • Run make mdi again.
  • Run make and hope that the old patches still apply.

    ... => Applying pkgsrc patches for rapidsvn-0.9.4 1 out of 1 hunks failed--saving rejects to configure.rej Patch /home/roland/proj/NetBSD/pkgsrc/devel/rapidsvn/patches/patch-ab failed ERROR: Patching failed due to modified or broken patch file(s): ERROR: /home/roland/proj/NetBSD/pkgsrc/devel/rapidsvn/patches/patch-ab

  • They don't. :(

  • Look at patches/patch-ab to see which file it is applied to. It's the configure script, so have a look at that.
  • Hmmm, it looks quite different. Let's see if the old line still exists.
  • Look for "$GXX" ==. It's in line 19749.
  • So we need to remove one of the equals characters, since that's what the patch did.
  • Change to the work directory ($WRKOBJDIR) and edit the file(s) you need to patch with pkgvi, make the changes and save it. pkgvi will automatically create a diff.
  • Change back to the package directory (e.g. .../pkgsrc/devel/rapidsvn/ and run mkpatches. This will create the new patches from the new diffs in $WRKOBJDIR/.newpatches/
  • Now copy the new patch(es) from $WRKOBJDIR/.newpatches/ to patches/ in the package directory.
  • Run make patches to regenerate the patches from the working directory.
  • Look at the patches to see if they contain more than they should, or expanded strings (for example, /usr/pkg instead of @PREFIX@.
  • Run make mps to regenerate the checksums.
  • Run make clean; make to try again.
  • In the mean time, create a patch for configure.in, since that's the source of the code which got patched in the configure script.
  • Wait until the build has finished ...

Part 2: Install the new package

  • Run make deinstall to remove the old package.
  • Run make install PKG_DEVELOPER=yes, hoping that the list of installed files has not changed.
  • Meanwhile, look at the NEWS, ChangeLog and CHANGES files in the WRKSRC to see what has changed between 0.9.3 and 0.9.4. Save that text to later include it in the commit message.
  • The package has installed without any failures. That's good.
  • Run make package clean to finish the work.

Part 3: Committing the update

  • Run pkglint -Wall to see if the package looks good.
  • Run cvs ci to commit the update.

    • The text of the commit message is Updated rapidsvn to 0.9.4, followed by an empty line, followed by the text from the CHANGES file.
  • Run (cd ../../doc && cvs up)

  • Run make changes-entry
  • Edit ../../doc/TODO to remove the line of the requested update.
  • Run (cd ../../doc && cvs ci CHANGES-* TODO).

    • The text of the commit message is Updated rapidsvn to 0.9.4, without all the details.

Part 4: Sending the patches to the upstream authors

Posted late Sunday evening, November 20th, 2011

You can easily save all of your currently installed NetBSD packages for use on other machines by using pkg_tarup. Here's what to do.

  1. Install pkg_tarup from pkgsrc. ie.. cd /usr/pkgsrc/pkgtools/pkg_tarup ; make install (as root)
  2. Now you can now use pkg_tarup to create a tarball package from any currently installed package. Just do pkg_tarup [packagename]

Here is an example of how powerful this technique can be. With only a few lines of shell code we can put all the installed packages into a directory:


cd /var/tmp
mkdir packages
cd packages
for PKGNAME in `pkg_info -e "*" | sort`; do
    echo "Packaging $PKGNAME"
    pkg_tarup -d "." "$PKGNAME" || echo "WARNING: Packaging $PKGNAME failed." 1>&2
done
Posted late Sunday evening, November 20th, 2011

Contents

No X installed

Assuming you have just made a clean install of NetBSD and you have not installed the X11 sets, you should do the following:

/etc/mk.conf

Edit your /etc/mk.conf and add the following line:

X11_TYPE=modular

Install xorg

Provided that your pkgsrc tree lies in /usr/pkgsrc, type:

# cd /usr/pkgsrc/meta-pkgs/modular-xorg
# make install

Configure Xorg

Usually no configuration is necessary. To start X you use:

$ startx

Additional configuration

Additional adjustments can be made in ~/.xinitrc and ~/.Xresources. For example, you may want to use a different terminal emulator font, such as pkgsrc/fonts/overpass.

~/.Xresources

To do so, we can add this line to ~/.Xresources:

*font: xft:overpass mono:size=12

For this to take effect, we must do merge it into the database.

$ xrdb -merge ~/.Xresources

To do this at every X startup, we can add it to ~/.xinitrc.

~/.xinitrc

An example ~/.xinitrc to run a window manager and configure keyboard layout:

xrdb merge ~/.Xresources                        # respect ~/.Xresources configuration

setxkbmap -option grp:alt_shift_toggle us,il    # two keyboard options, US keyboard and IL,
                                        # with alt+shift as a toggle option

openbox                                 # run your favourite window manager.
Posted late Sunday evening, November 20th, 2011

Installing a PostgreSQL Server under NetBSD is pretty easy. The recommended way is to install the prebuilt postgresql binaries. The PostgreSQL Server depends on the postgreSQL Client. Contents

Setting PKG_PATH

Setting PKG_PATH enables us to easily download and install packages and it's dependencies.

# export PKG_PATH=https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/$(uname -m)/$(uname -r)/All/ 

or if you are using csh:

# setenv PKG_PATH https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/$(uname -m)/$(uname -r)/All/ 

You should keep this in your .profile, .bash_profile, if you mainly using binaries.

Installing the PostgreSQL Server

# pkg_add -v postgresql92-server

This will install the postgresql client and the server and any missing dependency (readline, perl) and will add the user pgsql to your user database.

That's it. Almost.

Configuring the Server

Please copy the pgsql example script to /etc/rc.d/

# cp /usr/pkg/share/examples/rc.d/pgsql /etc/rc.d

If you want to keep the database cluster other place than the default location, just change the user pgsql's home directory, before proceeding to the initialisation:

# usermod -d /var/db/pgsql pgsql

This directory must be owned by pgsql:pgsql.

and then

Starting the Server

If you want to initialise the database with a local other than "C", for example with unicode, invoke this before starting postgresql for the first time:

# /etc/rc.d/pgsql initdb -E unicode

please start the server entering:

# /etc/rc.d/pgsql start

This will create all necessary initial databases on the first start.

Start on every boot

To start the server on every boot add

pgsql=yes

to your /etc/rc.conf

Creating an example Database

To create a database switch to the user pgsql

# createdb -e -h 127.0.0.1 -U pgsql newdbname

and create the database

$ createdb testdb
$ exit

Using the Database

Postgres provides a Tool to manage the Database called psql.

# psql -U pgsql testdb
Welcome to psql 8.0.4, the PostgreSQL interactive terminal.

Type:  \copyright for distribution terms
       \h for help with SQL commands
       \? for help with psql commands
       \g or terminate with semicolon to execute query
       \q to quit

testdb=#

PHP and PostgreSQL

You may wish to install the postgres Module for PHP, e.g. for PHP 7.0 use:

# pkg_add -v php70-pgsql

Have fun.

Additional Information

Posted late Sunday evening, November 20th, 2011

Installing a MySQL server is easy. The fastest way is to install the binary package. If you wish to use Apache and PHP, please read the article How to install a LAMP Server.

# pkg_add -v mysql-server

If you decide to install via pkgsrc, because of a newer version, just enter:

# cd /usr/pkgsrc/databases/mysql57-server/
# make install clean

It will automatically compile the client and the server.

That's it... almost. Now copy the rc.d script into /etc/rc.d/:

# cp /usr/pkg/share/examples/rc.d/mysqld /etc/rc.d/

and start the MySQL server via NetBSD's rc.d(8) framework. Only one modification is needed to rc.conf(5):

# echo "mysqld=yes" >> /etc/rc.conf

If you want to copy the rc.d scripts automatically with pkgsrc, you can use:

PKG_RCD_SCRIPTS=YES 

into mk.conf(5).

If MySQL is not starting up, you may need to create and set permissions on the /var/mysql directory and set up the default MySQL DB:

# mkdir /var/mysql
# mysql_install_db
# chown -R mysql:mysql /var/mysql

The default MySQL server database root password is auto-generated and marked expired upon creation. For security reasons, you should set your root password as soon as possible.

If you need information about the MySQL server package, use the following:

# pkg_info mysql-server

Have fun.

See also

Posted late Sunday evening, November 20th, 2011

LAMP is a an acronym for a combined set of software to run a web server containing the following software products: Apache, MySQL, and Perl, Python, or PHP. The "L" stands for Linux, therefore there is also an acronym named WAMP representing the Windows operating system. This also means that the title of this article is misleading. The approach is to install the same combined set of software, but using NetBSD as the operating system instead of Linux.

In the following examples, we will install all components using pkgsrc, building all packages from source.

Contents

Installing the Apache web server

The new Apache 2.4 server comes with two different threading models from which prefork is installed by default. It is not recommended to use the Worker model, if you wish to use Apache and PHP. As that is the case, we will install a default Apache 2.4 server.

# cd /usr/pkgsrc/www/apache24
# make install clean clean-depends

This will install the Apache 2.4 server and all its dependencies. If your build was successful, you should now edit the Apache configuration file /usr/pkg/etc/httpd/httpd.conf to fit your needs. At least set the Listen Attribute and your ServerName. Please ensure that if your machine's hostname does not globally resolve, to put it into your /etc/hosts file, otherwise Apache will refuse to start.

If you wish to start the Apache web server at boot time, please copy the rc.d example script from /usr/pkg/share/examples/rc.d/apache to /etc/rc.d and then add apache=yes to your /etc/rc.conf file.

# cp /usr/pkg/share/examples/rc.d/apache /etc/rc.d

If you want to copy the rc.d scripts automatically with pkgsrc, you can use:

PKG_RCD_SCRIPTS=YES 

in your /etc/mk.conf

You can now start, stop, and restart the Apache web server using apachectl, or using boot script /etc/rc.d/apache

To start the server enter:

# apachectl start

or

# /etc/rc.d/apache start

To stop the server, substitute start with stop. If you're running a production server, pay attention to the apachectl graceful option.

Installing MySQL

You can skip this part, if you don't want to install a MySQL server. To install the MySQL server enter:

# cd /usr/pkgsrc/databases/mysql57-server
# make install clean clean-depends

This will install the MySQL server and all its dependencies, like the MySQL client.

Configuring the MySQL server

Please copy the example start script to /etc/rc.d

# cp /usr/pkg/share/examples/rc.d/mysqld /etc/rc.d

and add mysqld=yes to your /etc/rc.conf

You can now start, stop, and restart the MySQL server using

# /etc/rc.d/mysqld start

to start and respectively stop and restart.

The default MySQL server database root password is auto-generated and marked expired upon creation. For security reasons, you should set your root password as soon as possible.

You can pass most of the options to the server via the file /etc/my.cnf. If you want the server to listen only on localhost, for instance, create /etc/my.cnf and add

[mysqld]
port=3306
bind-address=127.0.0.1

and restart your MySQL server. To check if your MySQL server is really listening only on localhost, use ?sockstat.

# sockstat -l

For many more options, consider reading the MySQL Documentation.

Installing the PHP module for Apache

# cd /usr/pkgsrc/www/ap-php
# make install clean

This will install by default the latest version of PHP 7.x and the PHP7 module for Apache 2.4

Configuring PHP

You should now add the LoadModule and the PHP Handlers definitions to your Apache Configuration File /usr/pkg/etc/httpd/httpd.conf

Add following lines:

LoadModule php7_module /usr/pkg/lib/httpd/mod_php7.so

and

AddType application/x-httpd-php .php

and if you wish

DirectoryIndex index.html index.php

Installing the MySQL module for PHP

This step is important and enables you to make MySQL database connections from your PHP script.

cd /usr/pkgsrc/databases/php-mysql/
make install clean

Now edit /usr/pkg/etc/php.ini and add the line

extension=mysql.so

You need this to enable MySQL functions in your PHP module.

Now restart your Apache web server. To test if PHP is working, create a small file called test.php in your document root directory, which is by default /usr/pkg/share/httpd/htdocs, containing only one line with the function phpinfo().

<?php phpinfo(); ?>

If you use PHP7 and wish to use short tags like <? phpinfo() ?>, then edit your /usr/pkg/etc/php.ini file and change option short_open_tag = Offto On to make this line work. In PHP7 short_open_tag is off by default.

Open your browser and point it to this URL:

http://127.0.0.1/test.php

You should now see a website with information regarding your PHP installation and a table named mysql, in the middle of the document, with MySQL information.

That's it. You can now install software like a phpMyAdmin, or a Wiki. Have fun.

See also

Commands

  • ?sockstat
Posted late Sunday evening, November 20th, 2011

First, I bootstrapped pkgsrc into the directory /usr/pkg/2006Q4.

$HOME/proj/pkgsrc/bootstrap/bootstrap --prefix=/usr/pkg/2006Q4 --unprivileged --compiler=sunpro
mkdir /usr/pkg/2006Q4/etc
mv work/mk.conf.example /usr/pkg/2006Q4/etc/mk.conf

Then I made a backup of that directory. You can never know. :)

cd /usr/pkg && gtar cfz 2006Q4.tar.gz 2006Q4

Since in a previous try, the pre-build program had removed the pkgsrc directory, I just commented it out by inserting an exit 0 in line 2. Since now there is no danger that anything in that directory is deleted, I could edit the mk.conf file and adjust some local settings.

/usr/pkg/2006Q4/mk.conf


# Example /usr/pkg/2006Q4/etc/mk.conf file produced by bootstrap-pkgsrc
# Tue Jan  9 13:01:49 CET 2007

.ifdef BSD_PKG_MK   # begin pkgsrc settings

PKGSRC_COMPILER=    sunpro
UNPRIVILEGED=       yes
PKG_DBDIR=      /usr/pkg/2006Q4/var/db/pkg
LOCALBASE=      /usr/pkg/2006Q4
VARBASE=        /usr/pkg/2006Q4/var
PKG_TOOLS_BIN=      /usr/pkg/2006Q4/sbin
PKGMANDIR=      man

TOOLS_PLATFORM.awk?=        /usr/pkg/2006Q4/bin/nawk
TOOLS_PLATFORM.sed?=        /usr/pkg/2006Q4/bin/nbsed
FETCH_CMD=          /usr/pkg/2006Q4/bin/ftp
TOOLS_PLATFORM.pax?=        /usr/pkg/2006Q4/bin/pax
TOOLS_PLATFORM.tar?=        /usr/pkg/2006Q4/bin/tar
TOOLS_PLATFORM.mtree?=      /usr/pkg/2006Q4/sbin/mtree

DISTDIR=        /usr/pkg/distfiles
PACKAGES=       /usr/pkg/2006Q4-packages

PKG_DEVELOPER=      yes

CC= cc
CXX=    CC
CPP=    cc -E
CXXCPP= CC -E

SUNWSPROBASE=       /local/SUNWspro
USE_LANGUAGES+=     c99

TOOLS_PLATFORM.nroff=   /opt/bin/groff

.endif          # end pkgsrc settings

/usr/pkg/2006Q4/build.conf


# $NetBSD: how_to_do_an_unprivileged_bulk_build_on_solaris.mdwn,v 1.2 2012/02/05 07:14:36 schmonz Exp $
#

osrev=`uname -r`
arch=`uname -m`
USR_PKGSRC="$HOME/proj/pkgsrc"
MAKECONF="/usr/pkg/2006Q4/etc/mk.conf"

PRUNEDISTFILES=no
NICE_LEVEL="nice -n 20"
LINTPKGSRC_CACHE=no

ADMIN="1illig@informatik.uni-hamburg.de"
ADMINSIG="- Roland"

REPORTS_DIR="/usr/pkg/2006Q4-pkgstat"
REPORT_BASEDIR=`date +%Y%m%d.%H%M`
REPORT_HTML_FILE="report.html"
REPORT_TXT_FILE="report.txt"
REPORTS_URL="file://$REPORTS_DIR"

UPDATE_VULNERABILITY_LIST=yes
PRUNEPACKAGES=yes
MKSUMS=yes
MKSUMMARY=no
RSYNC_DST=ftp.NetBSD.org:/pub/NetBSD/packages/pkgsrc-200xQy/NetBSD-a.b.c/i386
RSYNC_OPTS='-e ssh'

I began to setup the bulk build environment in a screen.

$ screen -S <ulk-2006Q4
$ sh ./proj/pkgsrc/mk/bulk/build -c /usr/pkg/2006Q4/etc/build.conf

It seems to work, and I'm waiting for the package databases to be generated. But since I have a T2000 here, I'm thinking about parallelizing the work. For generating the databases, the two "make" processes could be run completely in parallel, making use of 12000 processors. :)

In the main phase, all packages that have the exact same dependencies can be built in parallel, but must be still installed one after another.

Help, my files are being removed

When I had started the bulk build, it complained that it couldn't find /usr/pkg/2006Q4/bin/nawk anymore. But looking at the backup I made above, I say that it had been there after bootstrapping. The simple solution to this was to change the tools definitions in mk.conf so that they point to tools outside ${LOCALBASE}:


TOOLS_PLATFORM.awk= /usr/pkg/current/bin/nawk
TOOLS_PLATFORM.sed?=    /usr/pkg/current/bin/nbsed
FETCH_CMD=      /usr/pkg/current/bin/ftp
TOOLS_PLATFORM.pax?=    /usr/pkg/current/bin/pax
TOOLS_PLATFORM.tar?=    /usr/pkg/current/bin/tar
TOOLS_PLATFORM.mtree?=  /usr/pkg/current/sbin/mtree

Luckily, bin/bmake is not registered as belonging to any package, so it hasn't been removed yet.

The bulk build also tried to deinstall the infrastructure packages, so I had to protect them against that:


BULK_PREREQ+=       pkgtools/bootstrap-mk-files
BULK_PREREQ+=       pkgtools/tnftp
BULK_PREREQ+=       pkgtools/mtree
BULK_PREREQ+=       pkgtools/pax
BULK_PREREQ+=       pkgtools/pkg_install
BULK_PREREQ+=       sysutils/mtree

BULK_PREREQ+=       sysutils/checkperms
Posted late Sunday evening, November 20th, 2011

First, I bootstrapped pkgsrc into $HOME/bulk:


$ cd
$ env \
     CC=cc \
     CXX=CC \
     MIPSPROBASE=$HOME/mipspro-wrapper \
     ./proj/pkgsrc/bootstrap/bootstrap \
         --prefix=$HOME/bulk \
         --unprivileged \
         --compiler=mipspro \
         --quiet
...

Then, make a backup copy of LOCALBASE:

 $ tar cfz bulk.tar.gz bulk

It is a good idea to store all configuration files outside of LOCALBASE:

 $ mkdir bulk-etc
 $ cp bulk/etc/mk.conf bulk-etc/.

Like on Solaris, I needed to comment out the pre-build script. The configuration files look similar to the ones from the Solaris build.

Contents

build.conf


osrev=`uname -r`
arch=`uname -m`
USR_PKGSRC="$HOME/proj/pkgsrc"
MAKECONF="$HOME/bulk-etc/mk.conf"

PRUNEDISTFILES=no
NICE_LEVEL="nice -n 20"
LINTPKGSRC_CACHE=no
ADMIN="rillig@localhost"
ADMINSIG="- Roland"

REPORTS_DIR="$HOME/bulk-reports"
REPORT_BASEDIR=`date +%Y%m%d.%H%M`
REPORT_HTML_FILE="report.html"
REPORT_TXT_FILE="report.txt"
REPORTS_URL="file://$REPORTS_DIR"

UPDATE_VULNERABILITY_LIST=no
PRUNEPACKAGES=no
MKSUMS=no
MKSUMMARY=no
RSYNC_DST=none
RSYNC_OPTS=none

mk.conf


# Example /usr/people/rillig/bulk/etc/mk.conf file produced by bootstrap-pkgsrc
# Wed Feb 21 07:42:55 EST 2007

.ifdef BSD_PKG_MK   # begin pkgsrc settings

OPSYS=          IRIX
ABI=            64
PKGSRC_COMPILER=    mipspro
UNPRIVILEGED=       yes
PKG_DBDIR=      /usr/people/rillig/bulk/var/db/pkg
LOCALBASE=      /usr/people/rillig/bulk
VARBASE=        /usr/people/rillig/bulk/var
PKG_TOOLS_BIN=      /usr/people/rillig/bulk/sbin
PKGMANDIR=      man

TOOLS_PLATFORM.install?=    /usr/people/rillig/pkg/bin/ginstall
TOOLS_PLATFORM.sed?=        /usr/people/rillig/bulk/bin/nbsed
FETCH_CMD=          /usr/people/rillig/bulk/bin/ftp
TOOLS_PLATFORM.pax?=        /usr/people/rillig/bulk/bin/pax
TOOLS_PLATFORM.tar?=        /usr/people/rillig/bulk/bin/tar
TOOLS_PLATFORM.mtree?=      /usr/people/rillig/bulk/sbin/mtree
IMAKEOPTS+=     -DBuild64bit -DSgiISA64=4

DISTDIR=        /usr/people/rillig/distfiles
PKG_DEVELOPER=      yes
CC=         cc
CXX=            CC
CPP=            cc -E
CXXCPP=         CC -E

MIPSPROBASE=        /usr/people/rillig/mipspro-wrapper
TOOLS_PLATFORM.nroff=   /usr/people/rillig/pkg/bin/groff
BULKFILESDIR=       /usr/people/rillig/bulk-logs
PACKAGES=       /usr/people/rillig/bulk-packages
WRKOBJDIR=      /usr/people/rillig/bulk-tmp

.endif          # end pkgsrc settings

To get a usable environment, I have built some package before in $HOME/pkg, so that I can link to them (for example ginstall and groff).


$ screen -S bulk
$ sh proj/pkgsrc/mk/bulk/build -c /usr/people/rillig/bulk-etc/build.conf
...
BULK> Package bzip2-1.0.4 not built yet, packaging...
/usr/people/rillig/pkg/bin/bmake bulk-package PRECLEAN=no
###
### Wed Feb 21 08:56:12 EST 2007
### pkgsrc build log for bzip2-1.0.4
###

bmake: exec(/bin/sh) failed (Arg list too long)
*** Error code 1

Grmpf. IRIX has only 20k for the command line and the environment, and that gets filled pretty quickly. Let's see what's the cause for that. It's the command starting with "${RUN} set +e;" and going until the lonely "fi" in line 534. That's pretty much. Since I don't know how to break that into smaller chunks, I stop here.

Some hours later

I've made some progress. I didn't rewrite the bulk builds but simply patched bmake to create a temporary file and write the overly long command lines there when an execve fails due to E2BIG. I applied this patch in the devel/bmake directory and re-ran bootstrap. Things seem to work now.

At least, lang/perl is building, which takes some time. Not to speak of the database generation that follows. Meanwhile, I'm adding the following to the mk.conf file, just as in the Solaris bulk build:


BULK_PREREQ+=       pkgtools/bootstrap-mk-files
BULK_PREREQ+=       pkgtools/tnftp
BULK_PREREQ+=       pkgtools/mtree
BULK_PREREQ+=       pkgtools/pax
BULK_PREREQ+=       pkgtools/pkg_install
BULK_PREREQ+=       sysutils/mtree

BULK_PREREQ+=       sysutils/checkperms

See also

Posted late Sunday evening, November 20th, 2011

When working with pkgsrc, pkgsrc leaves working directories called work in the application directory. If you invoke the build with clean, then this directory is cleaned, but not the working directories of the dependencies. To avoid this problem, you should set DISTDIR and WRKOBJDIR in mk.conf.

# make install clean

If you want to clean the directories of the dependencies aswell you have to run:

# make install clean clean-depends

Either way, sometimes it is not possible or wanted to clean the working directories. You can clean them all using this one line shell command:

# find /usr/pkgsrc -name work -exec rm -r {} +

or this one:

# find /usr/pkgsrc -maxdepth 3 -mindepth 3 -name work -exec rm -r {} +

You can also change the place where the work directories are created by setting WORKOBJDIR in /etc/mk.conf.

You can clean them using make clean in the parent directory, but this is not advised. It takes a very long time. Using pkgclean is another option.

See also

Posted late Sunday evening, November 20th, 2011

Contents

Questions & Answers

How can I get a list of all ?make variables that are used by pkgsrc?

That's difficult. But you can get a very good approximation by changing to a package directory and running the following command:

make -dv show-var VARNAME=VARNAME \
| sed -n 's,^Global:\([^ ]*\) =.*,\1,p' \
| sed 's,\..*,.*,' \
| sort -u \
| grep ^\[A-Z\] \
| less

Another possibility is to run bmake show-all. This will list many (but not all) variables.

If you need more information about a specific variable, run bmake help topic=VARNAME or have a look at pkglint's variable definition file

When patching a GNU-style configure script, where should I add changes?

If you want your changes to override everything else, then look for "ac_config_files=" and put it somewhere before that line.

I'm going to make incompatible changes to pkgsrc. Where should I document it?

In the file doc/CHANGES-*.

What's the difference between ${TEST}, test and [?

There is practically no difference. All the standard options are supported on all platforms. See also ?The pkgsrc portability guide.

See also

Posted late Sunday evening, November 20th, 2011

This howto explains how NetBSD current can use thumb mode on ARM architecture (evbarm).

Introduction

While normal ARM instructions are 32 bits wide, thumb provides a subset of ARM instructions in 16 bits, and thus reduces executable sizes both in memory and on filesystem 1. See Wikipedia for more details.

On large codebases like NetBSD, ARM and thumb mode code can, and must, co-exists. This is because some parts of ARM specific code can not be programmed with the 16 bit thumb instructions. Luckily GCC makes this relatively easy with the thumb-interwork option.

As an overview, most machine independent (MI) C code can be compiled to thumb mode and most machine dependent (MD) C and assembly code can only be compiled to ARM mode. Some parts of the ARM port specific code in NetBSD have support for thumb mode, but most of them do not.

NetBSD's CPUFLAGS build variable is used tell the compiler, when an object file is compiled to thumb mode and when thumb interworking needs to enabled. The GCC options are -mthumb and -mthumb-interwork. By default ARM target binaries are of course compiled to ARM mode.

In a large codebase like NetBSD it becomes difficult to manually check if any one object file can be compiled to thumb mode. Luckily brute force works with the help of make option -k, as in keep going even one object file does not compile. By compiling whole tree with CPUFLAGS=-mthumb and MAKEFLAGS=-k, all of the build time failing machine dependent object files can be found, and marked with the help of Per file build options override to be compiled to ARM mode with thumb interworking.

Build time failures are of course not the only things can go wrong with thumb support. At run time, some parts of the kernel or userspace are expected to be in ARM mode. In userspace, for example, this means that the dynamic linker (ld.elf_so), userspace locking and C runtime initialization (CSU) parts need to be in ARM mode.

Userspace

Userspace in current compiles to userspace, but doesn't work due to a linker bug 23.

If the binutils package is upgraded to 2.18.50 from 2.16.1, then thumb mode works for userspace. After this, only a build script marking the ARM and thumb mode files is needed.

Following patches do it all for current snapshot from Oct 22nd 2008:

  • binutils upgrade and tuning for NetBSD in 4 and 5
  • thumb build script 6

After the patches have been applied on top of current, the build process continues like this:

  • build tools

    ./build.sh -U -O obj -m evbarm -j 3 tools

  • build userspace to thumb, note that the script expects to find tools directory under obj

    ./thumb.sh

  • build a normal ARM mode kernel, where CONFIG is the configuration file name like TISDP2420

    ./build.sh -U -O obj -m evbarm -j 3 kernel=CONFIG

The next step is to verify that the thumb mode userspace works by booting into it. On OMAP 2420 development board this requires setting up a TFTP server for the kernel executable netbsd.bin and then setting up the root filesystem over NFS (and configuring the kernel accordingly).

Known problems

While the thumb mode userspace boots and most programs and the shell work, some programs may have issues. Known regressions at this time are:

  • thumb mode gcc segfaults on target

    # cat > test.c << EOF
    > int main(void){
    > return 0;
    > }
    > EOF
    # gcc --verbose -Wall test.c
    Using built-in specs.
    Target: arm--netbsdelf
    Configured with: /usr/src/tools/gcc/../../gnu/dist/gcc4/configure --enable-longtThread model: posix
    gcc version 4.1.3 20080704 prerelease (NetBSD nb1 20080202)
     /usr/libexec/cc1 -quiet -v test.c -quiet -dumpbase test.c -auxbase test -Wall s#include "..." search starts here:
    #include <...> search starts here:
     /usr/include
    End of search list.
    GNU C version 4.1.3 20080704 prerelease (NetBSD nb1 20080202) (arm--netbsdelf)
            compiled by GNU C version 4.1.3 20080704 (prerelease) (NetBSD nb1 20080.GGC heuristics: --param
            ggc-min-expand=34 --param ggc-min-heapsize=7808
    Compiler executable checksum: c67c46e1fc2de869e7015b2c172bd073
    test.c: In function 'main':
    test.c:3: internal compiler error: Segmentation fault
    Please submit a full bug report,
    with preprocessed source if appropriate.
    See  for instructions.
  • thumb mode gdb segfaults on target

    # gdb --verbose /bin/cat
    [1]   Segmentation fault (core dumped) gdb --verbose /bin/cat

These issues may be solved by upgrading to newer versions of GCC compiler and gdb debugger.

Another problem is that newer binutils adds more symbols in thumb mode than older binutils, which makes thumb mode binaries file size larger than the with old binutils and larger than ARM mode binaries 7. These extra symbols only affect the file size and sections loaded to memory are smaller in thumb mode than in ARM mode. Also, all of these extra sections and symbols can be easily stripped by providing NetBSD build tools with a STRIPFLAG=-s build variable, though without this patch libraries are not stripped.

Posted late Sunday evening, November 20th, 2011

Contents

Supported hardware

Creative Music System

Very old and rare synthesizer.

See cms(4).

PC speaker

It has one-voice polyphony and sounds just awful. Useful only for testing MIDI input devices.

See pcppi(4).

Roland MPU-401

MIDI interface by Roland. It became popular thanks to excessive cloning.

Supported on many ISA cards, and following PCI cards:

Usually MPU interfaces are conncted to MIDI/Joystick port on sound cards. You won't be able to play/receive anything unless you connect some external MIDI device to such port. Though, in some rare cases MPU interface is connected to on-board/daughterboard WaveTable MIDI engine.

See mpu(4)

Simple MIDI interfaces

Simple MIDI interfaces are supported on many ISA cards, and following PCI cards:

  • Cirrus Logic CS4280 - clcs(4)
  • Creative Labs SoundBlaster PCI (Ensoniq AudioPCI based) - eap(4)
  • Trident 4DWAVE and compatibles - autri(4)

Usually simple MIDI interfaces are connected to MIDI/Joystick port on sound cards. You won't be able to play/receive anything unless you connect some external MIDI device to such port.

Note: MIDI port and synth on SoundBlaster Live! and newer cards by Creative is unsupported.

USB MIDI devices

Many USB MIDI devices are supported. Synth modules, keyboards and MIDI interfaces are handled well.

See umidi(4)

Yamaha OPL2 and OPL3

Popular single-chip FM synthesizer. Almost all ISA cards come with such chip.

Some of the newer cards have compatbile FM engine too. PCI cards based on following chipsets have it:

NetBSD opl driver has built-in General MIDI instrument definitions, so your system is ready to play without additional configuration.

Note: New PCI cards by Creative Labs do not have this chip.

See opl(4)

Identifying MIDI devices

You can easily discover what kind of MIDI devices are available - try grepping dmesg:

dmesg | grep midi

Sample output:

midi0 at pcppi1: PC speaker (CPU-intensive output)
midi1 at opl0: Yamaha OPL3 (CPU-intensive output)
umidi0 at uhub1 port 2 configuration 1 interface 1
umidi0: Evolution Electronics Ltd. USB Keystation 61es, rev 1.00/1.13, addr 2
umidi0: (genuine USB-MIDI)
umidi0: out=1, in=1
midi2 at umidi0: <0 >0 on umidi0

In this case three MIDI devices are detected - PC speaker, Yamaha OPL3 and USB MIDI device (Keystation 61es keyboard in this case).

Connecting MIDI devices

Connecting MIDI devices is very simple. For example if you want to drive OPL3 using USB MIDI keyboard try:

cat /dev/rmidi2 > /dev/rmidi1

You can now play :).

MIDI software for NetBSD

Utility called midiplay(1) comes with NetBSD.

Posted late Sunday evening, November 20th, 2011

Contents

Introduction

This page describes howto setup NetBSD to be able to use linux lvm tools and libdevmapper for lvm. For now my work was done on haad-dm branch in main netbsd repository I want to merge my branch back to main repository as soon as it will be possible.

LVM support has been merged to NetBSD-current on the 23rd of December 2008.

Details

Tasks needed to get LVM on NetBSD working

  • Get latest sources
  • Compile new kernel and tools
  • Create PV, LV and VG's and enjoy :)

Howto update/checkout sources

You can checkout the latest sources with this command

$ export CVS_RSH="ssh"
$ export CVSROOT="anoncvs@anoncvs.se.netbsd.org:/cvsroot"
$ cvs checkout -dfP src

Or update by command

$ export CVS_RSH="ssh"
$ export CVSROOT="anoncvs@anoncvs.NetBSD.org:/cvsroot"
$ cvs update -dfP

There are only 3 directories which were changed in my branch

  1. sys/dev/dm
  2. external/gpl2/libdevmapper
  3. external/gpl2/lvm2tools

Howto setup NetBSD system to use lvm

The easiest way is to build distribution and sets. You need to build with flag MKLVM set to yes.

$ cd /usr/src $ ./build.sh -u -U -V MKLVM=yes tools distribution sets 

After successful build update your system with them. You can also run make install (as root) in src/external/gpl2/lvm2 to install userland part of LVM into the existing NetBSD system. There is also simple driver used by LVM2 tools in our kernel in src/sys/modules/dm you have to install and load this driver before you test lvm. The NetBSD LVM uses same tools as Linux and therefore we have the same user interface as is used in many common linux distributions.

Howto compile new kernel

Kernel compilation procedure is described here http://www.netbsd.org/docs/guide/en/chap-kernel.html#chap-kernel-build.sh To get device-mapper compiled in kernel you have to add this option to kernel config file

pseudo-device dm

Using new MODULAR modules

There are two versions of modules in NetBSD now old LKM and new MODULAR modules. New modules are build in sys/modules/. All GENERIC kernels are compiled with support for new modules. There is reachover makefile for dm driver in sys/modules/dm. You can use it to build dm module.

For loading new style modules new module utils are needed. You need to add

MKMODULAR=YES

to /etc/mk.conf and build modload, modstat and modunload.

Compile lvm2tool and libdevmapper

To get lvm working it is needed to compile and install linux lvm tools. They are located in

  • external/gpl2/libdevmapper
  • external/gpl2/lvm2tools

Only make/make install is needed to build/install tools to machine. Tools are not integrated in to lists and build system now. Therefore it is possible that when you try to add them to build process it will fail(with error to many file in destdir).

Using lvm on NetBSD

lvm2tools are used to manage your lvm devices.

lvm pvcreate /dev/raw_disk_device # create Physical Volume 
lvm vgcreate vg00 /dev/raw_disk_device #create Volume Group-> pool of available disk space.
lvm lvcreate -L20M -n lv1 vg00 # create Logical volume aka Logical disk device
newfs  /dev/vg00/rlv1 # newfs without -F and -s doesn't work
mount /dev/vg00/lv1 /mnt # Enjoy

After reboot, you can activate all existing Logical Volumes (LV) in the system with the command:

lvm vgchange -a y

You can use:

lvm lvdisplay vgdisplay pvdisplay to see the status of your LVM devices.

You can also use lvm lvextend/lvreduce to change size of LV.

I haven't tested my driver on SMP system there are probably some bugs in locking so be aware of it :) and do not try to extend/reduce partition during I/O (something bad can happend).

TODO:

  • Review locking - I will probably allow only one ioctl call to be inside dm driver at time. DONE

  • Write snapshot driver -only skeleton was written yet.

  • Add ioctls needed to correct newfs functionality. I tried to implement them in device-mapper.c::dmgetdisklabel but it doesn't work yet. DONE

  • Write lvm rc.d script to enable LVM before disk mount so we can use lvm for system partitions. DONE

Posted late Sunday evening, November 20th, 2011

Contents

Overview

Apple Time Machine is a backup mechanism which uses an Apple sparse filesystem model, with extended attributes to store the files and directories of your OSX system as an incremental state. You can walk 'back in time' against the history of the filesystem and recover older versions of files, or files which have subsequently been deleted. Unlike a dump, its not a complete independent collection of files: it prunes back in time against what it knows has changed, and so cannot be relied on in the same way as offline disk or tape backup written in a rigorous system. Nor is it an archive: if you want to preserve something, you need to manage that independently of a time machine setup. But, having said that, its enormously useful and very user friendly, and lots of OSX users are very happy with it. Most of them use directly attached media like a firewire or USB disk, or an Apple appliance like the Time Capsule. However, it is technically possible to do Time Machine over a network, to a network-mounted filesystem.

NetBSD does not support HFS+ format filesystems directly in a way which can be exposed to an OSX host over the network. Normally, its UNIX filesystems are mounted on clients by protocols like NFS, or SMB, or AFP (Apple File Protocol) through either the built-in facilities of mount_nfs(8), mount_smbfs(8) or a package like netatalk. These are all provided by userspace daemons.

If you want to use these, there are documented ways to do this, such as the apple time machine freebsd in 14 steps page Rui Paulo wrote. They each have advantages and disadvantages, noting the need for special file support and extended attributes. Its probable that making the correct Apple sparse filesystem as a single file image, and moving this to the network-backed filestore gets round most of the problems, if you set the correct magic flag in your OSX to permit non-standard filesystems to be used to 'home' the time machine.

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

However, the NetBSD iSCSI implementation is robust, and efficient, and will provide arbitrary client-side filesystems (such as HFS+, or Windows filesystems) because its presenting SCSI disk as raw blocks. These raw blocks are typically provided from a sparse file, in the NetBSD filesystem tree.

iSCSI talks about targets (which is what a provider of an iSCSI disk is) and initiators (which is what a client connecting to a given target is). -In this situation, NetBSD will be running as the target, via the iscsi-target(8) daemon. The client has to have an initiator, which can present the target as a device.

In order to use this on OSX, you need an iSCSI initiator. One which is freely available is Studio Network Solutions globalSAN iSCSI initiator for OSX. At the time of writing, the version which works for me on Snow Leopard is 3.3.0.43 which is distributed as a ZIP'ed DMG file.

Initialize a target

To create a target, you edit the targets(5) file.

The default example shipped with NetBSD 5.0 and later found in /etc/iscsi/targets is:

#
# Structure of this file:
#
# + an extent is a straight (offset, length) pair of a file or device
#   it's the lowest common storage denominator
#   at least one is needed
# + a device is made up of one or more extents or other devices
#   devices can be added in a hierachical manner, to enhance resilience
# + in this example, no device definitions are necessary, as the target
#   will just use a simple extent for persistent storage
# + a target is made up of 1 or more devices
# The code does not support RAID1 recovery at present

# Simple file showing 1 extent, mapped straight into 1 target 

# extent        file or device          start           length
extent0        /tmp/iscsi-target0      0               100MB

# target        flags   storage         netmask
target0         rw      extent0         0.0.0.0/0

The target should be the name of a file in the filesystem you want it to reside in. When the iscsi-target daemon first runs over this configuration, targets are initialized as UNIX sparse files. Be very careful not to touch this file with a cp or any other operation which tickles NetBSD into filling in the 'holes' in the sparce file: dump and tar should be safe, as is rsync if the -S flag is given to it. (see sparse files below.)

The /etc/iscsi/target file can implement simple ACL models for the IP network/prefix of the client(s) you wish to permit. The default in this file as distributed is a fully open iSCSI target. There is an associated /etc/iscsi/auth file which can be used to instantiate CHAP and other protections over the iSCSI attachment by the client.

Be warned that the target name has to include a digit which will be translated to the SCSI 'LUN' number of the device for the client. Unless you know what you are doing, its best to leave this as 0 since the client may well demand only LUN0 is used.

Once you have configured a target, you need to enable and start the iscsi-target(8) daemon. Edit /etc/rc.conf and enable iscsi-target=YES, and then you can run /etc/rc.d/iscsi_target start (it will start automatically on reboot once enabled in /etc/rc.conf)

Install iSCSI initiator

  1. fetch the DMG, mount, install (reboot required)

Configure the initiator

  1. open 'System Preferences' and select globalSAN. It will throw a warning about restart. Don't worry, it just flicks you to the correct system Preference pane.
  2. select the '[Targets]' tab and use the [+] button to add a new target.
  3. provide the name or IP address of your NetBSD iSCSI target. If you didn't alter the port it serves from, don't alter the port here.
  4. select 'Header Digest' by default
  5. enable the mount. This will cause OSX to detect a new (raw) device, and ask you what to do with it. You should select [Initialize...] to enter the OSX 'disk utilities' pane. OSX is then initializing this SCSI LUN over the wire, using the iSCSI protocol to read and write the disk blocks. If you have a large back-end iSCSI target file, this operation will take quite a long time: its really just like newfs on a real raw disk, except its done a) over a network protocol and b) into a single file on the host/target filesystem.
  6. in Disk Utilities you can use any Volume Scheme you like. I selected one partition, of HFS+ with journalling. This is what a locally attached USB or firewire external disc is typically configured as, but 'its just a disk' -you can sub partition it any way you like.

If you look at this disk image on the target, its interesting:

$ ls -ltr /tmp/iscsi-target0 
-rw-r--r--  1 root  wheel  104857600 Nov 18 22:45 /tmp/iscsi-target0

It reports as a large file (the 100mb of the /etc/iscsi/target file) but, its not. DF on the filesystem will show its not all occupied.

$ file /tmp/iscsi-target0 
/tmp/iscsi-target0: x86 boot sector; partition 1: ID=0xee, starthead 254, startsector 1, 204799 sectors, extended partition table (last)\011, code offset 0x0

$

Its detected as a file which contains a boot sector and disk structure. In principle you could probably use vnconfig to mount and examine this file. Not advised if its being served by an iscsi-target daemon...

Thats it! You can enable automatic mounting, or not (as you please) in the globalSAN preferences pane.

Once this is completed, the disk is a normal SCSI device, in HFS+ mode, and can be configured for any operation including Time Machine.

Issues

  • It can be slow. Bear in mind that the speed you get for iSCSI is strongly limited by the client's ethernet speed, and any switching fabric you have. Real SAN use dedicated high speed switches and protocols like infiniband to maximise disk throughput. if you do iSCSI on a home network, its not going to look like your corporate SAN at work.
  • Wireless is a poor choice of network to initialize your time-machine backup on: in my case, the first dump is 160GB (!) and this would not complete in a sensible time.
  • Even on a wired network, a home router is going to throttle you. I had 100mbit on my server and 100mbit on my router-switch combo, and achieved throughput of the order 10GBytes/hr which is 2MBytes/sec. thats well below apparent switch speed. (I will be experimenting with back-to-back connection)
  • raidframe also imposes speed constraints. I was using a software RAID of 1+0 over (notionally) 6 SATA disks, but two of the mirror-sets were running in 'degraded' mode.

Sparse Files

A sparse file is a special file-type which reports its 'size' in ls -l as the total it could be, if it was fully populated. However, until actual file blocks at given offsets are written, they aren't there: its a linked list. This permits a very fast growable file to a limit, but at some risk: if you accidentally touch it the wrong way, it fills in the holes. You just have to be careful. FTP for instance, doesn't honour sparse files. if you copy a file with FTP, the holes are filled in.

Sparse files were implemented in UNIX FFS a long time ago, and are in HFS+ as well. The behaviour of the tools like pax, rsync, tar, ftp has to be verified on a case-by-case basis. Its not clear if NetBSD pax(1) is safe or not. rsync has the -S flag to preserve sparse files.

Posted late Sunday evening, November 20th, 2011

Contents

Using the FUSE in NetBSD -current

Requirements

You will need a current kernel and userland, and up to date pkgsrc-current (pkgsrc/filesystems)

Introduction

The sources we refer to in this HowTo can exist anywhere on your system (normally in /usr)
That's why we use src/foo/bar (i.e src/lib/libpuffs) instead of full path.

Setup

Make sure that your Kernel configuration file contains the following option under filesystem

  file-system PUFFS  

And this under pseudo-devices

  pseudo-device putter

Add this to your /etc/mk.conf

  MKPUFFS=yes 

Build your kernel and update your userland.

Make sure you run "make includes" in your source tree in src/sys if you at some point update your NetBSD sources so that you have proper header files.
Most importantly the src/lib/libpuffs headers.

  # cd src/sys
  # make USETOOLS=no includes 
  # cd src/lib/libpuffs 
  # make USETOOLS=no includes

Check if libpuffs is properly installed

  # cd src/lib/libpuffs 
  # make USETOOLS=no cleandir dependall
  # make USETOOLS=no install   

Check if librefuse is properly installed.

  # cd src/lib/librefuse 
  # make USETOOLS=no cleandir dependall
  # make USETOOLS=no install

Check if fusermount is up to date and installed

  # cd src/usr.sbin/fusermount
  # make USETOOLS=no cleandir dependall
  # make USETOOLS=no cleandir install

Check if puffs is up to date and installed

  # cd src/usr.sbin/puffs
  # make USETOOLS=no cleandir dependall
  # make USETOOLS=no install

  This will compile and install mount_9p, mount_portal, mount_psshfs and mount_sysctlfs

Install and Usage

Mount_psshfs

Mounting a remote filesystem over ssh

   # mount_psshfs user@host:/path/to/directory /mountpoint 
   # umount /mountpoint 

Stuff from /usr/pkgsrc/filesystems.

Fuse-ntfs-3g

Mounting a ntfs filesystem

   # ntfs-3g /dev/device /mountpoint 
   # umount /mountpoint

Fuse-obexftp

Mounting an obexfs filesystem (Make sure your bluetooth connection is established with your device)

   # obexfs -b 00:11:22:33:44:55 -B 10 /mnt/mountpoint 

(Where the 00:11:22:33:44:55 is the address of your Bluetooth device )

Fuse-encfs

Fuse-cryptofs

Fuse-cddfs

Fuse-curlftps

Posted late Sunday evening, November 20th, 2011

Contents

THIS PAGE NEEDS AN UPDATE BECAUSE netbsd-10 vm.swap_encrypt=1, default on most platforms today, obsoletes swapping to cgd

Summary

It's getting more and more popular to use encrypted swap. This is however not a trivial task with nfs-swap. Swap over nfs is supported like this:

server:/usr/swapfile none swap sw,-w=8192,nfsmntpt=/swap 0 0

But this can not be encrypted. We will however cheat and use a vnd(4) on a nfs-share.
This is how I did it on my Jornada 680 running 3.99.15.

Things needed

A kernel with both vnd(4) and cgd(4) support.

Creation

Making the swapspace

First we need to create the swapfile to be used. It's important that the swapfile is in a directory that is mounted when /etc/rc.d/swap2 runs. Either add the directory to $critical_filesystems_remote, or just put it in /usr.
Now run:

# dd if=/dev/zero of=/usr/swapfile bs=1m count=64

This will create a 64MB swapfile. Make sure it has the right permissions and owner.

# chown root:wheel /usr/swapfile
# chmod 600 /usr/swapfile

Configuring the swapspace the first time

Now we just have to configure it so the system can use it.
Configure the paramsfile for cgd(4).

cgdconfig -g -o /etc/cgd/swapfile -V none -k randomkey blowfish-cbc

Now we can configure the device.

# vnconfig vnd0 /usr/swapfile
# cgdconfig cgd0 /dev/vnd0c /etc/cgd/swapfile

Replace /dev/vnd0c with /dev/vnd0d if necessary.
Disklabel the cgd with disklabel -I -e cgd0, it will should look something like this.

# /dev/rcgd0c:
type: cgd
disk: cgd
label: default label
flags:
bytes/sector: 512
sectors/track: 2048
tracks/cylinder: 1
sectors/cylinder: 2048
cylinders: 64
total sectors: 131072
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0           # microseconds
track-to-track seek: 0  # microseconds
drivedata: 0 

3 partitions:
#        size    offset     fstype [fsize bsize cpg/sgs]
 c:    131072         0       swap                     # (Cyl.      0 -     63)

Note: Depending on which archictecture you use, you may need a different layout.
Like this on an i386:

 a:    131072         0       swap                     # (Cyl.      0 -    63)
 d:    131072         0     unused      0     0        # (Cyl.      0 -    63)

Depending on which partition your architecture uses as raw partition. If unsure, check with:

# sysctl kern.rawpartition
kern.rawpartion=3

Back it up so it can be used later.

# disklabel cgd0 > /etc/cgd/swapfile.disklabel

Use it (finally).

# swapctl -a /dev/cgd0c

Now you have working encrypted swap over nfs. To check its status:

# swapctl -l
Device      512-blocks     Used    Avail Capacity  Priority
/dev/cgd0c      131072     9696   121376     7%    0

Use the swapspace at every reboot

Using this swapspace automatically at every reboot is a little tricky since it can not be put int /etc/fstab, but it can be done in another way. And I have already done the work for you. ;-) Check that the variables make sense on your system. E.g that you used vnd0 and cgd0 and RAW_PART is right for your architecture. Create the file /etc/rc.conf.d/swap containing the following.

# Initialize cgd over vnd swap, suitable for nfs-swap.
#
# Note: We can NOT put this swapfile in /etc/fstab, this is why
# this is relatively complicated. 
#
# If this is the only swapspace you have configured then you can set
# no_swap=YES in rc.conf, otherwise the system will complain every boot.
#
# IMPORTANT:
# $swapfile has to be in $critical_filesystems_remote. /usr is by default
#
vnd_device="vnd0"
cgd_device="cgd0"
swapfile="/usr/swapfile"
paramsfile="/etc/cgd/swapfile"
swap_disklabel="/etc/cgd/swapfile.disklabel"
RAW_PART="c"                    # <- change to suit your arch
SWAP_PART="c"                   # <- change to same as the disklabel
start_postcmd="cryptovnd_swap"
stop_cmd="cryptovnd_stop"       # Note: We have to override stop_cmd

cryptovnd_swap()
{
        # Since there is only one swap-variable in rc.conf we have to
        # check that we are being called from swap2.
        if [ $name = "swap1" ]; then
                return
        fi
        if [ -f $swapfile ]; then
                echo "Configuring cgd over vnd swap."
                eval `stat -L -s $swapfile`
                if [ `echo $st_uid+$st_gid|bc` != 0 ]; then
                        echo "$swapfile MUST be owned by root and group wheel"
                        echo "$swapfile not used as swap."
                        return 1
                else
                        if [ ! -f $swap_disklabel ]; then
                                echo "No $swap_disklabel."
                                echo "$swapfile can not be used as swap."
                                return 1
                        fi
                        if [ $st_mode != "0100600" ]; then
                                echo "$swapfile MUST have permission 600"
                                echo "$swapfile not used as swap."
                                return 1
                        fi
                fi
                vnconfig $vnd_device $swapfile
                cgdconfig $cgd_device /dev/${vnd_device}$RAW_PART $paramsfile
                disklabel -R -r $cgd_device $swap_disklabel
                swapctl -a /dev/${cgd_device}$SWAP_PART
        fi
}

cryptovnd_stop()
{
        if [ $name = "swap2" ]; then
                swapctl -d /dev/${cgd_device}$SWAP_PART
                cgdconfig -u $cgd_device
                vnconfig -u $vnd_device
                swapctl -U -t noblk
        else
                swap1_stop
        fi
}

Some issues and notes

  • Do not include this cgd in /etc/cgd/cgd.conf
  • It could happen that there isn't enough entropy in the kernel to initialize the swap partition. If so, you can add your NIC to the entropy pool in /etc/rc.conf with /sbin/rndctl -ced ne0 if you have a ne(4) NIC.
  • If this is the only swapspace configured, set the variable no_swap=YES in /etc/rc.conf or the system will complain every boot.

Additional Information

Posted late Sunday evening, November 20th, 2011 Tags:

MTP mode

You can use fuse-gphotofs if you have a fuse/refuse capable NetBSD version to acces you player. You can also try libmtp which comes with command line tools, or simply gphoto.

UMS mode

You can try upgrading to the korean firmware to enable umass mode for your player.

Posted late Sunday evening, November 20th, 2011

What it does:

Offer a system to generate binary updates for NetBSD (NOT patches), install, remove with full "rollback to previous state" support, and manage them. Simple dependencies are allowed. It also allows auto-generating security advisories from update data.

Without further ado, here's how to use it.

Update producer: (people who create updates)

  1. Write the .plist file to describe the update, and put it online.
     Update the index file to include the update id. (for example,
     NetBSD-UP2007-0001)


     Examples for updates are available:


     <http://www.NetBSD.org/~elad/haze/Updates/2006/NetBSD-UP2006-0001.plist>
     <http://www.NetBSD.org/~elad/haze/Updates/2006/NetBSD-UP2006-0002.plist>


     The index file looks like this:


     <http://www.NetBSD.org/~elad/haze/Updates/2006/INDEX>


     Note the directory hierarchy: the year is important; everything
     else can be tuned, but the structure. For now.


  2. Maintain a build directory for ports you want to generate updates
     for. This is "obj-dir" in the config file (/etc/haze/haze.conf):

obj-dir /usr/netbsd/objdir/xphyre/destdir.%m

     Haze knows to replace "%m" with the machine type. The default is
     "/usr/obj/%s-%r/destdir.%m", where "%s" will be replaced by the
     string "NetBSD", and "%r" will be replaced by the release, for
     example, "4.0".


     To keep this directory up-to-date, all the producer has to do is
     run the build after the source is updated with the fix.


  3. Maintain a list of "targets" you want to monitor. Haze calls a
     combination of OS-release-machine a "target". For example,
     "NetBSD-4.0-amd64" is a target. By default, Haze will only generate
     updates for the host it runs on. You can override that, though:

generate-targets NetBSD-4.0-amd64 NetBSD-4.0-i386 NetBSD-3.0-amd64 NetBSD-3.0-i386

  4. After the new files are built, generate updates. This is done using
     the -G flag. For example, if you just rebuilt for
     NetBSD-UP2007-0001, and want to generate updates for it:

haze -G -U NetBSD-UP2007-0001

     The updates will show up in the output dir, /tmp by default, and
     will be in the form of NetBSD-UP2007-0001-4.0-amd64.tar.gz.


  5. Put the updates online, in the Updates/ directory. For example,
     this would be a valid URL to an update package:


      <http://www.NetBSD.org/~elad/haze/Updates/NetBSD-UP2006-0001-3.0-amd64.tgz>

Update consumer: (people who apply updates)

  1. Make sure there's a /etc/haze directory and that it's writable by
     the user Haze is running as. I would elaborate on this too much,
     but you *should* be able to tell Haze to perform updating on a
     specified root directory, and then do the sync yourself, if you
     don't trust running Haze as root. If you do:

mkdir /etc/haze

     Everything else, including the configuration file and local
     database, will be created by Haze automagically. You can inspect
     the default values used in haze.h.


  2. By default, things should pretty much Just Work. Therefore, here
     are some usage examples:

Show the batch of updates waiting to be installed:

haze -B

Install all pending updates:

haze

Show locally known updates (including installed, ignored, and not applicable updates):

haze -L

Install a specific update:

haze -U NetBSD-UP2006-0001

Rollback an update:

haze -R -U NetBSD-UP2006-0001

View details about an update:

haze -V -U NetBSD-UP2006-0001

Explicitly ignore an update:

haze -i -U NetBSD-UP2006-0001

Operate in dummy mode, and just print stuff you'd do:

haze -x ...

Generate a security advisory skeleton for an update:

haze -S -U NetBSD-UP2006-0001

List available command line options:

haze -h

See also

NetBSD updating system

Posted late Sunday evening, November 20th, 2011

Sometimes it is necessary to be more quiet, when using the shell. For example, when working in groups, or if you're just annoyed of the beep.

The console beep can be effectively disabled by disabling the pcppi driver in the kernel. This is done during kernel configuration by finding and commenting out the lines referring to the pcppi(4) driver and its attached child devices in the kernel configuration file, e.g. "sysbeep at pcppi" on i386. See the NetBSD Guide for information on compiling customised kernels.

In the event the user must use a precompiled kernel that has the sysbeep device enabled, the pcppi driver and attached devices can be interactively disabled at boot time, via userconf(1), as follows:

  1. Select "Drop to boot prompt" at the boot menu.

  2. Enter "boot -c" to begin interactive boot using userconf(1).

  3. Disable the pcppi driver and its attached devices using the "disable" command. Type "?" or "help" at the "uc>" prompt for a list of the userconf(1) commands and their syntax. You may find the device numbers of "sysbeep at pcppi" and associated pcppi device drivers using the "list" command. Type "quit" when done disabling the pcppi devices.

These methods should work on any hardware.

Alternatively, there are several ways to disable the beep after the system has booted and the user has logged in, as described below.

On the console you can set the volume of the console beep with ?wsconsctl.

To display the current volume enter:

# wsconsctl bell.volume

You can control the volume setting different values. To turn the volume off enter:

# wsconsctl -w bell.volume=0

This may not work on every hardware. The solution is to set the the bell pitch to 0 Hz.

# wsconsctl -w bell.pitch=0

The bell pitch defaults to 1500 Hz. Sometimes a lower value is more comfortable. Just keep trying.

To apply these settings at startup, add them to /etc/wscons.conf. For example:

setvar wskbd bell.volume 0
setvar wskbd bell.pitch  0

X11

If you are working with X11, you can turn off the console beep using the command ?xset.

If you are using modular Xorg you need to install ?xset, it is under x11/xset.

To turn the console beep off enter:

$ xset b off

To turn it back on, enter:

$ xset b on

You can also set the volume and the pitch of the beep. This is not always supported, depending on the hardware.

However if you want to set volume and pitch you can do something like that:

$ xset b 10 1000 100

See also

Posted late Sunday evening, November 20th, 2011

Screen

One can take screenshots of a console session with screen.

Run the program in screen and use ^A-h to make a "hardcopy".

Posted late Sunday evening, November 20th, 2011

Introduction

The best way to share a partition with Linux is to create an EXT2 filesystem and use revision 0 (aka GOOD_OLD_REV) as the filesystem format. By using the GOOD_OLD_REV (revision 0) format you can avoid the dir_index and large_file issues encountered with DYNAMIC_REV (revision 1).

Example

Here is an example of how you can create, format and check an EXT2 filesystem using the Linux native tools:

# mkfs -t ext2 -r 0 /dev/hda3
# fsck -C -t ext2 /dev/hda3
# tune2fs -c 20 -i 6m -L Share /dev/hda3
# fsck -C -t ext2 /dev/hda3
# tune2fs -l /dev/hda3

If you want to use the revision 1 ext2 fs format it is possible but you need to format it like this:

# mke2fs -I 128 -L "Choose a name here" -O ^dir_index,^has_journal -v /dev/partition

(you can get mke2fs from the e2fsprogs package).

Partitions formatted this way are compatible with the windows and OS X ext2 drivers:

http://www.fs-driver.org/ http://ext2fsx.sourceforge.net/

Posted late Sunday evening, November 20th, 2011

Contents

Introduction

The Common Unix Printing System (CUPS) is a modular printing system for Unix-like computer operating systems that allows a computer to act as a powerful print server. A computer running CUPS is a host which can accept print jobs from client computers, process them, and send them to the appropriate printer. More information regarding the architecture of CUPS can be found here.

To check whether your printer works with CUPS and to what extent, you may visit the OpenPrinting database.

Install CUPS

Assuming your pkgsrc tree lies at /usr/pkgsrc, type:

# cd /usr/pkgsrc/print/cups
# make install

Modify rc.conf

Next copy cupsd startup script at /etc/rc.d:

# cp /usr/pkg/share/examples/rc.d/cupsd /etc/rc.d

Then add the following line to your /etc/rc.conf file:

cupsd=YES

And start the CUPS daemon, with:

# /etc/rc.d/cupsd start
Starting cupsd.
#

Install Foomatic PPD collection

The Foomatic PPD collection includes suitable PPDs for printers listed in the Foomatic printer/driver database. Together with the foomatic-filters package, this collection of PPDs allows many non-PostScript printers to function as if they were PostScript printers:

cd /usr/pkgsrc/print/foomatic-ppds

# make install

The following package adds Foomatic PPDs to the CUPS PPD database:

# cd /usr/pkgsrc/print/foomatic-ppds-cups
# make install

Setup CUPS

Open a web browser and type in the address/location bar:

http://localhost:631

You will enter a web-based interface where you can manage your printing system (add printers, etc).

To print using cups from the command line without further modifications you have to use /usr/pkg/bin/lpr. Using simply lpr will not print using cups but the default NetBSD printing system. The same apply to the spool queue examination command lpq.

Network configuration

Server side

Client side

Documentation

Notes

Permissions

Just for the record (I'm not sure where this fits in): To make it print, I had to

chown lp /dev/ulpt0 # usb        

and install print/hpijs for my Deskjet (it is used by gs via the foomatic thing), then set some additional options in the web interface, e.g. a4 paper size. Conclusion: The web interface sucks, looking into the logs is indispensable.

BSD lpr

I think I no longer need the BSD lpr, so i just did this to avoid confusion:

# chmod -x /usr/bin/lp*
# chmod -x /usr/sbin/lp*

HP Printer

To use HP printers (like DeskJet, OfficeJet, PhotoSmart, Business InkJet, and some LaserJet models) and to not have this kind of error /usr/pkg/libexec/cups/filter/foomatic-rip failed, you need to install another package named hpijs :

# cd /usr/pkgsrc/print/hpijs
# make install clean clean-depends

See also

Posted late Sunday evening, November 20th, 2011

Contents

Introduction

The NetBSD guide contains a chapter on printing. The basic parts of this printing system are the line printer spooler daemon known as lpd(8) and the associated tools such as lpr(1). Practically all printer-specific configuration is done with a special printcap(5) database. This guide will not replicate the information presented in the official guide, but instead tries to offer some examples on how this printing system can be configured with modern printers.

Alternatives

It appears that the Common Unix Printing System (CUPS) has gradually replaced other more traditional printing systems (see also How to setup CUPS in NetBSD). However, there is still room for lpd(8) and friends. If not for anything else, CUPS is relatively complex piece of software with quite bad security record. Some prefer also to always use tools that come with an operating system instead of relying on external packages.

Configuring and using a printer with lpd(8) can be easier than with CUPS, once you wrap your head around it.

Example: HP DeskJet

The following steps were needed for a low-cost HP DeskJet printer. First two packages were installed: print/hpijs and print/foomatic-filters:

cd /usr/pkgsrc/print/hpijs
make install package clean
cd ../foomatic-filters
make install package clean

The former is essential for high-quality output with HP printers and the latter is convenient. The print/hpijs-package contains various .ppd-files compressed with bzip(1). You need to pick one suitable for your printer and unpack it to a desired location. An example:

mkdir /etc/print
cd /etc/print
cp /usr/pkg/share/ppd/HP-DeskJet_5550-hpijs.ppd.gz .
gunzip *.gz

The next step is to configure the printer using the /etc/printcap file. An example that uses parallel port:

# This requires the following packages:
# print/hpijs and print/foomatic-filters.
#
lp|hp|HP Deskjet 5550:\
        :lp=/dev/lpt0:\
        :af=/etc/print/HP-DeskJet_5550-hpijs.ppd:\
        :if=/usr/pkg/bin/foomatic-rip:\
        :sd=/var/spool/output/lpd:\
        :lf=/var/log/lpd-errs:\
        :rg=print:\
        :rs=true:\
        :sh:

This will use the mentioned filter. The curiously named foomatic Perl script takes PostScript as standard input and generates the printer's page description language as standard output. This is pretty much all that is needed. For additional parameters in the configuration file, such as limiting the access to a group named print, please refer to the manual page, printcap(5).

The final step is to enable the lpd(8) daemon. The usual conventions with rc.conf(5) apply:

lpd=YES
lpd_flags="-s"

The -s flag tells the daemon to use UNIX domain socket instead of listening on all interfaces. This is probably a good idea if you are not configuring a printing server (in which case you probably already know what you are doing).

After starting the daemon you should be able to print files from the command line using the lpr(1). Printing from GUIs should also work. At least print/xpdf and www/firefox3 were both capable of printing.

Example: HP LaserJet & network

Before start you have to enable printer daemon in rc.conf file, set:

lpd=YES

To enable printing for HP Laser Jet network printers add into /etc/printcap following:

# HP-4250
HP-4250:\
        :lp=:sh:sd=/var/spool/lpd/lp:\
    :rm=192.168.170.193:\
    :lf=/var/log/lpd-errs:mx#0:

Where: HP-4250 is basic name of your printer which you can change per desire. Instead of 192.168.170.193 you should set IP address of your printer.

Check and create directory path if needed.

See also

Posted late Sunday evening, November 20th, 2011

Contents

What is CVS

CVS is an abbreviation for Concurrent Versions System.

Requirements

Nothing. CVS comes with the NetBSD base installation.

Setting up the repository

First decide where you want to store all cvs repositories. Let's take /usr/cvsrepositories

# mkdir /usr/cvsrepositories

Now you can create directories for different projects.

# cd /usr/cvsrepositories
# mkdir mycompany
# mkdir myprivatestuff

This projects must have chmod 770 to separate them from each other.

# chmod 770 mycompany
# chmod 770 myprivatestuff

Creating user groups

You should create a group for projects, where people are working together.

# group add mycompanyname

You should now assign this group to the project directory they belong.

# cd /usr/cvsrepositories
# chgrp mycompanyname mycompany/

cvs init

Before you can either checkout or import anything, you have to init your projects root directory. To keep the path short for the CVSROOT environment variable, I recommend using symlinks to the repository from the root /.

# cd /
# ln -s /usr/cvsrepositories/mycompany mycompany

Now create the cvs repository using

# cvs -d /mycompany/ init

Creating users

Now create users that are allowed to check out from your repository. Keep company workers in the group you have created before.

# useradd -G mycompanyname -m john

And set a password for the user john

# passwd john

It's your decision if you want to grant the users shell access or not.

Setting environment variables

Please set the environment variables CVSROOT and CVS_RSH. Bash or ksh users please use export.

# export CVSROOT=username@yourserver.com:/mycompany
# export CVS_RSH=ssh

csh users set it like this:

# setenv CVSROOT username@yourserver.com:/mycompany
# setenv CVS_RSH ssh

As you can see we use ssh as the transport protocol. This is recommended. Keep your transfers encrypted.

Initial check in

You should now proceed with the the initial check in of your (code)work. You can do this from another server aswell. Don't forget to set the environment variables there too.

Now please change into the directory you wish to import initially.

# cd myproject

and import it to your cvs server.

# cvs import -m "myproject initial import" myproject myproject initial

this should produce an output like this:

N myproject/test.c
N myproject/test.h
N myproject/mi/mi.h
N myproject/md/md.h

No conflicts created by this import

checkout

To checkout the work, set your environment variables and enter

# cvs co -PA myproject

This will checkout myproject.

ssh config

Please configure your ssh client to fit your need in .ssh/config, like using a different ssh port than 22.

That's it. For more information about cvs(1) and using it please read the manpage.

See also

Posted late Sunday evening, November 20th, 2011

Besides setting up the global system timezone by symlinking /etc/localtime to a file in /usr/share/zoneinfo, you can also set a timezone that applies only for one user. This is done by setting the environment variable TZ.

You can set it in your startup file like this:

$ echo 'export TZ=Europe/Amsterdam' >> ~/.profile

From this shell all subsequent [date] calls will use the /usr/share/zoneinfo/Europe/Amsterdam file for translating the system's UTC time to your local time.

To run a single process with a specific timezone, try something like this:

$ env TZ=Canada/Eastern xclock -d -strftime "Toronto: %a, %d %b, %H:%M" &

This will start an environment with the TZ variable set to Canada/Eastern, and run a digital (-d) xclock with the time formatted as instructed by -strfime, including putting a note about which timezone it belongs to ("Toronto"). This process will detach from the terminal (because of the &), but leave the environment you ran it from with the same timezone it began with. With a setup like this, one can run an xclock (or many xclocks) displaying the localtime of various timezones around the world.

References

Posted late Sunday evening, November 20th, 2011

Contents

Introduction

This little article will try to make sense of the jungle that is NFS and NIS. For our example we will use NFS for keeping /home on a server, allowing us to work on the same files in our homedir from any computer in the network.

NIS

NIS (Network Information Service) is a directory system which is used to centralise configuration files like /etc/hosts and /etc/passwd. By using NIS for passwd, you can have the same users on each host in the network without the hassle of keeping the passwd file of all hosts synchronised.

We will need NIS (or another directory service) to make sure the NFS user ids/group ids are the same on the server as on all clients. Otherwise, bad things will happen, as you can probably imagine (especially in our example of mounting /home over NFS). Note that using NIS with NFS is not mandatory, you can also keep the server and client's passwd in synch.

NIS used to be called the "Yellow Pages", or YP for short. Because of trademarks it had to be renamed, but the programs are all still prefixed with yp.

Kernel options

Before doing anything with NFS, ensure that your kernel has support for NFS sharing. This means your clients and servers must have NFS kernel support enabled. This is the case for GENERIC Kernels. For custom Kernels, the following lines must be in the kernel file:

 file-system     NFS             # Network File System client

Your server also must have the following option:

 options         NFSSERVER       # Network File System server

If you want to get funky and boot from NFS (not discussed in this article), your clients need these options as well:

 options         NFS_BOOT_DHCP,NFS_BOOT_BOOTPARAM

Creating a NIS setup

The first thing we should do is decide on a NIS domain name. This has nothing to do with your machine's Internet domain name. It is just a unique name that is used to identify machines in the same NIS block.

The domainname is set (as root) using the domainname(1) program, or can be set in the /etc/mydomain file.
Alternatively, in most BSD systems, it can be set in /etc/rc.conf under the variable domainname.

root@earth# domainname planets

After this, we must initialise all files needed for the server to do its work. For this, we use the ypinit utility.

 root@earth# ypinit -m

The -m means we are creating a master server. On more complex networks, you can even want slave servers. The tool will ask you for a list of YP servers to bind to.
Since we're only using one server, just press RETURN (make sure your own server's internal address is in the list).

Before we run make in /var/yp, as the tool says, we must enable the NIS daemons: rpcbind, ypserv and ypbind (in that order). After that, we can run make in /var/yp.

To test if your setup is working, try yptest. It should spew out the passwd file among others, so don't panic ;)

To get stuff working on your client, you need to enable the yppasswdd, rpcbind and ypbind daemons as well. In order to do that, edit the /etc/rc.conf file and add there following:

#NIS server
ypserv="YES"
ypbind="YES"
yppasswdd="YES"
rpcbind="YES"

Then just run

# /etc/rc.d/rpcbind start
# /etc/rc.d/ypserv start
# /etc/rc.d/ypbind start
# /etc/rc.d/yppasswdd start

rpc.yppasswdd(8) must be running on the NIS master server to allow users to change information in the password file.
ypserv(8) provides information from NIS maps to the NIS clients on the network.
ypbind(8) finds the server for a particular NIS domain and stores information about it in a "binding file".

After that, you can use ypinit:

 root@mars# ypinit -c

Then, add your NIS server's address to the list. To test if everything is working, use yptest on the client as well. Note that ypbind will HANG if it can't find the server!

If everything is working, you are ready to go! Just edit /etc/nsswitch.conf and put in some nis keywords. For example:

 passwd:        files nis

would first look up usernames/passwords/uids in /etc/passwd, and if it can't find it, it would look it up using NIS. Right after changing this file, you should be able to log in on your system using a username which is only in /etc/passwd on the server. That's all there is to it.

The daemons

What are all those daemons for? Well, here's a quick rundown:

Portmap/rpcbind is the program which maps RPC (Remote Procedure Call) program numbers to port numbers (hence, portmapper). Any program which wishes to know on what port a certain RPC program is listening can ask this from the portmapper daemon (rpcbind). Each RPC service has its own number, which can be looked up in /etc/rpc. These numbers are how rpcbind can match the running RPC services to the ports. In short: If rpcbind is not running, not a single RPC program will work.

Ypserv is an authentication daemon for the RPC services, I believe. Ypbind is the daemon which can find the YP server for the specified domain.

NFS

Setting up NFS is a piece of cake. Just enter all directories you wish to export in /etc/exports and start the NFS daemon. In our example we would have:

 /home      -network 192.168.0.0 -mask 255.255.0.0 -maproot=root

This exports /home only on the LAN 192.168.x.x. The maproot line is needed, because otherwise the client's root will not have superuser access. Now, start the mount daemon and the NFS daemons (mountd and nfsd) as root on your server, in that order. For that type:

 root@mars# /etc/rc.d/rpcbind onestart
 root@mars# /etc/rc.d/mountd onestart
 root@mars# /etc/rc.d/nfsd onestart
 root@mars# /etc/rc.d/nfslocking onestart

If you wish to start the NFS server on boot, add following lines to your /etc/rc.conf

nfs_server=yes
rpcbind=yes
mountd=${nfs_server}
lockd=${nfs_server}
statd=${nfs_server}

Now, try to mount from the client and type:

 root@mars # mount -t nfs earth:/home /home

Voila, you're done. Just add all NFS volumes you want to mount to your /etc/fstab like this

 earth:/home   /home    nfs   rw

and have them mounted at system startup.

NOTE: I had much trouble with NFS which was caused by UDP packet fragmentation. This made all writes extremely slow (and other outgoing network traffic as well!) while reads were at an acceptable speed. To solve this, I added the (undocumented?) tcp option to fstab to mount NFS over TCP. You'll probably also need to add

nfsd_flags='-t'

to rc.conf so the NFS server serves up TCP exports.

If you just want to run NFS, you need to run the following daemons on your server: rpcbind, mountd, nfsd (in that order)

Notes

Concerning NFS

If you find NFS is not suitable for you, you could try Coda. The Coda filesystem tries to overcome some of the drawbacks of NFS:

  • Handling of (sudden) disconnections
  • Its own authentication system

And some others. The latest NFS versions are of course trying to integrate some of Coda's features as well.

Concerning NIS

A disadvantage of NIS is that it is not very secure. If security is a big concern, have a look at LDAP and NIS+, which are more complex directory services. For networks where security isn't that important (like most home networks), NIS will do. It is also much easier to set up than NIS+ or LDAP.

On NetBSD (probably on other systems as well), the NIS server consults /etc/hosts.allow and /etc/hosts.deny (from Wietse Venema's tcpwrappers package) to determine if the requesting host is allowed to access the NIS directory. This can help you in securing NIS a little.

My /etc/hosts.deny looks like this:

 ypserv: ALL
 rpcbind: ALL
 ypbind: ALL
 nfsd: ALL

In my /etc/hosts.allow I have my LAN hosts.

References

See also

Posted late Sunday evening, November 20th, 2011

Introduction

SWAT - the Samba Web Administration Tool provides a really quick and easy way to set up a Samba server, with more powerful configuration options available to those who need them. It's already part of the Samba package that is available from pkgsrc, and you don't even need to install and configure an HTTP server like Apache to use it.

In this tutorial I will go through the steps I took to set up a share on my NetBSD machine, so that the machine's hard drive could be used for storage by the Windows PC's on the network. The 'guest' account is used for access and I won't go into how to set up users and passwords, so this solution would probably be more suitable for home networks.

Install Samba and enable SWAT

The first step is to fetch, build and install Samba:

 # cd /usr/pkgsrc/net/samba
 # make install clean

Next, put scripts in /etc/rc.d so that smbd and nmbd will be started automatically when NetBSD boots up. I simply used the example scripts that came with NetBSD.

 # cp /usr/pkg/share/examples/rc.d/smbd /etc/rc.d/
 # cp /usr/pkg/share/examples/rc.d/nmbd /etc/rc.d/

You also need to add the following lines to /etc/rc.conf

 smbd=YES
 nmbd=YES

SWAT can be enabled by adding the following line to /etc/inetd.conf

 swat stream tcp nowait.400 root /usr/pkg/sbin/swat swat

Now, restart inetd to enable SWAT

 # /etc/rc.d/inetd restart

Use SWAT to configure the Samba server

You should now be able to access SWAT by surfing to http://:901/ where is the IP for your NetBSD machine (or localhost if you are accessing SWAT locally). Login as 'root' with your system's root password. You will be taken to SWAT's main menu.

First, click on the 'Globals' icon, and use that menu to configure global options such as the 'workgroup' your Samba server is to be part of. If you don't understand what an option does, click the 'help' link next to it. Use the 'commit changes' button to save your work.

Next, click on the 'Shares' icon. To create a share, type a name into the text box and click 'create share'. You will now be able to specify the path to the folder that you want to share. To make the share accessible to anyone on the network without a password, change 'guest ok' to 'yes' using the drop-down menu. Change 'read only' to 'no' if you want other machines on the network to have read/write access to your shared folder.

Finally click on the 'Status' icon. From here you can start/stop/restart the Samba server without having to issue any commands at the command line. Just click 'start smbd' and 'start nmbd'.

Your shared folder should now be accessible by the other machines on the network...To check this out, use 'Network Neighbourhood' in Windows or KDE users can surf to smb:/ in Konqueror...

Posted late Sunday evening, November 20th, 2011

Setting up a samba server on your NetBSD box for WindowsXP clients is really simple.

Install samba via pkgsrc:

# cd /usr/pkgsrc/net/samba
# make install clean

Start the services via /etc/inetd.conf then uncomment the next two lines.

#netbios-ssn stream tcp nowait root /usr/pkg/sbin/smbd
#netbios-ns dgram udp wait root /usr/pkg/sbin/nmbd

Change it to this:

netbios-ssn stream tcp nowait root /usr/pkg/sbin/smbd
netbios-ns dgram udp wait root /usr/pkg/sbin/nmbd

Save the changes and restart inetd:

/etc/rc.d/inetd restart

Now add the following lines to /etc/rc.conf:

smbd=YES
nmbd=YES
samba=YES

You have to create a /usr/pkg/etc/samba/smb.conf with the following basic configuration:

workgroup="some_group"
server string="NetBSD Samba Server"
hosts allow = 192.168.1. , 192.168.0.
encrypt passwords = yes


[shared]
comment = Shared
path = /home/ficovh/mp3
browseable = yes
writable = no
valid users = samba

Add a valid user to the NetBSD system:

# useradd samba

Add a windows user to samba and set the password:

# smbpasswd -a -U samba

Now test the server with your Windows machine.

You can also browse the content from a windows machine with NetBSD smbclient:

# smbclient //ip_windows/shared_name_resource

ip_windows is the IP for the windows machine and shared_name_resource is the directory shared.

You can also test if your local samba server is working.

# smbclient -Usamba -L localhost

Thats it, a basic samba server on your NetBSD box.

See also

Posted late Sunday evening, November 20th, 2011

Contents

Meta

Note that there is also a Xen HOWTO. Arguably this content could be folded in there.

Requirements

Xen3 is supported from NetBSD-4.0 onward. If you plan on using NetBSD-CURRENT, please read the article ?How to build NetBSD-current to do so. Guest operating systems can run from their own partitions, or from image files in the main (DOM0) install.

This tutorial describes how to:

  • Install and configure NetBSD as a DOM0
  • Install and run a NetBSD as a DOMU
  • Install and run a Windows XP system as as DOMU
  • Install and run a Debian system as a DOMU

Installing Xen tools and kernels

Xen tools

To run and administer xen domains, we need the xentools3 or xentools33 packages which are available in pkgsrc.

Xen 3.1 packages are under sysutils/xentools3 for traditional xentools, and sysutils/xentools3-hvm for the additional HVM support to run un- modified OSes such as Windows XP.

Xen 3.3 packages are under sysutils/xentools33. Unlike Xen 3.1, no extra package is required for HVM support. Note, it is not possible to install Xen 3.1 and Xen 3.3 packages at the same time. They conflict with each other.

HVM stands for Hardware Virtualization Managed. The benefit of hardware virtualization is that you can run OSes that don't know they are being virutalized like Windows XP, for example. However, you must have a CPU which supports this. Intel CPUs must have the 'VT' instruction. AMD CPUs will have the 'SVM' instruction. You can find out if your CPU supports HVM by taking a look at this page:

http://wiki.xensource.com/xenwiki/HVM_Compatible_Processors

In NetBSD 5.0 there's a new cpuctl command. This is an example output of an AMD CPU:

 # cpuctl identify 0
 cpu0: AMD Unknown K8 (Athlon) (686-class), 2210.22 MHz, id 0x60f82
 cpu0: features 0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
 cpu0: features 0x178bfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX>
 cpu0: features 0x178bfbff<FXSR,SSE,SSE2,HTT>
 cpu0: features2 0x2001<SSE3,CX16>
 cpu0: features3 0xebd3fbff<SCALL/RET,NOX,MXX,FFXSR,RDTSCP,LONG,3DNOW2,3DNOW>
 cpu0: features4 0x11f<LAHF,CMPLEGACY,SVM,EAPIC,ALTMOVCR0,3DNOWPREFETCH>
 cpu0: "AMD Turion(tm) 64 X2 Mobile Technology TL-64"
 cpu0: I-cache 64KB 64B/line 2-way, D-cache 64KB 64B/line 2-waycpu0: L2 cache 1MB 64B/line 16-way
 cpu0: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
 cpu0: DTLB 32 4KB entries fully associative, 8 4MB entries fully associativecpu0: Initial APIC ID 0
 cpu0: AMD Power Management features: 0x7f<TS,FID,VID,TTP,HTC,STC,100>
 cpu0: family 0f model 08 extfamily 00 extmodel 06

Note the SVM feature flag in the features4 line indicating that HVM support is present on this CPU. However, this feature may be disabled in the BIOS. In this case since NetBSD 5.0 there will show up this dmesg line on AMD CPUs only:

 cpu0: SVM disabled by the BIOS

Xen 3.1 (the xentools3-hvm package will automatically bring in the base xentools3):

# cd /usr/pkgsrc/sysutils/xentools3-hvm
# make install

Xen 3.3:

# cd /usr/pkgsrc/sysutils/xentools33
# make install

Xen kernel

Next, we will get the xen hypervisor kernel itself. For NetBSD 4.x and 5.x the i386 port does not support 'PAE' kernels and must run the Xen 3.1 package. This restriction has been removed in -current and is not relevant for the amd64 port.

For Xen 3.1, in pkgsrc this is sysutils/xenkernel3, for Xen 3.3, this is sysutils/xenkernel33:

# cd /usr/pkgsrc/sysutils/xenkernel3
# make install

And copy it into / directory, like this:

# cp /usr/pkg/xen3-kernel/xen.gz /

Xen DOM0 kernel

Lastly, we need a XEN-enabled kernel for our DOM0 domain. There are two possibilities: downloading the binary version, or building it from source.

Downloading the binary version

From NetBSD-4.0 onward, NetBSD supports Xen, and provides some XEN-enabled kernel directly from ?1[36], in each binary/kernel directory associated with a particular release.

For example, with NetBSD-4.0, You can grab one from (/netbsd-XEN3_DOM0.gz):

# ftp -a ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-4.0/i386/binary/kernel/netbsd-XEN3_DOM0.gz

The netbsd-XEN3_DOM0.gz file contains a gzipped version of the kernel. Just copy and move it into the root directory like this:

# cp netbsd-XEN3_DOM0.gz /

Building it from source

Building a kernel from source is out of the scope of this section. Please consult How to build a kernel from NetBSD's FAQ for more details.

Once building is done, you can find them in /usr/src/obj/_releasedir_/i386/binary/kernel/. Identically to binary version, copy or move netbsd-XEN3_DOM0 in the root directory /.

Selecting a bootloader

In NetBSD 5.0 the native boot loader, /boot, can load Xen directly. The NetBSD 5.0 bootloader can be easily dropped into a NetBSD 4.x system by coping them into /boot and running installboot(8) properly.

Updating /boot

For full details refer to installboot(8), but for a standard configuration with VGA console and an IDE or SATA drive with an FFSv1 root file system use the following:

# cp /usr/mdec/boot /boot
# installboot -v -o timeout=5 /dev/rwd0a /usr/mdec/bootxx_ffsv1

Updating /boot.cfg

NetBSD 5.0 or later will already have a /boot.cfg file with a basic configuration. Enabling Xen support only requires one additional line in this case. If you're upgrading from an earlier version or do not have an existing /boot.cfg use the following example:

banner=Welcome to NetBSD
banner==================
banner=
banner=Please choose an option from the following menu:
menu=Boot normally:boot netbsd
menu=Boot single-user:boot netbsd -s
menu=Boot backup kernel:boot onetbsd
menu=Drop to boot prompt:prompt
menu=Boot Xen with 256MB for dom0:load /netbsd-XEN3_DOM0 console=pc;multiboot /usr/pkg/xen3-kernel/xen.gz dom0_mem=256M
menu=Boot Xen with 256MB for dom0 (serial):load /netbsd-XEN3_DOM0 console=com0;multiboot /usr/pkg/xen3-kernel/xen.gz dom0_mem=256M console=com1 com1=115200,8n1
menu=Boot Xen with dom0 in single-user mode:load /netbsd-XEN3_DOM0 -s;multiboot /usr/pkg/xen3-kernel/xen.gz dom0_mem=256M
timeout=5
default=1

Make sure you update the "load /netbsd-XEN3_DOM0" and the "dom0_mem=256M" arguments to match your setup. On next boot select the 'Boot Xen with 256MB for dom0' option and make sure you see Xen kernel messages and the normal NetBSD kernel messages. Once you're satisfied it is working you can change the "default=1" line to "default=5" to automatically boot Xen on reboot.

Setting up DOM0

Creating xen devices

To create all xen devices, change to /dev and run ./MAKEDEV xen

cd /dev
./MAKEDEV xen

This should create the devices xencons, xenevt and xsd_kva. If any of these are missing you may not have updated to CURRENT using the latest sources and you will have to create the missing device files.

Configuring the bridge interface

The bridge(4) interface is used to provide network access to DOMUs.

To use one, edit (or create) the file /etc/ifconfig.bridge0 and insert following lines to:

create
!brconfig $int add bge0 stp bge0 up

Where 'bge0' should be changed to the name of the interface you want to use with your guest operating systems. use ifconfig(8) to get more details about your actual interfaces.

Rebooting into DOM0

Time to reboot:

# shutdown -r now

If all has gone well, you should have booted into the XEN3_DOM0 kernel. Check this with uname(1):

# uname -v
NetBSD 4.0 (XEN3_DOM0) #0: Sun Dec 16 01:20:31 PST 2007
builds@wb34:/home/builds/ab/netbsd-4-0-RELEASE/i386/200712160005Z-obj/home/builds/ab/netbsd-4-0-RELEASE/src/sys/arch/i386/compile/XEN3_DOM0

You should have XEN3 DOM0 mentioned somewhere.

Configuring rc scripts

Copy or symlink xend, xenbackendd and xendomains from /usr/pkg/share/examples/rc.d to /etc/rc.d.

# cp /usr/pkg/share/examples/rc.d/xend /etc/rc.d/
# cp /usr/pkg/share/examples/rc.d/xenbackendd /etc/rc.d/
# cp /usr/pkg/share/examples/rc.d/xendomains /etc/rc.d/

Edit /etc/rc.conf and add the following lines:

xend=YES
xenbackendd=YES
xendomains="dom1"

Later on, when you have created a configuration file for 'dom1', the xendomains variable specified above will trigger 'dom1' to be started when the system is booted. At this point no configuration exists for dom1, therefore it does nothing at this point. If you choose to name your configuration file something else, adapt the name accordingly.

To avoid rebooting a second, start all three services:

# /etc/rc.d/xend start
# /etc/rc.d/xenbackendd start
# /etc/rc.d/xendomains start

Run ifconfig -a to ensure the bridge interface is present and issue a ps ax | grep xen to ensure you have a similar output.

12 ?     DK   0:00.00 [xenwatch]
13 ?     DK   0:00.00 [xenbus]
411 ?     I    0:00.24 xenstored --pid-file=/var/run/xenstore.pid
594 ?     IWa  0:00.26 xenconsoled
629 ?     IW   0:00.00 /usr/pkg/bin/python2.3 /usr/pkg/sbin/xend start
631 ?     IWsa 0:00.02 /usr/pkg/sbin/xenbackendd
639 ?     IWa  0:00.52 /usr/pkg/bin/python2.3 /usr/pkg/sbin/xend start

The DOM0 configuration is now done. We will proceed to configuring DOMU domains.

Configuring DOMU

Configuring and installing a NetBSD DOMU

Create (or modify) /usr/pkg/etc/xen/dom1 and include this:

kernel = "/usr/src/obj/releasedir/i386/binary/kernel/netbsd-INSTALL_XEN3_DOMU.gz"
#kernel = "/netbsd-XEN3_DOMU"
memory = 64
name = "dom1"
#vcpus = 1
disk = [ 'phy:/dev/wd0g,0x03,w','file:/usr/src/obj/releasedir/i386/installation/cdrom/netbsd-i386.iso,0x04,r' ]
vif = [ 'bridge=bridge0' ]
root = "/dev/wd0d"

This configuration boots into the NetBSD sysinst program and allows you to install a NetBSD DOMU using the normal sysinst method. This configuration uses a DOMU_INSTALL kernel and an ISO image provided by a successful 'build release' and 'build iso-image'. You may be able to locate a valid Xen3 DOMU_INSTALL kernel from ftp://ftp.netbsd.org/pub/NetBSD-daily/ but if not, building a release is your best bet.

In this configuration file, /dev/wd0g is the reserved partition for the guest operating system. This should be changed to the partition you reserved prior to following the instructions within this document.

If you would like to use a physical CDROM instead of an ISO image, change the disk line to:

disk = [ 'phy:/dev/wd0g,0x03,w','phy:/dev/cd0a,0x04,r' ]

Now boot into sysinst using the command:

xm create dom1 -c

The reserved partition will appear as /dev/xbd0. Proceed as you would with a normal NetBSD installation using xbd0 as the target drive and xbd1 as the CDROM.

When you have finished, run shutdown -hp now to dom1.

Now edit /usr/pkg/etc/xen/dom1. Comment the INSTALL kernel and uncomment the DOMU kernel.

You should now have a working NetBSD DOMU (dom1). Boot into dom1 again with the command:

xm create dom1 -c

and ensure the file, /etc/ttys contains only this line or has all other lines commented:

console "/usr/libexec/getty Pc"         vt100   on secure

and the file /etc/wscons.conf is completely empty or has all lines commented out. These last two steps ensure no errors should be present on boot. Setting wscons=NO in /etc/rc.conf may effectively do the same thing.

From here, configure /etc/rc.conf and all runtime configuration files as you would normally. The network interface name should be xennet0. Use this name when configuring an IP address.

More information can be obtained by referencing the Xen user guide and the official NetBSD Xen Howto. Questions can be addressed to the port- xen@NetBSD.org mailling list.

Configuring and installing a Windows XP DOMU

This requires an HVM capable processor and xentools (see sections above).

This assumes you have a copy of the Windows install CD in /home/xen/winxp.iso, and wish to create a file /home/xen/winxp.img to hold the install. First create a blank file to hold the install. This assumes a size of 4GB (4096M). If you want a different size adjust the numbers to match:

# dd if=/dev/zero of=/home/xen/winxp.img bs=1m count=4096

Create /usr/pkg/etc/xen/win01:

kernel = '/usr/pkg/lib/xen/boot/hvmloader'
builder = 'hvm'
memory = '400'
device_model='/usr/pkg/libexec/qemu-dm'

disk = [ 'file:/home/xen/winxp.img,ioemu:hda,w',
          'file:/home/xen/winxp.iso,ioemu:hdb:cdrom,r', ]

# Hostname
name = "win01"

vif = [ 'type=ioemu, bridge=bridge0' ]

boot= 'd'
vnc = 1
usbdevice = 'tablet' # Helps with mouse pointer positioning

You may want to modify the amount of memory and pathnames.

Ensure you have a vncviewer installed, such as net/tightvncviewer or net/vncviewer from pkgsrc.

Then start the XENU and connect to it via VNC.

# xm create /usr/pkg/etc/xen/win01
# vncviewer :0

This will boot the Windows ISO image and let you install Windows as normal. As Windows reboots during install you may need to restart vncviewer.

After install change the boot d to boot c to have the system boot directly from the disk image.

Configuring and installing a GNU/Linux DOMU

We will do this in two steps:

  • install a GNU/Linux system, from a livecd or any installation media
  • configure the DOM0 so that it can create and start the Linux DOMU.

Installing a Linux distribution (soon-to-be DOMU)

Before proceeding with DOMU configuration, we will install our favorite GNU/Linux distribution on the computer.

In order to do it, we need at least two partitions (only one, if you do not consider the swap). These partitions must reside outside of the NetBSD slice, and may be either of primary or extended type. Of course, you can use more than two, but adapt your labels and partitions accordingly.

We do not cover the partition/slices manipulations through fdisk(8) and disklabel(8), as it depends strongly on how you manage your hard drive's space.

For this tutorial, we will use this partitioning:

# fdisk /dev/wd0d                                                      
fdisk: removing corrupt bootsel information
fdisk: Cannot determine the number of heads
Disk: /dev/wd0d
NetBSD disklabel disk geometry:
cylinders: 486344, heads: 16, sectors/track: 63 (1008 sectors/cylinder)
total sectors: 490234752

BIOS disk geometry:
cylinders: 1023, heads: 255, sectors/track: 63 (16065 sectors/cylinder)
total sectors: 490234752

Partition table:
0: Linux native (sysid 131)
    start 63, size 20482812 (10001 MB, Cyls 0-1274)
        PBR is not bootable: All bytes are identical (0x00)
1: Linux swap or Prime or Solaris (sysid 130)
    start 20482875, size 1959930 (957 MB, Cyls 1275-1396)
        PBR is not bootable: All bytes are identical (0x00)
2: NetBSD (sysid 169)
    start 61464690, size 428770062 (209360 MB, Cyls 3826-30515/178/63), Active
3: <UNUSED>
Drive serial number: -286527765 (0xeeebeeeb)

Here, you notice that we decide to use two primary partitions for our future Linux DOMU:

  • partition 0 (for the root directory /)
  • partition 1 (for the swap)

Labels:

16 partitions:
#        size    offset     fstype [fsize bsize cpg/sgs]
 a:  30720816  61464690     4.2BSD   2048 16384     0  # (Cyl.  60976*-  91453*)
 b:   1049328  92185506       swap                     # (Cyl.  91453*-  92494*)
 c: 428770062  61464690     unused      0     0        # (Cyl.  60976*- 486343)
 d: 490234752         0     unused      0     0        # (Cyl.      0 - 486343)
 e:  20480000  93234834     4.2BSD      0     0     0  # (Cyl.  92494*- 112812*)
 f:  20480000 113714834     4.2BSD      0     0     0  # (Cyl. 112812*- 133129*)
 g:  20480000 134194834     4.2BSD      0     0     0  # (Cyl. 133129*- 153447*)
 h: 335559918 154674834     4.2BSD      0     0     0  # (Cyl. 153447*- 486343)
 i:  20482812        63 Linux Ext2      0     0        # (Cyl.      0*-  20320*)
 j:   1959930  20482875       swap                     # (Cyl.  20320*-  22264*)

Bear in mind that we added two labels here, namely i and j, which maps respectively to partition 0 and partition 1 of the disk. We will use these labels later for DOMU configuration.

Now that we have partitioned the disk, proceed with installing your Linux distribution. We will not cover that part in this tutorial. You can either install it from an installation media (a cdrom from example), or copy files from an already installed distribution on your computer.

Tip: to manipulate ext2/3 filesystems (the traditional fs under Linux) from NetBSD, you can use sysutils/e2fsprogs from pkgsrc:

# cd /usr/pkgsrc/sysutils/e2fsprogs
# make install

And then use e2fsck, mke2fs and mount_ext2fs(8) directly from NetBSD.

Getting XEN aware Linux kernels

Once installation is done, reboot your computer and return to our Xen-NetBSD system.

To boot our Linux DOMU, we will need a Linux kernel supporting the XENU virtualisation. Depending on your Linux distribution, you can grab one from its repository (it is up to you to find it through aptitude, yum or whatever package manager you use), or get one from the Xen binary distribution.

To get a XENU Linux kernel from Xen binary distribution, get it directly from Xen website download page. Download the tarball and extract the vmlinuz-*-xen from it. In our case, with a 2.6.18 Linux kernel:

# ftp -a http://bits.xensource.com/oss-xen/release/3.1.0/bin.tgz/xen-3.1.0-install-x86_32.tgz
# cd /tmp
# tar -xzf xen-3.1.0-install-x86_32.tgz dist/install/boot/vmlinuz-2.6.18-xen

vmlinuz-2.6.18-xen is the kernel that Xen will use to start the DOMU. Move it to any directory you like (just remember it when configuring the kernel entry in the DOMU configuration file):

# mv dist/install/boot/vmlinuz-2.6.18-xen /vmlinuz-XEN3-DOMU

Configuring DOMU

Configuring the Linux DOMU is a bit different than a NetBSD one; some options tend to differ.

Edit (or create) the configuration file domu-linux, in /usr/pkg/etc/xen/:

# vi /usr/pkg/etc/xen/domu-linux

Here's a typical config file for a Linux DOMU:

#----------------------------------------------------------------------------
# Kernel image file. This kernel will be loaded in the new domain.
kernel = "/vmlinuz-XEN3-DOMU"

# Memory allocation (in megabytes) for the new domain.
memory = 256

# A handy name for your new domain. This will appear in 'xm list',
# and you can use this as parameters for xm in place of the domain
# number. All domains must have different names.
#
name = "domu-linux"

# Which CPU to start domain on (only relevant for SMP hardware).  CPUs
# numbered starting from ``0''.
#
cpu = "^1"   # leave to Xen to pick

#----------------------------------------------------------------------------
# Define network interfaces for the new domain.

# Number of network interfaces (must be at least 1). Default is 1.
vif = [ '' ]

# Define MAC and/or bridge for the network interfaces.
#
# The MAC address specified in ``mac'' is the one used for the interface
# in the new domain. The interface in domain0 will use this address XOR'd
# with 00:00:00:01:00:00 (i.e. aa:00:00:51:02:f0 in our example). Random
# MACs are assigned if not given.
#
# ``bridge'' is a required parameter, which will be passed to the
# vif-script called by xend(8) when a new domain is created to configure
# the new xvif interface in domain0.
#
# In this example, the xvif is added to bridge0, which should have been
# set up prior to the new domain being created -- either in the
# ``network'' script or using a /etc/ifconfig.bridge0 file.
#
vif = [ 'mac=aa:00:00:50:02:f0, bridge=bridge0' ]

#----------------------------------------------------------------------------
# Define the disk devices you want the domain to have access to, and
# what you want them accessible as.
#
# Each disk entry is of the form:
#
#       phy:DEV,VDEV,MODE
#
# where DEV is the device, VDEV is the device name the domain will see,
# and MODE is r for read-only, w for read-write.  You can also create
# file-backed domains using disk entries of the form:
#
#       file:PATH,VDEV,MODE
#
# where PATH is the path to the file used as the virtual disk, and VDEV
# and MODE have the same meaning as for ``phy'' devices.
#
# /dev/wd0i will be seen as "hda1" under DOMU (the root partition)
# /dev/wd0j will be seen as "hda2" under DOMU (the swap)
#
disk = [ 'phy:/dev/wd0i,hda1,w','phy:/dev/wd0j,hda2,w' ]

#----------------------------------------------------------------------------
# Set the kernel command line for the new domain.

# Set root device.
root = "/dev/hda1"

Now, you should be able to start your first Linux DOMU!

# xm create -c /usr/pkg/etc/xen/domu-linux

Possible caveats

If you intend to have more than one box configured with the above configuration on the same network, you will most likely have to specify a unique MAC address per guest OS, otherwise it is likely you will have a conflict. I'm not sure if the MAC assignment is random, incremental or if Xen is able to check for the existence of the proposed MAC address, so specifying the MAC address is recommended.

Here is a method to assign a MAC address to a newly created Xen Domu. First as described before, use the following vif parameter in your config file :

vif = [ 'bridge=bridge0' ]

Then, run the Xen DomU and, once logged run the following command :

# ifconfig xennet0

Output Sample : 

xennet0: flags=8863<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
        enabled=0
        address: 00:16:3e:2e:32:5f
        inet 192.168.4.81 netmask 0xffffff00 broadcast 192.168.4.255
        inet6 fe80::216:3eff:fe2e:325f%xennet0 prefixlen 64 scopeid 0x2

A MAC address is automaticaly generated, use it in your config file with the following syntax :

vif = [ 'mac=00:16:3e:2e:32:5f, bridge=bridge0' ]

And now you can restart the Xen DomU.

Please, note that the MAC Address MUST start with "00:16:3e".

See also

Posted late Sunday evening, November 20th, 2011

SMB aka CIFS (common internet file system) is a ubiquitous file sharing mechanism, but unfortunately it is very insecure. All files are sent clear over the line, and if you don't config password encryption, even passwords are sent as cleartext.

There is currently no built-in encryption or security in the CIFS protocol, nor is there any available as an extension to Samba, so we'll have to resort to external methods.

One of the nicer ways to secure Samba is by using stunnel. This is a little tool which listens on a port on the client machine and forwards all data sent to that port to another port/machine, encrypted via (Open)SSL.

Contents

Setting up the server

Configure samba

You set up the server just as you would normally, as described in How to set up a Samba Server.

If you wish to allow only secure traffic, you can let it listen on localhost with the following statement in smb.conf:

# Only listen on loopback interface
socket address=127.0.0.1

Configure stunnel

You can install security/stunnel from ?pkgsrc. Then you can copy /usr/pkg/share/examples/stunnel/stunnel.conf-sample and modify it to your needs. The following will be sufficient if you only need the bare minimum to get a secure samba setup:

Simple stunnel configuration for a secure samba setup

# OpenSSL certificate
cert = /usr/pkg/etc/stunnel/stunnel.pem
# Run chrooted as nobody
chroot = /var/run/stunnel
setuid = nobody
setgid = nobody
# This file is created after chrooting
pid = /stunnel

# Accept connections on port 800, on any interface
[smb]
accept  = 0.0.0.0:800
# instead of port 139, port 445 will also work, unless you're using Mac OS X clients
connect = localhost:139

As you can see, you'll need an SSL certificate/key. This can be generated like this:

# openssl req -new -nodes -x509 -out stunnel/stunnel.pem -keyout /etc/stunnel/stunnel.pem

Run stunnel

Just add stunnel=yes to your /etc/rc.conf:

# echo "stunnel=yes" >> /etc/rc.conf
# /etc/rc.d/stunnel start

Warning: stunnel is very silent. Even if it gets an error it will just fail silently. Check with pgrep if it's running.

Configuring your clients

Unix clients

On a Unix client you simply install and run security/stunnel as described above. You'll need to swap the port numbers and put it in client mode. ie, your stunnel.conf should look like this:

client=yes; [smb] accept=localhost:139 connect=servername:800

This makes your client act as a samba server, to which you can connect. As soon as you connect to your machine, the data is encrypted and forwarded to servername. You can run stunnel from rc.conf just like on the server side.

Of course you can easily test it by connecting to localhost:

# smbclient -U yoda //localhost/myshare

Windows clients

Connecting a Windows client to samba over stunnel is a major hassle. Some background on why this is a problem is in order.

Apparently, when Windows is booted, the kernel binds a socket to port 445 on every real (this is important as we'll see later on) network interface. This means that no other process can ever bind this port. (try it, you'll get a "permission denied" message). This would mean we need to use another port for our fake "shared folder". Unfortunately, the Windows filemanager has no way to specify which port to use when you click "map network drive", so that's not an option.

Luckily for us, Windows has the following odd behaviour: When you click "map network drive" in the filemanager, it will first try to connect to port 445. When it finds no service listening there, it will try to fall back to port 139. Only when that has no service listening either, it will tell the user it couldn't connect. We will "abuse" this behaviour by tricking it into using this port.

Simply binding stunnel to port 139 is impossible, because of the Windows behaviour where it binds ports 139 and 445 on every interface, even if no actual files are being shared. It turns out that it doesn't do this on loopback network devices. To install one, follow this set of instructions:

  1. Open the "add hardware" wizard from the control panel.
  2. Wait for it to search in vain for new hardware.
  3. Tell it "yes, I've already connected my hardware" or the wizard will end...
  4. Pick "add a new device" from the bottom of the list.
  5. Don't let windows search for the hardware but choose it from a list ("Advanced").
  6. Pick the category "Network adapters".
  7. Choose "Microsoft loopback adapter".

When our new "hardware" is installed, you need to assign it an IP and disable NetBIOS activity on it:

  1. Open the "properties" dialog from the contextmenu in the "network connections" overview.
  2. Deselect all bindings except the TCP/IP ones. Typically you'll need to deselect "client for Microsoft networks" and "File and printer sharing".
  3. Select "TCP/IP", and then "settings" (or "properties")
  4. Choose any private network IP address you'll never see in any real network. (10.232.232.232 is a good example)
  5. Click "Advanced..."
  6. Choose the tab titled "WINS"
  7. Under "NetBIOS settings", click on "Disable NetBIOS over TCP/IP"

Finally, we can install stunnel (Windows binaries are available from ?1(http://www.stunnel.org). Put this in the stunnel.conf file:

client=yes [smb] accept=10.232.232.232:139 connect=servername:800

It is advisable to install the stunnel service so it will start on system boot, which means it will be (semi-)transparent to the user.

To connect to the server, just open up the "map network drive" dialog and enter \\10.232.232.232\sharename in the "computer name" box. To make this process a little more userfriendly, think up a hostname and stick it in \winnt\system32\drivers\etc\hosts (windows NT/XP) or \windows\hosts (Windows '9x). The format of this file is exactly like /etc/hosts on Unix.

References

Posted late Sunday evening, November 20th, 2011

Contents

What TET is

The TETware family of tools are Test Execution Management Systems that takes care of the administration, sequencing, reporting and portability of all of the tests that you develop. Freeing up developers to concentrate on test development and helping testers by providing them with a single, standard, test harness. Enabling you to deliver your software projects on time and across multiple operating systems. They are all available off-the-virtual-shelf. Easily accessed by ftp download. So stop re-inventing the wheel, take the drudge out of test development and use TETware.

Reason of this article

There is an artcile describes how to build and run TET framework from scratch. The pkgsrc version has several restrictions:

  • the recent version not available until now
  • necessary to have the pkgsrc infrastructure
  • only distributed version of TET is provided

However, you may apply patches from the pkgsrc tree (for example the patch-ad still can be applied for distributed TET).

Porting to NetBSD

Current opensourced version is 3.7a. To download or check new versions look to the Links section below. (The guide was tested at least on the NetBSD-current/i386 and NetBSD-current/ARMv6)

Patchset

The simply patch would be applied before compilation is started.

Prebuild

  • After applying patch you'll need to copy the proper makefile

    cp src/defines/UNTESTED/freebsd.mk src/defines/netbsd.mk

  • Make the $TET_ROOT directory like /opt/tet3

  • Copy content of TET sources tree to $TET_ROOT

Notice to get distributed TET:

  • Create regular user tet on the target system with home path as $TET_ROOT without real shell

Build

  • Change working directory to $TET_ROOT
  • Build framework

    ./configure -t cd src make make install

where means Lite (-t lite) or Distributed (-t inet) version of TET framework.

Notice to get distributed TET:

  • Change owner of the $TET_ROOT to tet recursively

Update user's files

  • Put to your ~/.profile

    TET

    export TET_ROOT="/opt/tet3" export PATH=$PATH:$TET_ROOT/bin

System files correction (distributed TET)

  • Start tccd daemon from the /etc/rc.conf

    export TET_ROOT="/opt/tet3" if test -x "$TET_ROOT/bin/tccd" then $TET_ROOT/bin/tccd && echo "TET3 tccd started" fi

Other files ($TET_ROOT/systems, $HOME/systems.equiv, /etc/services) are corrected in according to TET3 install guide.

Sample of required files

You'll need a minimal $TET_ROOT/systems file. The following content in the file is enough for this test suite.

000 localhost 60103

For the 'inet' version of TET, you'll need tccd running, and a $HOME/systems.equiv file in the directory of the user running tccd. The following contents will do for this file:

Note! In our case the $HOME is equivalent to /opt/tet3

localhost

And you'll need to append following lines to the /etc/services:

tcc             60103/tcp
tcc             60103/udp

Simple check

The directory contrib in the source tree contains a number of simplest examples of test cases. Let's start something from it step by step. Consider test cases under contrib/capi. Login as user tet. Be ensure you have set $TET_ROOT to actual path (/opt/tet3 in our case) and $TET_ROOT/bin is covered by $PATH.

$ echo $TET_ROOT
/opt/tet3
$ echo $PATH
/home/andy/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin:/usr/pkg/bin:/usr/pkg/sbin:/usr/games:/usr/local/bin:/usr/local/sbin:/opt/tet3/bin

Change current directory to $TET_ROOT/contrib/capi.

cd $TET_ROOT/contrib/capi

Patch test case build system (yeah, I know...):

--- tools/buildtool.c.orig      2008-11-24 13:15:58.000000000 +0000
+++ tools/buildtool.c   2008-11-24 13:16:15.000000000 +0000
@@ -51,7 +51,7 @@
 #define BUILDSUCCESS 0   /* successful return from buildtool */


-static void tbuild()
+void tbuild()
 {
 char command_line[BUFLEN];
 char bres_line[BUFSIZ];

To run the test suite, firstly apply profile setting, then run the Setup script and after use tcc.

source profile
sh ./Setup
tcc -p -bec contrib/capi

Links

Posted late Sunday evening, November 20th, 2011

Oracle Express on NetBSD

started by Emmanuel Kasper

Oracle Express is a free beer version of the Oracle Database. You are free to use it in production, and to redistribute it, but it has harcoded limits of use one CPU, 1 GB of RAM, and 4 GB for a table size.

Download the binary at http://www.oracle.com/technology/software/products/database/xe/index.html

Convert the rpm in cpio format

pkg_add -v rpm2cpio
rpm2cpio.pl oracle-xe-univ-10.2.0.1-1.0.i386.rpm > oracle-xe-univ-10.2.0.1-1.0.i386.cpio

Unpack the cpio archive (it installs in /usr/lib/oracle )

cpio -idv < oracle-xe-univ-10.2.0.1-1.0.i386.cpio

Install the Suse linux environnment. I don't need what exactly needed, so I prefer to install all.

pkg_add -v suse-9.1nb3

Add oracle libraries to ldconfig library cache

echo /usr/lib/oracle/xe/app/oracle/product/10.2.0/server/lib >> /emul/linux/etc/ld.so.conf
/emul/linux/sbin/ldconfig

Test sqlplus binary

sqlplus /nolog 

Right now we have a sqlplus binary that works and is abled to connect over the network to a database but creating a database fails with the error:

ERROR:
ORA-12549: TNS:operating system resource quota exceeded

I guess some kernel recompile is necessary. See http://n0se.shacknet.nu/

Posted late Sunday evening, November 20th, 2011

Contents

Introduction

Netbeans, along with Eclipse, is one of the most widely used Java IDE (Integrated Development Environment), also capable of Ruby and C/C++ development. The purpose of this document is to describe the steps needed to run Netbeans 6.0 on a NetBSD system, using Linux Java Virtual Machine and Linux compatibility mode.

Downloading Netbeans

The latest version of Netbeans may be downloaded from here. We will be using version 6. After having downloaded file netbeans-6.0.zip, we compare the SHA256 sums to ensure data integrity:

$ digest sha256 netbeans-6.0.zip
SHA256 (netbeans-6.0.zip) = fc80d6fd507c5bcc647db564fc99518542b9122d7ea6fcf90abb08156a26d549

XXX: This is from the NetBeans DVD, check if the downloadable version is the same.

which is equal to the one mentioned in the download page.

Next we extract the compressed archive:

$ unzip netbeans-6.0.zip
inflating (...)
$ ls -ld netbeans
drwxr-xr-x  11 user  users  1024 Dec 23 16:55 netbeans/

Verifying Linux compatibility mode

Kernel support

Since we are going to install a Linux binary JVM, we must also enable Linux compatibility mode in our system. If you are using a GENERIC kernel, you are already done since the Linux compatibility layer is enabled by default. If not, you will need to compile your kernel with the following options:

options COMPAT_LINUX

In case you are unfamiliar with the process of building a custom kernel, please refer to NetBSD Documentation.

A quick way to check whether you are ok as far as kernel support is concerned, is to invoke the following command:

$ config -x | grep COMPAT_LINUX
options         COMPAT_LINUX    # binary compatibility with Linux

(Configuration data will be available if the given kernel was compiled with either INCLUDE_CONFIG_FILE or INCLUDE_JUST_CONFIG options.)

Alternatively you can search across the Sysctl tree:

$ sysctl -a | grep emul emul.linux.kern.ostype = Linux emul.linux.kern.osrelease = 2.4.18 emul.linux.kern.osversion = #0 Wed Feb 20 20:00:02 CET 2002

Note that the NetBSD documentation covers extensively this topic, so if you run into trouble, please consult the respective pages.

Linux shared libraries

Most of the times, applications are linked against shared libraries, and for Linux applications, Linux shared libraries are needed. You could get the shared libraries from any Linux distribution theoretically, as long as they are not too outdated, but the suggested method is to use the pkgsrc system.

This package supports running ELF binaries linked with glibc2 which don't require X11 shared libraries.

cd /usr/pkgsrc/emulators/suse100_base/

# make install clean

This package contains some old shared libraries required for backwards compatibility.

# cd /usr/pkgsrc/emulators/suse100_compat/
# make install clean

This package supports running ELF binaries linked with glibc2 which require X11 shared libraries.

# cd /usr/pkgsrc/emulators/suse100_x11/
# make install clean

Installing Sun's JDK

NetBeans IDE 6.0 requires a Java SE JDK, version 5 or 6. lang/jdk6 unfrotunately fails to run NetBeans with segmentation fault errors, so we will use lang/jdk15.

Next we modify the make configuration file /etc/mk.conf and add ACCEPTABLE_LICENSES+=jdk13-license, to accept the jdk license. You can do that with the following command:

# echo "ACCEPTABLE_LICENSES+=jdk13-license" >> /etc/mk.conf

We will manually download the Linux self-extracting jdk file jdk-1_5_0_12-linux-i586.bin from here, and put it in /usr/pkgsrc/distfiles. But since jdk15 depends on jre15, we have to download the Linux self-extracting jre file jre-1_5_0_11-linux-i586.bin from here and put it in /usr/pkgsrc/distfiles aswell.

$ ls -l .bin -rwxr-xr-x 1 user wheel 49622107 Dec 23 15:44 jdk-1_5_0_12-linux-i586.bin -rwxr-xr-x 1 user wheel 17138633 Dec 23 15:18 jre-1_5_0_12-linux-i586.bin*

Now we are ready to install jdk:

# cd /usr/pkgsrc/lang/sun-jdk15/
# make install clean

/proc filesystem

Some Linux programs like Netbeans rely on a Linux-like /proc filesystem. The NetBSD procfs filesystem can emulate a /proc filesystem that contains Linux-specific pseudo-files. We now manually mount the proc file system.

# mount_procfs -o linux procfs /usr/pkg/emul/linux/proc

and verify it with:

# mount
/dev/wd0g on / type lfs (local)
ptyfs on /dev/pts type ptyfs (local)
tmpfs on /tmp type tmpfs (local)
procfs on /usr/pkg/emul/linux/proc type procfs (local)

You may have NetBSD mounted it automatically during the booting process, by modifying the /etc/fstab file by adding the following line:

procfs /usr/pkg/emul/linux/proc procfs rw,linux

Please be careful with your /etc/fstab. Wrong entries may lead to an unbootable system.

Data segment size

In usr/pkgsrc/lang/sun-jdk15/MESSAGE.NetBSD it is stated that "The maximum data segment size assigned to your user must be at least 262144". It is known, that this value isn't enough. Thus enter:

$ ulimit -d 400000

You may want to put this into your .profile

In case you encounter runtime errors of the following form:

[stathis@netbsd ~] ./netbeans/bin/netbeans --jdkhome /usr/pkg/java/sun-1.5
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
[stathis@netbsd ~]

try increasing the data segment size.

Running Netbeans

You may now run Netbeans, by typing:

[stathis@netbsd ~] ./netbeans/bin/netbeans --jdkhome /usr/pkg/java/sun-1.5

Note that you can edit netbeans/etc/netbeans.conf and add the following line:

netbeans_jdkhome="/usr/pkg/java/sun-1.5"

so you won't have to explicitly set the location of J2SE JDK everytime you run Netbeans.

Posted late Sunday evening, November 20th, 2011

Contents

Things needed

the latest stable NetBSD/i386 installation. The archivers/{zip,unzip} packages. The linux emulation packages suse_base and suse_compat.

If you want to run binary modules (MEX), you also need the suse_devel-7.3 package; that is not (yet?) available as suse9.x version. Matlab 14.3 ships with a JDK 1.5 which runs nicely on netbsd-3, so no additional JRE needed. The three Matlab 7 / R14 Unix installation CDROMs. The patches listed below. A few GBytes of diskspace.

Work to do

A Short Cut

While the following steps are kept for documentation and your own experiments, there is a tar-ball available with the patches and a shell script to apply them.

Unpack, change to the patch_matlabcd directory, call sh patch-matlab.sh -m /path/to/cdcontent, optionally add '-v' for verbose output.

Footwork

  • Copy the three CDROMs to the harddisk (directories CD1, CD2, CD3).
  • mount_union(8) a scratch directory on top of the directory that contains CD[123]. This scratch directory will contain all the changes that you make to the content of the CDs.
  • chdir to CD1
  • Apply the patch
    
--- /u/software/payware/unix/matlab-r14/CD1/install     2004-09-13 14:56:28.000000000 +0200
+++ install     2004-09-14 11:04:26.000000000 +0200
@@ -516,6 +516,19 @@
                Darwin)                                 # Mac OS X
                     ARCH="mac"
                     ;;
+               NetBSD)                                 # NetBSD (Linux emul)
+                   case "`/usr/bin/uname -m`" in
+                   i*86)
+                       ARCH="glnx86"
+                       ;;
+                   ia64)
+                       ARCH="glnxi64"
+                       ;;
+                   x86_64)
+                       ARCH="glnxa64"
+                       ;;
+                   esac
+                   ;;
             esac
         fi
        return 0
@@ -730,6 +743,19 @@
            Darwin)                                     # Mac OS X
                 ver=`/usr/bin/uname -r | awk '{print $1}'`
                 ;;
+           NetBSD)                                     # NetBSD (Linux emul)
+               case $Arch in
+                   glnx86)
+                       ver=`/emul/linux/lib/libc.so.6 | head -1 | sed -e "s/^[^0-9]*//" -e "s/[ ,].*$//"`
+                       ;;
+                   glnxi64)
+                       ver=2.2.4
+                       ;;
+                   glnxa64)
+                       ver=`/emul/linux/lib64/libc.so.6 | head -1 | sed -e "s/^[^0-9]*//" -e "s/[ ,].*$//"`
+                       ;;
+               esac
+               ;;
         esac
     fi
 #
    

to the ./install script (assuming /emul/linux is a symlink to the root of the linux emulation setup). Apply equally to CD2 and CD3. Apply the patch

--- /u/software/payware/unix/matlab-r14/CD1/update/install/arch.sh      2004-09-13 14:54:55.000000000 +0200
+++ update/install/arch.sh      2004-09-14 11:38:19.000000000 +0200
@@ -185,6 +185,19 @@
                Darwin)                                 # Mac OS X
                     ARCH="mac"
                     ;;
+               NetBSD)                                 # NetBSD (Linux emul)
+                   case "`/usr/bin/uname -m`" in
+                   i*86)
+                       ARCH="glnx86"
+                       ;;
+                   ia64)
+                       ARCH="glnxi64"
+                       ;;
+                   x86_64)
+                       ARCH="glnxa64"
+                       ;;
+                   esac
+                   ;;
             esac
         fi
        return 0

to ./update/install/arch.sh. Apply equally to CD2 and CD3. Pick the ./update/pd/install/tar.cmp zip archive from CD1, and unpack it.

Archive:  tar.cmp
  Length     Date   Time    Name
 --------    ----   ----    ----
    41864  09-14-04 13:10   install_matlab
        0  09-14-04 13:08   update/
        0  09-14-04 13:08   update/install/
        0  09-14-04 13:08   update/install/scripts/
      676  07-10-97 05:11   update/install/scripts/abort.sh
     2575  01-11-99 20:24   update/install/scripts/actualp.sh
     6230  09-14-04 13:11   update/install/scripts/arch.sh
     7329  07-27-00 21:42   update/install/scripts/batch1.sh
     2597  01-11-99 20:24   update/install/scripts/batch4.sh
     6985  01-29-03 21:28   update/install/scripts/bld_lbin.sh
     9466  12-11-03 19:21   update/install/scripts/bld_sbin.sh
      789  10-23-95 19:07   update/install/scripts/center.sh
      530  12-11-03 19:21   update/install/scripts/cleanup.sh
      486  01-11-99 20:24   update/install/scripts/clearsc.sh
      848  01-11-99 20:24   update/install/scripts/cont.sh
      449  01-11-99 20:24   update/install/scripts/echon.sh
     2167  11-01-02 03:33   update/install/scripts/fin.sh
     4590  03-22-04 06:07   update/install/scripts/genpathdef.sh
     3256  12-11-03 19:21   update/install/scripts/intro_l.sh
     5376  12-11-03 19:21   update/install/scripts/intro_s.sh
    20832  01-09-04 18:30   update/install/scripts/lm.sh
     6718  12-11-03 19:21   update/install/scripts/local.sh
     3590  12-11-03 19:21   update/install/scripts/main.sh
     1115  12-11-03 19:21   update/install/scripts/mapname.sh
     1502  01-11-99 20:24   update/install/scripts/netruser.sh
     1754  10-23-98 20:50   update/install/scripts/oldname.sh
     5267  11-01-02 03:33   update/install/scripts/options.sh
     6853  08-31-00 18:09   update/install/scripts/perm.sh
     6585  06-12-02 22:21   update/install/scripts/ruser.sh
      784  01-11-99 20:24   update/install/scripts/searchp.sh
     3303  12-11-03 19:21   update/install/scripts/ucleanpe.sh
     3295  12-11-03 19:21   update/install/scripts/update.sh
     5395  12-11-03 19:21   update/install/scripts/util.sh
     8565  12-11-03 19:21   update/install/scripts/verifyp.sh
 --------                   -------
   171771                   34 files

Apply the patch

--- install_matlab.orig 2005-08-12 10:23:52.000000000 +0200
+++ install_matlab  2005-09-30 14:27:23.000000000 +0200
@@ -552,6 +552,19 @@
            Darwin)                 # Mac OS X
                     ARCH="mac"
                     ;;
+       NetBSD)                 # NetBSD (Linux emul)
+           case "`/usr/bin/uname -m`" in
+           i*86)
+           ARCH="glnx86"
+           ;;
+           ia64)
+           ARCH="glnxi64"
+           ;;
+           x86_64)
+           ARCH="glnxa64"
+           ;;
+           esac
+           ;;
             esac
         fi
    return 0
@@ -795,6 +808,19 @@
        Darwin)                 # Mac OS X
                 ver=`/usr/bin/uname -r | awk '{print $1}'`
                 ;;
+       NetBSD)                 # NetBSD (Linux emul)
+       case $Arch in
+           glnx86)
+           ver=`/emul/linux/lib/libc.so.6 | head -1 | sed -e "s/^[^0-9]*//" -e "s/[ ,].*$//"`
+           ;;
+           glnxi64)
+           ver=2.2.4
+           ;;
+           glnxa64)
+           ver=`/emul/linux/lib64/libc.so.6 | head -1 | sed -e "s/^[^0-9]*//" -e "s/[ ,].*$//"`
+           ;;
+       esac
+       ;;
         esac
     fi
 #

to install_matlab, and the patch

--- orig/update/install/scripts/arch.sh 2004-04-04 10:20:31.000000000 +0200
+++ update/install/scripts/arch.sh      2004-09-14 13:11:17.000000000 +0200
@@ -185,6 +185,19 @@
                Darwin)                                 # Mac OS X
                     ARCH="mac"
                     ;;
+               NetBSD)                                 # NetBSD (Linux emul)
+                   case "`/usr/bin/uname -m`" in
+                   i*86)
+                       ARCH="glnx86"
+                       ;;
+                   ia64)
+                       ARCH="glnxi64"
+                       ;;
+                   x86_64)
+                       ARCH="glnxa64"
+                       ;;
+                   esac
+                   ;;
             esac
         fi
        return 0

(same as above) to update/install/scripts/arch.sh. Re-zip the contents of the archive

[hf@venediger] /var/tmp # zip -r tar.cmp install_matlab update
  adding: install_matlab (deflated 77%)
  adding: update/ (stored 0%)
  adding: update/install/ (stored 0%)
  adding: update/install/scripts/ (stored 0%)
  adding: update/install/scripts/abort.sh (deflated 53%)
  adding: update/install/scripts/actualp.sh (deflated 63%)
  adding: update/install/scripts/arch.sh (deflated 69%)
  adding: update/install/scripts/batch1.sh (deflated 70%)
  adding: update/install/scripts/batch4.sh (deflated 60%)
  adding: update/install/scripts/bld_lbin.sh (deflated 75%)
  adding: update/install/scripts/bld_sbin.sh (deflated 80%)
  adding: update/install/scripts/center.sh (deflated 55%)
  adding: update/install/scripts/cleanup.sh (deflated 42%)
  adding: update/install/scripts/clearsc.sh (deflated 41%)
  adding: update/install/scripts/cont.sh (deflated 53%)
  adding: update/install/scripts/echon.sh (deflated 39%)
  adding: update/install/scripts/fin.sh (deflated 78%)
  adding: update/install/scripts/genpathdef.sh (deflated 66%)
  adding: update/install/scripts/intro_l.sh (deflated 80%)
  adding: update/install/scripts/intro_s.sh (deflated 85%)
  adding: update/install/scripts/lm.sh (deflated 81%)
  adding: update/install/scripts/local.sh (deflated 78%)
  adding: update/install/scripts/main.sh (deflated 68%)
  adding: update/install/scripts/mapname.sh (deflated 54%)
  adding: update/install/scripts/netruser.sh (deflated 62%)
  adding: update/install/scripts/oldname.sh (deflated 53%)
  adding: update/install/scripts/options.sh (deflated 72%)
  adding: update/install/scripts/perm.sh (deflated 74%)
  adding: update/install/scripts/ruser.sh (deflated 82%)
  adding: update/install/scripts/searchp.sh (deflated 47%)
  adding: update/install/scripts/ucleanpe.sh (deflated 63%)
  adding: update/install/scripts/update.sh (deflated 70%)
  adding: update/install/scripts/util.sh (deflated 70%)
  adding: update/install/scripts/verifyp.sh (deflated 78%)

to the same name. Copy to CD1, CD2 and CD3. Rinse and repeat.

  • And that's about it. Provide a proper license.dat, and you are ready for installation. chdir to the installation directory, call the /path/to/install script from CD1, and click OK a few times.

At the end of the installation, there is a spurious rm -f with empty argument that can safely be ignored. If you find out where it hides, please tell me.

  • OK, there's still the license manager...

FlexLM comes with a byzantine set of shell script tools. Of course, there is one more arch.sh to patch, this time under {MATLABHOME}/etc/util/arch.sh. Then create /usr/tmp, preferably as a symlink to /var/tmp, because FlexLM and the MLM backend have that hard-coded for logs, locks, preferences. Make sure you have got a proper license file in place - the procedure is well-documented. After that, you can either start the flexlm daemon from /etc/rc.local by using {MATLABHOME}/etc/rc.lm.glnx86 start|stop, or with the following rc.d script :

#!/bin/sh
#
# $Id: how_to_run_matlab_r14.3_on_netbsd__92__i386.mdwn,v 1.2 2012/02/05 07:14:36 schmonz Exp $
#
# PROVIDE: flexlm
# REQUIRE: DAEMON

. /etc/rc.subr

name="flexlm"
rcvar=$name
flexlm_user="flexlm"
matlabroot="/path/to/matlabr14"
lm_license="${matlabroot}/etc/license.dat"
required_files="${lm_license}"
command="${matlabroot}/etc/glnx86/lmgrd"
command_args="-c ${lm_license} -2 -p -l +/var/log/flexlmlog"

load_rc_config $name
run_rc_command "$1"

-- you need to create a user to run flexlm as, and own the logfile to him.

Other patches

arch.sh

For whatever reason there may be, a Matlab 14 installation has three instances of the arch.sh script that works out what platform Matlab is running on:

> find . -name arch.sh -print
./bin/util/arch.sh
./etc/util/arch.sh
./update/install/scripts/arch.sh

And (of course) the one that is sourced by {MATLABROOT}/bin/matlab is the original version which you should replace by e.g. {MATLABROOT}/etc/util/arch.sh once you get tired of adding -glnx86 to every matlab invocation.

MEX

If you want to build binary modules with mex you need to make sure those modules (libraries) are linux libraries since you cannot use native NetBSD libraries with a program running in emulation. Make sure you install the suse_devel package as listed above, and apply this patch to {MATLABROOT}/toolbox/matlab/general/mex.m. It makes sure the mex shell-script (which calls the compiler) is executed by the linux shell.

Sound

Add a symlink to the native /dev/sound like

[hf@dreispitz] /<1>linux/dev > pwd
/usr/pkg/emul/linux/dev
[hf@dreispitz] /<1>linux/dev > ll audio 
0 lrwx------  1 root  wheel  13 Sep 10 03:25 audio -> /../dev/sound

Linux pthread libs

Matlab 13, at least, expects the pthread shared library under /usr/lib whereas NetBSD's suse*_base package installs it under /lib. You need to manually add

[hf@dreispitz] /<2>usr/lib > ll libpthread.so*
0 lrwxr-xr-x  1 root  wheel  25 Jul 22 17:35 libpthread.so -> ../../lib/libpthread.so.0
0 lrwxr-xr-x  1 root  wheel  25 Jul 22 17:35 libpthread.so.0 -> ../../lib/libpthread.so.0

LD_LIBRARY_PATH

As a Linux binary, Matlab makes heavy use of the LD_LIBRARY_PATH environment variable to access shared libraries. For a shell exit, it populates LD_LIBRARY_PATH like

>> !echo $LD_LIBRARY_PATH
/opt/matlabr14/sys/os/glnx86:/opt/matlabr14/bin/glnx86:/opt/matlabr14/extern/lib/glnx86:/opt/matlabr14/sys/java/jre/glnx86/jre1.5.0/lib/i386/native_threads:/opt/matlabr14/sys/java/jre/glnx86/jre1.5.0/lib/i386/client:/opt/matlabr14/sys/java/jre/glnx86/jre1.5.0/lib/i386
>>

NetBSD ELF binaries do not normally use LD_LIBRARY_PATH; instead, NetBSD compiles fixed library paths into shared libraries and applications. Nevertheless, non-setuid NetBSD binaries still look at LD_LIBRARY_PATH. The above path apparently confuses complex NetBSD binaries like NEdit, XEmacs, Mozilla, which try to pick up Linux libraries instead of the native ones. As a workaround, unset LD_LIBRARY_PATH before calling the application, either with per-application wrapper scripts, or a generic wrapper script like

#!/bin/sh
unset LD_LIBRARY_PATH
exec "$@"

OpenGL

Commands like bench that use OpenGL result in a kernel panic . Christos Zoulas has provided a kernel patch. It is against 2005-10-10 NetBSD-current source and applies with offsets to the netbsd-3 branch; apply from within sys/arch/i386/.

The patch avoids the panic; what functionality exactly is missing remains to be seen.

Workaround: The Matlab command opengl software enforces software rendering which is slower than using hardware support - but appears to work, unlike the latter. You still need the kernel patch, though, as any OpenGL operation will panic otherwise.

Issues left

Why Mathworks duplicate their set of install scripts everywhere and still keep a zipped copy around to drop on top of the installation is beyond me. The diffs above are clean and minimal - it's just that you have to have five instances of each.

--Hauke Fath

Posted late Sunday evening, November 20th, 2011

Contents

Things needed

A NetBSD/i386 installation. A Maple install cd (hybrid version for Windows/Mac OS X/Linux).

A linux emulation package. procfs turned on.

I use Maple 10 on NetBSD 3.1 with the suse10 package from pkgsrc.

Install Maple

Mount the CD.

From the user that will be using the maple install: run installMapleLinuxSU from the root directory on the CD.

Follow through the steps. Remember to choose an install folder you have write access to. I will use my home folder.

Upon finishing the install process you will be asked to activate Maple. I advise that you activate it now instead of trying to later.

Quit the installer.

Tell Maple your OS

Maple uses ?uname to detect the system type. Running Maple now will result in it telling you that your operating system is unsupported. We need to tell Maple that our system is linux so we can run it under emulation.

Using your favorite text editor open the file ~/maple##/bin/maple.system.type

This file is a script that runs at startup. Looking at the file we see that many different system types can be detected and launched. The one we wish to use is bin.IBM_INTEL_LINUX

There are two ways of doing this:

1: We can add a NetBSD section to the script. Just sneak it in under the Darwin entry:

    "Darwin")
       # the OSX case
       MAPLE_BIN="bin.APPLE_PPC_OSX"
       ;;

# OUR ADDED SECTION
    "NetBSD")
       MAPLE_BIN="bin.IBM_INTEL_LINUX"
       ;;
# END OF OUR SECTION

    *)

2: Add one line just above the bottom:

# OUR ADDED LINE
$MAPLE_BIN="bin.IBM_INTEL_LINUX"
# END LINE

echo $MAPLE_BIN
exit 0

Launch Maple

From the ~/maple##/bin directory launch either maple or xmaple.

Enjoy your NetBSD Maple math fun!

Posted late Sunday evening, November 20th, 2011

Contents

Introduction

The NetBSD installation is not big, but it can be made smaller. Here are a few tricks to make the base of user space, the C standard library libc, a bit smaller for dynamically linked systems. These trials were done on a cross compiled NetBSD current ARMv6 branch.

First the user space is built with default options. Then the cross compiler script $TOOLDIR/bin/nbmake-evbarm was used to clean and rebuild the libc with special options. The new libc binary was then copied to the target file system and a smoke test of booting the ARM development board was done. If /sbin/init and /bin/sh managed with the new libc, the test was a success while everything else was a failure.

The result is a crippled libc and /lib/libc.so.12.159 file size reduction from 1164 to 692 kilobytes. Run time memory usage is harder to predict since it depends on what parts of in memory libc are actually used by the processes, but at least text and data sections reported by the size utility give some idea.

Build options

Default -O2, file size 1164025

-O2 optimization is used by default and the libc file size after a ./build.sh -U -m evbarm build is:

-r--r--r-- 1 test test 1164025 2008-04-18 08:23 obj/destdir.evbarm/lib/libc.so.12.159

Sections reported by size utility:

   text    data     bss     dec     hex filename
 931608   25728   64332 1021668   f96e4 obj/libc.so.12.159

-O1, file size 1159845

If the libc is build with CFLAGS=-O1, the ELF shared object file size is:

-rw-r--r-- 1 test test 1159845 Apr 19 09:20 lib/libc.so.12.159

Sections reported by size utility:

   text    data     bss     dec     hex filename
 927436   25728   64332 1017496   f8698 obj/libc.so.12.159

Linker strip, file size 1065624

If the -O1 build is stripped and size optimized by the linker with LDFLAGS=-Wl,-O1\ -Wl,-s, the file size reduces to:

-rwxr-xr-x 1 test test 1065624 2008-04-19 13:28 obj/libc.so.12.159 

Sections reported by size utility:

   text    data     bss     dec     hex filename
 923316   25728   64332 1013376   f7680 obj/libc.so.12.159

-Os, file size 1094281

The gcc compiler can optimize binaries for size with CFLAGS=-Os:

-rwxr-xr-x 1 test test 1094281 2008-04-19 10:56 obj/libc.so.12.159

Sections reported by size utility:

   text    data     bss     dec     hex filename
 861864   25736   64332  951932   e867c obj/libc.so.12.159

Manual strip, file size 1004180

The binary can then be stripped manually:

$ $TOOLDIR/bin/arm--netbsdelf-strip -s obj/libc.so.12.159 
$ ls -l obj/libc.so.12.159
-rwxr-xr-x 1 test test 1004180 2008-04-19 11:02 obj/libc.so.12.159

Sections reported by size utility:

   text    data     bss     dec     hex filename
 861864   25736   64332  951932   e867c obj/libc.so.12.159

Linker strip, file size 1000060

The -Os compiled binary is smaller with linker based strip and optimization where LDFLAGS=-Wl,-O1\ -Wl,-s than with a manual strip:

-rwxr-xr-x 1 test test 1000060 2008-04-19 12:07 obj/libc.so.12.159

Sections reported by size utility:

   text    data     bss     dec     hex filename
 857744   25736   64332  947812   e7664 obj/libc.so.12.159

Feature removal

In addition to compiler flags CFLAGS=-Os LDFLAGS=-Wl,-O1\ -Wl,-s, special feature flags can be used to strip features out of libc and reduce its size. Some feature flags, as documented by BUILDING and share/mk/bsd.README, are supported for the whole user space.

SCCS version strings, file size 953136

SCCS version strings are normally embedded into object file, but they be removed by following changes in lib/libc/Makefile:

--- lib/libc/Makefile.inc       3 Jun 2007 17:36:08 -0000       1.3 
+++ lib/libc/Makefile.inc       19 Apr 2008 11:01:23 -0000
@@ -24,7 +24,8 @@
 .include <bsd.own.mk>

 WARNS=4
-CPPFLAGS+=     -D_LIBC -DLIBC_SCCS -DSYSLIBC_SCCS -D_REENTRANT
+#CPPFLAGS+=    -D_LIBC -DLIBC_SCCS -DSYSLIBC_SCCS -D_REENTRANT
+CPPFLAGS+=     -D_LIBC -D_REENTRANT

 .if (${USE_HESIOD} != "no")
 CPPFLAGS+=     -DHESIOD

The resulting libc binary finally goes below the one megabyte mark:

-rwxr-xr-x 1 test test 953136 2008-04-19 13:54 obj/libc.so.12.159

Sections reported by size utility:

   text    data     bss     dec     hex filename
 857744   25736   64332  947812   e7664 obj/libc.so.12.159

Hesiod name service, file size 942468

Hesiod), a DNS based database service, support can be removed from libc with USE_HESIOD=no MKHESIOD=no build variables and result is:

-rwxr-xr-x 1 test test 942468 2008-04-19 14:16 obj/libc.so.12.159

Sections reported by size utility:

   text    data     bss     dec     hex filename
 847625   25252   62180  935057   e4491 obj/libc.so.12.159

Yellow Pages (YP), file size 917368

Yellow Pages (YP) or Network Information Service (NIS) directory service support can be removed with USE_YP=no MKYP=no variables:

-rwxr-xr-x 1 test test 917368 2008-04-19 14:29 obj/libc.so.12.159

Sections reported by size utility:

   text    data     bss     dec     hex filename
 824328   24488   58944  907760   dd9f0 obj/libc.so.12.159

IPv6 support, file size 909272

IPv6 support can be removed with USE_INET6=no:

-rwxr-xr-x 1 test test 909272 2008-04-19 14:48 obj/libc.so.12.159

Sections reported by size utility:

   text    data     bss     dec     hex filename
 816537   24368   58944  899849   dbb09 obj/libc.so.12.159

Stack smashing protection (SSP), file size 894764

SSP buffer overflow protection from the GCC compiler can be disabled with USE_SSP=no and the libc binary size goes below 900k:

-rwxr-xr-x 1 test test 894764 2008-04-19 15:02 obj/libc.so.12.159

Sections reported by size utility:

   text    data     bss     dec     hex filename
 802029   24368   58944  885341   d825d obj/libc.so.12.159

Remote procedure call (RPC), file size 806036

RPC support can be disabled with MKRPC=no variable and a patch like this:

--- lib/libc/Makefile 9 Jan 2008 01:33:52 -0000 1.131.4.1 +++ lib/libc/Makefile 23 Apr 2008 13:04:42 -0000 @@ -74,7 +80,10 @@ .endif .include "${.CURDIR}/regex/Makefile.inc" .include "${.CURDIR}/resolv/Makefile.inc" +MKRPC?= yes +.if (${MKRPC} != "no") .include "${.CURDIR}/rpc/Makefile.inc" +.endif .include "${.CURDIR}/ssp/Makefile.inc" .include "${.CURDIR}/stdio/Makefile.inc" .include "${.CURDIR}/stdlib/Makefile.inc"

As a result the libc size goes down to 806 kilobytes:

-rw-r--r-- 1 test test 806036 2008-04-23 16:00 lib/libc.so.12.159

Sections reported by size utility:

   text    data     bss     dec     hex filename
 717964   22624   58500  799088   c3170 obj/libc.so.12.159

Execution profiling control, file size 801720

Profiling control can be removed from libc with MKGMON=no and a patch like:

--- lib/libc/Makefile 9 Jan 2008 01:33:52 -0000 1.131.4.1 +++ lib/libc/Makefile 24 Apr 2008 08:07:08 -0000 @@ -58,7 +58,10 @@ .include "${.CURDIR}/dlfcn/Makefile.inc" .include "${.CURDIR}/gdtoa/Makefile.inc" .include "${.CURDIR}/gen/Makefile.inc" +MKGMON?= yes +.if (${MKGMON} != "no") .include "${.CURDIR}/gmon/Makefile.inc" +.endif .include "${.CURDIR}/hash/Makefile.inc" .include "${.CURDIR}/iconv/Makefile.inc" .include "${.CURDIR}/inet/Makefile.inc"

And libc size goes down around 4 kilobytes:

-rwxr-xr-x 1 test test 801720 2008-04-24 10:57 obj/libc.so.12.159 

Sections reported by size utility:

   text    data     bss     dec     hex filename
 713959   22500   58436  794895   c210f obj/libc.so.12.159

Citrus I18N, file size 767560

Citrus I18N support can be removed with MKCITRUS=no, lib/libc/locale/setlocale.c patch and a makefile patch:

--- Makefile 9 Jan 2008 01:33:52 -0000 1.131.4.1 +++ Makefile 25 Apr 2008 07:51:20 -0000 @@ -53,12 +53,18 @@

 .include "${.CURDIR}/../../common/lib/libc/Makefile.inc"
 .include "${.CURDIR}/db/Makefile.inc"
+MKCITRUS?= yes
+.if (${MKCITRUS} != "no")
 .include "${.CURDIR}/citrus/Makefile.inc"
+.endif
 .include "${.CURDIR}/compat-43/Makefile.inc"
 .include "${.CURDIR}/dlfcn/Makefile.inc"
 .include "${.CURDIR}/gdtoa/Makefile.inc"

The libc binary is now below 800 kilobytes:

-rwxr-xr-x 1 test test 767560 2008-04-25 10:01 obj/libc.so.12.159 

Sections reported by size utility:

   text    data     bss     dec     hex filename
 685150   18696   56968  760814   b9bee obj/libc.so.12.159

MD2, RMD160, SHA1, SHA2, MD4 and MD5 cryptographic hash functions, file size 723780

All cryptographic hash functions can be removed from the libc with MKMD2=no MKRMD160=no MKSHA1=no MKSHA2=no MKMD=no and lib/libc/hash/Makefile.inf and lib/libc/Makefile patches:

--- lib/libc/hash/Makefile.inc 27 Oct 2006 18:29:21 -0000 1.11 +++ lib/libc/hash/Makefile.inc 30 Apr 2008 09:10:21 -0000 @@ -4,8 +4,20 @@ # hash functions .PATH: ${ARCHDIR}/hash ${.CURDIR}/hash

+MKMD2?= yes
+.if (${MKMD2} != "no")
 .include "${.CURDIR}/hash/md2/Makefile.inc"
+.endif
+MKRMD160?= yes
+.if (${MKRMD160} != "no")
 .include "${.CURDIR}/hash/rmd160/Makefile.inc"
+.endif
+MKSHA1?= yes
+.if (${MKSHA1} != "no")
 .include "${.CURDIR}/hash/sha1/Makefile.inc"
+.endif
+MKSHA2?= yes
+.if (${MKSHA2} != "no")
 .include "${.CURDIR}/hash/sha2/Makefile.inc"
+.endif


--- lib/libc/Makefile   9 Jan 2008 01:33:52 -0000       1.131.4.1
+++ lib/libc/Makefile   30 Apr 2008 09:11:25 -0000

 .include "${.CURDIR}/inet/Makefile.inc"
 .include "${.CURDIR}/isc/Makefile.inc"
 .include "${.CURDIR}/locale/Makefile.inc"
+MKMD?= yes
+.if (${MKMD} != "no")
 .include "${.CURDIR}/md/Makefile.inc"
+.endif
 .include "${.CURDIR}/misc/Makefile.inc"
 .include "${.CURDIR}/net/Makefile.inc"
 .include "${.CURDIR}/nameser/Makefile.inc"

libc size reduces to:

-rwxr-xr-x 1 test test 723780 2008-04-30 12:03 obj/libc.so.12.159

Sections reported by size utility:

   text    data     bss     dec     hex filename
 642476   18456   56968  717900   af44c obj/libc.so.12.159

Misc, Native Language Support (NLS), Regular Expression and Stack Smashing Protection, file size 691884

Indeed, misc object, NLS, regular expression (IEEE Std 1003.2-1992 (“POSIX.2”) regular expressions) and Stack Smashing Protection (SSP) support library are easily removed from the build process with MKMISC=no MKNLS=no MKREGEX=no MKSSP=no and an obvious patch to lib/libc/Makefile:

.include "${.CURDIR}/md/Makefile.inc" +.endif +MKMISC?= yes +.if (${MKMISC} != "no") .include "${.CURDIR}/misc/Makefile.inc" +.endif .include "${.CURDIR}/net/Makefile.inc" .include "${.CURDIR}/nameser/Makefile.inc" +MKNLS?= yes +.if (${MKNLS} != "no") .include "${.CURDIR}/nls/Makefile.inc" +.endif .if (${MACHINE_ARCH} != "alpha") && (${MACHINE_ARCH} != "sparc64") .include "${.CURDIR}/quad/Makefile.inc" .endif +MKREGEX?= yes +.if (${MKREGEX} != "no") .include "${.CURDIR}/regex/Makefile.inc" +.endif .include "${.CURDIR}/resolv/Makefile.inc" ... .include "${.CURDIR}/rpc/Makefile.inc" +.endif +MKSSP?= yes +.if (${MKSSP} != "no") .include "${.CURDIR}/ssp/Makefile.inc" +.endif .include "${.CURDIR}/stdio/Makefile.inc" .include "${.CURDIR}/stdlib/Makefile.inc" .include "${.CURDIR}/string/Makefile.inc"

Boot with sbin/init still works as does bin/sh, but user friendly programs like bin/ps and bin/ls now fail due to missing symbols:

# ls -l lib/libc.so.12.159
/lib/libc.so.12: Undefined PLT symbol "__fgets_chk" (symnum = 156)
# ls lib/libc.so.12.159
lib/libc.so.12.159
# ps aux
/lib/libc.so.12: Undefined PLT symbol "__strcat_chk" (symnum = 207)
# ps
ps: warning: /var/run/dev.db: /lib/libc.so.12: Undefined PLT symbol "_catopen" )

File size now:

-rwxr-xr-x 1 test test 691884 2008-04-30 14:18 obj/libc.so.12.159

Segment sizes reported by size utility:

   text    data     bss     dec     hex filename
 614238   17284   56928  688450   a8142 obj/libc.so.12.159

Ideas

While a few compiler and feature options were a tried out, a number of new ideas were found. Some of these were quickly tried out, but they resulted in the build or smoke/boot test failures.

Compiler options

Thumb instruction set, file size 585440

Compile to 16 bit THUMB instruction set instead of normal 32 bit

  • Whole user space needs to be build with -mthumb-interwork
  • CPUFLAGS build variable should contain -mthumb and -mthumb-interwork
  • If some files (like atomic_init_testset.c) need arm32 code due to embedded ARM assembly or other tool chain issues, the CPUFLAGS build variable can be overridden: Per file build options override
  • libc from matt-armv6 branch builds with -mthumb but fails to run with SIGILL: Thumb compilation discussion on port-arm

After a successfull compile with atomic_init_testset.o compiled with -mthumb-interwork only, the file size is:

-rwxr-xr-x 1 mira mira 585440 2008-05-12 11:20 obj/libc.so.12.159

size utility reports:

   text    data     bss     dec     hex filename
 507462   17616   56928  582006   8e176 obj/libc.so.12.159

Feature removals

Database support from lib/libc/db

  • sbin/init and possibly others depend on Berkeley DB support
  • a custom init and bin/sh propably work without it

Regular memory allocator instead of JEMALLOC

  • USE_JEMALLOC: (share/mk/bsd.README) If "no", disables building the "jemalloc" allocator designed for improved performance with threaded applications.

    • seems to require rebuild of user space, but even after that init dies with:

    warning: no /dev/console panic: init died (signal 0, exit 12) Stopped in pid 1.1 (init) at netbsd:cpu_Debugger+0x4: bx r14

Reduce resolver size in lib/libc/resolver

  • remove IPv6 support
  • remove debug features

Use libhack

  • Use size optimized distrib/utils/libhack with/instead of libc

Alternatives

Crunchgen

crunchgen can be used to create statically (at compile time) linked executables, which include only those object files, which are really used by the executable.

  • Use crunchgen or related build infra to build a dynamic library from only those object files that are actually required by the applications

References

Posted late Sunday evening, November 20th, 2011

Contents

Introduction

The NetBSD kernel is not big, but it can be made smaller. Generic size reduction steps are:

  • compile with size optimizations
  • removing debugging and logging features
  • remove non-essentian functionality

Here is an example which shows how these steps were applied to a NetBSD kernel for an ARM development board. As a result the kernel size got reduced from 2251 to 1124 kilobytes.

Since kernel executable size is not the only measure of kernel memory usage, some tools and their example output are also shown.

Example: OMAP 2420

The board receives the kernel via TFTP and the root filesystem is on NFS. Most important components on the board are the serial ports and the Ethernet adapter.

NetBSD current has a default kernel configuration for the board: sys/arch/evbarm/conf/TISDP2420. This file includes a number of other configuration files, but after compilation the resulting configuration is available as sys/arch/evbarm/compile/obj/TISDP2420/config_file.h.

The configuration file structure and most options are explained in options(4) manual page. In this example options were changed by adding a no options OPTION_NAME directive or an options OPTION_NAME directive to the end of the default configuration file. By adding these to end of the file, previous declarations from included options files were easily replaced without changing the included files.

Default TISDP2420, 2251 kb

Default kernel is build with -O2 optimization and stripped with objdump. Many debugging options are still enabled.

$ ls -l sys/arch/evbarm/compile/obj/TISDP2420/netbsd*
-rwxr-xr-x 1 test test  2614515 2008-06-23 11:32 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test  2305536 2008-06-23 11:32 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rwxr-xr-x 1 test test 13908834 2008-06-23 11:32 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.gdb
-rw-r--r-- 1 test test  3440335 2008-06-23 11:32 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


$ size sys/arch/evbarm/compile/obj/TISDP2420/netbsd
   text    data     bss     dec     hex filename
1902956  339456  214740 2457152  257e40 sys/arch/evbarm/compile/obj/TISDP2420/netbsd

Dmesg shows how much memory is available after boot, when the kernel executable been loaded to RAM and most of the RAM based data structures have been initialized:

# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58032 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82536000, paddr=0x80710DISPC: omap2_lcd_alloc_fb(): memory  
allocated at vaddr=0x8255c000, paddr=0x80740DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82582000, 
paddr=0x80760gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

Optimize for size with -Os, 2059 kb

Compiler can optimize for size when build variable DEFCOPTS is set to -Os instead of the default -O2.

-rwxr-xr-x 1 test test  2458323 2008-06-25 09:48 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test  2109012 2008-06-25 09:48 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rwxr-xr-x 1 test test 13390766 2008-06-25 09:48 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.gdb
-rw-r--r-- 1 test test  3446815 2008-06-25 09:48 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1764840  339540  214768 2319148  23632c sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg | egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58220 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82537000, paddr=0x806e0DISPC: omap2_lcd_alloc_fb(): memory
allocated at vaddr=0x8255d000, paddr=0x80710DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82583000, 
paddr=0x80730gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

Remove debugging symbols, without -g, 2059 kb

Build variable DEBUG contains the -g flag. Build without it. Results show that the kernel was already stripped of debug symbols by objdump, so this step is not usefull.

-rwxr-xr-x 1 test test 2502667 2008-06-25 09:58 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 2109012 2008-06-25 09:58 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 3238409 2008-06-25 09:58 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1764840  339540  214768 2319148  23632c sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58220 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82537000, paddr=0x806e0DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255d000, paddr=0x80710DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82583000, 
paddr=0x80730gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

DIAGNOSTIC, 1867 kb

Kernel build without DIAGNOSTIC support.

-rwxr-xr-x 1 test test 2304678 2008-06-25 10:05 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1912404 2008-06-25 10:05 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 3208060 2008-06-25 10:05 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1567336  339540  214768 2121644  205fac sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58408 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82538000, paddr=0x806b0DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255e000, paddr=0x806e0DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82584000, 
paddr=0x80700gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

VERBOSE_INIT_ARM, 1867 kb

Kernel build without verbose ARM specific boot messages. VERBOSE_INIT_ARM seems to depend on DDB support.

-rwxr-xr-x 1 test test 2304616 2008-06-25 10:22 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1912404 2008-06-25 10:22 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 3207741 2008-06-25 10:22 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1565468  339540  214768 2119776  205860 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58408 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82538000, paddr=0x806b0DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255e000, paddr=0x806e0DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82584000, 
paddr=0x80700gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

KTRACE, 1867 kb

Kernel without KTRACE system call tracing support. KTRACE seems to depend on DDB kernel debugger support, since boot hangs without it.

-rwxr-xr-x 1 test test 2303792 2008-06-25 10:39 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1912372 2008-06-25 10:39 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 3198575 2008-06-25 10:39 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1557776  339508  214704 2111988  2039f4 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58408 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82538000, paddr=0x806b0DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255e000, paddr=0x806e0DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82584000, 
paddr=0x80700gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

DDB, 1867 kb

Build kernel without DDB in kernel debugger.

-rwxr-xr-x 1 test test 2260235 2008-06-25 10:42 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1911944 2008-06-25 10:42 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 3076696 2008-06-25 10:42 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1512444  339080  209440 2060964  1f72a4 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


total memory = 62464 KB
avail memory = 58416 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82538000, paddr=0x806b0DISPC: omap2_lcd_alloc_fb(): memory  
allocated at vaddr=0x8255e000, paddr=0x806e0DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82584000, 
paddr=0x80700gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

COMPAT_30, 1867 kb

Kernel without NetBSD 3.0 compatibility.

-rwxr-xr-x 1 test test 2259424 2008-06-25 10:55 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1911944 2008-06-25 10:55 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 3031468 2008-06-25 10:55 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1509828  339080  209440 2058348  1f686c sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58416 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82538000, paddr=0x806b0DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255e000, paddr=0x806e0DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82584000, 
paddr=0x80700gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

ksyms, 1510 kb

Kernel without /dev/ksyms support.

-rwxr-xr-x 1 test test 1925248 2008-06-25 10:59 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1546376 2008-06-25 10:59 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 3023265 2008-06-25 10:59 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1503148   39048  209056 1751252  1ab8d4 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58764 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x8253a000, paddr=0x80660DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x82560000, paddr=0x80680DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82586000, 
paddr=0x806b0gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

PTRACE, 1510 kb

Kernel without process tracing support.

-rwxr-xr-x 1 test test 1924948 2008-06-25 11:04 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1546376 2008-06-25 11:04 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 3015392 2008-06-25 11:04 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1499312   39048  209056 1747416  1aa9d8 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58764 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x8253a000, paddr=0x80660DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x82560000, paddr=0x80680DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82586000, 
paddr=0x806b0gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

FFS, 1510 kb

Kernel without fast filesystem support.

-rwxr-xr-x 1 test test 1924948 2008-06-25 11:09 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1546376 2008-06-25 11:09 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 3015329 2008-06-25 11:09 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1499248   39048  209056 1747352  1aa998 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58764 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x8253a000, paddr=0x80660DISPC: omap2_lcd_alloc_fb(): memory  
allocated at vaddr=0x82560000, paddr=0x80680DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82586000, 
paddr=0x806b0gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

MSDOSFS, 1509 kb

Kernel without FAT filesystem support.

-rwxr-xr-x 1 test test 1887407 2008-06-25 11:21 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1546152 2008-06-25 11:21 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2955657 2008-06-25 11:21 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1463132   38824  208672 1710628  1a1a24 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58764 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82538000, paddr=0x80660DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255e000, paddr=0x80680DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82584000, 
paddr=0x806b0gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

PTYFS, 1509 kb

Kernel without pseudo TTY filesystem and pseudo-device pty.

-rwxr-xr-x 1 test test 1883462 2008-06-25 11:37 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1545916 2008-06-25 11:37 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2911049 2008-06-25 11:37 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1446512   38588  208536 1693636  19d7c4 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58764 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82536000, paddr=0x80660DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255c000, paddr=0x80680DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82582000, 
paddr=0x806b0gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

bpfilter, 1445 kb

Kernel without Berkeley packet filter pseudo-device support.

-rwxr-xr-x 1 test test 1848968 2008-06-25 11:46 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1480372 2008-06-25 11:46 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2896335 2008-06-25 11:46 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1435984   38580  208272 1682836  19ad94 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58828 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82536000, paddr=0x80650DISPC: omap2_lcd_alloc_fb(): memory  
allocated at vaddr=0x8255c000, paddr=0x80670DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82582000, 
paddr=0x806a0gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

INSECURE, 1445 kb

Kernel without integrity protection.

-rwxr-xr-x 1 test test 1848968 2008-06-25 11:54 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1480372 2008-06-25 11:54 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2896335 2008-06-25 11:54 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1435984   38580  208272 1682836  19ad94 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58828 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82536000, paddr=0x80650DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255c000, paddr=0x80670DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82582000, 
paddr=0x806a0gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

INET6, 1316 kb

Kernel without IPv6 support.

-rwxr-xr-x 1 test test 1666318 2008-06-25 11:57 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1348532 2008-06-25 11:57 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2666559 2008-06-25 11:57 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1271320   37812  199760 1508892  17061c sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58960 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82537000, paddr=0x80630DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255d000, paddr=0x80650DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82583000, 
paddr=0x80680gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

MALLOC_NOINLINE, 1316 kb

Kernel without inlined malloc functions.

-rwxr-xr-x 1 test test 1666318 2008-06-25 12:01 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1348532 2008-06-25 12:01 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2666559 2008-06-25 12:01 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1271320   37812  199760 1508892  17061c sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58960 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82537000, paddr=0x80630DISPC: omap2_lcd_alloc_fb(): memory  
allocated at vaddr=0x8255d000, paddr=0x80650DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82583000, 
paddr=0x80680gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

PIPE_SOCKETPAIR, 1316 kb

Kernel with smaller but slower pipe implementation.

-rwxr-xr-x 1 test test 1665096 2008-06-25 12:04 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1348468 2008-06-25 12:04 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2658886 2008-06-25 12:04 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1264788   37748  199760 1502296  16ec58 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58964 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82537000, paddr=0x80630DISPC: omap2_lcd_alloc_fb(): memory  
allocated at vaddr=0x8255d000, paddr=0x80650DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82583000,  
paddr=0x80670gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

VMSWAP, 1316 kb

Kernel without swap support. Requires a build patch, but may still crash the kernel when uvm statistics are called from top, ps etc. programs.

--- a/sys/uvm/uvm_pdpolicy_clock.c
+++ b/sys/uvm/uvm_pdpolicy_clock.c
@@ -262,12 +262,13 @@ uvmpdpol_balancequeue(int swap_shortage)
                /*
                 * if there's a shortage of swap slots, try to free it.
                 */
-
+#if defined(VMSWAP)
                if (swap_shortage > 0 && (p->pqflags & PQ_SWAPBACKED) != 0) {
                        if (uvmpd_trydropswap(p)) {
                                swap_shortage--;
                        }
                }
+#endif /* defined(VMSWAP) */

                /*
                 * if there's a shortage of inactive pages, deactivate.


-rwxr-xr-x 1 test test 1662968 2008-06-25 12:09 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1348392 2008-06-25 12:09 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2642042 2008-06-25 12:09 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1248392   37672  199248 1485312  16aa00 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58968 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82537000, paddr=0x80630DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255d000, paddr=0x80650DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82583000, 
paddr=0x80670gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

SYSVMSG, SYSVSEM, SYSVSHM, 1252 kb

Kernel without system V message queues, semaphores and shared memory.

-rwxr-xr-x 1 test test 1627140 2008-06-25 12:15 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1282748 2008-06-25 12:15 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2612942 2008-06-25 12:15 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1225204   37564  198904 1461672  164da8 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 59032 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82537000, paddr=0x80620DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255d000, paddr=0x80640DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82583000, 
paddr=0x80660gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

rnd, 1252 kb

Kernel without /dev/random pseudo-device.

-rwxr-xr-x 1 test test 1625313 2008-06-25 12:25 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1282684 2008-06-25 12:25 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2599464 2008-06-25 12:25 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1219556   37500  197944 1455000  163398 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 59032 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82537000, paddr=0x80620DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255d000, paddr=0x80640DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82583000, 
paddr=0x80660gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

md, 1252 kb

Kernel without support for memory devices like ramdisks.

-rwxr-xr-x 1 test test 1624021 2008-06-25 12:29 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1282588 2008-06-25 12:29 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2581393 2008-06-25 12:29 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1214024   37404  197752 1449180  161cdc sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 59032 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82537000, paddr=0x80620DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255d000, paddr=0x80640DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82583000,  
paddr=0x80660gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

maxusers 2, 1252 kb

Kernel with maxusers set to two instead of the default 32.

-rwxr-xr-x 1 test test 1624021 2008-06-25 12:34 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1282588 2008-06-25 12:34 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2581393 2008-06-25 12:34 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1214024   37404  197752 1449180  161cdc sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 59548 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x824b6000, paddr=0x805a0DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x824dc000, paddr=0x805c0DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82502000, 
paddr=0x805e0gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

MFS, 1124 kb

Kernel without memory filesystem.

-rwxr-xr-x 1 test test 1483269 2008-06-25 12:37 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1151292 2008-06-25 12:37 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2437216 2008-06-25 12:37 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1106936   37180  197688 1341804  14796c sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 59672 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x824b3000, paddr=0x80580DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x824d9000, paddr=0x805a0DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x824ff000, 
paddr=0x805d0gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

COREDUMP, 1124 kb

Kernel without core dump support.

-rwxr-xr-x 1 test test 1482841 2008-06-25 13:34 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1151292 2008-06-25 13:34 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2428587 2008-06-25 13:34 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1102708   37180  197688 1337576  1468e8 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 59672 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x824b3000, paddr=0x80580DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x824d9000, paddr=0x805a0DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x824ff000, 
paddr=0x805d0gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

Kernel memory consumption

The kernel executable size is not the only measure of kernel memory consumption. A number of userspace tools are able to show how the kernel and it's executable threads use memory while the system is running. Following chapters list some example tool output on the ARM OMAP 2420 board.

Using these tools is easy, but interpreting the numbers seems to require some indepth knowledge of the NetBSD kernel's memory management system uvm. More details on the meaning of these numbers as well as examples of found problems and countermeasures like kernel configurations and sysctl setting would be appreciated.

top

top shows overall memory usage in the system and an interesting process called system.

# top
load averages:  0.14,  0.03,  0.01             09:20:54
4 processes:   3 sleeping, 1 on CPU
CPU states:     % user,     % nice,     % system,     % interrupt,     % idle
Memory: 1916K Act, 208K Wired, 996K Exec, 428K File, 52M Free
Swap:
  PID USERNAME PRI NICE   SIZE   RES STATE      TIME   WCPU    CPU COMMAND
   51 root      43    0   976K  784K CPU        0:00 14.00%  0.68% top
    0 root     125    0     0K 1008K schedule   0:00  0.00%  0.00% [system]
   40 root      85    0   980K  956K wait       0:00  0.00%  0.00% sh
    1 root      85    0    32K  500K wait       0:00  0.00%  0.00% init


# dmesg|grep -i mem
total memory = 62464 KB
avail memory = 58032 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82536000, paddr=0x80710DISPC: omap2_lcd_alloc_fb(): memory  
allocated at vaddr=0x8255c000, paddr=0x80740DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82582000, 
paddr=0x80760gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0


# top -t
load averages:  0.00,  0.00,  0.00          up 0 days,  1:54   11:14:37
23 threads:    5 idle, 17 sleeping, 1 on CPU
CPU states:     % user,     % nice,     % system,     % interrupt,     % idle
Memory: 3844K Act, 208K Wired, 1308K Exec, 2044K File, 50M Free
Swap:
  PID   LID USERNAME PRI STATE      TIME   WCPU    CPU COMMAND      NAME
    0     6 root     223 IDLE       0:00  0.00%  0.00% [system]     softser/0
    0     3 root     222 IDLE       0:00  0.00%  0.00% [system]     softnet/0
    0     4 root     221 IDLE       0:00  0.00%  0.00% [system]     softbio/0
    0     5 root     220 IDLE       0:00  0.00%  0.00% [system]     softclk/0
    0     2 root       0 IDLE       0:00  0.00%  0.00% [system]     idle/0
   77     1 root      43 CPU        0:00  0.00%  0.00% top          -
    0     7 root     127 xcall      0:00  0.00%  0.00% [system]     xcall/0
    0    25 root     126 pgdaemon   0:00  0.00%  0.00% [system]     pgdaemon
    0     1 root     125 schedule   0:00  0.00%  0.00% [system]     swapper
    0    28 root     125 vmem_reh   0:00  0.00%  0.00% [system]     vmem_rehash
    0    27 root     125 aiodoned   0:00  0.00%  0.00% [system]     aiodoned
    0     9 root     125 cachegc    0:00  0.00%  0.00% [system]     cachegc
    0     8 root     125 vrele      0:00  0.00%  0.00% [system]     vrele
    0    26 root     124 syncer     0:00  0.00%  0.00% [system]     ioflush
    0    11 root      96 iicintr    0:00  0.00%  0.00% [system]     iic0

ps

ps can show the system light weight processes/threads too with -s flag and lname displays a more sesible name:

# ps -awxs -o uid,pid,lid,nlwp,pri,ni,vsz,rss,command,lname
UID PID LID NLWP PRI NI VSZ RSS COMMAND LNAME
  0   0  28   20 125  0   0 976 [system vmem_rehash
  0   0  27   20 125  0   0 976 [system aiodoned
  0   0  26   20 124  0   0 976 [system ioflush
  0   0  25   20 126  0   0 976 [system pgdaemon
  0   0  24   20  96  0   0 976 [system nfsio
  0   0  23   20  96  0   0 976 [system nfsio
  0   0  22   20  96  0   0 976 [system nfsio
  0   0  21   20  96  0   0 976 [system nfsio
  0   0  12   20  96  0   0 976 [system iic1
  0   0  11   20  96  0   0 976 [system iic0
  0   0  10   20  96  0   0 976 [system pmfevent
  0   0   9   20 125  0   0 976 [system cachegc
  0   0   8   20 125  0   0 976 [system vrele
  0   0   7   20 127  0   0 976 [system xcall/0
  0   0   6   20 223  0   0 976 [system softser/0
  0   0   5   20 220  0   0 976 [system softclk/0
  0   0   4   20 221  0   0 976 [system softbio/0
  0   0   3   20 222  0   0 976 [system softnet/0
  0   0   2   20   0  0   0 976 [system idle/0
  0   0   1   20 125  0   0 976 [system swapper
  0   1   1    1  85  0  32 500 init    -
  0  40   1    1  85  0 980 968 -sh     -
  0  62   1    1  42  0 976 676 ps -awx -
  0  63   1    1  85  0 980 784 less -r -

pmap

Displaying process 0 memory map with pmap shows the kernel's memory map.

# pmap 0
80000000  17680K read/write/exec     [ anon ]
81144000      4K read/write/exec     [ anon ]
81145000     20K read/write/exec     [ anon ]
8114A000   7328K read/write/exec     [ kmem_map ]
81872000   4096K read/write/exec     [ pager_map ]
81C72000    304K read/write/exec     [ anon ]
81CBE000      4K read/write/exec     [ anon ]
81CBF000    580K read/write/exec     [ anon ]
81D50000   4096K read/write/exec     [ exec_map ]
82150000   1200K read/write/exec     [ phys_map ]
8227C000   2096K read/write/exec     [ mb_map ]
82488000     20K read/write/exec     [ anon ]
8248D000      8K read/write/exec     [ uvm_aobj ]
8248F000    524K read/write/exec     [ anon ]
82512000     48K read/write/exec     [ uvm_aobj ]
8251E000     32K read/write/exec     [ anon ]
82526000      8K read/write/exec     [ uvm_aobj ]
82528000     48K read/write/exec     [ anon ]
82534000      8K read/write/exec     [ uvm_aobj ]
82536000    456K read/write/exec     [ anon ]
825A8000     16K read/write/exec     [ uvm_aobj ]
825AC000     12K read/write/exec     [ anon ]
825AF000     48K read/write/exec     [ uvm_aobj ]
825BB000      4K read/write/exec     [ anon ]
825BC000     16K read/write/exec     [ uvm_aobj ]
825C0000     56K read/write/exec     [ anon ]
825CE000   8192K read/write/exec     [ ubc_pager ]
82DCE000     52K read/write/exec     [ anon ]
82DDB000      8K read/write/exec     [ uvm_aobj ]
82DDD000     20K read/write/exec     [ anon ]
82DE2000      8K read/write/exec     [ uvm_aobj ]
82DE4000      4K read/write/exec     [ anon ]
82DE6000      4K read/write/exec     [ anon ]
82DE9000      8K read/write/exec     [ uvm_aobj ]
82DEB000      8K read/write/exec     [ anon ]
82DED000      8K read/write/exec     [ uvm_aobj ]
82DEF000     16K read/write/exec     [ anon ]
82DF7000     16K read/write/exec     [ uvm_aobj ]
82DFB000     12K read/write/exec     [ anon ]
82DFE000      8K read/write/exec     [ uvm_aobj ]
82E00000    400K read/write/exec     [ anon ]
82E68000     12K read/write/exec     [ anon ]
82E70000    576K read/write/exec     [ anon ]
 total    48064K

vmstat

vmstat can display a lot of NetBSD memory management details and statistics, especially if the kernel is compiled with KMEMSTAT options. Output of vmstat -C -m on the ARM board.

References

Posted late Sunday evening, November 20th, 2011

System Requirements

  • a DVD compatible reader
  • any multimedia player (for example, xine or mplayer)
  • an internet connection to fetch libdvdcss files required for descrambling

Introduction

CSS (Content Scrambling System) is a scheme mainly used by copyrights holders to protect commercial instructional dvds from unauthorized copying, by literally scrambling the DVDs' content through a (weak) encryption process.

By default, under NetBSD, when installing xine or mplayer to read commercial's DVDs, the libdvdcss (needed by libavcodec to decipher the DVD's content) is not built. This guide will show you how to make and install the libdvdcss package in order to use it.

Please note that under certain legislations, such method is forbidden (especially in the US thanks to DMCA). Use libdvdcss at your own risk!

Building and using libdvdcss

Libdvdcss is already present for installation in pkgsrc. As it is illegal in certain states, pkgsrc can not install it directly without some user interaction.

Before making it, we will first need to add some information to mk.conf(5), namely the master sites from which we should fetch the libdvdcss source code. There are many, including the videolan project, which hosts libdvdcss:

# echo "LIBDVDCSS_MASTER_SITES='the website you found'" >> /etc/mk.conf

Now, start building libdvdcss (should not take more than a couple minutes):

# cd /usr/pkgsrc/multimedia/libdvdcss
# make install clean

That's it. From now on, libavcodec (and consequently, your favorite multimedia reader) should automagically use libdvdcss to read your commercial's DVDs, through dlopen(3), when required.

Posted late Sunday evening, November 20th, 2011

Contents

Introduction

NetBSD lets you mount ISO images using the vnd(4) disk driver.
The vnd driver provides a disk-like interface to a file.

Mounting the Image

# vnconfig vnd0 v7x86-0.8a.iso                      
# mount -t cd9660 /dev/vnd0a /mnt
# cd /mnt
# ls
COPYRIGHT        README           boot.cat         v7x86-0.8a.tar   version
INSTALL          RELNOTES         boot.img         v7x86intro.pdf

Unmounting the Image

# umount /mnt
# vnconfig -u vnd0

Additional Information

Posted late Sunday evening, November 20th, 2011

Contents

Verify UFS support

To check whether your Linux kernel supports the UFS filesystem you may execute the following command:

$ cat /proc/filesystems nodev sysfs nodev rootfs nodev proc . . . ext3 nodev usbfs vfat ufs

The keyword nodev in the first column means that filesystem does not require a block device to be mounted, that's why it is also called virtual filesystem. The support is either compiled inside the kernel or as a module:

$ ls -l /lib/modules/2.6.21-ARCH/kernel/fs/ufs/ufs.ko -rw-r--r-- 1 root root 84828 2007-05-25 20:11 /lib/modules/2.6.21-ARCH/kernel/fs/ufs/ufs.ko

Mount

In order to find the device that corresponds to your FFS partition, run:

  1. sfdisk -l
    
Disk /dev/hda: 155061 cylinders, 16 heads, 63 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = cylinders of 516096 bytes, blocks of 1024 bytes, counting from 0
  Device Boot Start     End   #cyls    #blocks   Id  System
/dev/hda1   *      0+  34536-  34537-  17406396    7  HPFS/NTFS
               end: (c,h,s) expected (1023,15,63) found (1023,254,63)
/dev/hda2      34536+ 134767- 100231-  50516392+   f  W95 Ext'd (LBA)
               start: (c,h,s) expected (1023,15,63) found (1023,255,63)
               end: (c,h,s) expected (1023,15,63) found (1023,254,63)
/dev/hda3     134767+ 144935-  10169-   5124735   a5  FreeBSD
               start: (c,h,s) expected (1023,15,63) found (1023,255,63)
               end: (c,h,s) expected (1023,15,63) found (1023,254,63)
/dev/hda4     144935+ 155060   10126-   5103189   a9  NetBSD
               start: (c,h,s) expected (1023,15,63) found (1023,255,63)
               end: (c,h,s) expected (1023,15,63) found (1023,80,63)
/dev/hda5      34536+ 102366-  67830-  34186288+  83  Linux
               start: (c,h,s) expected (1023,15,63) found (0,1,1)
               end: (c,h,s) expected (1023,15,63) found (1023,254,63)
/dev/hda6     102366+ 104294    1929-    971901   82  Linux swap / Solaris
               start: (c,h,s) expected (1023,15,63) found (0,1,1)
               end: (c,h,s) expected (1023,15,63) found (120,254,63)
/dev/hda7     104295+ 134767-  30473-  15358108+  83  Linux
               start: (c,h,s) expected (1023,15,63) found (0,1,1)
               end: (c,h,s) expected (1023,15,63) found (1023,254,63)
/dev/hda8     134767+ 143910-   9143-   4608000
/dev/hda9     143910+ 144935-   1026-    516735
/dev/hda10    144935+ 154078-   9143    4608072
/dev/hda11    154078+ 155060     983-    495117
/dev/hda12         0+  34536-  34537-  17406396
/dev/hda13     34536+ 102366-  67830-  34186288+
/dev/hda14    102366+ 104294    1929-    971901
/dev/hda15    104295+ 144935-  40641-  20482843+

So for FreeBSD (FFSv2), we have /dev/hda3 which is equivalent to /dev/ad0s3

And for NetBSD (FFSv1), we have /dev/hda4 which is equivalent to /dev/wd0c

But these devices are whole BSD slices (BIOS partitions), not BSD partitions.

By examinating carefully sfdisk - l output, we find that: /dev/hda3 (134767+,144935-) includes /dev/hda8 (134767+,143910-) and /dev/hda9 (143910+,144935-) /dev/hda4 (144935+,155060) includes /dev/hda10 (144935+,154078-) and /dev/hda11 (154078+,155060)

And we may deduce that for FreeBSD: /dev/hda8 is equivalent to /dev/ad0s3a (FreeBSD root partition) /dev/hda9 is equivalent to /dev/ad0s3b (FreeBSD swap)

And for NetBSD: /dev/hda10 is equivalent to /dev/wd0a (NetBSD root partition) /dev/hda11 is equivalent to /dev/wd0b (NetBSD swap)

Thus FreeBSD root partition lies at /dev/hda8. First create a directory to mount FFS partition and then mount it:

# mkdir /mnt/freebsd
# mount -t ufs -o ro,ufstype=ufs2 /dev/hda8 /mnt/freebsd/

And NetBSD root partition lies at /dev/hda10. First create a directory to mount FFS partition and then mount it:

# mkdir /mnt/netbsd
# mount -t ufs -o ro,ufstype=44bsd /dev/hda10 /mnt/netbsd/

Let's browse it:

# ls /mnt/*bsd
/mnt/freebsd:
bin   cdrom   COPYRIGHT  dist     etc   lib      media  proc    root  sys  usr
boot  compat  dev        entropy  home  libexec  mnt    rescue  sbin  tmp  var
/mnt/netbsd:
altroot  etc                 gnome-screensave.core  mnt      root   var
bin      GENERIC             kern                   netbsd   sbin
boot     GENERIC-DIAGNOSTIC  lib                    onetbsd  stand
CUSTOM   GENERIC-LAPTOP      libdata                proc     tmp
dev      GENERIC-NOACPI      libexec                rescue   usr

Edit /etc/fstab

Add the following line to your /etc/fstab file:

/dev/hda8       /mnt/freebsd     ufs     ufstype=ufs2,ro        0       2
/dev/hda10      /mnt/netbsd      ufs     ufstype=44bsd,ro       0       2

Now you can mount the FFS partitions by typing:

# mount /mnt/freebsd
# mount /mnt/netbsd

and verify with:

$ mount
[...]
/dev/hda8 on /mnt/freebsd type ufs (ro,ufstype=ufs2)
/dev/hda10 on /mnt/netbsd type ufs (ro,ufstype=44bsd)
[...]

Write support

Write support is available given several conditions are satisfied: - ufs write support option compiled in Linux kernel (CONFIG_UFS_FS_WRITE=y): it is disabled by default. - FFSv1 filesystem (FFSv2 not yet supported)

Please note that as I do not really need write support on NetBSD partitions from GNU/Linux, I did not bother to rebuild my Linux kernel and hence have not tested this feature.

Remarks

  • If you forget the ro option, you will get the following message at dmesg:

    $ dmesg | grep ufs ufs was compiled with read-only support, can't be mounted as read-write

  • If you forget to set the ufstype option, you will get the following message at dmesg:

    $ dmesg | grep ufstype mount -t ufs -o ufstype=sun|sunx86|44bsd|ufs2|5xbsd|old|hp|nextstep|nextstep-cd|openstep ...

    WARNING<<< Wrong ufstype may corrupt your filesystem, default is ufstype=old

So, extra care should be taken.

People have reported crashes using FFS partitions access under GNU/Linux (even in read-only mode, that is very strange). I am half convinced that has been caused by accessing a whole BSD slice (BSD dedicated BIOS partition) instead of a BSD partition.

Posted late Sunday evening, November 20th, 2011

Why use PXELINUX

You may have a running Linux server supporting network boot via PXELINUX, because it allows a nice selection menu of bootable images, and want to add a NetBSD kernel to the menu.

In an experimental environment, this would allow to boot diagnostic tools like memtest, Linux kernels like GRML, a variety of Linux installers, and also to install NetBSD without removable devices.

However, if the networked machine is already dedicated to NetBSD, using pxeboot_ia32.bin directly to boot a NetBSD kernel for repairs or boot a diskless workstation with NetBSD would be better.

Using PXELINUX to chain boot

PXELINUX can not only boot Linux kernels, but also a boot sector. Thus, we can instruct PXELINUX to load the NetBSD netboot loader pxeboot_ia32.bin, which then can load a NetBSD kernel.

The only tricky thing is where to put which files. The two files needed are e.g. on a bootable CD-ROM.

  • copy pxeboot_ia32.bin to the same directory where pxelinux.0 resides. The DHCP config file may contain filename="/lts/i386/pxelinux.0", then copy pxeboot_ia32.bin to e.g. /tftpboot/lts/i386/
  • copy the kernel, e.g. netbsd-INSTALL.gz, to the directory denoted option root-path "/opt/ltsp/i386" in the DHCP config file, possibly renaming it to the default netbsd.
  • Edit the pxelinux.cfg/default (or any other file there) e.g. like this

# NetBSD
 label netbsd
 kernel pxeboot_ia32.bin

In case of a NetBSD installation, proceed as in Example installation, using probably FTP, HTTP or NFS as installation source.

Remarks:

  • The kernel(s) can and should stay gzipped
  • Unless an installation kernel is used, it will try to NFS-mount the root-path from the DHCP config file as root and assume it is writable, i.e. try to be a diskless workstation, see link below
  • For NetBSD 5, use NETBSD_INSTALL_FLOPPY.GZ as installation kernel

See also

Posted late Sunday evening, November 20th, 2011

Contents

Introduction

The standard kernel that you build for the NSLU2 requires that the Slug be modified to provide a serial port so that you can interact with the Slug during the boot process and to log in. This page will show you how to boot your Slug into NetBSD using only an ethernet connection. The strategy is to configure and build the kernel so that it automatically finds and mounts the root disk through DHCP and NFS without requiring that we type in the location of the root drive through the serial port. The root disk is a modification of the typical setup that allows an insecure telnet login to the NetBSD kernel running on the NSLU2. Once logged in, you can set up a username and password, enable ssh, and close out the insecure telnet connection.

The command line instructions that follow are for a Linux system using bash. They should be pretty much the same for another *nix system, except for the differences due to the shell.

Get the source code

To get the source code (current):

$ mkdir ~/net
$ export CVS_RSH="ssh"
$ export CVSROOT="anoncvs@anoncvs.NetBSD.org:/cvsroot"
$ cd ~/net
$ cvs checkout -A -P src

*/!\ The CVS address didnt work for me. I used: export CVSROOT="anoncvs@anoncvs.se.NetBSD.org:/cvsroot" /!*

This will create a directory ~/net/src with the source tree in it.

See the section below (Versions that are known to work) to get an older vers