Annotation of wikisrc/rumpkernel.mdwn, revision 1.1

1.1     ! pooka       1: **Contents**
        !             2: 
        !             3: [[!toc levels=2]]
        !             4: 
        !             5: # The Anykernel and Rump Kernels
        !             6: 
        !             7: ## About
        !             8: 
        !             9: A driver abstracts an underlying entity. For example, a TCP/IP driver
        !            10: abstracts the details required to perform networking, the Fast File
        !            11: System (FFS) driver abstracts how the file system blocks are laid out on
        !            12: the storage medium, and a PCI network card driver abstracts how to
        !            13: access device registers to send and receive packets. The kernel
        !            14: architecture controls how kernel drivers run with respect to other
        !            15: system components. Some examples of kernel architectures are the
        !            16: monolithic kernel, microkernel and exokernel. In contrast to the above
        !            17: architectures, NetBSD is an *anykernel*. This means that some kernel
        !            18: drivers can be run according to any kernel architecture instead of being
        !            19: limited to a single one.
        !            20: 
        !            21: <!--
        !            22:     TODO
        !            23:     ![](rumparch.png)
        !            24: -->
        !            25: 
        !            26: When a driver is not run as part of the monolithic kernel, e.g. when it
        !            27: is run as a microkernel style server, the driver is hosted in a *rump
        !            28: kernel*. A rump kernel is an ultralightweight virtualized kernel running
        !            29: on top of high-level hypercall interface. Instead of a low-level
        !            30: hypercall API typically seen with operating systems with operations such
        !            31: as "modify page table", the rump kernel hypercall API provides
        !            32: high-level operations such as "run this code in a thread".
        !            33: 
        !            34: Currently, three implementations of the rump kernel hypercall interface
        !            35: exist.
        !            36: 
        !            37: -   The POSIX implementation is included in the NetBSD tree and makes
        !            38:     rump kernels to run as userspace processes on most operating systems
        !            39:     such as NetBSD, Linux and Solaris.
        !            40: -   The Xen implementation allows running rump kernels directly as Xen
        !            41:     DomU's without an intermediate operating system.
        !            42: -   The Linux kernel hypervisor allows rump kernels to run inside the
        !            43:     Linux kernel.
        !            44: 
        !            45: Rump kernels are radically different from OS virtualization technologies
        !            46: such as KVM, containers and usermode operating systems. A rump kernel
        !            47: does not support hosting application processes because a rump kernel is
        !            48: aimed at virtualizing kernel drivers and application virtualization
        !            49: would be pure overhead. Instead, existing entities such as processes
        !            50: from a hosting OS are used as clients for the rump kernel ("application"
        !            51: in the figure).
        !            52: 
        !            53: As a result of the above design choices, rump kernels are extremely
        !            54: lightweight. The bootstrap time for rump kernels on POSIX hosts is
        !            55: measured in milliseconds and memory footprint in 100kB's. This means
        !            56: that a rump kernel can be bootstrapped for example as part of a command
        !            57: line tool for virtually no cost or user impact. Rump kernels also
        !            58: mandate very little from the hypercall implementation meaning that rump
        !            59: kernels, and by extension NetBSD kernel drivers, can be hosted in
        !            60: virtually any environment.
        !            61: 
        !            62: Use cases for rump kernels include:
        !            63: 
        !            64: -   **Code reuse**: kernel drivers can be reused without having to run a
        !            65:     whole OS. For example, a full-featured TCP/IP stack (IPv6, IPSec,
        !            66:     etc.) can be included in an embedded appliance without having to
        !            67:     write the stack from scratch or waste resources on running an entire
        !            68:     OS.
        !            69: -   **Kernel driver virtualization**: every rump kernel has its own
        !            70:     state. Furthermore, the functionality offered by multiple rump
        !            71:     kernels running on the same host does not need to be equal. For
        !            72:     example, multiple different networking stacks optimized for
        !            73:     different purposes are possible.
        !            74: -   **Security**: when hosted on a POSIX system, a rump kernel runs in
        !            75:     its own instance of a userspace process. For example, it is widely
        !            76:     published that file system drivers are vulnerable to untrusted file
        !            77:     system images. Unlike on other general purpose operating systems, on
        !            78:     NetBSD it is possible to mount untrusted file systems, such as those
        !            79:     on a USB stick, in an isolated server with the kernel file system
        !            80:     driver. This isolates attacks and prevents kernel compromises while
        !            81:     not requiring to maintain separate userspace implementations of the
        !            82:     file system drivers or use other resource-intensive approaches such
        !            83:     as virtual machines.
        !            84: -   **Easy prototyping and development**: kernel code can be developed
        !            85:     as a normal userspace application. Once development is finished, the
        !            86:     code can simply be complied into the kernel. This is a much more
        !            87:     convenient and straightforward approach to kernel development than
        !            88:     the use of virtual machines.
        !            89: -   **Safe testing**: kernel code can be tested in userspace on any host
        !            90:     without risk of the test host being affected. Again, virtual
        !            91:     machines are not required.
        !            92: 
        !            93: ## Further Reading
        !            94: 
        !            95: ### Dissertation
        !            96: 
        !            97: The following is the definitive guide to the anykernel and rump kernels
        !            98: and supercedes all earlier publications and terminology on the subject.
        !            99: 
        !           100: -   [Flexible Operating System Internals: The Design and Implementation
        !           101:     of the Anykernel and Rump
        !           102:     Kernels](http://lib.tkk.fi/Diss/2012/isbn9789526049175/isbn9789526049175.pdf)
        !           103: 
        !           104: ### Software using rump kernels
        !           105: 
        !           106: These links are interesting for people who want to use rump kernels in
        !           107: addition to reading about them.
        !           108: 
        !           109: -   [Scripts for building rump kernels for POSIX
        !           110:     systems](https://github.com/anttikantee/buildrump.sh)
        !           111: -   [Rump kernel hypercall implementation for Xen; rump kernels as Xen
        !           112:     DomU's](https://github.com/anttikantee/rumpuser-xen)
        !           113: -   [fs-utils: File system image access
        !           114:     utilities](https://github.com/stacktic/fs-utils)
        !           115: -   Fast userspace packet processing: TCP/IP stack for use with
        !           116:     [DPDK](https://github.com/anttikantee/dpdk-rumptcpip) or
        !           117:     [netmap](https://github.com/anttikantee/netmap-rumptcpip)
        !           118: 
        !           119: ### Articles, Tutorials & Howtos
        !           120: 
        !           121: -   [Running rump kernels and applications on Xen without a full
        !           122:     OS](http://blog.NetBSD.org/tnf/entry/running_applications_on_the_xen)
        !           123: -   [PCI device driver support in rump kernels on
        !           124:     Xen](http://blog.NetBSD.org/tnf/entry/pci_driver_support_for_rump)
        !           125: -   [Experiment with a rump kernel hypervisor for the Linux
        !           126:     kernel](http://blog.NetBSD.org/tnf/entry/a_rump_kernel_hypervisor_for)
        !           127:     (allows rump kernels to run *in* the Linux kernel)
        !           128: -   [Experiment on compiling rump kernels to javascript and running them
        !           129:     in the
        !           130:     browser](http://blog.NetBSD.org/tnf/entry/kernel_drivers_compiled_to_javascript)
        !           131: -   [Kernel Servers using
        !           132:     Rump](http://www.NetBSD.org/docs/rump/sysproxy.html)
        !           133: -   [Tutorial On Rump Kernel Servers and
        !           134:     Clients](http://www.NetBSD.org/docs/rump/sptut.html)
        !           135: -   [Revolutionizing Kernel Development: Testing With
        !           136:     Rump](http://blog.NetBSD.org/tnf/entry/revolutionizing_kernel_development_testing_with)
        !           137: 
        !           138: ### Conference publications and talks
        !           139: 
        !           140: -   "The Anykernel and Rump Kernels" gives a general overview and
        !           141:     demonstrates rump kernels on Windows and in Firefox. The
        !           142:     [video](http://video.fosdem.org/2013/maintracks/K.1.105/The_Anykernel_and_Rump_Kernels.webm),
        !           143:     [slides](https://fosdem.org/2013/schedule/event/operating_systems_anykernel/attachments/slides/244/export/events/attachments/operating_systems_anykernel/slides/244/fosdem2013_rumpkern.pdf)
        !           144:     and an
        !           145:     [interview](https://archive.fosdem.org/2013/interviews/2013-antii-kantee/)
        !           146:     are available. Presented at FOSDEM 2013 (Operating Systems track).
        !           147: -   "Rump Device Drivers: Shine On You Kernel Diamond" describes device
        !           148:     driver and USB. The
        !           149:     [paper](http://ftp.NetBSD.org/pub/NetBSD/misc/pooka/tmp/rumpdev.pdf)
        !           150:     and [video presentation](http://www.youtube.com/watch?v=3AJNxa33pzk)
        !           151:     are available. Presented at AsiaBSDCon 2010.
        !           152: -   "Fs-utils: File Systems Access Tools for Userland" describes
        !           153:     fs-utils, an mtools-like utility kit which uses rump kernel file
        !           154:     systems as a backend. The
        !           155:     [paper](http://www.ukuug.org/events/eurobsdcon2009/papers/ebc09_fs-utils.pdf)
        !           156:     is available. Presented at EuroBSDCon 2009.
        !           157: -   "Rump File Systems: Kernel Code Reborn" describes kernel file system
        !           158:     code and its uses in userspace. The
        !           159:     [paper](http://usenix.org/events/usenix09/tech/full_papers/kantee/kantee.pdf)
        !           160:     and
        !           161:     [slides](http://usenix.org/events/usenix09/tech/slides/kantee.pdf)
        !           162:     are available. Presented at the 2009 USENIX Annual Technical
        !           163:     Conference.
        !           164: -   "Kernel Development in Userspace - The Rump Approach" describes
        !           165:     doing kernel development with rump kernels. The
        !           166:     [paper](http://www.bsdcan.org/2009/schedule/attachments/104_rumpdevel.pdf)
        !           167:     and
        !           168:     [slides](http://www.bsdcan.org/2009/schedule/attachments/105_bsdcan09-kantee.pdf)
        !           169:     are available. Presented at BSDCan 2009.
        !           170: -   "Environmental Independence: BSD Kernel TCP/IP in Userspace"
        !           171:     describes networking in rump kernels. The
        !           172:     [paper](http://2009.asiabsdcon.org/papers/abc2009-P5A-paper.pdf) and
        !           173:     [video presentation](http://www.youtube.com/watch?v=RxFctq8A0WI) are
        !           174:     available. Presented at AsiaBSDCon 2009.
        !           175: 
        !           176: ### Manual pages
        !           177: 
        !           178: The manpages provide the usual type of information. Start from
        !           179: [rump.3](http://man.NetBSD.org/cgi-bin/man-cgi?rump++NetBSD-current) and
        !           180: follow the cross-references in "SEE ALSO".
        !           181: 
        !           182: ## Discuss
        !           183: 
        !           184: Any topic related to rump kernels can be discussed on the
        !           185: [rumpkernel-users mailing
        !           186: list](https://lists.sourceforge.net/lists/listinfo/rumpkernel-users).
        !           187: Alternatively, you can use a NetBSD mailing which is related to a
        !           188: specific subtopic.
        !           189: 
        !           190: The IRC channel for rump kernels is **\#rumpkernel** on
        !           191: **irc.freenode.net**.
        !           192: 
        !           193: ## Availability
        !           194: 
        !           195: The anykernel and rump kernels were first introduced as a prototype in
        !           196: NetBSD 5.0. A stable version with numerous new features and improvements
        !           197: was shipped with NetBSD 6.0.
        !           198: 
        !           199: ## Source Code
        !           200: 
        !           201: All of the source code is available from the NetBSD source tree and can
        !           202: be obtained with the usual methods.
        !           203: 
        !           204: You can also [browse](http://cvsweb.NetBSD.org/bsdweb.cgi/src/) the
        !           205: source code history online. Code is found from all areas of the source
        !           206: tree. Some examples of where to look include
        !           207: [src/lib](http://cvsweb.NetBSD.org/bsdweb.cgi/src/lib/),
        !           208: [src/usr.bin](http://cvsweb.NetBSD.org/bsdweb.cgi/src/usr.bin/) and
        !           209: [src/sys/rump](http://cvsweb.NetBSD.org/bsdweb.cgi/src/sys/rump/).

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb