This page is a blog mirror of sorts. It pulls in articles from blog's feed and publishes them here (with a feed, too).
FOSDEM 2014 will take place on 1–2 February, 2014, in Brussels, Belgium. Just like in the last years, there will be both a BSD booth and a developer's room (on Saturday).
The topics of the devroom include all BSD operating systems. Every talk is welcome, from internal hacker discussion to real-world examples and presentations about new and shiny features. The default duration for talks will be 45 minutes including discussion. Feel free to ask if you want to have a longer or a shorter slot.
If you already submitted a talk last time, please note that the procedure is slightly different.
To submit your proposal, visit
and follow the instructions to create an account and an “event”. Please select “BSD devroom” as the track. (Click on “Show all” in the top right corner to display the full form.)
Please include the following information in your submission:
- The title and subtitle of your talk (please be descriptive, as titles will be listed with ~500 from other projects)
- A short abstract of one paragraph
- A longer description if you wish to do so
- Links to related websites/blogs etc.
The deadline for submissions is December 20, 2013. The talk committee, consisting of Daniel Seuffert, Marius Nünnerich and Benny Siegert, will consider the proposals. If yours has been accepted, you will be informed by e-mail before the end of the year.
The following report is by Manuel Wiesinger:
First of all, I like to thank the NetBSD Foundation for enabling me to successfully complete this Google Summer of Code. It has been a very valuable experience for me.
My project is a defragmentation tool for FFS. I want to point out at the beginning that it is not ready for use yet.
What has been done:
Fragment analysis + reordering. When a file is smaller or equal than the file system's fragment size, it is stored as a fragment. One can think of a fragment as a block. It can happen that there are many small files that occupy a fragment. When the file systems changes over time it can happen that there are many blocks containing fewer fragments than they can hold. The optimization my tool does is to pack all these fragments into fewer blocks. This way the system may get a little more free space.
Directory optimization. When a directory gets deleted, the space for that directory and its name are appended to the previous directory. This can be imagined like a linked list. My tool reads that list and writes all entries sequentially.
Non-contiguous files analysis + reordering strategy. This is what most other operating systems call defragmentation - a reordering of blocks, so that blocks belonging to the same file or directory can be read sequentially.
What did not work as expected
Testing: I thought that it is the most productive and stable to work with unit tests. Strictly test driven development. It was not really effective to play around with rump/atf. Although I always started a new implementation step by generating a file system in a state where it can be optimized. So I wrote the scripts, took a look if they did what I intended (of course, they did not always).
I'm a bit disappointed about the amount of code. But as I said before, the hardest part is to figure out how things work. The amount it does is relatively much, I expected more lines of code to be needed to get where I am now.
Before applying for this project, I took a close look at UFS. But it was not close enough. There were many surprises. E.g. I had no idea that there are gaps in files on purpose, to exploit the rotation of hard disks.
Time management, everything took longer than I expected. Mostly because it was really hard to figure out how things work. Lacking documentation is a huge problem too.
Things I learned
A huge lesson learned in software engineering. It is always different than expected, if you do not have a lot of experience.
I feel more confident to read and patch kernel code. All my previous experiences were not so in-depth. (e.g., I worked with pintos). The (mental) barrier of kernel/system programming is gone. For example I see a chance now to take a look on ACPI, and see if I can write a patch to get suspend working on my notebook.
I got more contact with the NetBSD community, and got a nice overview how things work. The BSD community here is very mixed. There are not many NetBSD people.
CVS is better than most of my friends say.
I learned about pkgsrc, UVM, and other smaller things about NetBSD too, but that's not worth mentioning in detail.
How I intend to continue:
After a sanity break, of the whole project, there are several possibilities.
In the next days I will speak to a supervisor at my university, if I can continue the project as a project thesis (I still need to do one). It may even include online defragmentation, based on snapshots. That's my preferred option.
I definitely want to finish this project, since I spent so much time and effort. It would be a shame otherwise.
What there is to do technically
Once the defragmentation works, given enough space to move a file. I want to find a way where you can defragment it even when there is too little space. This can be achieved by simply moving blocks piece by piece, and use the files' space as 'free space'.
Online defragmentation. I already skimmed hot snapshots work. It should be possible.
Improve the tests.
It should be easy to get this compiling on older releases. Currently it compiles only on -current.
Eventually it's worth it to port the tests to atf/rump on the long run.
I will continue to work, and definitely continue to use and play with NetBSD! :)
It's a stupid thing that a defrag tool is worth little (nothing?) on SSDs. But since NetBSD is designed to run on (almost) any hardware, this does not bug me a lot.
Thank you NetBSD! Although it was a lot of hard work. It was a lot of fun!
The NetBSD Project is pleased to announce NetBSD 6.1.2, the second security/bugfix update of the NetBSD 6.1 release branch, and NetBSD 6.0.3, the third security/bugfix update of the NetBSD 6.0 release branch. They represent a selected subset of fixes deemed important for security or stability reasons, and if you are running a prior release of NetBSD 6.x3, you are recommended to update.http://www.NetBSD.org/mirrors/.
The NetBSD Project is pleased to announce NetBSD 5.2.1, the first security/bugfix update of the NetBSD 5.2 release branch, and NetBSD 5.1.3, the third security/bugfix update of the NetBSD 5.1 release branch. They represent a selected subset of fixes deemed important for security or stability reasons, and if you are running a release of NetBSD prior to 5.1.3, you are recommended to update to a supported NetBSD 5.x or NetBSD 6.x version.http://www.NetBSD.org/mirrors/.
Updates to NetBSD 6.x will be coming in the next few days.
All open issues (wrong colours on scaled images, failing https, ...) have been resolved.
Here is a new screeenshot:
Yesterday I wrote a serious, user-oriented post about running applications directly on the Xen hypervisor. Today I compensate for the seriousness by writing a why-so-serious, happy-buddha type kernel hacker post. This post is about using NetBSD kernel PCI drivers in rump kernels on Xen, with device access courtesy of Xen PCI passthrough.
I do not like hardware. The best thing about hardware is that it gives software developers the perfect excuse to blame something else for their problems. The second best thing about hardware is that most of the time you can fix problems with physical violence. The third best thing about hardware is that it enables running software. Apart from that, the characteristics of hardware are undesirable: you have to possess the hardware, it does not virtualize nicely, it is a black box subject to the whims of whoever documented it, etc. Since rump kernels are targeting reuse and virtualization of kernel drivers in an environment-agnostic fashion, needless to say, there is a long, non-easy truce between hardware drivers and rump kernels.
Many years ago I did work which enabled USB drivers to run in rump kernels. The approach was to use the ugen device node to access the physical device from userspace. In other words, the layers which transported the USB protocol to and from the device remained in the host kernel, while the interpretation of the contents was moved to userspace; a USB host controller driver was written to act as the middleman between these two. While the approach did allow to run USB drivers such as umass and ucom, and it did give me much-needed exercise in the form of having to plug and unplug USB devices while testing, the whole effort was not entirely successful. The lack of success was due to too much of the driver stack, namely the USB host controller and ugen drivers, residing outside of the rump kernel. The first effect was that due to my in-userspace development exercising in-kernel code (via the ugen device) in creative ways, I experienced way too many development host kernel panics. Some of the panics could be fixed, while others were more in the department "well I have no idea why it decided to crash now or how to repeat the problem". The second effect was being able to use USB drivers in rump kernels only on NetBSD hosts, again foiling environment-agnostism (is that even a word?). The positive side-effect of the effort was adding ioconf and pseudo-root support to config(1), thereby allowing modular driver device tree specifications to be written in the autoconf DSL instead of having to be open-coded into the driver in C.
In the years that followed, the question of rump kernels supporting real device drivers which did not half-hide behind the host's skirt became a veritable FAQ. My answer remained the same: "I don't think it's difficult at all, but there's no way I'm going to do it since I hate hardware". While it was possible to run specially crafted drivers in conjuction with rump kernels, e.g. DPDK drivers for PCI NICs, using any NetBSD driver and supported device was not possible. However, after bolting rump kernels to run on top of Xen, the opportunity to investigate Xen's PCI passthrough capabilities presented itself, and I did end up with support for PCI drivers. Conclusion: I cannot be trusted to not do something.
The path to making PCI devices work consisted of taking n small steps. The trick was staying on the path instead of heading toward the light. If you do the "imagine how it could work and then make it work like that" development like I do, you'll no doubt agree that the steps presented below are rather obvious. (The relevant NetBSD man pages are linked in parenthesis. Also note that the implementations of these interfaces are MD in NetBSD, making for a clean cut into the NetBSD kernel architecture)
- passing PCI config space read and writes to the Xen hypervisor (pci(9))
- mapping the device memory space into the Xen guest and providing access methods (bus_space(9))
- mapping Xen events channels to driver interrupt handlers (pci_intr(9))
- allocating DMA-safe memory and translating memory addresses to and from machine addresses, which are even more physical than physical addresses (bus_dma(9))
On the Xen side of things, the hypercalls for all of these tasks are more or less one-liner calls into the Xen Mini-OS (which, if you read my previous post, is the layer which takes care of the lowest level details of running rump kernels on top of Xen).
And there we have it, NetBSD PCI drivers running on a rump kernel on Xen. The two PCI NIC drivers I tested both even pass the allencompassing ping test (and the can-configure-networking-using-dhcp test too). There's nothing like a dmesg to brighten the day.
Closing thoughts: virtual machine emulators are great, but you lose the ability to kick the hardware.
There are a number of motivations for running applications directly on top of the Xen hypervisor without resorting to a full general-purpose OS. For example, one might want to maximally isolate applications with minimal overhead. Leaving the OS out of the picture decreases overhead, since for example the inter-application protection offered normally by virtual memory is already handled by the Xen hypervisor. However, at the same time problems arise: applications expect and use many services normally provided by the OS, for example files, sockets, event notification and so forth. We were able to set up a production quality environment for running applications as Xen DomU's in a few weeks by reusing hundreds of thousands of lines of unmodified driver and infrastructure code from NetBSD. While the amount of driver code may sound like a lot for running single applications, keep in mind that it involves for example file systems, the TCP/IP stack, stdio, system calls and so forth -- the innocent-looking open() alone accepts over 20 flags which must be properly handled. The remainder of this post looks at the effort in more detail.
Shortly after I had published the above, I was contacted by Justin Cormack with whom I had collaborated earlier on his ljsyscall project, which provides system call interfaces to Lua programs. He wanted to run the LuaJIT interpreter and his ljsyscall implementation directly on top of Xen. However, in addition to system calls which were already handled by the rump kernel, the LuaJIT interpreter uses interfaces from libc, so we added the NetBSD libc to the mix. While currently the libc sources are hosted on github, we plan to integrate the changes into the upstream NetBSD sources as soon as things settle (see the repo for instructions on how to produce a diff and verify that the changes really are tiny). The same repository hosts the math library libm, but it is there just for convenience reasons so that these early builds deliver everything from a single checkout. It has been verified that you can alternatively use libm from a standard NetBSD binary distribution, as supposedly you could any other user-level library. The resulting architecture is depicted below.
The API and ABI we provide are the same as of a regular NetBSD installation. Apart from some limitations, such as the absense of fork() -- would it duplicate the DomU? -- objects compiled for a regular NetBSD installation can be linked into the DomU image and booted to run directly as standalone applications on top of Xen. As proofs of concept, I created a demo where a Xen DomU configures TCP/IP networking, mounts a file system image, and runs a httpd daemon to serve the contents, and Justin's demo runs the LuaJIT interpreter and executes the self-test suite for ljsyscall. Though there is solid support for running applications, not all of the work is done. Especially the build framework needs to be more flexible, and everyone who has a use case for this technology is welcome to test out their application and contribute ideas and code for improving the framework.
In conclusion, we have shown that it is straightforward to reuse both kernel and library code from an existing real-world operating system in creating an application environment which can run on top of a bare-metal type cloud platform. By being able to use in the order of 99.9% of the code -- that's 1,000 lines written per 1,000,000 used unmodified -- from an existing, real-world proven source, the task was quick to pull off, the result is robust, and the offered application interfaces are complete. Some might call our work a "yet another $fookernel", but we call it a working result and challenge everyone to evaluate it for themselves.
The NetBSD Project is pleased to announce NetBSD 6.1.1, the first security/bugfix update of the NetBSD 6.1 release branch. It represents a selected subset of fixes deemed important for security or stability reasons.
For more details, please see the 6.1.1 release notes.
Complete source and binaries for NetBSD 6.1.1 are available for download at many sites around the world. A list of download sites providing FTP, AnonCVS, SUP, and other services may be found at http://www.NetBSD.org/mirrors/.
SX is more or less a vector processor built into a memory controller. The 'or less' part comes from the fact that it can't actually read instructions from memory - the CPU has to feed them one by one. This isn't quite as bad as it sounds - SX has plenty of registers ( 128 - eight of them have special functions, the rest is free for all ) and every instruction takes a count to operate on several subsequent registers or memory locations ( ALU ops can use up to 16, memory accesses up to 32 ). SX supports some parallelism too - the ALUs can do up to two 16bit multiplications and two other arithmetic or logical ops per clock cycle ( 32bit multiplications use both ALUs ). The thing runs at 50MHz - not a whole lot by today's standards but it can be a significant help to any CPUs you can put into these machines.
The kernel part hs been committed a while ago ( see cgfourteen at obio ), always runs in 8bit colour with everything ( scrolling, character drawing etc. ) done by SX. Substantially faster than software only operation.
The Xorg driver ( xf86-video-suncg14 that is, it needs a recent cgfourteen kernel driver to map SX registers ) uses EXA, by default it enables basic acceleration ( Solid and Copy, hardware cursor support has been added years ago ), Xrender acceleration is incomplete and can be enabled by adding Options "Xrender" "true" to the cg14's device section in xorg.conf. For now it implements what's needed for anti-aliased font drawing, assembling alpha maps in video memory and operations KDE3 uses for drawing icons, buttons and so on. As it is now KDE3 looks mostly right - there are a few rendering errors ( for example the corners of the white frame in Konqueror's startup page, also some text is drawn with red and blue channels switched ) but nothing that would interfere with functionality.
Speed isn't great ( well, the hardware is 20 years old ) but the improvement over unaccelerated X is more than noticeable. Some benchmark numbers, all taken on an SS20 with a single 125MHz HyperSPARC:
- x11perf -comppixwin went from 26.3 to 93.7
- x11perf -compwinwin went from 8.8 to 93.5
- x11perf -aaftext went from about 4000 ( 5000 with shadow fb but that's technically cheating since not everything is drawn into video memory ) to 16000
There is still room for improvement - for now all Xrender operations work on one pixel at a time, SX has enough registers to do at least four in most cases. Also, functionality is added as needed whenever I run into it, so other things are likely broken ( I know gtk2 is, but that's easy to fix ) which is why Xrender acceleration is disabled by default.
Google Code-In (GCi) is a project like Google Summer Of Code (GSoC), but for younger students. While GSoC is aimed at university students, i.e. for people usually of age 19 or older, GCi wants to recruit pupils for Open Source projects.
When applying for participation, every project had to create a large number of potentially small tasks for students. A task was meant to be two hours of work of an experienced developer, and feasible to be done by a person 13 to 18 years old. Google selected ten participating organisations (this time, NetBSD was the only BSD participating) to insert their tasks into Google Melange (the platform which is used for managing GCi and GSoC).
Then, the students registered at Google Melange, chose a project they wanted to work on, and claimed tasks to do. There were many chats in the NetBSD code channel for students coming in and asking questions about their tasks.
After GCi was over, every organisation had to choose their two favourite students who did the best work. For NetBSD, the choice was difficult, as there were more than two students doing great work, but in the end we chose Mingzhe Wang and Matthew Bauer. These two "grand price winners" were given a trip to Mountain View to visit the Google headquarters and meet with other GCi price winners.
You can see the results on the corresponding wiki page
There were 89 finished tasks, ranging from research tasks (document how other projects manage their documentation), creating howtos, trying out software on NetBSD, writing code (ATF tests and Markdown converters and more), writing manpages and documentation, fixing bugs and converting documentation from the website to the wiki.
Overall, it was a nice experience for NetBSD. On the one hand, some real work was done (for many of them, integration is still pending). On the other hand, it was a stressful time for the NetBSD mentors supervising the students and helping them on their tasks. Especially, we had to learn many lessons (you will find them on the wiki page for GCi 2012), but next time, we will do much better. We will try to apply again next year, but we will need a large bunch of new possible tasks to be chosen again.
So if you think you have a task which doesn't require great prior knowledge, and is solvable within two hours by an experienced developer, but also by a 13-18 year old within finite time, feel free to contact us with an outline, or write it directly to the wiki page for Code-In in the NetBSD wiki.