Nov 2014
S M T W T F S
           
24 25
           

Archives

This page is a blog mirror of sorts. It pulls in articles from blog's feed and publishes them here (with a feed, too).

When I found out that Ingenic is giving away some of their MIPS Creator CI20 boards I applied, and to my surprise they sent me one. Of course, the point was to make NetBSD work on it. I just finished the first step.

That is, make it load a kernel, identify / setup the CPU, attach a serial console. This is what it looks like:

U-Boot SPL 2013.10-rc3-g9329ab16a204 (Jun 26 2014 - 09:43:22)
SDRAM H5TQ2G83CFR initialization... done


U-Boot 2013.10-rc3-g9329ab16a204 (Jun 26 2014 - 09:43:22)

Board: ci20 (Ingenic XBurst JZ4780 SoC)
DRAM: 1 GiB
NAND: 8192 MiB
MMC: jz_mmc msc1: 0
In: eserial3
Out: eserial3
Err: eserial3
Net: dm9000
ci20# dhcp
ERROR: resetting DM9000 -> not responding
dm9000 i/o: 0xb6000000, id: 0x90000a46
DM9000: running in 8 bit mode
MAC: d0:31:10:ff:7e:89
operating at 100M full duplex mode
BOOTP broadcast 1
DHCP client bound to address 192.168.0.47
*** Warning: no boot file name; using 'C0A8002F.img'
Using dm9000 device
TFTP from server 192.168.0.44; our IP address is 192.168.0.47
Filename 'C0A8002F.img'.
Load address: 0x88000000
Loading: #################################################################
##############
284.2 KiB/s
done
Bytes transferred = 1146945 (118041 hex)
ci20# bootm
## Booting kernel from Legacy Image at 88000000 ...
Image Name: evbmips 7.99.1 (CI20)
Image Type: MIPS NetBSD Kernel Image (gzip compressed)
Data Size: 1146881 Bytes = 1.1 MiB
Load Address: 80020000
Entry Point: 80020000
Verifying Checksum ... OK
Uncompressing Kernel Image ... OK
subcommand not supported
ci20# g 80020000
## Starting applicatpmap_steal_memory: seg 0: 0x30c 0x30c 0xffff 0xffff
Loaded initial symtab at 0x802502d4, strtab at 0x80270cb4, # entries 8323
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014
The NetBSD Foundation, Inc. All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.

NetBSD 7.99.1 (CI20) #113: Sat Nov 22 09:58:39 EST 2014
ml@blackbush:/home/build/obj_evbmips32/sys/arch/evbmips/compile/CI20
Ingenic XBurst
total memory = 1024 MB
avail memory = 1001 MB
kern.module.path=/stand/evbmips/7.99.1/modules
mainbus0 (root)
cpu0 at mainbus0: 1200.00MHz (hz cycles = 120000, delay divisor = 12)
cpu0: Ingenic XBurst (0x3ee1024f) Rev. 79 with unknown FPC type (0x330000) Rev. 0
cpu0: 32 TLB entries, 16MB max page size
cpu0: 32KB/32B 8-way set-associative L1 instruction cache
cpu0: 32KB/32B 8-way set-associative write-back L1 data cache
com0 at mainbus0: Ingenic UART, working fifo
com0: console

root device:

What works:

  • CPU identification and setup
  • serial console via UART0
  • reset ( by provoking a watchdog timeout )
  • basic timers - enough for delay(), since the CPUs don't have MIPS cycle counters
  • dropping into ddb and poking around

What doesn't work (yet):

  • interrupts
  • everything else

Biggest obstacle - believe it or not, the serial port. The on-chip UARTs are mostly 16550 compatible. Mostly. The difference is one bit in the FIFO control register which, if not set, powers down the UART. So throwing data at the UART by hand worked but as soon as the com driver took over the line went dead. It took me a while to find that one.
Posted Sunday afternoon, November 23rd, 2014 Tags: blog

The NetBSD Project is pleased to announce NetBSD 5.1.5, the fifth security/bugfix update of the NetBSD 5.1 release branch, and NetBSD 5.2.3, the third security/bugfix update of the NetBSD 5.2 release branch. They represent a selected subset of fixes deemed important for security or stability reasons, and if you are running a prior release of either branch, we strongly suggest that you update to one of these releases.

For more details, please see the NetBSD 5.1.5 release notes or NetBSD 5.2.3 release notes.

Complete source and binaries for NetBSD are available for download at many sites around the world. A list of download sites providing FTP, AnonCVS, SUP, and other services may be found at http://www.NetBSD.org/mirrors/.

Posted Friday evening, November 21st, 2014 Tags: blog
Those following the source-changes mailing list closely may have noticed several evbarm kernels getting "options MULTIPROCESSOR" in the last few days. This is due to those configurations now running properly in SMP mode, thanks to work mostly done by Matt Thomas and Nick Hudson.

The list of supported multiprocessor boards currently is:

  • Banana Pi (BPI)
  • Cubieboard 2 (CUBIEBOARD)
  • Cubietruck (CUBIETRUCK)
  • Merrii Hummingbird A31 (HUMMINGBIRD_A31)
  • CUBOX-I
  • NITROGEN6X

Details how to create bootable media and various other information for the Allwinner boards can be found on the NetBSD/evbarm on Allwinner Technology SoCs wiki page.

The release engineering team is discussing how to bring all those changes into the netbsd-7 branch as well, so that we can call NetBSD 7.0 "the ARM SoC release".

While multicore ARM chips are mostly known for being used in cell phones and tablet devices, there are also some nice "tiny PC" variants out there, like the CubieTruck, which originally comes with a small transparent case that allows piggybacking it onto a 2.5" hard disk:


Image from cubieboard.org under creative commons license.

This is nice to put next to your display, but a bit too tiny and fragile for my test lab - so I reused an old (originally mac68k cartridge) SCSI enclosure for mine:


Image by myself under creative commons license.

This machine is used to run regular tests for big endian (!) arm, the results are gathered here. Running it big-endian is just a way to trigger more bugs.

The last test run logged on the page is already done with an SMP kernel. No regressions were found so far, and the other bugs (sligtly more than 30 failures in the test run is way too much) will be addressed one by one.

Now happy multi-ARM-ing everyone, and I am looking forward to a great NetBSD 7.0 release!

Posted at lunch time on Thursday, November 6th, 2014 Tags: blog

The NetBSD Project is pleased to announce NetBSD 6.1.5, the fifth security/bugfix update of the NetBSD 6.1 release branch, and NetBSD 6.0.6, the sixth security/bugfix update of the NetBSD 6.0 release branch. They represent a selected subset of fixes deemed important for security or stability reasons, and if you are running a prior release of either branch, we strongly suggest that you update to one of these releases.

For more details, please see the NetBSD 6.1.5 release notes or NetBSD 6.0.6 release notes.

Complete source and binaries for NetBSD are available for download at many sites around the world. A list of download sites providing FTP, AnonCVS, SUP, and other services may be found at http://www.NetBSD.org/mirrors/.

Posted in the wee hours of Monday night, October 7th, 2014 Tags: blog

There was a nice, friendly and informal summit of NetBSD developers (and interested users) in Sofia on Friday, September 26, 2014.

Bernd Ernesti (veego@) took this photo:

In the back row from left to right:
Masao Uebayashi (uebayasi), Thomas Klausner (wiz), Yann Sionneau, Marc Balmer (mbalmer), Justin Cormack (justin), Jaap Boender (jaapb), Adrian Steinmann (ast), Martin Husemann (martin), Taylor R Campbell (riastradh), Michael van Elst (mlelstv), Sevan Janiyan, Alexander Nasonov (alnsn).
In the front row, from left to right:
Julian Coleman (jdc), Joerg Sonnenberger (joerg), Valeriy E. Ushakov (uwe), Christoph Badura (bad), S.P.Zeidler (spz), Pierre Pronchery (khorben), Stephen Borrill (sborrill)

Some Developers made it to the conference only after the summit, so Emmanuel Dreyfus (manu), Luke Mewburn (lukem) and Lourival Neto (lneto) are unfortunately missing on that picture.

Marc Balmer presented some slides prepared by Masanobu SAITOH about ongoing work at IIJ.

Marc also proposed some extensions to the in-tree httpd (aka bozohttpd) to allow creation of simple dynamic content via "Lua templates". The question whether it would be possible to serve www.netbsd.org by bozohttpd was discussed, and only administrative reasons seem to prevent it - which was overall considered "good enough".

Pierre Pronchery presented his work on EdgeBSD - and why he does not consider it a fork, but more a playground for experimentation. He also reported on some of his experiences with using git on the NetBSD source tree.

Two more slightly TNF internal issues were discussed, and after that the whole crowd moved to dinner (including a bit of Bulgarian wine).

Having an in-person meeting of a relative huge number of NetBSD developers was considered very useful and we will try to repeat it at other occasions. Next time, if similar attendance is likely, we will plan for more time (like a full day) and also create a schedule of talks/presentations up front, still with the option to add ad-hoc ones.

Posted Tuesday evening, September 30th, 2014 Tags: blog

The most time-consuming part of operating system development is obtaining enough drivers to enable the OS to run real applications which interact with the real world. NetBSD's rump kernels allow reducing that time to almost zero, for example for developing special-purpose operating systems for the cloud and embedded IoT devices. This article describes an experiment in creating an OS by using a rump kernel for drivers. It attempts to avoid going into full detail on the principles of rump kernels, which are available for interested readers from rumpkernel.org. We start by defining the terms in the title:

  • OS: operating system, i.e. the overhead that enables applications to run
  • internet-ready: supports POSIX applications and talks TCP/IP
  • a week: 7 days, in this case the period between Wednesday night last week and Wednesday night this week
  • from scratch: began by writing the assembly instructions for the kernel entry point
  • rump kernel: partial kernel consisting of unmodified NetBSD kernel drivers
  • bare metal: what you get from the BIOS/firmware

Why would anyone want to write a new OS? If you look at our definition of "OS", you notice that you want to keep the OS as small as possible. Sometimes you might not care, e.g. in case of a desktop PC, but other times when hardware resources are limited or you have high enough security concerns, you actually might care. For example, NetBSD itself is not able to run on systems without a MMU, but the OS described in this article does not use virtual memory at all, and yet it can run most of the same applications as NetBSD can. Another example: if you want to finetune the OS to suit your application, it's easier to tune a simple OS than a very complicated general purpose OS. The motivation for this work came in fact from someone who was looking to provision applications as services on top of VMWare, but found that no existing solution supported the system interfaces his applications needed without dragging an entire classic OS along for the ride.

Let's move on to discussing what an OS needs to support for it to be able to host for example a web server written for a regular OS such as Linux or the BSDs. The list gets quite long. You need a file system where the web server reads the served pages from, you need a TCP/IP stack to communicate with the clients, and you need a network interface driver to be able to send and receive packets. Furthermore, you need the often overlooked, yet very surprisingly complicated system call handlers. For example, opening a socket is not really very complicated to handle. Neither is reading and writing data. However, when you start piling things like fcntl(O_NONBLOCK) and poll() on top, things get trickier. By a rough estimate, if you run an httpd on NetBSD, approximately 100k lines of code from kernel are used just to service the requests that the httpd makes. If you do the math (and bc did), there are 86400 seconds in a week. The OS we are discussing is able to run an off-the-shelf httpd, but definitely I did not write >1 line of code per second 24/7 during the past week.

Smoke and Mirrors, CGI Edition

The key to happiness is not to write 100k lines of code from scratch, nor to port it from another OS, as both are time-consuming and error-prone techniques, and error-proneness leads to even more consumption of time. Rump kernels come into the picture as the key to happiness and provide the necessary drivers.

As the old saying goes: "rump kernels do not an OS make", and we need the rest of the bits that make up the OS side of the software stack from somewhere. These bits need to make it seem like the drivers in a rump kernel are running inside the NetBSD kernel, hence "smoke and mirrors". What is surprising is how little code needs to exist between the drivers and the hardware, just some hundreds of lines of code. More specifically, in the bare metal scenario we need support for:

  • low level machine dependent code
  • thread support and a scheduler
  • rump kernel hypercall layer
  • additionally: bundling the application into a bootable image

The figure below illustrates the rump kernel software stack. The arrows correspond to the above list (in reverse order). We go over the list starting from the top of the list (bottom of the figure).

rump kernel software stack

Low level machine dependent code is what the OS uses to get the CPU and devices to talking terms with the rest of OS. Before we can do anything useful, we need to bootstrap. Bootstrapping x86-32 is less work than one would expect, which incidentally is also why the OS runs only in 32bit mode (adding 64bit support would not likely be many hours of work — and patches are welcome). Thanks to the Multiboot specification, the bootstrap code is more or less just a question of setting the stack pointer and jumping to C code. In C code we need to parse the amount of physical memory available and initialize the console. Since NetBSD device drivers mainly use interrupts, we also need interrupt support for the drivers to function correctly. On x86, interrupt support means setting up the CPU's interrupt descriptor tables and programming the interrupt controller. Since rump kernels do not support interrupts, in addition we need a small interrupt stub that transfers the interrupt request to a thread context which calls the rump kernel. In total, the machine dependent code is only a few hundred lines. The OSDev.org wiki contains a lot of information which was useful when hammering the hardware into shape. The other source of x86 hardware knowledge was x86 support in NetBSD.

Threads and scheduling might sound intimidating, but they are not. First, rump kernels can run on top of any kinds of threads you throw at them, so we can just use the ones which are the simplest to implement: cooperative threads. Note, simple does not mean poorly performing threads, and in fact the predictability of cooperative threads, at least in my opinion, makes them more likely to perform better than preemptive threading in cases where you are honing an OS for a single application. Second, I already had access to an implementation which served as the basis: Justin Cormack's work on userspace fibers, which in turn has its roots in Xen MiniOS we use for running rump kernel on the Xen hypervisor, could be re-purposed as the threads+scheduler implementation, with the context switch code kindly borrowed from MiniOS.

The rump kernel hypercall interface is what rump kernels themselves run on. While the implementation is platform-specific, our baremetal OS shares a large portion of its qualities with the Xen platform that was already supported. Therefore, most of the Xen implementation applied more or less directly. One notable exception to the similarities is that Xen paravirtualized devices are not available on bare metal and therefore we access all I/O devices via the PCI bus.

All we need now is the application, a.k.a. "userspace". Support for application interfaces (POSIX syscalls, libc, etc.) readily exists for rump kernels, so we just use what is already available. The only remaining issue is building the bundle that we bootstrap. For that, we can repurpose Ian Jackson's app-tools which were originally written for the rump kernel Xen platform. Using app-tools, we could build a bootable image containing thttpd simply by running the app-tools wrappers for ./configure and make. The image below illustrates part of the build output, along with booting the image in QEMU and testing that the httpd really works. The use of QEMU, i.e. software-emulated bare metal, is due to convenience reasons.

thttpd booting on bare metal

Conclusions

You probably noticed that whole thing is just bolting a lot of working components together while writing minimal amounts of necessary glue. That is exactly the point: never write or port or hack what you can reuse without modification. Code reusability has always been the strength of NetBSD and rump kernels add another dimension to that quality.

The source code for the OS discussed in this post is available under a 2-clause BSD license from repo.rumpkernel.org/rumpuser-baremetal.

Posted at midnight, August 8th, 2014 Tags: blog
A recent discussion on the port-vax mailing list brought a problem with the default installation method (when booting from CD, which typically is the easiest way) to my attention: it would not work on machines with 16 MB RAM or less.

So, can we do better? Looking at the size of a GENERIC kernel:

   text    data     bss     dec     hex filename
2997389   67748  173044 3238181  316925 netbsd
it seems we can not easily go below 4 MB (and for other reasons we would need to compile the bootloader differently for that anyway). But 16MB is still quite a difference, so it should work.

Now at the time I started this quest, I only had one VAX machine in real hardware - a VaxStation 4000 M96a, one of the fastest machines, and with 128 MB RAM well equipped. This is nice if you try to natively compile modern gcc, but I did not feel like fiddling with my hardware to create a better test environment for small RAM installations.

Like a year (or so) ago, when I fixed the VAX primary boot blocks (with lots of help from various vaxperts on the port-vax mailing list), SIMH, found in pkgsrc as emulators/simh, proved helpful. Testing various configurations I found an emulated VAX 11/780 with 8 MB to be the smallest I could get working.

The first step of the tuning was obvious: the CD image used a ramdisk based kernel, with the ramdisk containing all of the install system. At the same time, most of the CD was unused. We already use different schemes on i386, amd64 and sparc64 - so I cloned the sparc64 one and adjusted it to VAX. Now we use the GENERIC kernel on CD and mount the ISO9660 filesystem on the CD itself as root file system. The VAX boot loader already could deal with this, only a minor fix was needed for the kernel to recognize some variants of CD drives as boot device.

The resulting CD did boot, but did not go far in userland. The CD did only contain a (mostly) empty /dev directory (without /dev/console), which causes init(8) to mount a tmpfs on /dev and run the MAKEDEV script there. But to my surprise, on the 11/780 mfs was used instead of tmpfs - and we will see why soon. Next step in preparation of the userland for the installer is creating additional tmpfs instances to deal with the read-only nature of the CD used as root. This did not work at all, the mount attempts simply failed - and the installer was very unhappy, as it could not create files in /tmp for example.

I checked, but tmpfs was part of the VAX GENERIC kernel. I tried the install CD on a simulated MicroVAX 3900 with 64 MB of RAM - and to my surprise all of /dev and the three additional tmpfs instances created later worked (as well as the installation procedure). I checked the source (stupid me) and then found the documentation: tmpfs reserved a hard coded 4 MB of RAM for the system. With the GENERIC kernel booted on a 8 MB machine, we had slightly less than 4 MB RAM free, so tmpfs never worked.

One step back - this explained why /dev ended up as a mfs instead of tmpfs. The MAKEDEV code is written to deal with kernels that do include tmpfs, but also with those that do not: it tried tmpfs, and falls back to mfs if that does not work. This made me think, I could do the same (but without even trying tmpfs): I changed the install CD scripts to use mfs instead of tmpfs. The main difference is: mfs uses a userland process to manage the swappable memory. However, we do not have any swap space yet. Checking when sysinst enables swapping for the first time, I found: it never did on VAX. Duh! I added the missing calls to machine dependent code in sysinst, but of course the installer can only enable swap after partitioning is done (and a swap partition got created).

Testing showed: we did not get far enough with four mfs instances. So let us try with fewer. One we do not need is the /dev one: I changed the CD content creation code to pre-populate /dev on the CD. This is not possible with all filesystems, including the original ISO9660 one, but with the so-called Rockridge Extensions it works. We know that it is a modern NetBSD kernel mounting the CD - so support for those extensions is always present. I made some errors and hit some bugs (that got fixed) on the way there, but soon the CD booted without creating a mfs (nor tmpfs) for /dev.

Still, three mfs instances did not survive until sysinst enabled swapping. The userland part was killed once the kernel ran out of memory. I needed tmpfs working with less than 4 MB memory free. After a slight detour and some discussion on the tech-kern mailing list, I changed tmpfs to deal (and only reserve a dynamically scaled amount of memory calculated bv the UVM memory management). With this change, a current install CD just works, and installation completes successful.

The following is just the start of the installation process, the sysinst part afterwards is standard stuff and left out for brevity.

Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014                 
    The NetBSD Foundation, Inc.  All rights reserved.   
Copyright (c) 1982, 1986, 1989, 1991, 1993           
    The Regents of the University of California.  All rights reserved.
                                                                      
NetBSD 6.99.43 (GENERIC) #1: Thu Jun  5 22:01:14 CEST 2014
        martin@night-owl.duskware.de:/usr/obj/vax/usr/src/sys/arch/vax/compile/GENERIC
VAX 11/780
total memory = 8188 KB
avail memory = 3884 KB
mainbus0 (root)       
cpu0 at mainbus0: KA80, S/N 1234(0), hardware ECO level 7(112)
cpu0: 4KB L1 cachen, no FPA                                   
sbi0 at mainbus0           
mem0 at sbi0 tr1: standard
mem1 at sbi0 tr2: standard
uba1 at sbi0 tr3: DW780   
dz1 at uba1 csr 160100 vec 304 ipl 15
mtc0 at uba1 csr 174500 vec 774 ipl 15
mscpbus0 at mtc0: version 5 model 5   
mscpbus0: DMA burst size set to 4  
uda0 at uba1 csr 172150 vec 770 ipl 15
mscpbus1 at uda0: version 3 model 6   
mscpbus1: DMA burst size set to 4  
de0 at uba1 csr 174510 vec 120 ipl 15: delua, hardware address 08:00:2b:cc:dd:ee
mt0 at mscpbus0 drive 0: TU81                                                  
mt1 at mscpbus0 drive 1: TU81
mt2 at mscpbus0 drive 2: TU81
mt3 at mscpbus0 drive 3: TU81
ra0 at mscpbus1 drive 0: RA92
ra1 at mscpbus1 drive 1: RA92
racd0 at mscpbus1 drive 3: RRD40
ra0: size 2940951 sectors       
ra1: no disk label: size 2940951 sectors
racd0: size 1331200 sectors             
boot device: racd0         
root on racd0a dumps on racd0b
root file system type: cd9660 
init: kernel secur           

You are using a serial console, we do not know your terminal emulation.
Please select one, typical values are:

        vt100
        ansi
        xterm

Terminal type (just hit ENTER for 'vt220'): xterm
                                                                                
 NetBSD/vax 6.99.43                                                             
                                                                                
 This menu-driven tool is designed to help you install NetBSD to a hard disk,   
 or upgrade an existing NetBSD system, with a minimum of work.                  
 In the following menus type the reference letter (a, b, c, ...) to select an   
 item, or type CTRL+N/CTRL+P to select the next/previous item.                  
 The arrow keys and Page-up/Page-down may also work.                            
 Activate the current selection from the menu by typing the enter key.          


                +---------------------------------------------+
                |>a: Installation messages in English         |
                | b: Installation auf Deutsch                 |
                | c: Mensajes de instalacion en castellano    |
                | d: Messages d'installation en fran├žais      |
                | e: Komunikaty instalacyjne w jezyku polskim |
                +---------------------------------------------+

Overall this improved NetBSD to better deal with small memory systems. The VAX specific install changes can be brought over to other ports as well, but sometimes changes to the bootloader will be needed.
Posted mid-morning Sunday, June 8th, 2014 Tags: blog

The NetBSD Project is pleased to announce NetBSD 6.1.4, the fourth security/bugfix update of the NetBSD 6.1 release branch, and NetBSD 6.0.5, the fifth security/bugfix update of the NetBSD 6.0 release branch. They represent a selected subset of fixes deemed important for security or stability reasons, and if you are running a prior release of either branch, we strongly suggest that you update to one of these releases.

For more details, please see the NetBSD 6.1.4 release notes or NetBSD 6.0.5 release notes.

Complete source and binaries for NetBSD 6.1.4 and NetBSD 6.0.5 are available for download at many sites around the world. A list of download sites providing FTP, AnonCVS, SUP, and other services may be found at http://www.NetBSD.org/mirrors/.

Posted Saturday afternoon, April 19th, 2014 Tags: blog
In 2009 the playstation2 port was removed from the NetBSD sources, since it had not been compilable for months and no modern enough compiler was available.

Due to a strange series of events the code changes needed to support the (slightly unusual) MIPS CPU used in the playstation2 had never been merged into gcc nor binutils mainline. Only recently this has been fixed. Unfortunately the changes have not been pulled up to the gcc 4.8.3 branch (which is available in NetBSD-current), so an external toolchain from pkgsrc is needed for the playstation2.

To install this toolchain, use a pkgsrc-current checkout and cd to cross/gcc-mips-current, then do "make install" - that is all.

Work is in progress to bring the old code up to -current. Hopefully a bootable NetBSD-current kernel will be available soon.

Posted late Monday morning, March 31st, 2014 Tags: blog
After several months of preparation, the first ports (hppa, sparc and sparc64) have switched their compiler to gcc version 4.8 today. Amd64 and i386 should follow soon.

Work is ongoing to bring this modern toolchain to all other ports too (most of them already work, but some more testing will be done). If you want to try it, just add -V HAVE_GCC=48 to the build.sh invocation.

Note that in parallel clang is available as an alternative option for a few architectures already (i386, amd64, arm, and sparc64), but needs more testing and debugging at least on some of them (e.g. the sparc64 kernel does not boot).

For a project with diverse hardware support like NetBSD, all toolchain updates are a big pain - so a big THANK YOU! to everyone involved; in no particular order Christos Zoulas, matthew green, Nick Hudson, Tohru Nishimura, Frank Wille (and myself).

Posted late Thursday morning, March 6th, 2014 Tags: blog
Add a comment