[c2k8]: Hackathon Summary Part 1

Mark Uemura (mtu@), back by popular demand, has been gracious enough to share his experiences at c2k8, The OpenBSD General Hackathon in Edmonton. Here is his first summary:

c2k8 General Hackathon (Part 1) - June 7-15, 2008, Edmonton, Alberta, Canada The c2k8 hackathon is now over but tremors of the week long earthquake that hit cvs are still being felt from Theo's dinner table. That is what hackathons do. They are like fuel tanks that keep OpenBSD development moving forward and the Big Hackathon held annually is like a boost of nitrous oxide.

Mark's continues below.

As promised, I will try and highlight the development throughout the week long c2k8 hackathon in a series of summary parts. It will be an uphill battle for me as there is just no way that I would be able to impart what happened there through text and pictures. You will be missing so many really interesting conversations, get-to-the-right-solution debates and often heard and sometimes felt screams of joy emanating from Bob Beck (beck@) and others as if they had just won the lottery. I truly hope that some of you reading this will one day be able to experience this event and/or the many mini-hackathons leading up to this big annual hackathon. This is where comrades (drinking companions) get to spend time working on something they truly enjoy and amongst friends that have come for the same reason.

In the n2k8 series, I mentioned that there are many individuals that I follow on misc@. They are like heroes on the OpenBSD stage and people that I admire for what they have done in OpenBSD and for the community. Otto Moerbeek (otto@) is one of those guys. Theo has spoken very highly of Otto in the past and ever since then, I've paid closer attention to what Otto has to say on misc@ and what he has done in OpenBSD. His most recent work on ffs2 support in OpenBSD is really impressive. With drives getting bigger and cheaper, large partitions are becoming that much more important. I'm sure that I can speak for everyone and say, "We really appreciate your work!".

Here is what this kind and polite gentleman had to say about this work at the c2k8 hackathon:

During this hackathon I worked on two major things and a lot of small ones. The first major thing is malloc(3). Actually, most of the work on malloc has been done in the last two years: first lots of studying of the current malloc and thinking how to improve it. Then, spread over some months, I did implement it at home and here in Edmonton I worked on the final pieces.

The problem with current malloc is that its main datastructure (the page directory) originally was designed to deal with contiguous memory as obtained by sbrk(2). Some years ago, it was adapted to use mmap(2), which on OpenBSD returns pages with random addresses. The page directory does not handle that well: its overhead, both in time and memory usage, for random page addresses is pretty bad.

The new malloc implementation (omalloc for short) uses a page directory which does not degrade when handling random page addresses. It also handles various edge cases better: it never needs memory to free memory. It also randomizes sub-page sized allocations by default; the current malloc only does that when the 'G" options is used. I knew there was room for improvement, but I was pretty amazed that omalloc improves performance by tens of percents in some cases. The code is also quite a bit smaller and cleaner, in my opinion.

During the hackathon, I implemented randomized chunk allocation and a trick to detect certain types of buffer overflows more easily by shifting small allocations to the end of a page. Damien Miller (djm@) did a review of the code and came up with a couple of good suggestions. The nice thing about a hackathon (apart from the bbq, beer and being able to work full time on things I like 100%) is that these suggestions could be discussed face to face, immediately, and so I was able to finish omalloc in far less time than I expected. Of course omalloc needs more testing, if you are running -current (and only -current!), you can help by giving the latest version a spin.

The other thing I worked on is fsck_ffs(8) for really large filesystems. fsck_ffs needs to store a few bytes of data about each inode in the filesystem being checked. Depending on the limitations of the platform, this means that filesystems having lots of inodes cannot be checked. Using a diff originally started by Dale Rahn (drahn@), I could reduce memory consumption by 20%. This diff has been committed. I am also working on a diff to actually not store all inode data in memory simultaneously; but this work is not yet ready. Although, in a certain test setup, I could reduce memory usage by a factor of 4 -- and there's more to be gained.

One of the small things I worked on with Ken Westerback (krw@) is disklabel(8). We are taking steps to make it present less confusing data in interactive editing mode to make it more understandable. This is also work in progress: it needs more tweaking before it hits the tree.

The hackathon has been fun and fruitful for me!

---

Another person that I really was dying to meet was Marco Peereboom (marco@). I knew that he was working a lot in the past with David Gwynne (dlg@) who was sadly missing at this year's big hackathon but there in spirit. :-) David has learned so much from Marco and together they gave us blob free RAID management (bio(4)) and the sensors framework. Marco doesn't stop there. Now we have softraid(4), which is also really exciting; and more recently, support for softraid with crypto support! Oh, and that's not all!

He has done quite a lot of work with Jordan Hargrave (jordan@) and others to make ACPI work on OpenBSD for those poor souls that do not have an X40 and can't suspend. ;-) I brought along my evil ACPI machine, a Sony UX series computer, so that Marco could iron out some more issues with ACPI. I thought that it would eventually be a potential Zaurus replacement. Theo bought two of them last year to give to developers mainly because of the ACPI. Needless to say, it is pretty nicely supported now thanks to Marco and Jordan.

Here is what Marco had to say about his time at the c2k8 hackathon:

I came with the best intentions to do some softraid extensions that several folks have been requesting (rebuilds, marking dirty volumes clean, etc); however, ACPI took most of my time. Throughout the event, I was walking around debugging random laptops people handed me. The good news is that all laptops we debugged were no longer broken due to our code but due to hardware issues. I think we can declare a major victory in ACPI. Now what's left to do is get some drivers extended and port/write some others. Well, there is also the suspend thing that people seem to want. :-)

I did get to work closely with djm@ and hshoexer@ (Hans-Joerg Hoexer), which was an absolute privilege! Those guys are amazing. hshoexer@ wrote the softraid crypto implementation while djm@ added a new crypto algorithm that should work a whole lot better for block devices (AES-XTS). hshoexer@ pushed me to finally implement the delete softraid volume and thanks to that activity, I found an ancient bug that had been plaguing us (disks are shut down normally yet, they complain that they have unclean metadata upon reboot).

The other large project I started working on is rewriting the softraid metadata from scratch because we want to be able to deal with foreign metadata. Since it needs a rewrite, djm@ and I completely rewrote the softraid on-disk format. I am dealing with the fallout of that which is quite the task. This will keep me busy for weeks to come.

(c2k8 hackathon summary to be continued)

Thanks to Mark, Otto and Marco for sharing their experiences at c2k8. We plan to publish several more installments from c2k8 over the next week or so. Please show your appreciation by donating, buying a CD set or convincing the company you work for, that benefits from OpenBSD development, to contribute to the project.