OpenBSD Journal

[c2k8]: Hackathon Summary Part 4

Contributed by mtu on from the long-live-biomem dept.

c2k8 General Hackathon (Part 4) - June 7-15, 2008, Edmonton, Alberta, Canada

All those of us who wished to go on the hike were told to be downstairs and ready by 06:00. Surprisingly, everyone made it in time; however, at about ten minutes to six, Theo came down from the hacking room looking seriously sleep deprived. He noticed that Artur Grabowski (art@) was alone, outside, waking up in the cold drizzly Canadian morning air whilst getting his morning dose of nicotine before the four hour long car ride. On his way out to talk to Art, Theo muttered, "Art's going to love to hear this." It was just the two of them outside now and they both looked concerned. Art lit up another one and by now most of the group had congregated outside ready to depart. Was there going to be a change in plans?

art, theo and bob

Read on to know what happened next:


The biomem diff that Bob Beck (beck@) committed on June 10 (the one that was worked on for about a year with Art), was going to be pulled out as it was crashing Theo's laptop. All it took was a NFS client copying large 200 MB files across from the server.

Interestingly enough, Bob had some foresight or an incredible premonition with this diff as the last line reads, "ok deraadt@, krw@, art@ - who promises to be around to deal with any fallout." He almost timed it perfectly. Almost ;-). Actually, it was a tough decision for Bob and Art to continue with the planned hike. They didn't want to call it off and disappoint a dozen or so guys that made the effort to wake up at the crack of dawn for the chance to eat buffalo burgers and relax in a famous Canadian Hot Spring.

bob and crew
I was in Bob's car along with Art and I can tell you that they must have argued and contemplated what could be the issue for almost an hour! It was painful to watch, as they both knew that they couldn't do anything until they returned later in the evening. They also knew that if the biomem diff wasn't fixed soon, it would most likely miss this next release. However, there was no question that the biomem diff was being pulled out until the NFS issue was resolved.

art, dale and bob
With great perseverance and sheer stubbornness, as you'll read below, these two veterans managed to do their magic and biomem was once more commited. This time for good, I'm sure :-).

On another note, I would like to thank Bob on four counts: 1) for working on biomem; 2) for hosting a wonderful hackathon in an ideal venue; 3) hosting an amazing barbecue and; 4) for taking us to the famous Miette Hot Springs! The whole week long event, or should I say, series of events, were extremely fun and memorable. Perhaps just as importantly, it was a chance for everyone to increase their skills and knowledge, as well as contribute immensely to the advancement of quality free software!

Here is what Bob had to say about his time at the c2k8 hackathon:

Well, what can I say about the hackathon?

First I suppose, is all the stuff around organising and putting it all together. It's quite a bit of work and time spent sorting things out, everything from the rooms, the room lists, the hacking room, power, network, barbeque at my place, etc. etc. etc. - typically I feel basically run off my feet at these things.

Fortunately I had a lot of local help with some of the details to remove a little bit of that stress. Gordon Klok, Jason Meltzer, and Paul Greidanus all get a big THANK YOU from me for their help in setting up and getting stuff alive ahead of time. U of A conference services was also very very helpful in getting everything in place with very few hiccups. I was actually very pleased with the level of responsiveness I got from everyone at the U of A.

As far as what I was working on, I went into it with no goals to work on anything, actually. My main goal was just to ensure everything went well, and everyone else could be productive. However, everything went smoothly enough that I could spend time diving into what we affectionally know as the "biomem" diff, that has been comitted and backed out like a yoyo over the past year. Most of the guts of it was written by Art (Grabowski) and I've tortured it, pulled out a few bugs, and added some instrumentation over the past year. What this actually does is to enable us to map much more of our memory as filesystem buffer cache, by allowing pages to be mapped in only when used.

The problem is that when touching the buffer cache many other subsystems get affected - especially NFS, so I ended up spending much of my time hacking NFS in self defense - We found a nasty bug in how NFS uses buffers - and I spent some time fixing this and adding a queue mechanism to allow nfs to use a certain amount of buffers for asynchronous IO without consuming all of them.

During the process I also chased what I believed to be yet another NFS bug, which turned out to be an em(4) driver bug in the way interrupt coalescing was handled. This affected the ethernet card I was using - While I finally found it, let's just say that the process was a long and tortuous journey down the rabbit hole, involving much screaming and hair pulling.

The long and the short of it is this has allowed the biomem diff to go in with relatively few current side effects - some of which we are still chasing - and we hope to have this tuned up nicely for 4.4, which should move us along the way to better filesystem and memory performance.

Anyway, my hope was simply that everyone there had a good time, and got a lot done - I wasn't too focused on working on things myself.



Another OpenBSD icon is Artur Grabowski (art@). He has been there almost from the very beginning. When I think of Art, I can imagine him as one of the 300 fighting right beside Leonidas. Here's a man who's been through it all. Like Bob, he's got your back if you're in trouble. He's not on some high horse even though he is a long time OpenBSD veteran. He's still on the front line forging the path for people to follow. Art is a natural leader that people admire, respect and gravitate toward for who he is and what he has done.

coca cola adv
As we were hiking the mountain, Art was kind enough to entertain me with many stories of old. For example, I've never been to a USENIX conference so I asked him what it was like. He hasn't been to USENIX for many years now but during the height of the dot com bubble years when he did go, it was the event to attend. Apparently, all you needed to do was get there and the vendors with their big expense accounts paid for the rest; food, beer and lots of parties. Actually, the really interesting stories came out of those many parties :-). Then the dot com bubble burst and he said that it wasn't the same thereafter. I'd be curious what others thought of their Usenix experience.

bob and art demonstrating the hurl
So what is it that drives Art to stay so close to the OpenBSD Project over the past ten years? Perhaps, this post in the archives may give some hint. "Yes it is. If it wouldn't be fun (toy) I wouldn't sacrifice 1-10 hours a day, sacrifice my personal life, pay for network access, pay electricity bills [...], skip sleep, etc." This "fun" theme is key. Heck, you can't pay commercial developers to maintain the same standards. Perhaps it's their job rather than something that they are passionate about and have fun doing at the same time. Is there a correlation between fun and code quality? Perhaps not, but mix that with pride, reputation and a whole lot of OpenBSD best practices and then I would say yes. :-)

Here is what Art had to say about his work and time at the c2k8 hackathon:

My hackathon was as usual. Most of what I did was to cruise around and talk to people about their problems and try to find hacks to debug things. I also worked on a few projects within memory handling.

We resurrected one of my old diffs that has been maturing for a year that's another step in the direction of a bit smarter buffer cache that can actually use more physical memory and some time in the future even be coherent with mmap. Bob did some fixes for it to make it work on all architectures and with some minor fixes it started to work and went into the tree. As usual, when we touch anything around the buffer cache, this triggered an avalanche of nfs bugs. So we actually debugged and found some really ancient problems (12 years old) in the nfs code that we could fix.

In that process we ran into yet another bug that was causing distressingly bad performance in nfs, that turned out to be a bug in the interrupt mitigation in the em driver. None of this would be possible without having people nearby to bounce ideas with in real time.

I also did some work on a diff that mickey posted a few months ago that will make kernel malloc a bit less brain dead, but it still has problems on alpha, that I'm actually debugging in Bob's living room right now (we went for a hike after the hackathon and came back yesterday).


I would like to express my deepest appreciation to Bob and Art for what they have done and continue to do for OpenBSD. You are role models for the community to follow in code and leaders in your own right.

(c2k8 hackathon summary to be continued)

(Comments are closed)

  1. By tedu ( on

    yay. :)

    1. By Anonymous Coward ( on

      > yay. :)

      A hackathon without tedu is definitively missing something. Where else can I bounce dumb ideas as easily?



  2. By Anonymous Coward ( on

    Great stuff, very much appreciated! Thank you.

  3. By mk ( on

    An avalanche of nfs bugs -- haha, I can totally see this happen.

    1. By Anonymous Coward ( on

      If nfs bugs are found, it sounds like a good reason to touch the buffer cache more often.

      What happened to mickey@ ?

  4. By sthen@ (2a01:348:108:155:216:41ff:fe53:6a45) on

    I'll second the big thanks to Jason Gordon and Paul for their hard work with the infrastructure - besides the hack room setup, power and network, there were also a bunch of fast machines for people to work on that had to be transported, racked and configured before everything started, and of course reinstated to their normal home afterwards - lots of behind-the-scenes work and it is really appreciated.

  5. By art ( on

    Remember, you have to call Bob "King NFS" now. He really enjoys it.

  6. By Cobalt ( on

    Since I love numbers - are there any quantitative measures on how this buffer change improves things? Or it this an intermediate step that lets the 'real' work begin?

    1. By Anonymous Coward ( on

      > Since I love numbers - are there any quantitative measures on how this buffer change improves things? Or it this an intermediate step that lets the 'real' work begin?

      It improves things from the point of view of the code.

      I really don't want to benchmark every step we take since some steps can (and most likely will) make things slower and then people will whine without seeing (or caring about) the whole picture. I was afraid that this would have been such a step. For some reasons it actually made things faster.

      1. By Bob Beck ( on

        > > Since I love numbers - are there any quantitative measures on how this buffer change improves things? Or it this an intermediate step that lets the 'real' work begin?
        > It improves things from the point of view of the code.
        > I really don't want to benchmark every step we take since some steps can (and most likely will) make things slower and then people will whine without seeing (or caring about) the whole picture. I was afraid that this would have been such a step. For some reasons it actually made things faster.

        Actually It *can* make things a *lot* faster in certain situations
        where you twiddle to non-default numbers (like cranking up the cachepct and number of vnodes). However, since we're not ready for that yet I'm
        not going any further in terms of explaining that - you do so at your own

        Hopefully if we can however keep this in and stable for 4.4, we can proceed with making it somewhat dynamic, in order to get those sorts
        of performance wins without screwing around with your kernel - or breaking
        other things.


Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]