Contributed by mtu on from the long-live-biomem dept.
Read on to know what happened next:
The biomem diff that Bob Beck (beck@) committed on June 10 (the one that was worked on for about a year with Art), was going to be pulled out as it was crashing Theo's laptop. All it took was a NFS client copying large 200 MB files across from the server.
Interestingly enough, Bob had some foresight or an incredible premonition with this diff as the last line reads, "ok deraadt@, krw@, art@ - who promises to be around to deal with any fallout." He almost timed it perfectly. Almost ;-). Actually, it was a tough decision for Bob and Art to continue with the planned hike. They didn't want to call it off and disappoint a dozen or so guys that made the effort to wake up at the crack of dawn for the chance to eat buffalo burgers and relax in a famous Canadian Hot Spring.
I was in Bob's car along with Art and I can tell you that they must have argued and contemplated what could be the issue for almost an hour! It was painful to watch, as they both knew that they couldn't do anything until they returned later in the evening. They also knew that if the biomem diff wasn't fixed soon, it would most likely miss this next release. However, there was no question that the biomem diff was being pulled out until the NFS issue was resolved.
With great perseverance and sheer stubbornness, as you'll read below, these two veterans managed to do their magic and biomem was once more commited. This time for good, I'm sure :-).
On another note, I would like to thank Bob on four counts: 1) for working on biomem; 2) for hosting a wonderful hackathon in an ideal venue; 3) hosting an amazing barbecue and; 4) for taking us to the famous Miette Hot Springs! The whole week long event, or should I say, series of events, were extremely fun and memorable. Perhaps just as importantly, it was a chance for everyone to increase their skills and knowledge, as well as contribute immensely to the advancement of quality free software!
Here is what Bob had to say about his time at the c2k8 hackathon:
Well, what can I say about the hackathon?
First I suppose, is all the stuff around organising and putting it all together. It's quite a bit of work and time spent sorting things out, everything from the rooms, the room lists, the hacking room, power, network, barbeque at my place, etc. etc. etc. - typically I feel basically run off my feet at these things.
Fortunately I had a lot of local help with some of the details to remove a little bit of that stress. Gordon Klok, Jason Meltzer, and Paul Greidanus all get a big THANK YOU from me for their help in setting up and getting stuff alive ahead of time. U of A conference services was also very very helpful in getting everything in place with very few hiccups. I was actually very pleased with the level of responsiveness I got from everyone at the U of A.
As far as what I was working on, I went into it with no goals to work on anything, actually. My main goal was just to ensure everything went well, and everyone else could be productive. However, everything went smoothly enough that I could spend time diving into what we affectionally know as the "biomem" diff, that has been comitted and backed out like a yoyo over the past year. Most of the guts of it was written by Art (Grabowski) and I've tortured it, pulled out a few bugs, and added some instrumentation over the past year. What this actually does is to enable us to map much more of our memory as filesystem buffer cache, by allowing pages to be mapped in only when used.
The problem is that when touching the buffer cache many other subsystems get affected - especially NFS, so I ended up spending much of my time hacking NFS in self defense - We found a nasty bug in how NFS uses buffers - and I spent some time fixing this and adding a queue mechanism to allow nfs to use a certain amount of buffers for asynchronous IO without consuming all of them.
During the process I also chased what I believed to be yet another NFS bug, which turned out to be an em(4) driver bug in the way interrupt coalescing was handled. This affected the ethernet card I was using - While I finally found it, let's just say that the process was a long and tortuous journey down the rabbit hole, involving much screaming and hair pulling.
The long and the short of it is this has allowed the biomem diff to go in with relatively few current side effects - some of which we are still chasing - and we hope to have this tuned up nicely for 4.4, which should move us along the way to better filesystem and memory performance.
Anyway, my hope was simply that everyone there had a good time, and got a lot done - I wasn't too focused on working on things myself.
-Bob
Another OpenBSD icon is Artur Grabowski (art@). He has been there almost from the very beginning. When I think of Art, I can imagine him as one of the 300 fighting right beside Leonidas. Here's a man who's been through it all. Like Bob, he's got your back if you're in trouble. He's not on some high horse even though he is a long time OpenBSD veteran. He's still on the front line forging the path for people to follow. Art is a natural leader that people admire, respect and gravitate toward for who he is and what he has done.
As we were hiking the mountain, Art was kind enough to entertain me with many stories of old. For example, I've never been to a USENIX conference so I asked him what it was like. He hasn't been to USENIX for many years now but during the height of the dot com bubble years when he did go, it was the event to attend. Apparently, all you needed to do was get there and the vendors with their big expense accounts paid for the rest; food, beer and lots of parties. Actually, the really interesting stories came out of those many parties :-). Then the dot com bubble burst and he said that it wasn't the same thereafter. I'd be curious what others thought of their Usenix experience.
So what is it that drives Art to stay so close to the OpenBSD Project over the past ten years? Perhaps, this post in the archives may give some hint. "Yes it is. If it wouldn't be fun (toy) I wouldn't sacrifice 1-10 hours a day, sacrifice my personal life, pay for network access, pay electricity bills [...], skip sleep, etc." This "fun" theme is key. Heck, you can't pay commercial developers to maintain the same standards. Perhaps it's their job rather than something that they are passionate about and have fun doing at the same time. Is there a correlation between fun and code quality? Perhaps not, but mix that with pride, reputation and a whole lot of OpenBSD best practices and then I would say yes. :-)
Here is what Art had to say about his work and time at the c2k8 hackathon:
My hackathon was as usual. Most of what I did was to cruise around and talk to people about their problems and try to find hacks to debug things. I also worked on a few projects within memory handling.
We resurrected one of my old diffs that has been maturing for a year that's another step in the direction of a bit smarter buffer cache that can actually use more physical memory and some time in the future even be coherent with mmap. Bob did some fixes for it to make it work on all architectures and with some minor fixes it started to work and went into the tree. As usual, when we touch anything around the buffer cache, this triggered an avalanche of nfs bugs. So we actually debugged and found some really ancient problems (12 years old) in the nfs code that we could fix.
In that process we ran into yet another bug that was causing distressingly bad performance in nfs, that turned out to be a bug in the interrupt mitigation in the em driver. None of this would be possible without having people nearby to bounce ideas with in real time.I also did some work on a diff that mickey posted a few months ago that will make kernel malloc a bit less brain dead, but it still has problems on alpha, that I'm actually debugging in Bob's living room right now (we went for a hike after the hackathon and came back yesterday).
//art
I would like to express my deepest appreciation to Bob and Art for what they have done and continue to do for OpenBSD. You are role models for the community to follow in code and leaders in your own right.
(c2k8 hackathon summary to be continued)
(Comments are closed)
By tedu (74.68.146.146) on
Comments
By Anonymous Coward (76.250.126.209) on
A hackathon without tedu is definitively missing something. Where else can I bounce dumb ideas as easily?
:-)
/marco
By Anonymous Coward (212.20.215.132) on
By mk (193.128.72.68) on
Comments
By Anonymous Coward (219.90.214.40) on
--
What happened to mickey@ ?
By sthen@ (2a01:348:108:155:216:41ff:fe53:6a45) on
By art (84.55.71.48) on
By Cobalt (129.174.114.180) on
Comments
By Anonymous Coward (81.229.83.195) on
It improves things from the point of view of the code.
I really don't want to benchmark every step we take since some steps can (and most likely will) make things slower and then people will whine without seeing (or caring about) the whole picture. I was afraid that this would have been such a step. For some reasons it actually made things faster.
Comments
By Bob Beck (129.128.11.43) beck@openbsd.org on http://www.seizurerobots.com/
>
> It improves things from the point of view of the code.
>
> I really don't want to benchmark every step we take since some steps can (and most likely will) make things slower and then people will whine without seeing (or caring about) the whole picture. I was afraid that this would have been such a step. For some reasons it actually made things faster.
Actually It *can* make things a *lot* faster in certain situations
where you twiddle to non-default numbers (like cranking up the cachepct and number of vnodes). However, since we're not ready for that yet I'm
not going any further in terms of explaining that - you do so at your own
peril.
Hopefully if we can however keep this in and stable for 4.4, we can proceed with making it somewhat dynamic, in order to get those sorts
of performance wins without screwing around with your kernel - or breaking
other things.