OpenBSD Journal

[c2k8]: Hackathon Summary Part 1

Contributed by merdely on from the fscking-softraid-crypto dept.

Mark Uemura (mtu@), back by popular demand, has been gracious enough to share his experiences at c2k8, The OpenBSD General Hackathon in Edmonton. Here is his first summary:

c2k8 General Hackathon (Part 1) - June 7-15, 2008, Edmonton, Alberta, Canada

The c2k8 hackathon is now over but tremors of the week long earthquake that hit cvs are still being felt from Theo's dinner table. That is what hackathons do. They are like fuel tanks that keep OpenBSD development moving forward and the Big Hackathon held annually is like a boost of nitrous oxide.

otto, marco and other hackers hacking

Mark's continues below.

joris and otto
As promised, I will try and highlight the development throughout the week long c2k8 hackathon in a series of summary parts. It will be an uphill battle for me as there is just no way that I would be able to impart what happened there through text and pictures. You will be missing so many really interesting conversations, get-to-the-right-solution debates and often heard and sometimes felt screams of joy emanating from Bob Beck (beck@) and others as if they had just won the lottery. I truly hope that some of you reading this will one day be able to experience this event and/or the many mini-hackathons leading up to this big annual hackathon. This is where comrades (drinking companions) get to spend time working on something they truly enjoy and amongst friends that have come for the same reason.

In the n2k8 series, I mentioned that there are many individuals that I follow on misc@. They are like heroes on the OpenBSD stage and people that I admire for what they have done in OpenBSD and for the community. Otto Moerbeek (otto@) is one of those guys. Theo has spoken very highly of Otto in the past and ever since then, I've paid closer attention to what Otto has to say on misc@ and what he has done in OpenBSD. His most recent work on ffs2 support in OpenBSD is really impressive. With drives getting bigger and cheaper, large partitions are becoming that much more important. I'm sure that I can speak for everyone and say, "We really appreciate your work!".

Here is what this kind and polite gentleman had to say about this work at the c2k8 hackathon:

otto
During this hackathon I worked on two major things and a lot of small ones. The first major thing is malloc(3). Actually, most of the work on malloc has been done in the last two years: first lots of studying of the current malloc and thinking how to improve it. Then, spread over some months, I did implement it at home and here in Edmonton I worked on the final pieces.

The problem with current malloc is that its main datastructure (the page directory) originally was designed to deal with contiguous memory as obtained by sbrk(2). Some years ago, it was adapted to use mmap(2), which on OpenBSD returns pages with random addresses. The page directory does not handle that well: its overhead, both in time and memory usage, for random page addresses is pretty bad.

The new malloc implementation (omalloc for short) uses a page directory which does not degrade when handling random page addresses. It also handles various edge cases better: it never needs memory to free memory. It also randomizes sub-page sized allocations by default; the current malloc only does that when the 'G" options is used. I knew there was room for improvement, but I was pretty amazed that omalloc improves performance by tens of percents in some cases. The code is also quite a bit smaller and cleaner, in my opinion.

otto
During the hackathon, I implemented randomized chunk allocation and a trick to detect certain types of buffer overflows more easily by shifting small allocations to the end of a page. Damien Miller (djm@) did a review of the code and came up with a couple of good suggestions. The nice thing about a hackathon (apart from the bbq, beer and being able to work full time on things I like 100%) is that these suggestions could be discussed face to face, immediately, and so I was able to finish omalloc in far less time than I expected. Of course omalloc needs more testing, if you are running -current (and only -current!), you can help by giving the latest version a spin.

The other thing I worked on is fsck_ffs(8) for really large filesystems. fsck_ffs needs to store a few bytes of data about each inode in the filesystem being checked. Depending on the limitations of the platform, this means that filesystems having lots of inodes cannot be checked. Using a diff originally started by Dale Rahn (drahn@), I could reduce memory consumption by 20%. This diff has been committed. I am also working on a diff to actually not store all inode data in memory simultaneously; but this work is not yet ready. Although, in a certain test setup, I could reduce memory usage by a factor of 4 -- and there's more to be gained.

One of the small things I worked on with Ken Westerback (krw@) is disklabel(8). We are taking steps to make it present less confusing data in interactive editing mode to make it more understandable. This is also work in progress: it needs more tweaking before it hits the tree.

The hackathon has been fun and fruitful for me!


kettenis, deraadt and marco
Another person that I really was dying to meet was Marco Peereboom (marco@). I knew that he was working a lot in the past with David Gwynne (dlg@) who was sadly missing at this year's big hackathon but there in spirit. :-) David has learned so much from Marco and together they gave us blob free RAID management (bio(4)) and the sensors framework. Marco doesn't stop there. Now we have softraid(4), which is also really exciting; and more recently, support for softraid with crypto support! Oh, and that's not all!

He has done quite a lot of work with Jordan Hargrave (jordan@) and others to make ACPI work on OpenBSD for those poor souls that do not have an X40 and can't suspend. ;-) I brought along my evil ACPI machine, a Sony UX series computer, so that Marco could iron out some more issues with ACPI. I thought that it would eventually be a potential Zaurus replacement. Theo bought two of them last year to give to developers mainly because of the ACPI. Needless to say, it is pretty nicely supported now thanks to Marco and Jordan.

Here is what Marco had to say about his time at the c2k8 hackathon:

go skins
I came with the best intentions to do some softraid extensions that several folks have been requesting (rebuilds, marking dirty volumes clean, etc); however, ACPI took most of my time. Throughout the event, I was walking around debugging random laptops people handed me. The good news is that all laptops we debugged were no longer broken due to our code but due to hardware issues. I think we can declare a major victory in ACPI. Now what's left to do is get some drivers extended and port/write some others. Well, there is also the suspend thing that people seem to want. :-)

jordan, kettenis and marco
I did get to work closely with djm@ and hshoexer@ (Hans-Joerg Hoexer), which was an absolute privilege! Those guys are amazing. hshoexer@ wrote the softraid crypto implementation while djm@ added a new crypto algorithm that should work a whole lot better for block devices (AES-XTS). hshoexer@ pushed me to finally implement the delete softraid volume and thanks to that activity, I found an ancient bug that had been plaguing us (disks are shut down normally yet, they complain that they have unclean metadata upon reboot).

The other large project I started working on is rewriting the softraid metadata from scratch because we want to be able to deal with foreign metadata. Since it needs a rewrite, djm@ and I completely rewrote the softraid on-disk format. I am dealing with the fallout of that which is quite the task. This will keep me busy for weeks to come.

(c2k8 hackathon summary to be continued)

Thanks to Mark, Otto and Marco for sharing their experiences at c2k8. We plan to publish several more installments from c2k8 over the next week or so. Please show your appreciation by donating, buying a CD set or convincing the company you work for, that benefits from OpenBSD development, to contribute to the project.

(Comments are closed)


Comments
  1. By Kevin (70.75.1.233) on

    Awesome! Thanks for the update! Please keep them coming...

    Comments
    1. By Martin Toft (martintoft) on http://martintoft.dk

      > Awesome! Thanks for the update! Please keep them coming...

      +1 :-D

      In my world, they are bright spots in a month of exams.

  2. By Anonymous Coward (194.154.200.108) on

    Awesome reports.
    We really like these hackathons interviews and reports!!! Please keep posting them in the future!
    They are really interesting and give an idea of what's coming next in openbsd.

    Comments
    1. By Anonymous Coward (128.171.90.200) on

      The hackathon reports have been really great, undeadly has better articles, it puts a human face on the developers, everyone gets excited about features, and hopefully more people will donate more often in order to have more hackathons for more reports and so on.

      Comments
      1. By Luis Coronado (190.10.50.15) on

        > The hackathon reports have been really great, undeadly has better articles, it puts a human face on the developers, everyone gets excited about features, and hopefully more people will donate more often in order to have more hackathons for more reports and so on.
        >

        For sure, specially when I start to see things like this on my dmesg output:

        inteldrm0 at vga1
        info: [drm] Intel i845G GMCH (unit 0)
        [drm:pid0:drm_load]
        [drm:pid0:drm_addmap] offset = 0xff680000, size = 0x00080000, type = 1
        [drm:pid0:drm_ioremap] offset: 0xff680000 size: 0x80000 type: 1
        [drm:pid0:drm_ioremap] REGISTERS map
        [drm:pid0:drm_addmap] Added map 1 0xff680000/0x80000
        [drm:pid0:drm_agp_init] agp_available = 1
        info: [drm] AGP at 0xf0000000 128MB
        [drm:pid0:drm_ctxbitmap_next] drm_ctxbitmap_next bit : 0
        [drm:pid0:drm_ctxbitmap_init] drm_ctxbitmap_init : 0
        info: [drm] Initialized i915 1.6.0 20080312

        I hope this gets covered on a future post @undeadly with the corresponding bits on xenocara

        Great stuff.

  3. By Anonymous Coward (213.118.21.77) on

    This is Otto's post on tech@ about the new malloc implementation.
    http://marc.info/?l=openbsd-tech&m=120514094126123&w=2

    Otto is one of my favorite OpenBSD developers :-) (I'm sure that also has to do with he being Dutch)

    Comments
    1. By Miod Vallat (miod) on

      > Otto is one of my favorite OpenBSD developers :-) (I'm sure that also has to do with he being Dutch)

      Otto is no ordinary developer. He's the only developer whose name is written backwards (yet, for some reason, noone seems to notice this).

      Comments
      1. By Anonymous Coward (74.13.41.215) on

        > > Otto is one of my favorite OpenBSD developers :-) (I'm sure that also has to do with he being Dutch)
        >
        > Otto is no ordinary developer. He's the only developer whose name is written backwards (yet, for some reason, noone seems to notice this).
        >
        >

        He just does it backwards as a joke, like how I sometimes post things on message boards as though my board were a dvorak, but the map was qwerty.

      2. By Anonymous Coward (219.90.189.199) on

        ottO?

    2. By Anonymous Coward (155.212.216.106) on

      > This is Otto's post on tech@ about the new malloc implementation.
      > http://marc.info/?l=openbsd-tech&m=120514094126123&w=2
      >
      > Otto is one of my favorite OpenBSD developers :-) (I'm sure that also has to do with he being Dutch)

      That's interesting, how does it compare to FreeBSD 7's new(ish) jemalloc?

      Comments
      1. By Otto Moerbeek (otto) on http://www.drijf.net

        > That's interesting, how does it compare to FreeBSD 7's new(ish) jemalloc?

        I looked at jemalloc to see if was suitable for OpenBSD, but it lacks at least guard pages and randomization. Even after some effort to understand jemalloc, I failed. Maybe I gave up to soon, but to me jemalloc just left the impression of being overengineered.

        In contrast, djm@ was able to spot some improvements after reviewing omalloc for less than a day. For several reason I prefer a resoanable simpe and compact malloc implementation.

        Comments
        1. By Brynet (Brynet) on

          > > That's interesting, how does it compare to FreeBSD 7's new(ish) jemalloc?
          >
          > I looked at jemalloc to see if was suitable for OpenBSD, but it lacks at least guard pages and randomization. Even after some effort to understand jemalloc, I failed. Maybe I gave up to soon, but to me jemalloc just left the impression of being overengineered.
          >
          > In contrast, djm@ was able to spot some improvements after reviewing omalloc for less than a day. For several reason I prefer a resoanable simpe and compact malloc implementation.
          >

          So what kind of further work is required before omalloc will be imported into the tree? :)

          Nice job btw..

  4. By Venture37 (venture37) venture37<A>hotmail.com on www.geeklan.co.uk

    very cool guys, my system is now able to boot GENERIC.MP =)

  5. By Richard Toohey (121.72.23.169) knightofthecode@gmail.com on

    Thanks for the write-ups - will donate $100 as a sign of appreciation. Keep them coming!

  6. By Frank DENIS (82.224.188.215) on http://00f.net

    A lot of great things code was commited to OpenBSD lately. The hackathons have been very productive, congratulations to everyone.

    Unfortunately there is still one area where OpenBSD is clearly behind most other operating systems, and the gap is getting bigger and bigger : file system.

    A long time ago, there was an UBC branch. It was quite promizing, but unfortunately, it was abandonned because of obscure bugs that never were identified.

    tedu@ started to port UBC from NetBSD but the project was also aborted.

    So, in 2008, OpenBSD still has no unified buffer cache.

    Yes, UBC is difficult to get right and impossible to get perfect. But after some failures, Linux, FreeBSD and NetBSD now have something that's not perfect, but very efficient in most cases, and always better than no UBC in all cases.

    And to focus on the file system itself, while UFS is mature, stable, blabla, it's also getting obsolete.

    After a unclean shutdown, a foreground fsck that take ages to complete is lousy.

    No snapshot, no copy-on-write, no transaction. These features aren't gadgets, they are really useful, and sometimes required, if only to hot-backup a database.

    Yes, ZFS in FreeBSD is still experimental.
    Yes, HAMMER in DragonflyBSD is also not production-ready.
    Yes, BtrFS is still a work in progress.

    But still, the roots are there and UFS will probably not get any more attention in a few years.

    Comments
    1. By Ian McWilliam (202.7.166.181) on

      > A lot of great things code was commited to OpenBSD lately. The hackathons have been very productive, congratulations to everyone.
      >
      > Unfortunately there is still one area where OpenBSD is clearly behind most other operating systems, and the gap is getting bigger and bigger : file system.
      >
      > A long time ago, there was an UBC branch. It was quite promizing, but unfortunately, it was abandonned because of obscure bugs that never were identified.
      >
      > tedu@ started to port UBC from NetBSD but the project was also aborted.
      >
      > So, in 2008, OpenBSD still has no unified buffer cache.
      >
      > Yes, UBC is difficult to get right and impossible to get perfect. But after some failures, Linux, FreeBSD and NetBSD now have something that's not perfect, but very efficient in most cases, and always better than no UBC in all cases.
      >
      > And to focus on the file system itself, while UFS is mature, stable, blabla, it's also getting obsolete.
      >
      > After a unclean shutdown, a foreground fsck that take ages to complete is lousy.
      >
      > No snapshot, no copy-on-write, no transaction. These features aren't gadgets, they are really useful, and sometimes required, if only to hot-backup a database.
      >
      > Yes, ZFS in FreeBSD is still experimental.
      > Yes, HAMMER in DragonflyBSD is also not production-ready.
      > Yes, BtrFS is still a work in progress.
      >
      > But still, the roots are there and UFS will probably not get any more attention in a few years.

      Cool, so you have a BSD version of these filesystems for OpenBSD to implement. Can't wait for your inclusions.

      Let's all enjoy the revolution your going to provide. I can't wait.

    2. By tedu (74.68.146.146) on

      > A lot of great things code was commited to OpenBSD lately. The hackathons have been very productive, congratulations to everyone.
      >
      > Unfortunately there is still one area where OpenBSD is clearly behind most other operating systems, and the gap is getting bigger and bigger : file system.

      The entire hackathon hasn't been summarized yet...

    3. By Otto Moerbeek (otto) on http://www.drijf.net

      > Yes, UBC is difficult to get right and impossible to get perfect. But after some failures, Linux, FreeBSD and NetBSD now have something that's not perfect, but very efficient in most cases, and always better than no UBC in all cases.

      Funny, this hackathon saw code that lays the groundwork for changes to the buffer chache that might as well lead to a buffer cache having features similar to UBC.

      > And to focus on the file system itself, while UFS is mature, stable, blabla, it's also getting obsolete.

      Just the fact that there are other filesystems exist or are being developed does not make it obsolete.

      > After a unclean shutdown, a foreground fsck that take ages to complete is lousy.

      Sure, and we are aware of that.

      >
      > No snapshot, no copy-on-write, no transaction. These features aren't gadgets, they are really useful, and sometimes required, if only to hot-backup a database.

      IMO, by definition a real database has solutions to the problem of hot backups.

      >
      > Yes, ZFS in FreeBSD is still experimental.
      > Yes, HAMMER in DragonflyBSD is also not production-ready.
      > Yes, BtrFS is still a work in progress.
      >
      > But still, the roots are there and UFS will probably not get any more attention in a few years.

      For almost all uses, FFS/FFS2 is still pretty good. Call us conservative, but we are not jumping on any "great new filesytem" bandwagon before it is time.

      Comments
      1. By Anonymous Coward (81.83.46.216) on

        > Call us conservative, but we are not jumping on any "great new filesytem" bandwagon before it is time.

        That's why I like OpenBSD. Being a FOSS project doesn't mean you have to early-adopt every new bit on the 'market' IMHO. Maybe OpenBSD also doesn't have the resources to immediately implement all these bits too, but I for sure don't mind.

      2. By anonymous pedro (201.53.48.138) on

        > Funny, this hackathon saw code that lays the groundwork for changes to the buffer chache that might as well lead to a buffer cache having features similar to UBC.

        I'd like to make a small correction: the hackathon that first "saw" code laying the groundwork for what may eventually become OpenBSD's own implementation of UBC was f2k7, thanks to the efforts of two guys working for a German company. The diff was then committed at c2k7 (after 20-or-so attempts at getting it right).

        What was "seen" during c2k8 is, of course, just as important (or even more important, because it is a step further). But factually speaking (and it is always good to get the facts straight), it is a continuation of the work that was started back then.

      3. By anonymous pedro (201.53.48.138) on

        I don't buy this ditheistic view of development at all. There is a whole continuum between stability and innovation that can and should be explored.

    4. By anonymous pedro (201.53.48.138) on

      Frank,

      When I was a fellow at the project, I sent you an invitation to join us in the effort of changing precisely what you currently describe. We, the people with skills who believe such changes to be necessary, are the only ones who can make them come true.

      Unfortunately, you didn't reply to my mail. While I respect that position (and before people accuse you of blatant hypocrisy, let me assert that Frank has submitted a good deal of patches), it won't change anything.

      Comments
      1. By Anonymous Coward (212.129.63.1) on

        > Frank,
        >
        > When I was a fellow at the project, I sent you an invitation to join us in the effort of changing precisely what you currently describe. We, the people with skills who believe such changes to be necessary, are the only ones who can make them come true.
        >
        > Unfortunately, you didn't reply to my mail. While I respect that position (and before people accuse you of blatant hypocrisy, let me assert that Frank has submitted a good deal of patches), it won't change anything.

        Pedro,

        I probably don't have the required skills for such a critical code.

        While I made the initial QNXFS implementation and ReiserFS work for the Linux kernel, the VFS and the buffer cache of OpenBSD is something I really have very little knowledge about, and it looks complicated and easy to break. I admire that people like you, art@, otto@ and tedu@ were able to digg into it, but if after years, OpenBSD remains one of the last OS without UBC, it means that it requires really good skills to get it and I don't pretend to have those.

        Anonymous Coward, having COW filesystems and snapshots doesn't mean "early-adopt every new bit on the 'market'". Netapp and Linux have that for 10 years. Call that a toy if you never needed that, but it doesn't mean that it isn't a requirement for others.
        Yes, just talking about it just doesn't help at all. But I had to confess why I stopped a bit using OpenBSD lately. With regrets.

        Comments
        1. By Anonymous Coward (76.250.126.209) on

          NetApp has hundreds of engineers and their snapshotting is unique and unparalleled. OpenBSD has a couple of very smart guys writing code. I'd love to have WAFL but in the meantime I'll settle for something that works and is open source. Would I like snapshots; totally! would I trade something that works well for crap? never!

          Foreground fsck would be nice if done right; unfortunately TheotherBSD's foreground fsck cripples the box and is essentially the same as not having it.

          I am baffled that you even bring up crap like ReiserFS; why in the world would you trade a well working FS for some crap like that? That is simply feature chasing. ZOMGJOURNALING!!!

          ZFS is awesome but has a shit license. You could write a port though. It is not very hard to write a module that loads the kernel pieces.

          UBC is mostly overrated. I know that is an unpopular stance but UBC buys one less than advertised.

          Comments
          1. By Anonymous Coward (212.129.63.1) on

            Your troll is outdated.

  7. By Anonymous Coward (85.179.239.15) on

    Is someone currently working on getting SATA hotplug to work with ahci(4)? This would be quite a killer feature for me, to be able to use my eSATA disks with OpenBSD.

Latest Articles

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]