OpenBSD Journal

[c2k10] Interview with Henning Brauer (henning@) Part 11

Contributed by mtu on from the where-beer-spawns-daemons dept.

p1040376s Henning Brauer (henning@) has been an inspiration to me since I started using OpenBSD. On misc@, there are a few individuals that I follow. Nick Holland's (nick@) posts are very educational and informative and I try to read Henning's as well. His posts are usually to the point, authoritative and often funny. When you meet henning@ in person and get to know him better, you learn that he has a great sense of humour, that is, if it is not evident already on misc@. He also makes the most out of life even when he misses a step. He has the scars and crutches to prove it. Well, what doesn't kill you...

Read on to learn why I think that he's a great inspiration to me and what correlation there might be between beer and daemons:

[c2k10] Article Series: 1 2 3 4 5 6 7 8 9 10 11 (more to follow)

Henning started using OpenBSD around the 2.7 Release; a very popular release for many developers. At around the same time but after having spent four years as a software developer, he started his own ISP. Not long after starting this new venture, he was hit with a nasty attack on one of his Linux servers. He then turned to the other BSDs for options. He replayed the same attack on FreeBSD and OpenBSD, with the latter handling the attack much better. It didn't take him too long to realize that OpenBSD was a better fit for his needs.

P6280147s About two years later in 2002, he made is first commit as an OpenBSD developer and over 90% of his commits have been to src ever since. He got his cvs account because of pf(4). Back then, ipf had terrible performance problems, at least with his setup as he was seeing the CPU hit 100%. Then came pf but it was less stable, less consistent and didn't have as good docs. He thinks that he was the first to run pf in a bigger setup and as a result, he would see crashes seconds after boot. After Daniel Hartmeier (dhartmei@) submitted one bug fix that he and henning@ had worked on together, henning@ reported the success to misc@ and then got a reply from Theo with an invite to the hackathon. After that pf worked more reliably than ipf as he was seeing only 10% CPU utilization on the same hardware that he used to run ipf on with 100% CPU utilization before the switch to pf.

When I read of all that henning@ has done for OpenBSD, you would think that he was a child prodigy and at least one of his parents was a programmer but no this is not the case. In fact, when he started hacking on pf, he had only a basic understanding of C. He says that not knowing very much C has its advantages. He goes on to say that "OpenBSD is the best community to learn C". One of his most recent talks was on secure programming, the OpenBSD way.

The key to getting involved with OpenBSD is to find something that bothers you and work on it until it is fixed. Alternatively, find something to work on that you are passionate about. Theo and others can't tell you what to do. They know in what areas they could use help and they know what resources such as hardware that would help speed up the development process but that's about all. We now know how henning@ got involved but what motivates him to continue making OpenBSD better? He enjoys spending his free time doing things that are fun. Sounds simple enough, right? Though, not all of it is fun.

Bret Lambert (blambert@) would say, "the developers enjoy removing 20 odd years of bad Californian ditch weed from the kernel". Fixing issues is generally a monstrous endeavour and the success feeling when it is eventually fixed becomes a big part of the motivation. However, when you need to get lost in 20 years of old code with nasty and painful code/problems, you soon realize that it's not all fun. But there are really two different types of pain and it is all relative between the two. Redesigning is much harder and less rewarding than designing something from scratch. A good example of this is the redesign of pf(4) compared to writing bgpd(8), both of which henning@ was intimately involved in.

p1030678s Redesigning pf started in 2005 on a ferry in Canada. There were a couple of years of straightforward evolutionary development and then several years of very painful under-the-hood changes to pf. After the pf redesign pain, visible changes were possible. In contrast, bgpd was more fun, easy and less pain than the pf redesign. Yet despite the pf redesign pain, the long build up of changes paid back after so many years and that, he said, was very rewarding. Working on daemons was really pain relief for henning@. Of course, bgpd also helped to solve issues that he was dealing with at work but it was a chance to step away from the pf pain for a bit.

The funny bit about the origins of bgpd was that he had contemplated writing a new daemon but thought it was way over his head. Then he had a few beers with Theo and made the mistake of telling him that he was thinking of redoing a routing daemon. When I heard that story, I had an epiphany and then understood why drinking is a big part of the OpenBSD development process. Yes, beer is good for development. It's the reason OpenBSD keeps adding more daemons to every release.

These hackathons are also a big part of why OpenBSD is so unique and why the development community is so special, at least to henning@. Annually, he participates in his fair share of hackathons, big and small. You'll often hear him say, "I'm here for hacking and hurling." And what of the people? He says they are the best group to be around and he would even come to a hackathon just to socialise with these guys even if he couldn't hack. That's how fun they are to be around.

P6280111s He goes on to say that hackathons are very important as they help you understand the big picture and then you work through the details. It's a place where you are not afraid to make substantial changes. Yet, it is not all about the touch frequency of your fingers to the keyboard. Henning typically spends a long time designing before coding. A good example of this was when he was designing bgpd. He took a black box approach but he had a rough design in his head even down to the API. Before starting to code this, he wrote the buffer framework that is below the imsg framework used by so many daemons today, as well as the logging framework. After this, he finally wrote the session engine and a considerable part of the parent process. At this point, the session engine could talk to other bgp daemons but ignored the routing. Now, he had something to show others, so he sent it to a few people including Claudio Jeker (claudio@) who picked up the ball and by the end of December of the same year they had something working. bgpd was eventually added to OpenBSD in 3.5 Release thanks to beer, pain relief and the fine collaboration efforts of both henning@ and claudio@.

Not to be out done by Reyk Floeter (reyk@), at another hackathon and while drinking a local brew known to developers as Trad, henning@ was once again joking around with others and said that they should rewrite ntpd, the OpenBSD way. He said that he could do the network magic so long as someone else was able to do the time magic. Right then and there, another mini project was born. Unfortunately, the time magic developer didn't stick around for very long and henning@ was stuck having to complete both. Thank goodness he was still in pain relief mode. He eventually got help from others to complete this new daemon and in 3.6 Release, ntpd came to be. Other NTP implementations were very complex but the ntpd that henning@ rewrote was simple with much less code.

Asked why he helped to improve dhcpd (including dhcp client and dhcrelay), he said that the alternatives were bloated and without OpenBSD's signature privilege separation and revocation techniques.

Since the summer of 2002, henning@ has worked on pf, bgpd, ntpd, zlib, dhcpd, mopd, httpd and more.

As henning@ had his hands in the development of so many daemons, I asked what advice he could give to others about software development. When implementing change, he said that it was common to have an idea about the change and spend a lot of time thinking about it. He generally makes bigger complex problems into smaller simpler problems. He says it's best to write this down and then convert it into C. However, for henning@, rather than write it down, it stays in his head before converting it to C. Interestingly, when done, it vanishes from his memory. Also, if the basic design is wrong, then throw it away and start from scratch. However, he says writing new code like bgpd is way easier than changing existing code but reimplementing it is not the solution or even possible except when done in smaller steps. Needless to say, you will have bugs.

This is what henning@ had to say about his work at c2k10:


I didn't have all that many plans for c2k10. I wanted to go through the PR database and look for pf and networking related PRs and fix a good share of them. I did so for the pf related ones, but that was really mostly low-hanging fruit.

On the boilerplate was fixing pflog. I thought that it would take me 2 or 3 days and I could maybe resurrect my work for ifaddrs in a red-black-tree - walking that single-linked list for every packet isn't all that cheap with a few more addresses in the system. I didn't get there, I didn't even finish pflog.

pflog is a bit weird in the sense that we pass the live mbuf (-chain, really), as in, the one describing the packet pf is working on at that given time to bpf and bpf further to userland which in turn means that we should not (I'd even say must not) modify that mbuf. Before the NAT code rewrite, we were mostly ok. Whenever NAT changed an address, that was rewritten back to the mbuf pretty much instantly, before we passed it on to pflog. Nonetheless, the pflog code used to modify the live mbuf. I find this unacceptable. One of the many downsides of applying the address rewrite immediately is that we have to undo it later when it turns out that we have dropped the packet and need to quote parts of it in the icmp reply. That was error prone and we had bugs there more than once. Not to mention it is also slower than writing back late once.

With the new style NAT, we can nat and rdr more than once. Now it makes even less sense to copy the headers back for every change. Furthermore, since nat-to and rdr-to are now an integral part of the regular ruleset and with no separate step beforehand, there isn't that one natural point where the copyback should happen to be certain to do it before passing it on to pflog.

What happens right now with the pflog that I am working on is that we pass the mbuf up BEFORE doing the copyback. As in, we log with the original addresses and ports instead of the rewritten ones. Fixing this could be a matter of moving the logging later, but that conflicts with another change we want: whenever we run into a match log rule, we want to log immediately with the addresses as seen by pf just after that rule is applied. We can't do that for pass since only the last matching pass rule counts and we must not log on intermediate pass rules... can't change that behaviour without screwing the large installed base, so we won't.


So my idea: bpf is copying anyway. Hook the rewrite & copyback in there, operate on the copy bpf just made. First step was to refactor code, again. Namely, pull out the setup of the pf_pdesc structure from pf_test and pf_test6. That pays out regardless of the pflog changes, making the code clearer and easier to manage. In fact, when Ryan was reviewing my diff, he got me over to his screen pointing at remaining code in pf_test asking whether we can fold the 4 copies there into one... turns out we'll be able to. This was way more painful than I thought. The diff file sent around was named pain.diff for a reason.

Next, was to step into bpf which always hurts. I made a custom copy function and a new bpf entry point for pflog so that we can use it. I extended this function to handle the rewriting. I get one mbuf at attach time that I recycle over and over again, with its data pointer pointing to the bpf storage area, then calling pf_setup_pdesc and pf_translate on it. This mostly works, but has some bug (bugs?) left to be found & squashed. As in, I didn't finish. Once this is done we can make pf log immediately on match log, as mentioned above.

And then, I need to go back to ifaddr RB...

[Editor's note: henning@ completed the pflog work that includes the "log immediately" at j2k10 in Japan and added some other interesting goodies as well.]


Now if you were sitting on the fence wondering if you too could ever become an OpenBSD developer, then I hope that henning@ becomes your inspiration. If you know that you'll never be a developer but rather a devoted and appreciative user, then there are other ways you can contribute to the OpenBSD project besides buying T-Shirts, posters and CD releases or making donations. The developers give their time. You could do the same. I'm sure that Jason McIntyre (jmc@) can use help with documentation and Nick Holland (nick@) could use help with and undeadly could use some more editors. The project moves because people like henning@, jmc@, nick@ and many others who care and do what they can to make it better. Just get involved and make a difference. Thanks Henning for all that you have done and continue to do for the OpenBSD community. We appreciate this very much.

Mark T. Uemura

(Comments are closed)

  1. By brynet (Brynet) on

    He just won't stop staring!!

  2. By Nick Holland (NickHolland) on

    Many years ago, before either Henning or I were a part of the OpenBSD project, he and I privately chuckled about the two of us becoming members of the development team. Henning's statement to me: "I really don't have anything to contribute to the project".

    Silly Henning. :)

    1. By brynet (Brynet) on

      > Many years ago, before either Henning or I were a part of the OpenBSD project, he and I privately chuckled about the two of us becoming members of the development team. Henning's statement to me: "I really don't have anything to contribute to the project".
      > Silly Henning. :)

      Now that's perseverance, both of you wanted to be part of the project, and you made it happened.

  3. By Lennie (Lennie) on

    Hi mtu/other undeadly writers/editors,

    In the article it says:

    "One of his most recent talks was on secure programming, the OpenBSD way." which points to a presentation on PF, did you maybe mean this presentation ?:

Latest Articles


Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]