Date: 2015-10-30

On October 18th 20 years ago the first commits to the OpenBSD project landed in the CVS repository. Today on the anniversary the beastie.pl team invites all readers to a series of interviews that our staff conducted with the project developers.

We continue with our thirteenth interview - Henning Brauer.

1. For the readers who don't know you, can you shortly introduce yourself?

I'm Henning, not 20 any more, OpenBSD developer since 2002. I architected & wrote large parts of pf, started, architected and wrote large parts of bgpd and ntpd. The imsg & privsep framework I wrote for bgpd is in almost all newer OpenBSD daemons. I also worked a lot in the network stack, including many redesigns. One of the last bigger projects I did was the replacement of the queueing subsystem.

2. Why did you choose to run OpenBSD? How long have you been using it?

I started using OpenBSD around 2.6 or 2.7 times. Basically, we had a DoS attack against a linux webserver and it behaved poorly. Around the same time a colleague was telling me about their very positive experience with switching their Nameservers to FreeBSD, so I looked at FreeBSD, and OpenBSD + NetBSD while on it. OpenBSD behaved very well against the replayed attack and generally attracted me, things were just Done Right.

We had a pair of OpenBSD firewalls with ipf shortly after, with manual failover of course, nothing like carp was even remotely on the radar - at these times, a "cold standby" scheme was common for many things with higher availibility requirements. I had performance problems. When OpenBSD 3.0 with the first pf incarnation was published, I tried it right away. It was very fast - both at runtime and to crash. In a long debugging session with Daniel Hartmeier we eventually found the problem and fixed it; from that point on I was running pf instead of ipf and the very same machines that were pegged on CPU for hours every day peaked at some 10% CPU load. I wrote an email to misc@ giving a short rundown of the story, and received an invitation to the upcoming hackathon from theo in return. They haven't let me off the leash since then :)

Honestly, besides the incredible technical expertise in the OpenBSD hackers group, this is just and awesome crowd that is a lot of fun to work with, and more than just work with. It is still an honour for me to be part of this group.

3. For those readers that still haven't joined the OpenBSD community, why should they try OpenBSD?

Well, I'm not into marketing really, but I can easily tell why I use OpenBSD almost everywhere.

Reliability.

That goes much further than "not crashing". Running internet-facing services with often high availability requirements means you need a software stack that just works, all the time, and doesn't surprise you. "Just works" here includes a very very important bit: not getting taken over by third parties. Security is critical.

Reliability also means that the software stack has to be user friendly in the sense of not surprising the admin - you neither want sudden reboots, sudden death of daemons providing essential services and so on. The consistency that OpenBSD delivers gives me that, whatever I do in the scope of the OpenBSD base system doesn't lead to surprises but works exactly how you expect it to (and minor derivations are getting fixed when encountered). Having to add a tiny bit to a production system that shall not fail and not having to fear that this addition causes havoc is worth a lot.

I could write on and on, there is so much I count on the positive side. Very noteworthy before I stop to not digress too much of course is the security OpenBSD delivers. I don't have to worry about machines being taken over while I have beer with friends, sleep, sit in an airplane, or whatnot. OpenBSD's defense in depth approach, last not least through the many mitigation techniques, helps a lot there, even with 3rd-party software that is sometimes questionable from the security standpoint but you can't escape from - php probably being the prime example. Getting all that without major performance impact is impressive.

4. Is OpenBSD your daily driver at home & at work?

Yes, my workstation at work runs OpenBSD (and even hosts my personal homepage), so do all my laptops. I have work to do and don't want to fiddle with the OS in the process, it needs to Just Work. OpenBSD gives me that.

The vast majority of the many many many many machines at work also run OpenBSD. The fact that these don't need babysitting but generally just require attention for upgrades every now and then makes a real difference. Of course we have extensive monitoring to not run into surprises, but generally, OpenBSD machines required very very little manual maintenance.

But there's so much more. Even my little media server at home runs OpenBSD. The microcontrollers driving the door locks at work interface with OpenBSD, so do the ones controlling the lights. All money runs through OpenBSD - the bank interfaces run on it. So does all billing including invoice generation.

I think it is pretty appropriate to call OpenBSD my daily driver.

5. How did you become an OpenBSD developer? What do you think is required in order to join the OpenBSD project as a developer?

Well my entry point was, as mentioned above, pf - me apparently being one of the first ones to try it in a bigger production setup.

A common saying in OpenBSD is that you receive your account when other developers get annoyed by the number of diffs of yours that they commit. At some point you get the commit bit to do it yourself. So to get involved, you just get involved. Fix problems, annoyances, inconsistencies when you encounter them and send the diff to tech@, same goes for missing functionality. Your diffs usually either get committed or you get valuable feedback.

6. Can you tell us about some OpenBSD-related areas you work on?

"everything networking" is probably closest.

When I run into shortcomings (from my subjective PoV, of course) I try to work on those too where feasible - an example is the template support in disklabel for autoinstall with custom partitioning I did earlier this year.

7. Do you have an idea of the time you spend working on the OpenBSD project?

I don't do any time accounting - after all, OpenBSD hacking is supposed to be fun.

The amount of time I spend on OpenBSD varies a lot. In the last couple of months it was very very very little unfortunately due to extraordinary workload and some health issues, I hope that'll change soon.

In the good ol' days (tm) it was much easier to allocate time; I wrote the initial bgpd from zero over the buffer, imsg and privsep frameworks to the point where session management was reasonably there, i. e. could talk to peers, keep sessions up thru keepalives etc, in 2 or 3 weeks. Over the last years the bigger projects happened mostly at or around hackathons. These are very important for OpenBSD, the progress you can make by throwing a bunch of us in a room for a coupe of days straight is amazing - and it is a lot of fun.

8. OpenBSD tends to lead in development best practices does it work the other way around? Is there a process improvement the project started or aims to adapt from the oustide world?

We don't tend to have that much "process" written in stone. Last not least keep in mind that we're all volunteers and do this for fun. OpenBSD is not an isolated remote island for and by itself, so of course there is influence both from and to the outside world. I bet you'll get 10 different answers (being examples, really) on how OpenBSD influenced the outside world when you ask 10 developers. Similarly, almost every developer brings his own experience and ideas into OpenBSD.

9. It's been a long 20 years of amazing releases. What are you most proud of and what would you like to revisit/redo?

Personally probably most proud of pf, last not least on how far it spread. I can't envision something as great as pf evolving in another environment than OpenBSD - the sheer number of developers who have contributed/committed to pf is pretty impressive already. But it really is much more than just the hard facts, it's the environment of smart people that are easy (and fun!) to work with.

10. As a conclusion, can you tell us how you forecast OpenBSD's future? What's the next big challenge?

Forecasts are always hard - esp. covering a nontrivial timeframe. I see no indicators for OpenBSD losing traction or anything in that direction, rather the opposite. I am very sure that OpenBSD will stay OpenBSD and not allow i. e. a large corporate user to steer it. Basically the freedom to do "the right thing" versus having to follow commercial interests.

Shorter term one of the bigger challenges is better SMP in the kernel, from my PoV unsurprisingly last not least in the network stack. There has been great progress already, which I unfortunately couldn't contribute too much to, yet.

I do see a need for improvements in the file system area, foremost more flexibility.

Longer term it's much more about general trends than specifically OpenBSD. We see the environment we operate in changing quite drastically these days. With IT as a whole becoming more mature some of the "written in stone" rules count less and less, some will become irrelevant. The widely deployed line of thought that you set up a bunch of machines for a certain task and re-do everything a couple of years later vanishes; the need to replace hardware frequently is about to vanish (where it hasn't already) since you don't need to buy new hardware every couple of years due to performance requirements - in many areas, at least. The trend towards virtualization is part of that: a current entry-level server is way to beefy for many many many tasks, so one way to make efficient use of its resources is virtualization. I predict that we'll see setups being around MUCH longer than what we're used to today. The virtualization has another side-effect - shifting VMs from one host to another is trivial, while today it is pretty common to do fresh installs on new hardware when the hardware is changed. We'll see much longer-living installs not just due to longer hardware life, we'll see that to a whole new level with VMs that just get shifted to new/other hosts/platforms. That makes maintenance a much bigger deal, a system that is totally cluttered after a few years, with nobody being able to really grok what's going on is an increasing problem with age of the installation. OpenBSD's simplicity, consistency and great management infrastructure help a lot there.

At the same time, what used to be exclusively IT stuff spreads into many other areas. We see computers controlling all kinds of everyday things, from trains to airplanes, from your heating system to power plants and traffic lights. Especially in these roles we won't see frequent upgrades in the sense of replacing the setup - a plane usually flies for 20+ years, a train often reaches 40 before retirement, and even today we already have power plants controlled by ancient PDP11s. In these use cases you also seldom see an IT admin team taking care of the computers, they are just a part of the bigger picture. At the same time we see these things getting networked more and more, including being connected to the internet. That in turn makes security even more important than it already is. As bad as a taken over website is, a taken over power grid, train, or airplane is MUCH worse.

The increasing use of commodity IT equipment in these more or less embedded roles where we typically don't see an administrator team dealing with software upgrades etc has a lot of consequences. It doesn't have to be something as obviously important as the power grid - in the majority of cases, much lower profile, these systems will need to become way more "self maintaining", in the sense of applying updates and especially security fixes largely automated. There will be staff at least supervising the process for power plants, airplanes, trains and the like (at least I really really really hope that...), less so for less prominently exposed use cases. The alternatives for the majority of these cases won't be "automated maintenance" versus "manual maintenance", the option in way too many cases would be "unmaintained" (call me disenchanted if you like).

That all said, the maintenance automation doesn't necessarily have to be part of the base OS, so large parts of that might happen in an OS-independent way, of course with OS specific bits to keep care of the OS itself. But the OS needs to be automation friendly. And guess what: OpenBSD already is. Of course there is still room for improvement and I generally expect the automation bits to gain importance over the next decade; I think OpenBSD is well prepared for that.