OpenBSD Journal

Out With the Old, in With the New

Contributed by tbert on from the out-with-the-borrowed-in-with-the-blue dept.

Ted Unangst (tedu@) has given out a blog post detailing some of the recent work going into OpenBSD:

Notes and thoughts on various OpenBSD replacements and reductions. Existing functionality and programs are frequently rewritten and replaced for the sake of simplicity or security or whatever it is that OpenBSD is all about. This process has been going on for some time, of course, but some recent activity is worth highlighting.

Itís probably worth preemptively citing jwzís ďCascade of Attention-Deficit TeenagersĒ. It certainly is appealing to throw everything away as a bug disposal mechanism. As noted, this rarely has the intended effect and just replaces one set of bugs with another set. The rewrites mentioned here have a slightly different motivation. Instead of trying to fix known bugs, weíre trying to fix unknown bugs. Itís not based on the current buggy state of the code, but the anticipated future buggy state of the code. Past bugs are a bigger factor than current bugs.

An example that didnít receive much attention because itís not user visible is the replacement of the isp SCSI driver. The isp driver supported a wide variety of QLogic interface cards. Unfortunately, support for the newest generation of hardware didnít work, and efforts to improve it frequently broke other hardware models. The driver tried to do too much and as a result was difficult to maintain. Not all similar looking code is really meant to be shared. Eventually, the isp driver was replaced with three different drivers: qla, qle, and qlw. Each driver can now be modified in isolation without unintentional side effects on other hardware, or the need to consider if and where further special cases need to be added. Despite the fact that these three drivers duplicate all the common boilerplate code, combined they only amount to about half as much code as the old driver. More is less.

The reverse effect can be seen in the evolution of mandoc. Ingoís talk about mandoc becoming the main BSD manual toolbox has all the details, but itís interesting to see one of the most unixy parts of the system succeed by following a very nonunix path. What could be more unix than having to fork and exec and pipe the output of a half dozen programs to put some text on the screen? Now weíve got this slicing, dicing wunderprogram. Even so, itís smaller and simpler than the accreted mass of what it replaces, but it does more and does it better. Less is more.

Much has been said about httpd already, and how itís either the best or worst. Overlooked in Reykís httpd paper is a quote from another OpenBSD developer in regards to SPDY support. ďIs there a reason to believe this specific piece of code is worse than any other?Ē The unnamed developer was actually me. Not too long after I asked that came the 1.5.10 SPDY memory corruption bug followed by the 1.5.11 SPDY buffer overflow. To their credit, nginx did mark the feature experimental, and it was OpenBSD that enabled it, but the lesson for me was to ask the right question. Is there a reason to believe that if a feature exists, it has bugs? Yes.

(I asked the question without knowing SPDY was experimental. ďItís experimentalĒ would certainly have been a good answer, but communicating was hard. Inside OpenBSD, there was a fair amount of confusion about what features were enabled, or should be enabled. Once you know a feature exists and can be enabled, how can you not choose all the Pikachus?)

From the beginning, the most commonly requested httpd feature has been rewriting. Reyk resisted adding it, though, since it more or less required regex support, and now you have two problems. At BSDCan it came up again, and I mentioned Lua patterns. Theyíre much simpler than regular expressions and can be learned quickly, but in my experience are more than powerful enough for most tasks. True, if youíre already a regex swashbuckler, itís yet another thing to learn, but itís really just a matter of looking up the syntax for character classes. On the other hand, for a user setting up a web site with simple needs... Compare re_format(7) and Lua Patterns. It only took Reyk about a day to hack something together, and shortly after commit.

The file utility is a curious beast. Almost by definition, its sole input will be untrusted input. Perversely, people will then trust what file tells them and then go about using that input, as if file somehow sanitized it. You donít trust the source to tell you what something is, but nevertheless trust whatís inside it? Regardless, if a program should be hardened against attack, file would be a good choice. So Nicholas rewrote it. The new version endeavors to be safe first, accurate second. Given the ease with which chimeric files can be created, Iím not even sure accuracy belongs on the scoring rubric.

These examples have been about code, but as often as not itís about user experience as well. Stop and consider XKCD: Manuals. The best tools are the ones which do not require (extensive) manuals. I like to think signify is in this category. The xkcd hovertext regarding sudoers is certainly appropriate, since sudo is moving to ports. As hinted later in the thread, I have a replacement utility doas which is very tiny. Perhaps too tiny. Weíll see. This is not the doas retrospective post.

Popularly cited is also Joelís never rewrite article. A few comparisons. First, business customers may revolt if future releases have fewer features, but many OpenBSD users welcome less if it means simpler. Second, ďthey had never stopped working on the old code base, so they had something to ship, making it merely a financial disaster, not a strategic one.Ē Excellent point. Most OpenBSD rewrites are skunkworks kinds of projects, at least to start. Todd didnít stop updating sendmail while smtpd was in development. New code can reduce the enthusiasm for maintaining the old, but frequently the old and new are worked on by different developers.

Joelís warning anecdote about the untidy two page long function does parallel something that happened early in doas development. Instead of copying the sudo privilege switching code, I wrote my own. As a result, there was a bug because I forgot to mess with the supplemental groups. I should perhaps have copied the sudo function, but I couldnít find it. The good news is that the replacement code was easily read and the bug quickly identified. Code does accumulate fixes for obscure edge cases over time, and there is a real cost when a rewrite loses them.

As with jwzís post, Joelís article focuses on feature equivalent rewrites. One can try simply reducing functionality in an existing codebase, but then thereís a lot of scaffolding left over. YAGNI (you arenít gonna need it) in reverse. You did need it, but now you donít. Like extreme liposuction, the end result is better than before, but itís still not pretty. Start from scratch, add things back, and maybe find even more unexpected fat.

Pruning and Polishing covered some complementary material. In particular, the polishing section mentions the benefits of homogenization. All of the above examples (isp, nginx, file, sudo) were stylistic outsiders. Letís pretend rewrites are extreme polishing, like sandblasting. The replacement source code now feels like OpenBSD code.

(Comments are closed)

  1. By Anonymous Coward ( on

    Here's a link to tedu@'s original blog post:

    1. By tbert (tbert) on

      > Here's a link to tedu@'s original blog post:

      Ouch; this one's on us. The intention was never to not attribute this to Herr Unangst, merely an editorial mistake.

  2. By Anonymous Coward ( on

    The link to jwz's "Cascade of Attention-Deficit Teenagers" is incorrect. The correct link is

  3. By Blake (2a01:e34:ec06:8f90:5a55:caff:fef5:2a75) on

    That was awesome, thank you.

  4. By Anonymous Coward ( on

    Obligatory comment. I hope you understand :)


Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]