The OpenSMTPD audit, a debrief

EHLO folks,

As you can probably figure this has been a rough week and we didn't even have time to debrief on our own mailing-list, so now is the time.

I've been asked a year ago if I was interested in having a code audit by the guys from Qualys. Having people actively searching for issues in the code for us and helping us fix them proactively was just too good not to accept given our small man-power.

They found several issues, some that were introduced in early days, long before the project was committed to OpenBSD, some others that were quite stupid, and some that were subtle enough that I had to reproduce them to be sure that the issues were for real. They did a _real_ good job.

The report taught us a few things and helped us spot weak points that we will work on hardening to make sure. I'll summarize a bit, the audit was disclosed publically and I encourage you to read it.

First of all, on the positive side, privileges separation, chrooting and the message passing design have proven fairly efficient at protecting us from a complete disaster. Worst attacks resulted in unprivileged process being compromised, the privileged process remained untouched, so did the queue process which runs as a separate user too, preventing data loss.

On OpenBSD some attacks were further mitigated at system-level thanks to the stack canary and the hardened malloc. Moving some of our stack based buffers to the heap would have made some of the attacks even harder.

This is good news, we're not perfect and bugs will creep in, but we know that these lines of defense work, and they do reduce considerably how we will suffer from a bug, turning a bug into a nuisance rather than a full catastrophe. No root were harmed during this audit as far as we know

Then, the report has identified some weak points that we are going to be working on primarily from today. The offline queue, for instance was not designed correctly. The attacks succeeded because of missing checks, but with a proper design these missing checks would have been harmless. This required a bit of refactor and is now fixed in both cvs and git. I'll do a release just for this in the next few days.

The report also identified a lot of indirect issues, ones that aren't an issue until they find another bug first. A lot of these are based on the trust between our processes, each one assuming the messages being passed by another were correct.

We had already started work to remove these assumptions and perform very strict inter-process checking, unfortunately we have not converted every imsg call to this API yet. We're working on that so this should never be a problem in the future.

Other bugs, well, errors from 2 developers working on a big project over an extensive period of time. No mechanism but peer-review will avoid it.

Now, what to expect from us from now:

I was waiting for the audit to come out before bringing the filters code to OpenBSD. The refactor was huge and caused us to maintain way too many different branches. We're now going to essentially freeze features, work on bringing master back to OpenBSD progressively by micro-diffs, then we can work back on new features using the OpenBSD tree as our master as we used to do before filters. Bonus: other hackers get to peer-review.

This was discussed before the Qualys advisory came out, it is clear that we have been off-tree for too long, and it is putting a lot of burden on our team.

We're going to release 5.7.4 shortly with some security improvements and a few assorted minor bug fixes, then freeze the 5.7.x branch.