OpenBSD Journal

OpenSMTPD Update

Contributed by tbert on from the back-from-the-dead-letter-office dept.


Artwork by Marie Mantopoulos

Gilles Chehade (gilles@) sends us the following:

I was asked if I could write a small article about OpenSMTPD, what we've been up to since last article and what we plan to do still.

It turns out that last article, if you omit the small SQLite blog post, is from 2009 and there's been waaaaaay too many things to mention.

So here's a short overview of the last few months, the current state of things, the most important features we came up with and the things that are planned for the next weeks and months ;)

He's taken the time to round up the entire mail-moving crew, with Eric Faurot (eric@) and Charles Longeau (chl@) pitching in to get us up to speed.

Gilles:

Obviously the main work I've done on OpenSMTPD was to bring it back to life.

During almost all 2010, the project had seen no commits besides very occasional bugfixes; testers and contributors had left and by the end of the year, people were starting to ask if the project was dead and if it was going to be removed from the tree.

Early 2011, I really got annoyed with the situation and decided to get back maintainership of the project. I started by removing some optimizations that were always getting in the way of moving things forward; I broke the project for days to put it back in a shape where I could hack on it without wasting days working around a method to save a few bytes here and there.

Eric:

My involvement in OpenSMTPD is a bit accidental. It started when someone (I don't remember who exactly) told me that the OpenSMTPD developers might be interested in the async resolver I have been writing.

So I contacted gilles@ and told him about my code. He was really excited about it, as the existing code for DNS lookup was a real bottleneck, resource- and performance-wise. So, it got imported in no time, and we quickly fixed the few remaining issues.

Gilles:

At that time, we did not have an asynchronous resolver so any blocking I/O on DNS lookup could cause OpenSMTPD to hang sessions. To avoid this, we had a fork()-based hack to run each lookup in a new process and have it communicate back the result through imsg when it was available.

The cool part was that it was planned that this hack would go away some day and we had it hidden behind an API that allowed us to integrate the new resolver without too much pain.

Eric:

At that time I had really no idea about OpenSMTPd internals. Since I was there, I started to dig further into it out of curiosity. I liked the overall design, but there were lots of places where the code was overly complex. It felt like the code was designed with provision for future features, but the problem is that it lead to a point where the existing bugs were not easy to fix, and new features were too complex to add. At that time, developement had basically ground to a halt.

So I started to send lots of small patches to clean things up: stop passing the global environnement around everywhere, simplify statistic counters, add traces to better diagnose the flow of messages between the different processes. I also fixed some reliability issues, such as preventing the server from fork-bombing itself at startup of it had lots of offline messages to enqueue.

Gilles:

People started mailing me to ask if we had diffs they could test, and after a few weeks the project became as active as it used to be, with testers and contributors sending diffs every now and then. I can't stress enough how much we rely on feedback, because there are so many different setups out there that it is impossible to test reliability without an active community letting us know if it works for them.

Charles then got in touch with me to ask if I had moved the portable version forward. At that time I had a portable version that was lagging behind as my main interest was OpenBSD. He asked me if it bothered me that he was working on a portable version and quite frankly it was a relief :-)

A portable version means more users to test and report bugs, precisely what we needed !

Charles:

My main work on OpenSMTPD is to maintain the portable version. I had been working on it for some time, but only announced around September 2011 that it was supposed to work on *BSD and Linux. It's mainly based on the "compat glue" stuff in portable OpenSSH.

The goal of having a portable OpenSMTPD version is to bring new testers to spot bugs in the non-portable version.

After a few months, we had reports about users using it on *BSD and Linux, and even on a Nokia N900 (hi todd@!), which makes portable running on portable :)

Gilles:

I then focused on simplifying and abstracting some APIs to allow experiments and custom backends to be written more easily. I started with the maps API, which supported plain text files and db(3). Both were already hidden behind a common interface, but adding new backends meant modifying a set of functions and making them less readable. I changed the API so that each backend was isolated and the interface would select the proper backend based on configuration.

Once it was done for maps, I started adapting the same method for all of the mechanisms in OpenSMTPD where a user would enjoy having custom backends. It started with the queue, that allows us to replace the filesystem storage with different layouts or completely different storage. Then moved to the delivery where we could easily write a new backend to replace mbox, maildir and mda.

Eric:

I managed to convince gilles@ to dump envelopes on disk in plain text. This was especially important as the old binary structure made it almost impossible to debug the queue correctly.

Gilles:

After Eric and I worked on getting plain text envelopes working, during a hackathon at Miod's(miod@), I wrote the last abstraction which allowed us to write custom backends for the delivery scheduler.

Eric:

I rewrote most of the MTA and SMTP code, on top of a simple io API that hides away low-level connection logic (io events, buffering, ssl context) to focus on protocol logic. In the process the way envelope update notifications are sent to the runner and queue during the delivery process was very much streamlined. The code that handles incoming and outgoing messages is now much simpler to follow than it used to be, and it will allow us to implement new features much more easily.

The queue protocol and the file-system queue were also largely rewritten and simplified. The queue protocol used to define different kind of queues where messages could live: incoming, queued, offline, bounce, corrupt, purge. This ended up being too complex for no real gain, and caused lots of problems. For example the bounce queue is where the bounce messages were supposed to be queued, but an envelope already had a type field for that. Another problem was that offline messages were seen as a specific kind of message in the queue, and it was the queue process which was responsible for reading them back from disk and re-enqueuing them. So the queue walk interface was awkward as it had to handle both offline messages (which are inherently local) and commited messages (which can reside in remote storage). This also made the runner very tricky. The offline messages are now completely handled by the smtpd master process, which re-injects them at startup.

Gilles:

While Eric was improving offline messages handling, I worked on an API for session filters. The idea was that we didn't want shared objects, we didn't want to fork a filter for every session and ... we wanted to stay coherent with our asynchronous design.

Todd (Fries) suggested that I should inspire from login scripts. That meant a filter would be a new process sharing a descriptor with the daemon and we would use imsg to communicate back and forth.

I was seduced by the idea because though it required some work to be done right and provide a simple interface to filter writers, it was a solution technically superior to anything I've thought of. With this design filters can run with different privileges, they run in their own memory space, and with little work and the proper API exposed to filter writers, they can be be turned into little daemons thus not having overhead besides the initial start.

I wrote a little library which makes writing filters trivial. A filter will simply register callbacks for any step of a session, then start the event loop which will turn it into a daemon. Its functions will be called with appropriate parameters, all the details behind the asynchronous exchanges with the daemon are hidden.

Below is an example of a filter to reject incoming mails from example.com:

 
    #include <sys/types.h>
    #include "filter.h"

    int
    rcpt_cb(u_int64_t id, struct filter_rcpt *rcpt, void *p)
    {
            if (strcmp(rcpt->domain, "examples.com") == 0)
                    return 0;
            return 1;
    }

    int
    main(int argc, char *argv[])
    {
            filter_init();
     
            filter_register_rcpt_callback(rcpt_cb, NULL);
     
            filter_loop();
     
            return 0;
    }

Charles:

In parallel, I also worked on adding async dns queries for filters. Basically, filters in OpenSMTPD are standalone programs. The filter API allows filters to register callbacks for some events (CONNECT, HELO, MAIL FROM, RCPT TO, DATA, ...) and also allows filters to make async DNS queries, which are rerouted to the async DNS resolver done by Eric.

For now I made a DNSBL POC. It works, but I still need to check more stuff before commit it.

Eric:

Altogether, my contributions to OpenSMTPD have not been about adding features, but rather cleaning things up and sanitizing the design and code where I thought it was really needed. It gave gilles@ the little boost of motivation that he needed to start working again on it and move forward. There are still plenty of room and ideas for improvements. Among the various things that I plan to do, as time permits, there are three major points that are worth noting:

  • The MTA logic must be updated. As it works now, it creates one SMTP connection for each message sent, and it cannot send multiple messages over the same connection. Another problem, which is actually more in the scheduler, is that an outgoing message with multiple recipients results in different batches for the different recipient domains, even if they are all relayed through the same host. This largely sub-optimal, especially in the (supposedly very common) use-case where the smtpd server is relaying all mails through one's ISP smtp servers.
  • Another thing that needs to be solved is the multiple bounce issue. Currently, one failure in a envelope delivery generates one bounce. When deliveries fail for several recipients of the same message, the server must try to group multiple reports in a single notification message. The way to fix this is known; it just needs
  • The last major field of improvement is filtering. The current exerimental filter scheme ends up being too slow and not flexible enough. We also want filtering to occur on the outgoing messages. This is being discussed.

Gilles:

These days I work on several areas.

My current sandbox is full of code to allow the use of mappings in various places where they aren't usable right now. For example, we should be able to do stuff like:

    map "cmap" source db "/etc/mail/clients.db"
    map "vmap" source db "/etc/mail/virtual.db"
    map "rmap" source db "/etc/mail/relays.db"

    accept from map cmap for virtual map vmap relay via map rmap
allowing OpenSMTPD to dynamically lookup incoming clients, destinations and routes in dynamic maps that could be updated at runtime. This is mostly done in my sandbox.

I also have a diff to replace the current relaying syntax where all of the relay options are part of the rule, with a new syntax where the relay options are part of an url:

    accept for all relay via "tls+auth://host" 
this is much nicer and allows a map to contain different relays with different options:
    map "rmap" source plain "/etc/mail/relays.txt"
    accept for all relay via map rmap
with relays.txt containing:
    smtps://mx1.example.org
    tls://mx2.example.org
    smtps+auth://mx3.example.org 

Another feature I'm working on is the ability to use a mapping to store the source address OpenSMTPD will use for its outgoing connections. I had planned to implement them in a few weeks, but the spamhaus project forced me into writing the feature in a hurry when they blacklisted an entire netblock my server was in to block two spammers.

I also spend a lot of time cleaning up code by removing structures and constructs that we intended to use a long time ago and that we either replaced with a better solution or that we did not use after all this time. This leads to code that's simpler to read and bugs that are easier to track.

Finally, I'm focusing on making the daemon rock solid. Every now and then I ask for people to flood my instance to see how it resists. We're close to be rock solid, my instance copes with hundreds of concurrent connections flooding it with random sessions; however sometimes a simple session causes our scheduler to go nuts. That's the kind of bugs I'm tracking these days ;)

Charles:

Next, we are in contact with some people to have it included in pkgsrc, and more generally I'll try to get in touch with every main Linux distribution to include a package of portable OpenSMTPD.

Finally, I will try to port it for Mac OS X.

If you are interrested please have a look at: http://opensmtpd.org/portable.html

Gilles:

We receive mails regularly from people asking if we are production-ready and if we're going to be the default MTA for OpenBSD soon. This is really a matter of finding enough time to fix the few known show-stoppers.

Maybe now is a good opportunity to stress out that companies that would be interested in sponsoring work on OpenSMTPD should really get in touch with us if they want development to go at a faster pace. I know some freelancers that would love to work full-time on this project ;-)

Contact information is available on the OpenSMTDP website.

(Comments are closed)


Comments
  1. By Adam P (adamrt) adam@adamrt.com on

    Great read. Thank you to everyone for taking the time to do these write ups. They are extremely interesting to me. I wish we had OpenBSD news like this everyday.

    Also reminded me to buy a 5.1 cd set. Thanks!

    Comments
    1. By bgpepi (bgpepi) on

      > Great read. Thank you to everyone for taking the time to do these write ups. They are extremely interesting to me. I wish we had OpenBSD news like this everyday.
      >
      > Also reminded me to buy a 5.1 cd set. Thanks!
      I am also thanks!

  2. By Janne Johansson (jj) jj@stacken.kth.se on http://www.inet6.se

    I tried to edit this article to fix the source code example html rendering and broke the article in half instead. Working on getting it back up again.

    My sincere apologies in the mean time.

    Comments
    1. By Janne Johansson (jj) on http://www.inet6.se

      > I tried to edit this article to fix the source code example html rendering and broke the article in half instead. Working on getting it back up again.

      With Brets help, and cut-n-paste skills, I got it back into some kind of shape now.

  3. By Zach McGrew (zmcgrew) zmcgrew@gmail.com on http://zmcgrew.no-ip.com

    Just wanted to say thank you for this great article and renewing my interest in OpenSMTPD all over again!

    This is yet another great write up on this site. How about something on OpenCVS next? That's another one of the awesome projects in the Open{BSD, SMTPD, SSH} family I haven't heard anything about in a long time.

    Comments
    1. By Tamotsu (tamo) on http://tamo.tdiary.net/

      > How about something on OpenCVS next? That's another one of the awesome projects in the Open{BSD, SMTPD, SSH} family I haven't heard anything about in a long time.

      Don't forget OpenNTPd.

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]