OpenBSD Journal

Call for Testing: smtpd

Contributed by ray on from the bleeding-edge-mail dept.

Gilles Chehade has put out a small request for testing his smtpd,
On Wed, Dec 03, 2008 at 10:58:00AM -0700, Gilles Chehade wrote:
> CVSROOT:      /cvs
> Module name:  src
> Changes by:   gilles@  2008/12/03 10:58:00
>
> Modified files:
>       usr.sbin/smtpd : queue.c smtp.c smtp_session.c smtpd.c smtpd.h
>
> Log message:
> - fix event masking issues in smtp process which could lead to a fatal() if
> queue process did not answer fast enough to an imsg. spotted by
> Jacek Masiulaniec
> - queue layout was mostly to bootstrap the project, it does not behave good
> under load, it does complex things to stay in a recoverable state
> and it probably didnt do it too well. New queue code is simpler,
> smaller and allows for atomic submissions (a mail can never be in a
> state where it needs to be recovered). It still needs some work but
> works better than previous code, no regression.
>

If you are testing smtpd, make sure to make clean and start
with an empty /var/spool/smtpd before you run the new queue
code. If you run into a fatal please let me know as soon as
you can, I have time for hacking right now ;-)

Gilles
If you're feeling lucky and want to help out, please do!

(Comments are closed)


  1. By James (jturner) james@bsdgroup.org on http://bsdgroup.org

    All still seems to work here. I'm seriously debating putting this into production for my personal email to really give it some good testing. Keep up the good work gilles@!

    1. By Gilles CHEHADE (gilles) on http://www.poolp.org/~gilles/

      > All still seems to work here. I'm seriously debating putting this into production for my personal email to really give it some good testing. Keep up the good work gilles@!

      Well, to be honest I have been running opensmtpd for a while[0] on my own server to handle my personal email, and as long as you run sendmail or postfix on a secondary backup MX, you should be fine[1] ;-)

      The new queue code I commited ensures you will never ever tell a client you accepted mail before it is actually atomically moved from the incoming queue to the actual delivery queue. With this, even if opensmtpd fatal()-s or crashes at a bad time, mail you have accepted is safe.

      I currently do live tests by having opensmtpd on mx1, sendmail on mx2, and a cron job on mx1 which sends me a mail every 3 hours. If I don't see the mail, it means opensmtpd is no longer running and I can fix it[2]. mx2 just ensures I don't lose mails meanwhile.

      Gilles

      [0] I replaced sendmail with it weeks before it was imported in OpenBSD
      [1] Don't do it, I warned you ;-)
      [2] It used to happen every couple weeks, fortunately it got better

      1. By Anonymous Coward () on

        > > All still seems to work here. I'm seriously debating putting this into production for my personal email to really give it some good testing. Keep up the good work gilles@!
        >
        > Well, to be honest I have been running opensmtpd for a while[0] on my own server to handle my personal email, and as long as you run sendmail or postfix on a secondary backup MX, you should be fine[1] ;-)
        >
        > The new queue code I commited ensures you will never ever tell a client you accepted mail before it is actually atomically moved from the incoming queue to the actual delivery queue. With this, even if opensmtpd fatal()-s or crashes at a bad time, mail you have accepted is safe.
        >
        > I currently do live tests by having opensmtpd on mx1, sendmail on mx2, and a cron job on mx1 which sends me a mail every 3 hours. If I don't see the mail, it means opensmtpd is no longer running and I can fix it[2]. mx2 just ensures I don't lose mails meanwhile.
        >
        > Gilles
        >
        > [0] I replaced sendmail with it weeks before it was imported in OpenBSD
        > [1] Don't do it, I warned you ;-)
        > [2] It used to happen every couple weeks, fortunately it got better
        >

        Is there a way to set something similar to SMART_HOST in sendmail with this smtpd?

        I'd love to replace this at home, over the stock sendmail and be able to relay non-internal mail through my ISP as they block port 25 out unless it's from their SMTPd.

        1. By James (jturner) on http://bsdgroup.org

          > > > All still seems to work here. I'm seriously debating putting this into production for my personal email to really give it some good testing. Keep up the good work gilles@!
          > >
          > > Well, to be honest I have been running opensmtpd for a while[0] on my own server to handle my personal email, and as long as you run sendmail or postfix on a secondary backup MX, you should be fine[1] ;-)
          > >
          > > The new queue code I commited ensures you will never ever tell a client you accepted mail before it is actually atomically moved from the incoming queue to the actual delivery queue. With this, even if opensmtpd fatal()-s or crashes at a bad time, mail you have accepted is safe.
          > >
          > > I currently do live tests by having opensmtpd on mx1, sendmail on mx2, and a cron job on mx1 which sends me a mail every 3 hours. If I don't see the mail, it means opensmtpd is no longer running and I can fix it[2]. mx2 just ensures I don't lose mails meanwhile.
          > >
          > > Gilles
          > >
          > > [0] I replaced sendmail with it weeks before it was imported in OpenBSD
          > > [1] Don't do it, I warned you ;-)
          > > [2] It used to happen every couple weeks, fortunately it got better
          > >
          >
          > Is there a way to set something similar to SMART_HOST in sendmail with this smtpd?
          >
          > I'd love to replace this at home, over the stock sendmail and be able to relay non-internal mail through my ISP as they block port 25 out unless it's from their SMTPd.

          I believe would you want in your smtpd.conf is something similar to


          accept from 0.0.0.0 for domain "mydomain.com" deliver to maildir "/home/%u/Maildir"
          accept for all relay via "mx.example.org"


          This would then accept any mail destined for mydomain.com and relay the rest via mx.example.org (your smarthost). Anyone please correct me if I'm wrong.

        2. By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/

          > Is there a way to set something similar to SMART_HOST in sendmail with this smtpd?
          >
          > I'd love to replace this at home, over the stock sendmail and be able to relay non-internal mail through my ISP as they block port 25 out unless it's from their SMTPd.

          As long as your ISP doesn't expect you to authenticate you can use something like:

          accept for localhost deliver to mbox "/var/mail/%u"
          accept for all relay via mx1.yourisp.com

          Gilles

  2. By Brian Marshall () on

    Hello, I'm willing to try this in production for a very low-volume mail server. Is it possible to check out and build without tracking -current?
    Thanks!

    -Brian

    1. By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/

      > Hello, I'm willing to try this in production for a very low-volume mail server. Is it possible to check out and build without tracking -current?
      > Thanks!
      >

      /!\ Don't try this in production /!\

      This being said, if you *really* want to test it, you can checkout the src/usr.sbin/smtpd directory from -current and it will build on -stable.

      You'll also need to add a group _smtpd and a user _smtpd with /var/empty as its home directory. You will also need to create /var/spool/smtpd that's owned by root:whell and with permissions set to 711.

      1. By Anonymous Coward () on

        > > Hello, I'm willing to try this in production for a very low-volume mail server. Is it possible to check out and build without tracking -current?
        > > Thanks!
        > >
        >
        > /!\ Don't try this in production /!\
        >
        > This being said, if you *really* want to test it, you can checkout the src/usr.sbin/smtpd directory from -current and it will build on -stable.
        >
        > You'll also need to add a group _smtpd and a user _smtpd with /var/empty as its home directory. You will also need to create /var/spool/smtpd that's owned by root:whell and with permissions set to 711.
        >

        Will this possibly replace the old and crusty sendmail by 4.5? =)

        I'm excited to see smtpd in works, this is really awesome!

        1. By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/

          > > > Hello, I'm willing to try this in production for a very low-volume mail server. Is it possible to check out and build without tracking -current?
          > > > Thanks!
          > > >
          > >
          > > /!\ Don't try this in production /!\
          > >
          > > This being said, if you *really* want to test it, you can checkout the src/usr.sbin/smtpd directory from -current and it will build on -stable.
          > >
          > > You'll also need to add a group _smtpd and a user _smtpd with /var/empty as its home directory. You will also need to create /var/spool/smtpd that's owned by root:whell and with permissions set to 711.
          > >
          >
          > Will this possibly replace the old and crusty sendmail by 4.5? =)
          >
          > I'm excited to see smtpd in works, this is really awesome!

          I think sendmail still has quite some time in front of it ;-)

          It is quite unlikely that smtpd replaces sendmail before a while, it needs to be more mature, it needs to get tested a *lot* more and it needs to provide the features most users expect (and that is quite a lot of features). Only way to speed things is to help testing and spot bugs !

          1. By Lennie () on

            > It is quite unlikely that smtpd replaces sendmail before a while, it needs to be more mature, it needs to get tested a *lot* more and it needs to provide the features most users expect (and that is quite a lot of features). Only way to speed things is to help testing and spot bugs !
            >

            I would much rather see it be stable and in base with an easy way to switch instead of any bells and wistles.

            All I need is a way to get any cron-messages of my machines to a central mailbox and I'm sure it's the same for many.

            Having something simple and efficient to do this would be really nice.

            1. By ScottD () on

              > > It is quite unlikely that smtpd replaces sendmail before a while, it needs to be more mature, it needs to get tested a *lot* more and it needs to provide the features most users expect (and that is quite a lot of features). Only way to speed things is to help testing and spot bugs !
              > >
              >
              > I would much rather see it be stable and in base with an easy way to switch instead of any bells and wistles.
              >
              > All I need is a way to get any cron-messages of my machines to a central mailbox and I'm sure it's the same for many.

              All good sys admins do! I'd suggest something really ugly, low level, and 'frailware' but it doesn't require nor risk having an MTA. Script nc in cron. My crontab command line looks like:

              /usr/local/bin/server_status.sh | /usr/bin/nc servstat.domain.tld 25 > /dev/null

              where servstat is a CNAME alias to the central collector (an email receiving hub). The server_status.sh script collects a bunch of system info (for SOX type purposes) and then simulates an smtp session and upon reaching the DATA statement disgorges the status report and QUITs. This approach has been working for over 2 yrs on systems with no MTA .
              >
              > Having something simple and efficient to do this would be really nice.
              >
              Gilles this is a very good project. Looking forward to using smtpd in say... 4.8 ?

              This approach has been working for 2 years.

      2. By Brian Marshall () on

        > > Hello, I'm willing to try this in production for a very low-volume mail server. Is it possible to check out and build without tracking -current?
        > > Thanks!
        > >
        >
        > /!\ Don't try this in production /!\
        >
        > This being said, if you *really* want to test it, you can checkout the src/usr.sbin/smtpd directory from -current and it will build on -stable.
        >
        > You'll also need to add a group _smtpd and a user _smtpd with /var/empty as its home directory. You will also need to create /var/spool/smtpd that's owned by root:whell and with permissions set to 711.
        >

        Very nice. I like the smtpd.conf syntax. Any way to map all users from a domain to local users (eg. *@example.com -> *@localhost)?
        Easy compile/setup. Passed the abuse.net open relay test. I'll submit any bugs I run into.

        -Brian

        1. By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/

          > Very nice. I like the smtpd.conf syntax. Any way to map all users from a domain to local users (eg. *@example.com -> *@localhost)?
          > Easy compile/setup. Passed the abuse.net open relay test. I'll submit any bugs I run into.
          >

          accept from 0.0.0.0/0 for domain "poolp.org" deliver to mbox "/var/mail/%u"

          this maps foo@poolp.org to the user account foo

          Gilles

          1. By Anonymous Coward () on

            > > Very nice. I like the smtpd.conf syntax. Any way to map all users from a domain to local users (eg. *@example.com -> *@localhost)?
            > > Easy compile/setup. Passed the abuse.net open relay test. I'll submit any bugs I run into.
            > >
            >
            > accept from 0.0.0.0/0 for domain "poolp.org" deliver to mbox "/var/mail/%u"
            >
            > this maps foo@poolp.org to the user account foo
            >
            > Gilles
            >

            Any plans to make 0.0.0.0/0 short hand like 0/0 or what would also be nice is like pf, 'any' or 'all'. Just wondering.

            1. By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/

              > > > Very nice. I like the smtpd.conf syntax. Any way to map all users from a domain to local users (eg. *@example.com -> *@localhost)?
              > > > Easy compile/setup. Passed the abuse.net open relay test. I'll submit any bugs I run into.
              > > >
              > >
              > > accept from 0.0.0.0/0 for domain "poolp.org" deliver to mbox "/var/mail/%u"
              > >
              > > this maps foo@poolp.org to the user account foo
              > >
              > > Gilles
              > >
              >
              > Any plans to make 0.0.0.0/0 short hand like 0/0 or what would also be nice is like pf, 'any' or 'all'. Just wondering.

              Yeah, I just need to ... do it.

              Gilles

  3. By David Chisnall () on

    Does it support authenticated SMTP over SSL yet? It bugs me that on an operating system that's secure by default the first thing I need to do after updating is update my source tree and recompile sendmail to support this.

    1. By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/

      > Does it support authenticated SMTP over SSL yet? It bugs me that on an operating system that's secure by default the first thing I need to do after updating is update my source tree and recompile sendmail to support this.

      It supports both smtp over ssl and tls out of the box.
      SMTP AUTH is in the works, it will work out of the box as well.

      Gilles

      1. By Anonymous Coward () on

        > > Does it support authenticated SMTP over SSL yet? It bugs me that on an operating system that's secure by default the first thing I need to do after updating is update my source tree and recompile sendmail to support this.
        >
        > It supports both smtp over ssl and tls out of the box.
        > SMTP AUTH is in the works, it will work out of the box as well.
        >
        > Gilles
        >

        OMFG, this rocks!

      2. By David Chisnall () on

        > > Does it support authenticated SMTP over SSL yet? It bugs me that on an operating system that's secure by default the first thing I need to do after updating is update my source tree and recompile sendmail to support this.
        >
        > It supports both smtp over ssl and tls out of the box.
        > SMTP AUTH is in the works, it will work out of the box as well.
        >
        > Gilles

        Really excellent news. Please post something on undeadly again when this is implemented, and I will switch over.

        Do you have any plans for integrating spam filtering?

        1. By jirib () on

          > > > Does it support authenticated SMTP over SSL yet? It bugs me that on an operating system that's secure by default the first thing I need to do after updating is update my source tree and recompile sendmail to support this.
          > >
          > > It supports both smtp over ssl and tls out of the box.
          > > SMTP AUTH is in the works, it will work out of the box as well.
          > >
          > > Gilles
          >
          > Really excellent news. Please post something on undeadly again when this is implemented, and I will switch over.
          >
          > Do you have any plans for integrating spam filtering?
          >

          He declared in previous article it will be mitler compatible so milter is your friend probably :)

          1. By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/

            > > > > Does it support authenticated SMTP over SSL yet? It bugs me that on an operating system that's secure by default the first thing I need to do after updating is update my source tree and recompile sendmail to support this.
            > > >
            > > > It supports both smtp over ssl and tls out of the box.
            > > > SMTP AUTH is in the works, it will work out of the box as well.
            > > >
            > > > Gilles
            > >
            > > Really excellent news. Please post something on undeadly again when this is implemented, and I will switch over.
            > >
            > > Do you have any plans for integrating spam filtering?
            > >
            >
            > He declared in previous article it will be mitler compatible so milter is your friend probably :)

            Oh nooooooooo no no nooo, not milter-compatible !

            It will be a milter-like interface that provides a way for admins to plug filters that inspect various aspects of the session. But what I'd like is some kind of hybrid between milters and postfix's filters. Something that allows me to plug a shell script or a more complex C filter just as easily, and which allows any admin to write a filter in a matter of minutes.

            There hasn't been work done on this yet so I won't expand until I've given it enough thinking and discussing with other developers ;)

            Gilles

        2. By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/

          > > > Does it support authenticated SMTP over SSL yet? It bugs me that on an operating system that's secure by default the first thing I need to do after updating is update my source tree and recompile sendmail to support this.
          > >
          > > It supports both smtp over ssl and tls out of the box.
          > > SMTP AUTH is in the works, it will work out of the box as well.
          > >
          > > Gilles
          >
          > Really excellent news. Please post something on undeadly again when this is implemented, and I will switch over.
          >
          > Do you have any plans for integrating spam filtering?
          >

          There will be a milter-like interface at some point, we just need to get some of the more basic features to work flawlessly first, spam-filtering and friends will happen after ;-)

          Gilles

          1. By Peter van Oord van der Vlies () zork@pvo-it.nl on http://www.itisit.nl

            >
            > There will be a milter-like interface at some point, we just need to get some of the more basic features to work flawlessly first, spam-filtering and friends will happen after ;-)
            >
            > Gilles

            When spamfiltering and virus checking is working i will join the test.

          2. By Anonymous Coward () on

            > > > > Does it support authenticated SMTP over SSL yet? It bugs me that on an operating system that's secure by default the first thing I need to do after updating is update my source tree and recompile sendmail to support this.
            > > >
            > > > It supports both smtp over ssl and tls out of the box.
            > > > SMTP AUTH is in the works, it will work out of the box as well.
            > > >
            > > > Gilles
            > >
            > > Really excellent news. Please post something on undeadly again when this is implemented, and I will switch over.
            > >
            > > Do you have any plans for integrating spam filtering?
            > >
            >
            > There will be a milter-like interface at some point, we just need to get some of the more basic features to work flawlessly first, spam-filtering and friends will happen after ;-)
            >
            > Gilles

            Combined with PF and a "possible" communication between those 2 processes (somehow) a neat antispam solution could be set up wich adds autoblocking so that the SMTP hasn't to deal with any mails from offensiv hosts anymore.

            PF can take a look at connections/seconds wich do hit the box but in one connection serval junk mails could get transfered. That's the idea I've in mind wich would allow some kind of "deep packet inspection" (at leats for smtp).

            Same maybe could get done for the relayd or other things.
            It would turn PF into a IDS-like system somehow. :-)

          3. By Terrell Prude' Jr. () tprude@cmosnetworks.com (this is a spamtrap address) on http://www.cmosnetworks.com/

            > > > > Does it support authenticated SMTP over SSL yet? It bugs me that on an operating system that's secure by default the first thing I need to do after updating is update my source tree and recompile sendmail to support this.
            > > >
            > > > It supports both smtp over ssl and tls out of the box.
            > > > SMTP AUTH is in the works, it will work out of the box as well.
            > > >
            > > > Gilles
            > >
            > > Really excellent news. Please post something on undeadly again when this is implemented, and I will switch over.
            > >
            > > Do you have any plans for integrating spam filtering?
            > >
            >
            > There will be a milter-like interface at some point, we just need to get some of the more basic features to work flawlessly first, spam-filtering and friends will happen after ;-)
            >
            > Gilles


            I will *gladly* try this out on my home email server when SMTP AUTH is in place. That's where I try stuff out before I do it for my customers (usu. small businesses). I'm currently using Postfix for that reason.

            SMTP AUTH is what allows SOHO folks to do road-warrior-type stuff if they don't have a VPN gateway (most don't), and yep, I always do it with SSL/TLS. My customers ask me, "can MS Exchange do it?" I tell 'em "yep, if you want to get owned within 10 minutes." I then show them the price tag. Then they consider FOSS. :-)

            Why don't most small businesses have a VPN gateway? They usually have one "business grade" IP address on a broadband link, and for KISS principle reasons, I'd really, REALLY rather have that on a separate IP address. It's for their good. Very few people that come after me can handle the more complex port/protocol forwarding that this requires (they're usually MCSE-types).

            I already have nearly all of them on OpenBSD's "spamd" in front of their real MTA (again, currently Postfix), and they love the drastic reduction in spam. And so do I.

            --TP

  4. By Anonymous Coward () on

    This is really a very nice addition. I've always loathed configuring and debugging sendmail in the past, and this new smtpd seems simple, clear, and easy to work with. Even more so than qmail.

    A hearty "thank you" for all your work (and also to those who helped).

  5. By tamo () on

    Seems that the unpriv user _smtpd tries to read ~/.forward.
    This prevents opensmtpd from behaving like sendmail, i.e. reading ~/.forward even if ~/ is not world-accessable.

    Also, "\user" syntax is not allowed in ~/.forward yet and, "/home/user/Maildir" in ~/.forward is not recognized as a maildir.

    1. By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/

      > Seems that the unpriv user _smtpd tries to read ~/.forward.
      > This prevents opensmtpd from behaving like sendmail, i.e. reading ~/.forward even if ~/ is not world-accessable.
      >

      Yes, the forward file is read by _smtpd which means that a user can disable the forwarding of his mails by removing world-readable permissions.

      > Also, "\user" syntax is not allowed in ~/.forward yet and, "/home/user/Maildir" in ~/.forward is not recognized as a maildir.
      >

      The ~/.forward syntax is not (yet ?) fully compatible with sendmail's forward files. I'm still unsure if we need \user or if user will be sufficient due to how the resolution is done. Maildir support in ~/.forward shouldn't be too hard, I'll keep that in mind.

      Gilles

      1. By Jordi Espasa () jordi.espasa@opengea.org on

        Many thanks for it Gilles; keep up your work!
        żDo you already decided to call it 'opensmtpd'? żIt means that will be a dedicated site as OpenNTPd or OpenSSH OpenBSD-related projects?

        One more time, many thanks ;)

      2. By tamo () on

        > The ~/.forward syntax is not (yet ?) fully compatible with sendmail's forward files.

        That's okay for me. Thanks for your work and the kind reply!

        One more incompatibility as a note: Sendmail allows two formats (\n-delimited and ,-delimited) of ~/.forward, but that's just confusing IMO. OpenSMTPd assumes ~/.forward is \n-delimited.

        Testers should be careful.
        "smtpd -vd" doesn't show very much information for each failure like this.
        I'd like more perrors or something instead of just return 0.

        1. By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/

          > > The ~/.forward syntax is not (yet ?) fully compatible with sendmail's forward files.
          >
          > That's okay for me. Thanks for your work and the kind reply!
          >

          Actually, it will be compatible, it is just a matter of time.
          With regard to previous reply, ~/.forward file WILL be read by privileged process and evaluated even when ~/ is not world-readable. This will be fixed pretty soon.


          > One more incompatibility as a note: Sendmail allows two formats (\n-delimited and ,-delimited) of ~/.forward, but that's just confusing IMO. OpenSMTPd assumes ~/.forward is \n-delimited.
          >

          That's mostly because I needed ~/.forward files to work asap in my environment so I could replace sendmail, I still need to do a lot of work in the aliases/forward areas, compatibility is one of them.

          Feel free to mail me any incompatibility you run into so that I can keep track of them.


          > Testers should be careful.
          > "smtpd -vd" doesn't show very much information for each failure like this.
          > I'd like more perrors or something instead of just return 0.
          >

          Yup, I agree it could be more verbose, initially it simply returned 0 which was caught by the mailer daemon handler. The mailer daemon code is not done, so I ended up removing this, I'll probably slap a few log_debug()'s around there until mailer-daemon is back.

          Gilles

  6. By jkm () on

    This is really great! Im so looking forward to replacing sendmail with smtpd.

    Ill try to set up a test environment this weekend. I dont have a spare MX to use to test with my personal mail but i plan to forward all my spam to a separate box with smtpd.

    Let the spam actually do some good work instead of just being wasted with spamd :)

  7. By Anonymous Coward () on

    Can smtpd be used to relay emails to a remote MX listening on submission (port 587) with STARTTLS and PLAIN/LOGIN SMTP AUTH? I'd love to test smtpd under this configuration :)

    1. By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/

      > Can smtpd be used to relay emails to a remote MX listening on submission (port 587) with STARTTLS and PLAIN/LOGIN SMTP AUTH? I'd love to test smtpd under this configuration :)
      >

      It is not yet possible, but something for which some work has already be done and which is definitely not too low on the TODO list.

      Gilles

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]