Contributed by ray on from the bleeding-edge-mail dept.
If you're feeling lucky and want to help out, please do!On Wed, Dec 03, 2008 at 10:58:00AM -0700, Gilles Chehade wrote: > CVSROOT: /cvs > Module name: src > Changes by: gilles@ 2008/12/03 10:58:00 > > Modified files: > usr.sbin/smtpd : queue.c smtp.c smtp_session.c smtpd.c smtpd.h > > Log message: > - fix event masking issues in smtp process which could lead to a fatal() if > queue process did not answer fast enough to an imsg. spotted by > Jacek Masiulaniec > - queue layout was mostly to bootstrap the project, it does not behave good > under load, it does complex things to stay in a recoverable state > and it probably didnt do it too well. New queue code is simpler, > smaller and allows for atomic submissions (a mail can never be in a > state where it needs to be recovered). It still needs some work but > works better than previous code, no regression. > If you are testing smtpd, make sure to make clean and start with an empty /var/spool/smtpd before you run the new queue code. If you run into a fatal please let me know as soon as you can, I have time for hacking right now ;-) Gilles
(Comments are closed)
By James (jturner) james@bsdgroup.org on http://bsdgroup.org
Comments
By Gilles CHEHADE (gilles) on http://www.poolp.org/~gilles/
Well, to be honest I have been running opensmtpd for a while[0] on my own server to handle my personal email, and as long as you run sendmail or postfix on a secondary backup MX, you should be fine[1] ;-)
The new queue code I commited ensures you will never ever tell a client you accepted mail before it is actually atomically moved from the incoming queue to the actual delivery queue. With this, even if opensmtpd fatal()-s or crashes at a bad time, mail you have accepted is safe.
I currently do live tests by having opensmtpd on mx1, sendmail on mx2, and a cron job on mx1 which sends me a mail every 3 hours. If I don't see the mail, it means opensmtpd is no longer running and I can fix it[2]. mx2 just ensures I don't lose mails meanwhile.
Gilles
[0] I replaced sendmail with it weeks before it was imported in OpenBSD
[1] Don't do it, I warned you ;-)
[2] It used to happen every couple weeks, fortunately it got better
Comments
By Anonymous Coward (67.69.227.99) on
>
> Well, to be honest I have been running opensmtpd for a while[0] on my own server to handle my personal email, and as long as you run sendmail or postfix on a secondary backup MX, you should be fine[1] ;-)
>
> The new queue code I commited ensures you will never ever tell a client you accepted mail before it is actually atomically moved from the incoming queue to the actual delivery queue. With this, even if opensmtpd fatal()-s or crashes at a bad time, mail you have accepted is safe.
>
> I currently do live tests by having opensmtpd on mx1, sendmail on mx2, and a cron job on mx1 which sends me a mail every 3 hours. If I don't see the mail, it means opensmtpd is no longer running and I can fix it[2]. mx2 just ensures I don't lose mails meanwhile.
>
> Gilles
>
> [0] I replaced sendmail with it weeks before it was imported in OpenBSD
> [1] Don't do it, I warned you ;-)
> [2] It used to happen every couple weeks, fortunately it got better
>
Is there a way to set something similar to SMART_HOST in sendmail with this smtpd?
I'd love to replace this at home, over the stock sendmail and be able to relay non-internal mail through my ISP as they block port 25 out unless it's from their SMTPd.
Comments
By James (jturner) on http://bsdgroup.org
> >
> > Well, to be honest I have been running opensmtpd for a while[0] on my own server to handle my personal email, and as long as you run sendmail or postfix on a secondary backup MX, you should be fine[1] ;-)
> >
> > The new queue code I commited ensures you will never ever tell a client you accepted mail before it is actually atomically moved from the incoming queue to the actual delivery queue. With this, even if opensmtpd fatal()-s or crashes at a bad time, mail you have accepted is safe.
> >
> > I currently do live tests by having opensmtpd on mx1, sendmail on mx2, and a cron job on mx1 which sends me a mail every 3 hours. If I don't see the mail, it means opensmtpd is no longer running and I can fix it[2]. mx2 just ensures I don't lose mails meanwhile.
> >
> > Gilles
> >
> > [0] I replaced sendmail with it weeks before it was imported in OpenBSD
> > [1] Don't do it, I warned you ;-)
> > [2] It used to happen every couple weeks, fortunately it got better
> >
>
> Is there a way to set something similar to SMART_HOST in sendmail with this smtpd?
>
> I'd love to replace this at home, over the stock sendmail and be able to relay non-internal mail through my ISP as they block port 25 out unless it's from their SMTPd.
I believe would you want in your smtpd.conf is something similar to
accept from 0.0.0.0 for domain "mydomain.com" deliver to maildir "/home/%u/Maildir"
accept for all relay via "mx.example.org"
This would then accept any mail destined for mydomain.com and relay the rest via mx.example.org (your smarthost). Anyone please correct me if I'm wrong.
By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/
>
> I'd love to replace this at home, over the stock sendmail and be able to relay non-internal mail through my ISP as they block port 25 out unless it's from their SMTPd.
As long as your ISP doesn't expect you to authenticate you can use something like:
accept for localhost deliver to mbox "/var/mail/%u"
accept for all relay via mx1.yourisp.com
Gilles
By Brian Marshall (208.181.67.27) on
Thanks!
-Brian
Comments
By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/
> Thanks!
>
/!\ Don't try this in production /!\
This being said, if you *really* want to test it, you can checkout the src/usr.sbin/smtpd directory from -current and it will build on -stable.
You'll also need to add a group _smtpd and a user _smtpd with /var/empty as its home directory. You will also need to create /var/spool/smtpd that's owned by root:whell and with permissions set to 711.
Comments
By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/
^^^^^
wheel
By Anonymous Coward (96.21.15.58) on
> > Thanks!
> >
>
> /!\ Don't try this in production /!\
>
> This being said, if you *really* want to test it, you can checkout the src/usr.sbin/smtpd directory from -current and it will build on -stable.
>
> You'll also need to add a group _smtpd and a user _smtpd with /var/empty as its home directory. You will also need to create /var/spool/smtpd that's owned by root:whell and with permissions set to 711.
>
Will this possibly replace the old and crusty sendmail by 4.5? =)
I'm excited to see smtpd in works, this is really awesome!
Comments
By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/
> > > Thanks!
> > >
> >
> > /!\ Don't try this in production /!\
> >
> > This being said, if you *really* want to test it, you can checkout the src/usr.sbin/smtpd directory from -current and it will build on -stable.
> >
> > You'll also need to add a group _smtpd and a user _smtpd with /var/empty as its home directory. You will also need to create /var/spool/smtpd that's owned by root:whell and with permissions set to 711.
> >
>
> Will this possibly replace the old and crusty sendmail by 4.5? =)
>
> I'm excited to see smtpd in works, this is really awesome!
I think sendmail still has quite some time in front of it ;-)
It is quite unlikely that smtpd replaces sendmail before a while, it needs to be more mature, it needs to get tested a *lot* more and it needs to provide the features most users expect (and that is quite a lot of features). Only way to speed things is to help testing and spot bugs !
Comments
By Lennie (194.213.15.37) on
>
I would much rather see it be stable and in base with an easy way to switch instead of any bells and wistles.
All I need is a way to get any cron-messages of my machines to a central mailbox and I'm sure it's the same for many.
Having something simple and efficient to do this would be really nice.
Comments
By ScottD (192.107.131.105) on
> >
>
> I would much rather see it be stable and in base with an easy way to switch instead of any bells and wistles.
>
> All I need is a way to get any cron-messages of my machines to a central mailbox and I'm sure it's the same for many.
All good sys admins do! I'd suggest something really ugly, low level, and 'frailware' but it doesn't require nor risk having an MTA. Script nc in cron. My crontab command line looks like:
/usr/local/bin/server_status.sh | /usr/bin/nc servstat.domain.tld 25 > /dev/null
where servstat is a CNAME alias to the central collector (an email receiving hub). The server_status.sh script collects a bunch of system info (for SOX type purposes) and then simulates an smtp session and upon reaching the DATA statement disgorges the status report and QUITs. This approach has been working for over 2 yrs on systems with no MTA .
>
> Having something simple and efficient to do this would be really nice.
>
Gilles this is a very good project. Looking forward to using smtpd in say... 4.8 ?
This approach has been working for 2 years.
By Brian Marshall (208.181.67.27) on
> > Thanks!
> >
>
> /!\ Don't try this in production /!\
>
> This being said, if you *really* want to test it, you can checkout the src/usr.sbin/smtpd directory from -current and it will build on -stable.
>
> You'll also need to add a group _smtpd and a user _smtpd with /var/empty as its home directory. You will also need to create /var/spool/smtpd that's owned by root:whell and with permissions set to 711.
>
Very nice. I like the smtpd.conf syntax. Any way to map all users from a domain to local users (eg. *@example.com -> *@localhost)?
Easy compile/setup. Passed the abuse.net open relay test. I'll submit any bugs I run into.
-Brian
Comments
By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/
> Easy compile/setup. Passed the abuse.net open relay test. I'll submit any bugs I run into.
>
accept from 0.0.0.0/0 for domain "poolp.org" deliver to mbox "/var/mail/%u"
this maps foo@poolp.org to the user account foo
Gilles
Comments
By Anonymous Coward (96.21.15.58) on
> > Easy compile/setup. Passed the abuse.net open relay test. I'll submit any bugs I run into.
> >
>
> accept from 0.0.0.0/0 for domain "poolp.org" deliver to mbox "/var/mail/%u"
>
> this maps foo@poolp.org to the user account foo
>
> Gilles
>
Any plans to make 0.0.0.0/0 short hand like 0/0 or what would also be nice is like pf, 'any' or 'all'. Just wondering.
Comments
By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/
> > > Easy compile/setup. Passed the abuse.net open relay test. I'll submit any bugs I run into.
> > >
> >
> > accept from 0.0.0.0/0 for domain "poolp.org" deliver to mbox "/var/mail/%u"
> >
> > this maps foo@poolp.org to the user account foo
> >
> > Gilles
> >
>
> Any plans to make 0.0.0.0/0 short hand like 0/0 or what would also be nice is like pf, 'any' or 'all'. Just wondering.
Yeah, I just need to ... do it.
Gilles
By David Chisnall (82.7.199.50) on
Comments
By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/
It supports both smtp over ssl and tls out of the box.
SMTP AUTH is in the works, it will work out of the box as well.
Gilles
Comments
By Anonymous Coward (96.21.15.58) on
>
> It supports both smtp over ssl and tls out of the box.
> SMTP AUTH is in the works, it will work out of the box as well.
>
> Gilles
>
OMFG, this rocks!
By David Chisnall (82.7.199.50) on
>
> It supports both smtp over ssl and tls out of the box.
> SMTP AUTH is in the works, it will work out of the box as well.
>
> Gilles
Really excellent news. Please post something on undeadly again when this is implemented, and I will switch over.
Do you have any plans for integrating spam filtering?
Comments
By jirib (89.176.141.36) on
> >
> > It supports both smtp over ssl and tls out of the box.
> > SMTP AUTH is in the works, it will work out of the box as well.
> >
> > Gilles
>
> Really excellent news. Please post something on undeadly again when this is implemented, and I will switch over.
>
> Do you have any plans for integrating spam filtering?
>
He declared in previous article it will be mitler compatible so milter is your friend probably :)
Comments
By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/
> > >
> > > It supports both smtp over ssl and tls out of the box.
> > > SMTP AUTH is in the works, it will work out of the box as well.
> > >
> > > Gilles
> >
> > Really excellent news. Please post something on undeadly again when this is implemented, and I will switch over.
> >
> > Do you have any plans for integrating spam filtering?
> >
>
> He declared in previous article it will be mitler compatible so milter is your friend probably :)
Oh nooooooooo no no nooo, not milter-compatible !
It will be a milter-like interface that provides a way for admins to plug filters that inspect various aspects of the session. But what I'd like is some kind of hybrid between milters and postfix's filters. Something that allows me to plug a shell script or a more complex C filter just as easily, and which allows any admin to write a filter in a matter of minutes.
There hasn't been work done on this yet so I won't expand until I've given it enough thinking and discussing with other developers ;)
Gilles
By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/
> >
> > It supports both smtp over ssl and tls out of the box.
> > SMTP AUTH is in the works, it will work out of the box as well.
> >
> > Gilles
>
> Really excellent news. Please post something on undeadly again when this is implemented, and I will switch over.
>
> Do you have any plans for integrating spam filtering?
>
There will be a milter-like interface at some point, we just need to get some of the more basic features to work flawlessly first, spam-filtering and friends will happen after ;-)
Gilles
Comments
By Peter van Oord van der Vlies (2001:888:102e:0:21a:73ff:fec0:ab86) zork@pvo-it.nl on http://www.itisit.nl
> There will be a milter-like interface at some point, we just need to get some of the more basic features to work flawlessly first, spam-filtering and friends will happen after ;-)
>
> Gilles
When spamfiltering and virus checking is working i will join the test.
By Anonymous Coward (212.224.70.25) on
> > >
> > > It supports both smtp over ssl and tls out of the box.
> > > SMTP AUTH is in the works, it will work out of the box as well.
> > >
> > > Gilles
> >
> > Really excellent news. Please post something on undeadly again when this is implemented, and I will switch over.
> >
> > Do you have any plans for integrating spam filtering?
> >
>
> There will be a milter-like interface at some point, we just need to get some of the more basic features to work flawlessly first, spam-filtering and friends will happen after ;-)
>
> Gilles
Combined with PF and a "possible" communication between those 2 processes (somehow) a neat antispam solution could be set up wich adds autoblocking so that the SMTP hasn't to deal with any mails from offensiv hosts anymore.
PF can take a look at connections/seconds wich do hit the box but in one connection serval junk mails could get transfered. That's the idea I've in mind wich would allow some kind of "deep packet inspection" (at leats for smtp).
Same maybe could get done for the relayd or other things.
It would turn PF into a IDS-like system somehow. :-)
By Terrell Prude' Jr. (151.188.18.46) tprude@cmosnetworks.com (this is a spamtrap address) on http://www.cmosnetworks.com/
> > >
> > > It supports both smtp over ssl and tls out of the box.
> > > SMTP AUTH is in the works, it will work out of the box as well.
> > >
> > > Gilles
> >
> > Really excellent news. Please post something on undeadly again when this is implemented, and I will switch over.
> >
> > Do you have any plans for integrating spam filtering?
> >
>
> There will be a milter-like interface at some point, we just need to get some of the more basic features to work flawlessly first, spam-filtering and friends will happen after ;-)
>
> Gilles
I will *gladly* try this out on my home email server when SMTP AUTH is in place. That's where I try stuff out before I do it for my customers (usu. small businesses). I'm currently using Postfix for that reason.
SMTP AUTH is what allows SOHO folks to do road-warrior-type stuff if they don't have a VPN gateway (most don't), and yep, I always do it with SSL/TLS. My customers ask me, "can MS Exchange do it?" I tell 'em "yep, if you want to get owned within 10 minutes." I then show them the price tag. Then they consider FOSS. :-)
Why don't most small businesses have a VPN gateway? They usually have one "business grade" IP address on a broadband link, and for KISS principle reasons, I'd really, REALLY rather have that on a separate IP address. It's for their good. Very few people that come after me can handle the more complex port/protocol forwarding that this requires (they're usually MCSE-types).
I already have nearly all of them on OpenBSD's "spamd" in front of their real MTA (again, currently Postfix), and they love the drastic reduction in spam. And so do I.
--TP
By Anonymous Coward (24.172.134.210) on
A hearty "thank you" for all your work (and also to those who helped).
By tamo (60.47.223.46) on
This prevents opensmtpd from behaving like sendmail, i.e. reading ~/.forward even if ~/ is not world-accessable.
Also, "\user" syntax is not allowed in ~/.forward yet and, "/home/user/Maildir" in ~/.forward is not recognized as a maildir.
Comments
By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/
> This prevents opensmtpd from behaving like sendmail, i.e. reading ~/.forward even if ~/ is not world-accessable.
>
Yes, the forward file is read by _smtpd which means that a user can disable the forwarding of his mails by removing world-readable permissions.
> Also, "\user" syntax is not allowed in ~/.forward yet and, "/home/user/Maildir" in ~/.forward is not recognized as a maildir.
>
The ~/.forward syntax is not (yet ?) fully compatible with sendmail's forward files. I'm still unsure if we need \user or if user will be sufficient due to how the resolution is done. Maildir support in ~/.forward shouldn't be too hard, I'll keep that in mind.
Gilles
Comments
By Jordi Espasa (84.77.66.237) jordi.espasa@opengea.org on
żDo you already decided to call it 'opensmtpd'? żIt means that will be a dedicated site as OpenNTPd or OpenSSH OpenBSD-related projects?
One more time, many thanks ;)
By tamo (60.47.223.46) on
That's okay for me. Thanks for your work and the kind reply!
One more incompatibility as a note: Sendmail allows two formats (\n-delimited and ,-delimited) of ~/.forward, but that's just confusing IMO. OpenSMTPd assumes ~/.forward is \n-delimited.
Testers should be careful.
"smtpd -vd" doesn't show very much information for each failure like this.
I'd like more perrors or something instead of just return 0.
Comments
By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/
>
> That's okay for me. Thanks for your work and the kind reply!
>
Actually, it will be compatible, it is just a matter of time.
With regard to previous reply, ~/.forward file WILL be read by privileged process and evaluated even when ~/ is not world-readable. This will be fixed pretty soon.
> One more incompatibility as a note: Sendmail allows two formats (\n-delimited and ,-delimited) of ~/.forward, but that's just confusing IMO. OpenSMTPd assumes ~/.forward is \n-delimited.
>
That's mostly because I needed ~/.forward files to work asap in my environment so I could replace sendmail, I still need to do a lot of work in the aliases/forward areas, compatibility is one of them.
Feel free to mail me any incompatibility you run into so that I can keep track of them.
> Testers should be careful.
> "smtpd -vd" doesn't show very much information for each failure like this.
> I'd like more perrors or something instead of just return 0.
>
Yup, I agree it could be more verbose, initially it simply returned 0 which was caught by the mailer daemon handler. The mailer daemon code is not done, so I ended up removing this, I'll probably slap a few log_debug()'s around there until mailer-daemon is back.
Gilles
By jkm (194.237.142.10) on
Ill try to set up a test environment this weekend. I dont have a spare MX to use to test with my personal mail but i plan to forward all my spam to a separate box with smtpd.
Let the spam actually do some good work instead of just being wasted with spamd :)
By Anonymous Coward (121.96.216.19) on
Comments
By Gilles Chehade (gilles) on http://www.poolp.org/~gilles/
>
It is not yet possible, but something for which some work has already be done and which is definitely not too low on the TODO list.
Gilles