Contributed by deanna on from the cheque-is-in-the-mail dept.
I work as security/system admin at the University of Alberta. Every so often my boss (beck@) or I are called to swoop in like ninjas to rescue someone's mailserver from a joe-job or the mail virus du jour. This involves putting a spamd box inline with their mail processors. To make this work, we use a box with at least two network interfaces (names like Nexcom, Commell and Soekris come to mind because they're easy to carry around) and an IP address on the same subnet as the mail server to be protected.For this example, I am protecting my workstation from my laptop with a Nexcom. The interface closest to the edge of the case (fxp2) is designated the external interface, and is given an address (172.16.5.111) on the same subnet as the protected mail server.
At this point, the machine is forwarding ethernet frames, but is not doing any filtering. Thus, connections to the mailserver are passed unmolested. This is what we are trying to prevent. Enter pf. The net.inet.ip.forwarding sysctl must be set to 1 because network address translation and redirection involves routing - it's not just ethernet any more.ifconfig fxp0 up ifconfig fxp2 inet 172.16.5.111 netmask 255.255.255.0 up route add default 172.16.5.1 ifconfig bridge0 create brconfig bridge0 add fxp0 add fxp2 up
The stock pf.conf that ships with OpenBSD comes close, but it doesn't work on a bridge. The rdr statement rewrites the destination address, but it won't be routed properly. Actually, it won't be routed at all - the destination is rewritten but the routing table is not consulted. Thus, you get packets on the wire headed for localhost - which doesn't work. Pf of course has an answer for this. If you think you know better than the routing table where a packet should go, you can specify the interface where the packet should be sent from, you may specify it with route-to. Because the smtp connection is to be handled by spamd on localhost, it should be routed out the lo0 interface.sysctl net.inet.ip.forwarding=1 pfctl -ef /etc/pf.conf
Enough with the chatter, here's a pf.conf that will trap smtp connections passing through a bridge and send them to spamd on localhost.
There you have it - everything required to set up a bump-in-the-wire spam trap. While I have given commands which will produce the desired result, this configuration can be made permanent by editing the relevant configuration files:ext_if="fxp2" table <spamd> persist table <spamd-white> persist rdr on $ext_if inet proto tcp from <spamd> to port smtp \ -> 127.0.0.1 port spamd rdr on $ext_if inet proto tcp from !<spamd-white> to port smtp \ -> 127.0.0.1 port spamd # "log" so you can watch the connections getting trapped pass in log on $ext_if route-to lo0 inet proto tcp to 127.0.0.1 port spamd
(Comments are closed)
By Will Backman (216.220.225.229) on http://cisx1.uma.maine.edu/~wbackman/spamd.html
It works so far.
http://cisx1.uma.maine.edu/~wbackman/spamd.html
Comments
By Anonymous Coward (24.79.89.96) on
> It works so far.
>
> http://cisx1.uma.maine.edu/~wbackman/spamd.html
I first tried with your solution (when you sent me the link to that page) though I wasn't sure what interfaced to put the IP on. Though trivial, it did cause me to think about what was going on and ended up with the same solution as Chris described above. I was also using it for something other than spamd but expect more on that later.
Comments
By sean (24.79.89.96) on
By Anonymous Coward (195.29.148.236) on
1) sysctl net.inet.ip.forwarding=1 (this enables forwarding: packets not destined to "us" are forwarded, i.e. routed)
2) "The rdr statement rewrites the destination address, but it won't be routed properly. Actually, it won't be routed at all - the destination is rewritten but the routing table is not consulted."
Why are we dealing with forwarding if routing table is not consulted at all ? Forwarding should consult routing table. Please explain, I'm obviously missing something here.
Comments
By sean (24.79.89.96) on
>
> 1) sysctl net.inet.ip.forwarding=1 (this enables forwarding: packet not destined to "us" are forwarded, i.e. routed)
>
For packets to traverse different interfaces you need that sysctl enabled otherwise your bridge has a big hole in it.
> 2) "The rdr statement rewrites the destination address, but it won't be routed properly. Actually, it won't be routed at all - the destination is rewritten but the routing table is not consulted."
>
You put an IP on the bridge so you connect to it. The rdr statement is used because on the bridge the default action for any ingress traffic is to forward to the egress port on the bridge (and vice versa). With spamd you are changing that rule such that you are saying everything goes over with bridge EXCEPT these types of connections. This is an exception to the routing table so you reroute the packet to lo0 which has spamd bound to it (which then uses the default gateway to send packets out). Remember lo0 itself is an interface.
> Why are we dealing with forwarding if routing table is not consulted at all ? Forwarding should consult routing table. Please explain, I'm obviously missing something here.
>
You are redirecting outside of the bridge which is then subject to the routing table.
By Anonymous Coward (213.118.21.137) on
1 because network address translation and redirection involves routing -
it's not just ethernet any more.
At that point, you aren't forwarding anything. You need to set net.inet.ip.forwarding=1 first, as it has to do with forwarding data between interfaces. Ofcourse it result in enabeling routing too because it is on a higher level.
If you keep the OSI model in mind you see that:
A router routes IP packets on the network layer.
A switch forwards MAC frames on the data link layer.
Some people mix those terms up, and it's very confusing for beginners.
By Anonymous Coward (71.12.222.126) on
" http://lists.freebsd.org/pipermail/freebsd-pf/2005-December/001784.html "
..... a message from May 2003:Daniel Hartmeier:
Yes, a bridge operates on ethernet level.
For an rdr, pf will only replace the destination IP address/port, it
doesn't touch the destination MAC address. I assume that in your case,
the TCP SYN is sent to the MAC address of the internal host (not the
firewall). pf replaces the destination IP address/port and hands the
packet back to the bridge, which forwards it based on its destination
MAC address.
You can use 'route-to lo0' to cause pf to route the incoming packets to
the loopback interface (using 127.0.0.1 as replacement destination
address) instead of handing it back to the bridge after translation:
rdr on $ext_if inet proto tcp from $outside_system to any port smtp ->
127.0.0.1 port 8025
pass in on $ext_if route-to lo0 inet proto tcp from any to $ext_if port
8025 keep state
Also, if the bridge is transparent (no IP addresses assigned to the
interfaces), spamd won't work, as userland on the firewall is isolated
from all networks. You need to assign an IP address to the external
interface, otherwise there is no routing table entry which spamd needs
to send replies to the external client.
Many pf tricks work on bridges, but not all of them. Some require IP
addresses assigned to the interfaces, for some you even need to enable
IP forwarding. A bridge works very differently from a plain IP
forwarder, you'll have to think in terms of ethernet frames, not IP
packets. Don't use a bridge if you want the functionality of an IP
forwarder.
Comments
By Anonymous Coward (195.29.157.74) on
By Anonymous Coward (75.132.114.37) on
I'm having a problem visualizing the routing here, though, so let me see if this is correct.
The switch is plugged in to fxp0 (for example). fxp1 then would connect to the mail server via crossover cable, I would assume.
The bridge operates at layer 2, so the switch sees the MAC address of the mail server's NIC? And it will also see the MAC address of fxp0, which has a legitimate routable IP address?
And the TCP segments with spamd will have the address as the bridge's external IP? That is, if I attempt a connect to the mail server IP and get routed to lo0's spamd, what IP address am I talking to?
I was going to trace this out last night and was installing 4.0 on a machine connected via a Belkin KVM. Switched over to another machine to do some browsing. Switched back at the end of the install and had no keyboard access to type "done halt", so there it sits until tonight.....
Comments
By Luca Corti (81.208.36.86) luca@leenoox.net on http://luca.leenoox.net
A layer 2 (data-link) bridge forwards the frame out all ports but the port the frame were received on in the first place if it doesn't have the destination MAC address in its MAC Address table. If it has the destination MAC int the table, it forwards it out the port the MAC address was detected on.
Packets going to the mailserver will be encapsulated as ethernet frames and sent to the MAC address of the mailserver (resolved via ARP by the sending host/nearby router), received by the bridge interface and forwarded out on the port connected to the mailserver.
Any IP address eventually assigned to fxp0 is irrelevant wrt bridging mailserver traffic.
> And the TCP segments with spamd will have the address as the bridge's external IP? That is, if I attempt a connect to the mail server IP and get routed to lo0's spamd, what IP address am I talking to?
From your point of view you are talking to the mailserver address. This is what RDR/NAT is all about, translating network (layer 3) addresses on the gateway in a user transparent way.
Comments
By Anonymous Coward (75.132.114.37) on
My specific curiosity was about a situation where you have a drop-in box and the mailserver's switch port is configured to only allow one MAC address. For instance, on a Cisco switch:
(config-if)#switchport port-security
(config-if)#switchport port-security maximum 1
(config-if)#switchport port-security violation shutdown
But requiring that it have an IP address on the mailserver's subnet would imply that a little manual jiggling be required anyway. I'll try it out tonight.
Comments
By Luca Corti (81.208.36.86) luca.corti@infinito.it on http://luca.leenoox.net
In that case you could allow two MAC addresses on that port in the switch configuration
OR
you could try to make sure OpenBSD doesn't ever send ethernet frames using the MAC address of its interfaces as source MAC address (maybe avoiding to send any traffic out from the box and not assigning any IP address to it). Don't know if this would work anyway.
> But requiring that it have an IP address on the mailserver's subnet would imply that a little manual jiggling be required anyway. I'll try it out tonight.
I don't get your point here. You don't need any IP address on the bridge.
Comments
By Anonymous Coward (75.132.114.37) on
ifconfig fxp2 inet 172.16.5.111 netmask 255.255.255.0 up
I've been looking into this for a while now, because an anti-spam device sounds good. Just as Chris' info here does, it would be nice to plop down a low-end server in front of any mail server and just have it work. I'm just trying to figure out any requirements and caveats beforehand. And before this, the most I've seen on this was from the misc@ archives where Graham Toal was requesting the same info.
For instance, I'd like to get a Soekris 4801 and configure it more or less as above. But if I just sent it out to a site, what is the most they would have to do, and what might cause it to fail? So far, the only requirements look like: having a crossover cable, verifying speed/duplex on both interfaces, and adding an IP address/netmask in the same subnet as the mailserver to the external bridge interface. And the only thing I've thought of that could fail is having a switchport configured to shutdown on detection of an additional MAC address.
In fact, my first wish for such a bridge was to just load up some dnsbl's and filter them out, not doing any greylisting at all. But pf tables can't hold extremely large lists, like the CBL, due to memory constraints, and after hearing a couple of beck@'s talks, I've become a greylisting convert.
And thanks for the info, Chris.
Sincerely,
bigbutts@obtuse.com
By Martijn Rijkeboer (145.100.55.162) on http://www.bunix.org/
Nice article, but shouldn't spamd be told about good mailservers, so whitelisted addresses aren't removed after 30 days?
For example:
Comments
By phessler (209.204.157.100) on
> Nice article, but shouldn't spamd be told about good mailservers, so whitelisted addresses aren't removed after 30 days?
>
>
>
> For example:
>
>
>
> pass out log quick on $ext_if inet proto tcp from $mailserver_ip to any \
> port smtp keep state
>
everytime a server successfully sends mail through, the counter is reset. so servers that only send you the 'once a month mailing list password' aren't greylisted again.
you could add them again if you want, but there is no need for it.
Comments
By Martijn Rijkeboer (145.100.21.5) on http://www.bunix.org/
> reset. so servers that only send you the 'once a month mailing list
> password' aren't greylisted again.
> you could add them again if you want, but there is no need for it.
But when an IP-address is in the table spamd-white the traffic isn't redirected to spamd. So spamd only sees traffic from that IP-address the first time a connection is made an after 30 days when the IP-address is removed from the spamd-white table.
This way the mailserver will be subjected to gray-listing every 30 days.
Comments
By sthen (85.158.44.146) on
that's what spamlogd is for.
Comments
By Martijn Rijkeboer (145.100.21.5) on http://www.bunix.org/
True, but in this configuration, once the IP-address is in the spamd-white table, there's no rule with a 'log' statement that matches the traffic so spamlogd will not see the traffic.
By david (64.113.73.133) dlg+undeadly@dorkzilla.org on
Comments
By Anonymous Coward (75.132.114.37) on
> You mention using soekris machines. Do you just use the greylisting and DNSRBL capabilities of spamd, then, and count on that being sufficient to knock down the bulk of the mail? I can't imagine a soekris being able to handle something like spamassassin.
Listen to Bob Beck's talk on spamd from: http://www.fetissov.org/public/nycbsdcon06/
It's funny and informative (your question is answered at length).
Comments
By Chris Kuethe (129.128.11.75) ckuethe@ on
> soekris being able to handle something like spamassassin.
For emergency deployment we grab whatever's on my shelf - that usually ends up being a nexcom. In a more permanent configuration we get a couple of 1U machines and carp them together.
A soekris is not appropriate for running spamassassin, but if all you need is something to keep the connection rate down so your 4-brain mail server that does run spamassassin can catch up, a small box like that may be enough.
Comments
By Anonymous Coward (64.129.81.169) on
> > soekris being able to handle something like spamassassin.
>
Could/have you load the CBL list on this platform, to get it updated requires port rsync, but the CBL has over 3 million IP addresses to blacklist?
Comments
By Anonymous Coward (75.132.114.37) on
beck@ doesn't like dnsbl's, but here's his prototype for a greyscanner:
http://www.ualberta.ca/~beck/greyscanner
And listen to his presentation (linked above), if you haven't. All presentations should be so good.
spamhaus.org makes me happy (includes the CBL), though, so I'd include a check, a la the check_dnsbl Nagios plug-in (http://www.tblc.org/~ostrowb/check_dnsbl). If spamhaus says you have a problem, it's almost assured that you have a problem.
Knowing that spamd is never going to process a single legitimate email, I'd be curious to see how many other anomalies could be detected (e.g., early talkers).
Comments
By Bob Beck (129.128.11.43) beck@openbsd.org on http://www.ualberta.ca/~beck/nycbug06/spamd/
>
> beck@ doesn't like dnsbl's, but here's his prototype for a greyscanner:
>
> http://www.ualberta.ca/~beck/greyscanner
>
>
I don't like using DNSBL's as the sole source of blocking. I find
them useful to add scores to messages in spamassassin, I just
don't like trusting them for all or nothing, because it's DNS, so
by definition, it's spoofable, poisonable garbage.
-Bob
Comments
By Anonymous Coward (75.132.114.37) on
You said you rewrote spamd from scratch and threw it away. What in the current implementation would you be dissatisfied with?
Thanks a lot for your work.
By Anonymous Coward (75.132.114.37) on
I misread your comment, as well, so ignore my comment. Disregard. Sorry. Carry on.
By cnst (76.3.196.122) on
> You mention using soekris machines. Do you just use the greylisting and DNSRBL capabilities of spamd, then, and count on that being sufficient to knock down the bulk of the mail? I can't imagine a soekris being able to handle something like spamassassin.
Misinformed you are -- spamd has nothing to do with DNSRBL and with spamassassin.
By Jason L. Wright (134.20.35.80) jason@openbsd.org on http://www.thought.net/jason
Comments
By Bob Beck (129.128.11.43) beck@openbsd.org on http://bofh.cns.ualberta.ca/beck/pictures/MooseCamp_2006/DSCN1633.html
Does anyone but me have a hard time picturing beck@ swooping like a ninja?
They shouldn't. I swoop from the trees with my deadly ninja skills all the time ;)