OpenBSD Journal

A transparent spamd(8) bridge

Contributed by deanna on from the cheque-is-in-the-mail dept.

Chris Kuethe (ckuethe@) writes:

I work as security/system admin at the University of Alberta. Every so often my boss (beck@) or I are called to swoop in like ninjas to rescue someone's mailserver from a joe-job or the mail virus du jour. This involves putting a spamd box inline with their mail processors. To make this work, we use a box with at least two network interfaces (names like Nexcom, Commell and Soekris come to mind because they're easy to carry around) and an IP address on the same subnet as the mail server to be protected.

For this example, I am protecting my workstation from my laptop with a Nexcom. The interface closest to the edge of the case (fxp2) is designated the external interface, and is given an address (172.16.5.111) on the same subnet as the protected mail server.

ifconfig fxp0 up
ifconfig fxp2 inet 172.16.5.111 netmask 255.255.255.0 up
route add default 172.16.5.1
ifconfig bridge0 create
brconfig bridge0 add fxp0 add fxp2 up
At this point, the machine is forwarding ethernet frames, but is not doing any filtering. Thus, connections to the mailserver are passed unmolested. This is what we are trying to prevent. Enter pf. The net.inet.ip.forwarding sysctl must be set to 1 because network address translation and redirection involves routing - it's not just ethernet any more.

sysctl net.inet.ip.forwarding=1
pfctl -ef /etc/pf.conf
The stock pf.conf that ships with OpenBSD comes close, but it doesn't work on a bridge. The rdr statement rewrites the destination address, but it won't be routed properly. Actually, it won't be routed at all - the destination is rewritten but the routing table is not consulted. Thus, you get packets on the wire headed for localhost - which doesn't work. Pf of course has an answer for this. If you think you know better than the routing table where a packet should go, you can specify the interface where the packet should be sent from, you may specify it with route-to. Because the smtp connection is to be handled by spamd on localhost, it should be routed out the lo0 interface.

Enough with the chatter, here's a pf.conf that will trap smtp connections passing through a bridge and send them to spamd on localhost.

ext_if="fxp2"

table <spamd> persist
table <spamd-white> persist

rdr on $ext_if inet proto tcp from <spamd> to port smtp \
        -> 127.0.0.1 port spamd
rdr on $ext_if inet proto tcp from !<spamd-white> to port smtp \
        -> 127.0.0.1 port spamd

# "log" so you can watch the connections getting trapped
pass in log on $ext_if route-to lo0 inet proto tcp to 127.0.0.1 port spamd
There you have it - everything required to set up a bump-in-the-wire spam trap. While I have given commands which will produce the desired result, this configuration can be made permanent by editing the relevant configuration files:

(Comments are closed)


Comments
  1. Comments
    1. By Anonymous Coward (24.79.89.96) on

      > Here is formula for a drop-in spamd transparent bridge.
      > It works so far.
      >
      > http://cisx1.uma.maine.edu/~wbackman/spamd.html

      I first tried with your solution (when you sent me the link to that page) though I wasn't sure what interfaced to put the IP on. Though trivial, it did cause me to think about what was going on and ended up with the same solution as Chris described above. I was also using it for something other than spamd but expect more on that later.

      Comments
      1. By sean (24.79.89.96) on

        Ooops, forgot my name.

  2. By Anonymous Coward (195.29.148.236) on

    Could you please explain these two contradictive (to me) statements:

    1) sysctl net.inet.ip.forwarding=1 (this enables forwarding: packets not destined to "us" are forwarded, i.e. routed)

    2) "The rdr statement rewrites the destination address, but it won't be routed properly. Actually, it won't be routed at all - the destination is rewritten but the routing table is not consulted."

    Why are we dealing with forwarding if routing table is not consulted at all ? Forwarding should consult routing table. Please explain, I'm obviously missing something here.

    Comments
    1. By sean (24.79.89.96) on

      > Could you please explain these two contradictive (to me) statements:
      >
      > 1) sysctl net.inet.ip.forwarding=1 (this enables forwarding: packet not destined to "us" are forwarded, i.e. routed)
      >
      For packets to traverse different interfaces you need that sysctl enabled otherwise your bridge has a big hole in it.

      > 2) "The rdr statement rewrites the destination address, but it won't be routed properly. Actually, it won't be routed at all - the destination is rewritten but the routing table is not consulted."
      >
      You put an IP on the bridge so you connect to it. The rdr statement is used because on the bridge the default action for any ingress traffic is to forward to the egress port on the bridge (and vice versa). With spamd you are changing that rule such that you are saying everything goes over with bridge EXCEPT these types of connections. This is an exception to the routing table so you reroute the packet to lo0 which has spamd bound to it (which then uses the default gateway to send packets out). Remember lo0 itself is an interface.

      > Why are we dealing with forwarding if routing table is not consulted at all ? Forwarding should consult routing table. Please explain, I'm obviously missing something here.
      >
      You are redirecting outside of the bridge which is then subject to the routing table.

    2. By Anonymous Coward (213.118.21.137) on

      >At this point, the machine is forwarding ethernet frames, but is not doing any filtering. ... The net.inet.ip.forwarding sysctl must be set to
      1 because network address translation and redirection involves routing -
      it's not just ethernet any more.

      At that point, you aren't forwarding anything. You need to set net.inet.ip.forwarding=1 first, as it has to do with forwarding data between interfaces. Ofcourse it result in enabeling routing too because it is on a higher level.

      If you keep the OSI model in mind you see that:
      A router routes IP packets on the network layer.
      A switch forwards MAC frames on the data link layer.
      Some people mix those terms up, and it's very confusing for beginners.

    3. By Anonymous Coward (71.12.222.126) on


      " http://lists.freebsd.org/pipermail/freebsd-pf/2005-December/001784.html "

      ..... a message from May 2003:Daniel Hartmeier:

      Yes, a bridge operates on ethernet level.

      For an rdr, pf will only replace the destination IP address/port, it
      doesn't touch the destination MAC address. I assume that in your case,
      the TCP SYN is sent to the MAC address of the internal host (not the
      firewall). pf replaces the destination IP address/port and hands the
      packet back to the bridge, which forwards it based on its destination
      MAC address.

      You can use 'route-to lo0' to cause pf to route the incoming packets to
      the loopback interface (using 127.0.0.1 as replacement destination
      address) instead of handing it back to the bridge after translation:

      rdr on $ext_if inet proto tcp from $outside_system to any port smtp ->
      127.0.0.1 port 8025

      pass in on $ext_if route-to lo0 inet proto tcp from any to $ext_if port
      8025 keep state

      Also, if the bridge is transparent (no IP addresses assigned to the
      interfaces), spamd won't work, as userland on the firewall is isolated
      from all networks. You need to assign an IP address to the external
      interface, otherwise there is no routing table entry which spamd needs
      to send replies to the external client.

      Many pf tricks work on bridges, but not all of them. Some require IP
      addresses assigned to the interfaces, for some you even need to enable
      IP forwarding. A bridge works very differently from a plain IP
      forwarder, you'll have to think in terms of ethernet frames, not IP
      packets. Don't use a bridge if you want the functionality of an IP
      forwarder.

      Comments
      1. By Anonymous Coward (195.29.157.74) on

        Very good summary of pf, bridging, and IP routing interoperability. Thank you for pointing to Daniel Hartmeier's post.

  3. By Anonymous Coward (75.132.114.37) on

    How fortuitous; I was looking into this in the wee hours of the morning today.

    I'm having a problem visualizing the routing here, though, so let me see if this is correct.

    The switch is plugged in to fxp0 (for example). fxp1 then would connect to the mail server via crossover cable, I would assume.

    The bridge operates at layer 2, so the switch sees the MAC address of the mail server's NIC? And it will also see the MAC address of fxp0, which has a legitimate routable IP address?

    And the TCP segments with spamd will have the address as the bridge's external IP? That is, if I attempt a connect to the mail server IP and get routed to lo0's spamd, what IP address am I talking to?

    I was going to trace this out last night and was installing 4.0 on a machine connected via a Belkin KVM. Switched over to another machine to do some browsing. Switched back at the end of the install and had no keyboard access to type "done halt", so there it sits until tonight.....

    Comments
    1. By Luca Corti (81.208.36.86) luca@leenoox.net on http://luca.leenoox.net

      > The bridge operates at layer 2, so the switch sees the MAC address of the mail server's NIC? And it will also see the MAC address of fxp0, which has a legitimate routable IP address?

      A layer 2 (data-link) bridge forwards the frame out all ports but the port the frame were received on in the first place if it doesn't have the destination MAC address in its MAC Address table. If it has the destination MAC int the table, it forwards it out the port the MAC address was detected on.

      Packets going to the mailserver will be encapsulated as ethernet frames and sent to the MAC address of the mailserver (resolved via ARP by the sending host/nearby router), received by the bridge interface and forwarded out on the port connected to the mailserver.

      Any IP address eventually assigned to fxp0 is irrelevant wrt bridging mailserver traffic.


      > And the TCP segments with spamd will have the address as the bridge's external IP? That is, if I attempt a connect to the mail server IP and get routed to lo0's spamd, what IP address am I talking to?

      From your point of view you are talking to the mailserver address. This is what RDR/NAT is all about, translating network (layer 3) addresses on the gateway in a user transparent way.


      Comments
      1. By Anonymous Coward (75.132.114.37) on

        Thanks.

        My specific curiosity was about a situation where you have a drop-in box and the mailserver's switch port is configured to only allow one MAC address. For instance, on a Cisco switch:

        (config-if)#switchport port-security
        (config-if)#switchport port-security maximum 1
        (config-if)#switchport port-security violation shutdown

        But requiring that it have an IP address on the mailserver's subnet would imply that a little manual jiggling be required anyway. I'll try it out tonight.

        Comments
        1. By Luca Corti (81.208.36.86) luca.corti@infinito.it on http://luca.leenoox.net

          > My specific curiosity was about a situation where you have a drop-in box and the mailserver's switch port is configured to only allow one MAC address. For instance, on a Cisco switch:

          In that case you could allow two MAC addresses on that port in the switch configuration

          OR

          you could try to make sure OpenBSD doesn't ever send ethernet frames using the MAC address of its interfaces as source MAC address (maybe avoiding to send any traffic out from the box and not assigning any IP address to it). Don't know if this would work anyway.

          > But requiring that it have an IP address on the mailserver's subnet would imply that a little manual jiggling be required anyway. I'll try it out tonight.

          I don't get your point here. You don't need any IP address on the bridge.

          Comments
          1. By Anonymous Coward (75.132.114.37) on

            As above:
            ifconfig fxp2 inet 172.16.5.111 netmask 255.255.255.0 up

            I've been looking into this for a while now, because an anti-spam device sounds good. Just as Chris' info here does, it would be nice to plop down a low-end server in front of any mail server and just have it work. I'm just trying to figure out any requirements and caveats beforehand. And before this, the most I've seen on this was from the misc@ archives where Graham Toal was requesting the same info.

            For instance, I'd like to get a Soekris 4801 and configure it more or less as above. But if I just sent it out to a site, what is the most they would have to do, and what might cause it to fail? So far, the only requirements look like: having a crossover cable, verifying speed/duplex on both interfaces, and adding an IP address/netmask in the same subnet as the mailserver to the external bridge interface. And the only thing I've thought of that could fail is having a switchport configured to shutdown on detection of an additional MAC address.

            In fact, my first wish for such a bridge was to just load up some dnsbl's and filter them out, not doing any greylisting at all. But pf tables can't hold extremely large lists, like the CBL, due to memory constraints, and after hearing a couple of beck@'s talks, I've become a greylisting convert.

            And thanks for the info, Chris.

            Sincerely,
            bigbutts@obtuse.com


  4. By Martijn Rijkeboer (145.100.55.162) on http://www.bunix.org/

    Nice article, but shouldn't spamd be told about good mailservers, so whitelisted addresses aren't removed after 30 days?

    For example:

    pass out log quick on $ext_if inet proto tcp from $mailserver_ip to any \
          port smtp keep state
    

    Comments
    1. By phessler (209.204.157.100) on

      >
      > Nice article, but shouldn't spamd be told about good mailservers, so whitelisted addresses aren't removed after 30 days?
      >
      >
      >
      > For example:
      >
      >
      >
      > pass out log quick on $ext_if inet proto tcp from $mailserver_ip to any \
      > port smtp keep state
      >

      everytime a server successfully sends mail through, the counter is reset. so servers that only send you the 'once a month mailing list password' aren't greylisted again.

      you could add them again if you want, but there is no need for it.

      Comments
      1. By Martijn Rijkeboer (145.100.21.5) on http://www.bunix.org/

        > everytime a server successfully sends mail through, the counter is
        > reset. so servers that only send you the 'once a month mailing list
        > password' aren't greylisted again.
        > you could add them again if you want, but there is no need for it.
        But when an IP-address is in the table spamd-white the traffic isn't redirected to spamd. So spamd only sees traffic from that IP-address the first time a connection is made an after 30 days when the IP-address is removed from the spamd-white table.

        This way the mailserver will be subjected to gray-listing every 30 days.

        Comments
        1. By sthen (85.158.44.146) on

          > But when an IP-address is in the table spamd-white the traffic isn't redirected to spamd. So spamd only sees traffic from that IP-address the first time a connection is made an after 30 days when the IP-address is removed from the spamd-white table.

          that's what spamlogd is for.

          Comments
          1. By Martijn Rijkeboer (145.100.21.5) on http://www.bunix.org/

            > that's what spamlogd is for.
            True, but in this configuration, once the IP-address is in the spamd-white table, there's no rule with a 'log' statement that matches the traffic so spamlogd will not see the traffic.

  5. By david (64.113.73.133) dlg+undeadly@dorkzilla.org on

    You mention using soekris machines. Do you just use the greylisting and DNSRBL capabilities of spamd, then, and count on that being sufficient to knock down the bulk of the mail? I can't imagine a soekris being able to handle something like spamassassin.

    Comments
    1. By Anonymous Coward (75.132.114.37) on

      > You mention using soekris machines. Do you just use the greylisting and DNSRBL capabilities of spamd, then, and count on that being sufficient to knock down the bulk of the mail? I can't imagine a soekris being able to handle something like spamassassin.

      Listen to Bob Beck's talk on spamd from: http://www.fetissov.org/public/nycbsdcon06/

      It's funny and informative (your question is answered at length).

      Comments
      1. By Chris Kuethe (129.128.11.75) ckuethe@ on

        > You mention using soekris machines... I can't imagine a
        > soekris being able to handle something like spamassassin.

        For emergency deployment we grab whatever's on my shelf - that usually ends up being a nexcom. In a more permanent configuration we get a couple of 1U machines and carp them together.

        A soekris is not appropriate for running spamassassin, but if all you need is something to keep the connection rate down so your 4-brain mail server that does run spamassassin can catch up, a small box like that may be enough.

        Comments
        1. By Anonymous Coward (64.129.81.169) on

          > > You mention using soekris machines... I can't imagine a
          > > soekris being able to handle something like spamassassin.
          >


          Could/have you load the CBL list on this platform, to get it updated requires port rsync, but the CBL has over 3 million IP addresses to blacklist?

          Comments
          1. By Anonymous Coward (75.132.114.37) on

            They won't all fit in memory. However, assuming you're greylisting, you could run it against the greylist (or whitelist, I suppose) tables.

            beck@ doesn't like dnsbl's, but here's his prototype for a greyscanner:

            http://www.ualberta.ca/~beck/greyscanner

            And listen to his presentation (linked above), if you haven't. All presentations should be so good.

            spamhaus.org makes me happy (includes the CBL), though, so I'd include a check, a la the check_dnsbl Nagios plug-in (http://www.tblc.org/~ostrowb/check_dnsbl). If spamhaus says you have a problem, it's almost assured that you have a problem.

            Knowing that spamd is never going to process a single legitimate email, I'd be curious to see how many other anomalies could be detected (e.g., early talkers).

            Comments
            1. By Bob Beck (129.128.11.43) beck@openbsd.org on http://www.ualberta.ca/~beck/nycbug06/spamd/

              > They won't all fit in memory. However, assuming you're greylisting, you could run it against the greylist (or whitelist, I suppose) tables.
              >
              > beck@ doesn't like dnsbl's, but here's his prototype for a greyscanner:
              >
              > http://www.ualberta.ca/~beck/greyscanner
              >
              >

              I don't like using DNSBL's as the sole source of blocking. I find
              them useful to add scores to messages in spamassassin, I just
              don't like trusting them for all or nothing, because it's DNS, so
              by definition, it's spoofable, poisonable garbage.


              -Bob

              Comments
              1. By Anonymous Coward (75.132.114.37) on

                Excellent presentations, by the way. I'll be very disappointed if a "Porn is the power of greytrapping!" sample doesn't end up in the 4.1 song (bsdtalk068 @ 14:06).

                You said you rewrote spamd from scratch and threw it away. What in the current implementation would you be dissatisfied with?

                Thanks a lot for your work.

    2. By Anonymous Coward (75.132.114.37) on

      > You mention using soekris machines. Do you just use the greylisting and DNSRBL capabilities of spamd, then, and count on that being sufficient to knock down the bulk of the mail? I can't imagine a soekris being able to handle something like spamassassin.

      I misread your comment, as well, so ignore my comment. Disregard. Sorry. Carry on.

    3. By cnst (76.3.196.122) on

      > You mention using soekris machines. Do you just use the greylisting and DNSRBL capabilities of spamd, then, and count on that being sufficient to knock down the bulk of the mail? I can't imagine a soekris being able to handle something like spamassassin.

      Misinformed you are -- spamd has nothing to do with DNSRBL and with spamassassin.

  6. By Jason L. Wright (134.20.35.80) jason@openbsd.org on http://www.thought.net/jason

    Does anyone but me have a hard time picturing beck@ swooping like a ninja?

    Comments

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]