Appalachian Web Solutions chooses OpenBSD as its firewall

"Appalachian Web Solutions, a Carolina based hosting and web design company, decides to use OpenBSD as their firewall."

" My partner and I decided to evaluate different firewalls before actually putting anything into production. I have worked as a network and infrastructure security consultant for almost 10 years and have experience with various firewalls like Cisco PIX, CheckPoint Firewall-1, Gauntlet, and Raptor among a few lesser known firewalls. My partner has been a network, systems, and security administrator for 6 years and has worked with Linux, FreeBSD, Solaris, AIX and Windows Operating Systems as well as the firewalls CheckPoint Firewall-1 and Cisco PIX. We both knew that the firewall should be integrated into the system and if it was to be on an operating system, that operating system must be secure and have a good track record for security.

Our requirements were simple, we wanted a industry proven, stable, reliable, and most importantly secure firewall that was relatively easy to manage. We did not feel all these things were asking too much, so we also added speed. The firewall must be stateful, be able to watch for IP Options, Fragmented packets, and other strange anomalies such as port scans, etc. Both of us being comfortable with command line, we did not have a need for the firewall to have a graphical user interface (GUI).

We ruled windows out quickly with all the security issues released, which seems like a daily event. AIX, HP-UX, and Solaris were out of the question because the cost justification did not allow for such expensive platforms (especially being this hosting company was funded by us with no venture capitalist money). While CheckPoint does run on Linux we quickly ruled it out not only based on price but the number of vulnerabilities it has had in the past few years was among the highest of all the firewalls. It seemed as if it was between PIX and Linux running IPTABLES (we eliminated IPCHAINS because it is not stateful), we are familiar with the PIX interface and its strengths and weaknesses. We had someone willing to donate a PIX to us for our hosting company, and we did test it. The cost of maintenance and support was costly and we felt the cost justification for a commercial package was not feasible compared to the level of security that can be reached with an Open Source firewall. The technical merit of using a Cisco PIX was also greatly reduced when attempting to group similar objects. Cisco introduced grouped object into the 6.2.2 code for the PIX. The addition of objects into the groups is extremely antiquated compared to other firewalls that allow you to create group objects fairly easily. One who is familiar with writing basic shell scripts would have no problem creating objects (or variables) in Open Source based firewalls. PDM can be installed onto the PIX which add additional management capabilities via a web browser (and eases the pain of creating groups); however, we did not want to add any additional services to the firewall unless absolutely needed.

While we started looking into IPTABLES, I had used OpenBSD for a couple other projects and discussed with my partner the possibility of trying OpenBSD running PF as the firewall in comparison to Linux running IPTABLES. Both being security minded, a quick evaluation of the facts pointed out that OpenBSD has only had one exploit in over 7 years?very impressive by itself, but that was not quite enough to move that direction. Researching both OSes for quite sometime specifically with security in mind, everything pointed to OpenBSD. Several excerpts from "Building Linux and OpenBSD Firewalls" (Chapter 4) explain why OpenBSD is actually a better firewall than Linux, while the book may be a little dated on the versions of OSes and also limits its Linux distribution to only RedHat it does explain the fundamental reasons for choosing OpenBSD over Linux which is what we were looking for. A well written article at http://www.benzedrine.cx/pf-paper.html by Daniel Hartmeier explains that IPTABLES does not perform sequence number analysis. The sequence number analysis gives a little added security by reducing the amount of information an attacker knows about the systems. The fact that OpenSSH, OpenSSL and PF were all written by OpenBSD people, also assisted in the decision, but actually testing the firewall proving that it was more than capable of handling loads and still maintaining its composure was what changed our minds.

We implemented OpenBSD 3.1 into our environment and have never thought twice about it. We obviously had to patch a security hole (OpenSSH) and watch for any new vulnerabilities like any good administrator. The upgrade was done using CVS and there were no problems with the update.

While most hosting companies do not implement firewalls and only lock the systems down at the OS layer (or they should if they don?t), we did feel like this offers enough protection for our customers. We believe in best practices for security, which is to provide security in layers, the firewall being the first layer for our environment, second being the OS layer of each system, and third being the application level.

Now for the configuration (an example config will be listed at the end):

There are several things to note with PF, most importantly is the order of operations. Your /etc/pf.conf file must be in the following order or it will not work:

1. Options
2. Scrub
3. NAT and RDR
4. Rules or Filters

   The /etc/pf.conf file explained:

   1. Options:
      By default packets with IP options set will be blocked. This reduces the risk of ?OS fingerprinting? packages like nmap and queso. However if you have applications that require options (multicast, IGMP, etc) you can use allow-opt in the /etc/pf.conf file to enable the firewall to pass packets with options set.
   2. Scrub
      Packet normalization means clearing IP options and reassembling fragmented packets. Some operating systems and applications have trouble with fragmented packets, and typically there is no reason to have non-normalized packets allowed through a firewall. Normalization does come at a cost; it will require more memory and additional load on the system (this load is minimal).

      While scrubbing packets is a good idea, which direction do you scrub and on what interface? Always scrub on the outside (Internet) interface at a minimum. We suggest you scrub all inbound packets (inbound to each interface). Scrubbing outbound packets should not be necessary since that would be traffic originating from the firewall essentially.

      You will want to use the scrub in all as the second option.

   3. NAT and RDR
      Being we are a hosting company we did not require NAT since all our addresses are valid IP Addresses.
   4. Rules or Filters
      The meat of any firewall is the rules that protect the system but more importantly is the rules that allow access to systems. There are several philosophies on firewalls. The philosophy we use is to deny everything by default and only allow specifically what is needed. While many PF experts will explain how to do the rules we have tried to keep them logical in a FIFO type scenario such that the deny rule is last and all other rules are above. The first rule that it hits and matches it allows (if that is what the rules states) and the firewall stops with that packet other than to track its state.

      The first thing we must do is allow the system to talk to itself over the loopback interface. While this may seem unnecessary it has been my experience to allow the system to communicate with itself, otherwise some applications may not function correctly.

      Example:

      pass in quick on lo0 all
      pass out quick on lo0 all
      

      We also do not want to allow any IPv6 traffic to our site.

      Example:

      block in quick inet6 all
      block out quick inet6 all
      

      With many packet filter firewalls that are stateful many tools such as nmap can be used to determine what services are on systems behind the firewall. Utilizing FIN, URG and PSH flags some scanners can bypass a firewall looking for initializing flags such as SYN and allow attackers to see what is actually running on your servers. You will want to deny any packets that are FIN, URG, or PSH. Remember while you may want to restrict this on all interfaces the external interface is the most important place to block this type of traffic

      Example:

      block in quick on $external_if inet proto tcp from any to any flags FUP/FUP
      

      You will want to restrict non-routable packets from entering your network from the outside. Below is a listing of some of the types of non-routable networks and their purpose.

      RFC1918 compliant addresses used for internal addresses

      10.0.0.0/8
      172.16.0.0/12
      192.168.0.0/16
      

      Microsoft DHCP default addresses

      169.254.0.0/16
      

      Loopback Range

      127.0.0.0/8
      

      Resverved for DOCS

      192.0.2.0/24
      

      Sun cluster interconnect

      204.152.64.0/23
      

      Broadcast

      255.255.255.255/32
      0.0.0.0/8 
      

      You will see this on your network if you are doing DHCP. This comes from the client requesting and IP address.

      Multicast/Reserved Class D

      224.0.0.0/4
      

      You now can create a variable that encompasses all these network objects into one easy to use object:

      Example:

      nonroute={10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.0.0/16, 127.0.0.0/8, 
      		192.0.2.0/24, 204.152.64.0/23, 255.255.255.255/32, 0.0.0.0/8,  
      224.0.0.0/4}
      

      And your rule will look like the following to block all non-routable ip addresses:

      Example:

      block in quick on $ext_if from $nonroute to any
      block out quick on $ext_if from any to $nonroute
      

      Now you can create variables servers that have common services for example:

      Example:

      nameservers={208.192.4.1/32, 208.192.4.2/32}
      ftpservers={208.192.4.3/32}
      webservers={208.192.4.5/32, 208.192.4.6/32, 208.192.4.7/32}
      hosting={195.6.2.0/24}
      

      You can create variables for services that will be used across multiple servers (you can use the names of the services as they are listed in /etc/services, you can add your own services to /etc/services too):

      Example:

      hosting_services={pop, smtp, ftp, www, https, mysql}
      

      Once this is completed you can create rules like the following:

      Example

      pass in quick on $ext_if inet proto udp from any to $namservers port 53
      pass in quick on $ext_if inet proto tcp from any to $ftpservers port { ftp, ftp-data }
      pass in quick on $ext_if inet proto tcp from any to $webservers port { www, https }
      pass in quick on $ext_if inet proto tcp from any to $hosting port $hosting_services
      

      On the TCP protocol you will want to restrict the initial packet to a SYN only. The way you can do this is to add S/SA to the end of a tcp rule like:

      Example:

      pass in quick on $ext_if inet proto tcp from any to $ftpservers port { ftp, ftp-data } flags S/SA
      

      Now to add the stateful options to the firewall so it will remember what you have done:

      For UDP packets you will want to use "keep state" and on TCP packets you will want to utilize "modulate state". Because TCP packets are connection based, it is easy to hijack a TCP packet, for this reason PF has a modulate state option that increases the security over keep state. Modulate state generates a more random sequence number which makes hijacking packets more difficult (to find out more about OSes and sequence numbers visit http://razor.bindview.com/publish/papers/tcpseq.html).

      Example:

      pass in quick on $ext_if inet proto tcp from any to $ftpservers port {ftp, ftp-data} flags S/SA modulate state
      

      Your inbound rules are almost complete except for one thing, where is your default block rule?

      Example:

      block in quick on $ext_in all 
      

      You will want to add the following outgoing rules for the return traffic:

      block out on $ext_if all
      pass out on $ext_if inet proto tcp all flags S/SA keep state
      pass out on $ext_if inet proto udp all keep state
      

      Example /etc/pf.conf file

      # Comment section for when rulebase was created modified, etc.  It is suggested to use a standard format for this like the following:
      #
      # DATE - USER
      # Modification and why
      #
      #########################################################################################
      #-------------------------
      # Variables Section
      #-------------------------
      ext_if=?fxp0?
      int_if=?fxp1?
      dmz1_if=?fxp2?
      #-------------------------------
      # Non routable addresses
      #-------------------------------
      noroute={10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 255.255.255.255/32, 
                      169.254.0.0/16, 127.0.0.0/8, 0.0.0.0/8, 192.0.2.0/24, 
                      204.152.64.0/23 127.0.0.0/8}
      #------------------------------
      # Corporate Servers
      #------------------------------
      firewall={208.192.4.1/32}
      webservers={208.192.4.130/32}
      securewebservers={208.192.4.131/32, 208.192.4.132/32}
      nameservers={208.192.4.135/32, 208.192.4.136/32}
      mailservers={208.192.4.140/32}
      ftpservers={208.192.4.150/32}
      #------------------------------
      # Hosting/Virtual Servers
      #------------------------------
      hosting={208.192.5.0/24, 208.192.6.0/24, 208.192.7.0/24}
      virtual_servers={208.192.10.0/24, 208.192.11.0/24}
      #------------------------------
      # Hosting Services
      #------------------------------
      basic_services={imap, pop3, smtp, www, https, ftp, ftp-data, mysql}
      #************************************************************************
      #************************************************************************
      #	Firewall Rulebase Begin
      #************************************************************************
      #************************************************************************
      #
      #-----------------------------------------------------------------
      #       Packet Normalization (deny fragmented packets)
      #-----------------------------------------------------------------
      scrub in all
      #------------------------------------
      #       Allow Loopback Packets
      #------------------------------------
      pass in quick on lo0 all
      pass out quick on lo0 all
      #---------------------------------
      #       Drop Spoofed Packets
      #---------------------------------
      block in quick from $noroute to any
      #-------------------------------
      #   Drop wrong TCP Flags
      #-------------------------------
      block in quick on fxp0 inet proto tcp from any to any flags FUP/FUP
      #-----------------------------------------
      #       Corporate Firewall RULES
      #-----------------------------------------
      # Always document what the rules are doing you don?t know how long it will be before you look at them again ?
      pass in quick on $ext_if proto tcp from any to $mailservers port {25, 143} flags S/SA modulate state
      pass in quick on $ext_if proto tcp from any to $webservers port 80 flags S/SA modulate state
      pass in quick on $ext_if proto tcp from any to $securewebservers port 443 flags S/SA modulate state
      pass in quick on $ext_if proto udp from any to $nameservers port 53 keep state
      #---------------------------------------------------
      # FTP to webserver uncomment as needed
      #---------------------------------------------------
      pass in quick on $ext_if proto tcp from any to $ftpservers port {ftp, ftp-data} flags S/SA modulate state
      #----------------------------------------------
      # Webmin Rule uncomment as needed
      #----------------------------------------------
      #pass in quick on $ext_if proto tcp from any to $webservers port 10000 flags S/SA modulate state
      
      #-------------------------------------
      #       Hosting Firewall RULES
      #-------------------------------------
      pass in quick on $ext_if proto tcp from any to $host1vs port $basicservices flags S/SA modulate state
      pass in quick on $ext_if proto udp from any to $hostnameservers port 53 keep state
      #------------------------------
      #       Catch All Rule
      #------------------------------
      block in on $ext_if from any to any
      #--------------------------------------------------------------
      #  Allow return traffic and connection from firewall
      #--------------------------------------------------------------
      pass out on $ext_if from any to any keep state
      pass out on $int_if from any to any keep state
      pass out on $host1_if from any to any keep state
      

      Once you have the /etc/pf.conf file generated you will want to install the rulebase and enable PF.

      /sbin/pfctl -f /etc/pf.conf -e
      

      You may want to look at various aspects of what your firewall is doing?

      Show the statistics and counters for filter information

      /sbin/pfctl -s info
      

      Display hit count for each rule

      /sbin/pfctl -s rules -v 
      

      Display state table information

      /sbin/pfctl -s state
      

      Watch traffic

      tcpdump -i pflog0
      

      We are constantly looking for ways to be more secure and efficient. Our next project is VPN connection point between our remote offices and our facilities using IPSEC.

      We recently migrated our firewall from OpenBSD 3.1 to OpenBSD 3.2 with no issues. The added stability and performance of the new version of PF was worth the wait. Appalachian Web Solutions has been extremely pleased with OpenBSD in their production environment.

      My partner and I, both feel confident that we have made a wise technical and business decision for our hosting company to ensure the security of our customers and the continued success of our business.

      Notes:
      IP addresses used throughout this article are randomly picked addresses.

      I would like to point out a couple of great resources that helped us get started with our firewall project:

      http://www.muine.org/~hoang/openpf.html
      Securing Small Networks with OpenBSD (all the articles) http://www.onlamp.com/lpt/a//bsd/2002/02/28/openbsd.html
      http://www.openbsd.org/faq
      http://www.inebriated.demon.nl/pf-howto/
      http://www.benzedrine.cx/pf.html

      Special Thanks to the OpenBSD team and Daniel Hartmeier the creator of PF in addition to the many other people that assisted both these projects.

      ---

      Brandon is one of the principals of Appalachian Web Solutions, LLC he is responsible for maintaining the security of the systems. He received his CISSP in February of 2001 and his GCIA in September of 2001; he continues to try to improve the security of Appalachian Web Solutions. While not working he spends time with his wife, four dogs, and cat. He can be reached at bnewport@appws.com.

      http://www.appws.com "