OpenBSD Journal

Enterprise OpenBSD administration?

Contributed by jose on from the big-network dept.

Jeff asks: "I've used OpenBSD at home now for a couple years, but only recntly have I found a niche for it at work (we're mostly a Sun shop when it comes to Unix). Now that I will be admining several OpenBSD servers at work, including servers in our DMZ, I'm curious how people manage OpenBSD in an Enterprise setting.

In particular, I'm curious about the feasibility of having a system dedicated to compiling source/ports/etc and then installing on my production systems. Or does everyone just CVS source updates and recompile on all their systems? My thought here would be to reduce the number of systems that need access through the firewall for installing ports and grabbing the latest CVS.

I'd be interested to hear the approaches others use for this."

(Comments are closed)


Comments
  1. By Jim () on

    I am looking into the same topic. Since I lean towards consistency, I'm going to leverage the Release system. What I want to know is if anyone is doing this in conjunction with cross-compiling for multiple architectures. For example: my compile box is i386, I CVSup it and build releases for i386 and sparc64. The run around and install the release I just built...

    Thoughts.?.?.

    Comments
    1. By Brett Eisenberg () on

      Release?

      Comments
      1. By Anonymous Coward () on

        man release

    2. By Anonymous Coward () on

      Well I tryed to compile the 3.4-stable with DistCC( http://distcc.samba.org ) on my i386 box about one week after the release and it went strait to hell =(
      However the sources where just fine - there must have been a problem with the distcc configuration or a bug/problem with distcc ..

      It probably didn't succeed becaues I didn't try extremly hard - but it aint easy.
      And that is very sad because it would have been very cool to combine the force of all my boxes to compile OpenBSD =) - but I'll survive om my 1.4 Ghz workstation

    3. By Marc Espie () espie@openbsd.org on mailto:espie@openbsd.org

      We don't support cross-building stuff yet, sorry.

      But building the whole ports tree on a box, and reusing the resulting packages on several machines is supported.

      In current, you can grab packages through scp, and more improvements are going to happen before 3.5.

      Comments
      1. By Jim () on

        Thanks for the clarification Marc. I saw a posting that niklas@ had a goal of making openbsd able to cross build on any arch for any other arch. I was hoping to merge that ability with the release capability to justify one uber fast build host...

        I found the cross-tools et al in src/Makefile that led me astray. Ah well.

  2. By Jordan Dubuc () on

    We maintain three OpenBSD servers in our environment. The boxes are locked down tight at the firewall level and hence don't have access to CVS repositories. We don't have extra hardware for a dedicated build machine so when it's that time, I get the base source on the servers either from CD or by pushing it out to the servers using SCP/SFTP.

    If it's a kernel patch I'm going to be applying, I of course go back and re-apply any previous kernel patches to the source, along with the new one(s). Compile, make install (or copy new kernel over old), then remove source from machines.

    Our BSD firewall server doesn't have the compiler tools on it normally, so in order to do patching I first have to mount a "tools CD" with all the necessary binaries.

  3. By tom () tom@replic8.net on mailto:tom@replic8.net

    consider using a central release distribution server (can be just a mirror of ftp*openbsd.org) and a set of customized boot floppies having the distribution server and path preset.
    if your machines can be managed remotely (serial console for instance) netbooting is very handy to upgrade as you don't have to run around hassling with medias.
    upgrading packages can be done in a similiar way. keeping them on the distribution server and creating some scripts to wrap around pkg_add (...) can do the job using local cron jobs or centralised using public key authentication with openssh.
    there are many more approaches... just think :)

    Comments
    1. By Anonymous Coward () on

      How do you install it through netboot? Do you have a howto or something? 10x.

      Comments
      1. By tom () tom@replic8.net on mailto:tom@replic8.net

        see http://www.deadly.org/article.php3?sid=20030109015748 for more on netbooting.

        i'm using intel based nics (fxp) pxe.

  4. By Michael Sullenszino () no spam at sullenszino org on mailto:no spam at sullenszino org

    Well, I have been doing this for about 3 years. I use only i386 hardware and have one master server behind the firewall that has ssh keys on all 20 or so of my openbsd devices (firewalls, webservers, mailservers, etc). When a patch comes out, I manually compile and test on the master server, then have written a kludgy little perl script that ssh's to all of the servers, mount_nfs -T /usr/src and /usr/obj and runs make install on those directories that have been patched (plus a check to see if we need the Makefile.bsd-wrapper to properly install).

    I originally did this as a whim with no intenet of supporting more than 3 or 4 servers that were on site. But it has worked so well in practice (i.e. even replacing libc and /bin and /sbin has been a breeze) that I am now supporting off site boxen with fixed IPs and no problems to date.

    As for the firewall, the updating script has a routine that checks for the current mountd ports used and updates the firewall to open those up for a short period of time to the IP's that will be using the master server for a 'make install'. Then you need to keep a list of all the directories that have been patched with a note as to whether you need the Makefile.bsd-wrapper, put that in a perl hash and away you go. This also helps in rolling out a new server. When I bring out a new device I can just patch it to current from the master server...

    I would appreciate any input with people who have more NFS experience than as to any pitfalls that this might bring about that I have not run into yet.

    Comments
    1. By Anonymous Coward () on

      Could you post the script that opens the current ports used by mountd? I've had trouble configuring the firewalls on nfs servers in the past and I would be interested in seeing what you have come up with.

      Comments
      1. By Michael Sullenszino () no spam at sullenszino com on mailto:no spam at sullenszino com

        Sure, it is at the bottom of this post. If the wordwrap thing comes out funny, email me and I can provide this for you via email or http post.

        3 things to note:
        -----------------
        1) the CVS master has ssh dsa keys on all servers, including the firewall allowing for remote passwordless root access. The security implications should be clear, you want this box behind a firewall and locked up!
        2) I put this script at the beginning of the script that goes out and updates the server and then put it at the end with the "off" argument to close off the firewall again.
        3) The firewall has this as part of its pf.conf
        --------------------------
        # Current portmapped mountd ports
        nfs_tcp="937"
        nfs_udp="642"
        nfs_server="192.168.0.22"
        table { 192.168.0.1/32, ...}
        [more stuff here]
        # Port Forward traffic to CVS master
        rdr on $ext_if proto udp from any to $external_addr port $nfs_udp -> $nfs_server port $nfs_udp
        rdr on $ext_if proto tcp from any to $external_addr port $nfs_tcp -> $nfs_server port $nfs_tcp
        rdr on $ext_if proto tcp from any to $external_addr port 2049 -> $nfs_server port 2049
        rdr on $ext_if proto udp from any to $external_addr port 111 -> $nfs_server port 111
        ....
        # Pass in rdr nfs traffic
        pass in on $ext_if proto udp from to $nfs_server/32 port { 111, $nfs_udp } keep state
        pass in on $ext_if proto tcp from to $nfs_server/32 port { 2049, $nfs_tcp } flags S/FSRPAU keep state
        pass in on $ext_if proto tcp from to $nfs_server/32 port > 500 flags S/FSRPAU keep state
        pass in on $ext_if proto udp from to $nfs_server/32 port > 500 keep state
        ---------------------------------------------


        Here is the perl script I use:
        ----------------------------------------------
        #!/usr/bin/perl

        # Script to pull the current nfs portmapped ports for udp and tcp,
        # then suck down the current firewall pf.conf from firewall. It
        # then updates the nfs_udp and nfs_tcp ports gleaned from rpcinfo -p
        # Finally, it scp the temp file back up to firewall and executes pfctl to initialize the new firewall.

        # Usage: $0 [off]
        # If the off argument is given, the script shuts down the port forwarding

        $firewall = "firewall.yourdomain.com"; # or ip address


        # Get current pf.conf from firewall
        @pfconf = `ssh root@$firewall cat /etc/pf.conf`;
        # Create temp pf.conf in /root/tmp/
        $pf_tmp = "/root/tmp/pf.conf";
        open(PF_FH, ">$pf_tmp");


        # See if this is a request to turn on or off nfs on the firewall
        if ($ARGV[0] eq "off") {
        foreach $line (@pfconf) {
        chomp($line);
        if ($line =~ /^pass.+nfs_server/) {
        $line=~ s/^pass/#pass/;
        }
        print PF_FH "$linen";
        }
        } else {
        # Otherwise get current mountd ports and reload firewall
        # Get current NFS/mountd tcp and udp ports
        $rpcinfo = `rpcinfo -p`;
        @rpcinfo = split(/n+/, $rpcinfo);
        foreach $line (@rpcinfo) {
        if ($line =~ /udp.+mountd/) {
        chomp($line);
        $line =~ s/.+([t|u])wws+(d+).+/$2/;
        $udpport = $line;
        }
        if ($line =~ /tcp.+mountd/) {
        chomp($line);
        $line =~ s/.+([t|u])wws+(d+).+/$2/;
        $tcpport = $line;
        }
        }

        # update and print out new pf.conf
        foreach $line (@pfconf) {
        chomp($line);
        # replace tcp port with current
        if ($line =~ /^nfs_tcp/) {
        $line = "nfs_tcp="$tcpport"";
        }
        # replace udp port with current
        if ($line =~ /^nfs_udp/) {
        $line = "nfs_udp="$udpport"";
        }
        # uncomment any commented out nfs_server lines
        if ($line =~ /^#pass.+nfs_server/) {
        $line =~ s/^#pass/pass/;
        }
        print PF_FH "$linen";
        }
        close(PF_FH);

        }
        # Send the new file back to firewall
        system("scp $pf_tmp root@$firewall:/etc && ssh root@$firewall pfctl -f /etc/pf.conf");

  5. By Matt Van Mater () on

    look around for fenderq's easybakeoven88 release building script (it has been posted here on deadly before)

    I use it to automatically generate a new release every once in a while. I also and added a few things such as:

    -upload new release to local ftp
    -grab release from ftp and unpack to / which upgrades my system to stable (after a reboot)
    -apply patches and recompile (so you don't have to cvs up to stable)

    This isn't useful between versions of obsd (you have to update /etc by hand and things like that) but at least it makes it easy to keep all my systems up to date while only having to cvs and compile one time.

    I use it at home on my servers, as well as at work on a handful of boxes.

    Comments
    1. By Anonymous Coward () on

      I, personally, would be interested in your modified version of this script. Any chance you can post it somewhere to copy?

      Comments
      1. By Sleepy () on http://www.maximumunix.org

        I did not know openbsd supported netbooting.that is nice to hear.
        we are a linux dev box.but I run OpenBSD on the firewalls and all production(DNS,Mail,etc) servers.
        mainly because of the ports collection , security and consistency.I don't install compxx.tgz on the boxes, I do have one machine where I do the untar->configure->compile->install
        cycle.I have hacked around SlackBuild scripts(yeah I know, but slack is the best of that breed) and it creates packages for Me,this have been working well for me in the last couple of years.
        when it comes to kernel, I usually have 2 kernels
        one for the mainstream hardware, and one specific(I got few soekris around for internal usage)
        take a look at slackbuild. it would be nice to discover if something similar(or better) exists for OpenBSD.

        Comments
        1. By Anonymous Coward () on

          slackbuild looks interesting. how does it differ from pkg_create?

      2. By Matt Van Mater () on

        I want to clean it up a little first, but then I'll post a link to it. I'll have it up by this weekend.

    2. By Anonymous Coward () on

      > "look around for fenderq's easybakeoven88 release building script (it has been posted here on deadly before)"

      This has the juicy bits:
      http://www.deadly.org/article.php3?sid=20030319035045

    3. By andre () on http://rootshell.be/~andre

      > This isn't useful between versions of obsd
      > (you have to update /etc by hand and things
      > like that)

      You might want to have a look at updiff.sh (http://rootshell.be/~andre/software/updiff.sh).
      There's a README file at http://rootshell.be/~andre/software/README.updiff

  6. By Jascha () jascha@localareasecurity.com on mailto:jascha@localareasecurity.com

    I am looking for any benchmarking or performance numbers from people using OpenBSD in corporate enviorments as a firewall. It is for a paper I am putting together comparing offerings such as CheckPoint with solutions such as PF/OpenBSD. Please email me if you know of good sources or can provide this material from a production enviorment. . .

    Comments
    1. By Anonymous Coward () on

      A good start:
      http://www.benzedrine.cx/pf.html

      The presentation has some comparisons:
      http://www.benzedrine.cx/sucon/

    2. By Dom De Vitto () on

      Ask henning.

  7. By Willb () on

    We need a ports tree that has pre-numbered patches.
    We don't know what the patch will be, but the port is just a fake...it will then go out and download the real patch as if it were a dependency.

    When patch 16 comes out, we do a make 16. Ports then fetches the patch source from a repository, compliles, installs, etc.

    If only I knew anything about the ports system.

  8. By Christopher () on

    I've been curious if it's practical to maintain a central server that's kept up to date, and running a cvs server, then just checkin the whole system, binaries and all. On the production servers it would be a matter of updating the patched software via cvs. There must be drawbacks though as I've not seen it discussed, and unfortunately I haven't had the opportunity to try it out in practice.

    Comments
    1. By Peter Hessler () spambox@theapt.org on http://www.theapt.org

      I don't check in binaries to CVS, but I do keep a list of ports that are installed, and I also check in the port config files.

      Restoring the server (should be. I haven't tested it yet) hsould be as simple as: fetch cvs info, install all of the ports listed, install the config files, and reboot. I'm very careful to edit the cvs version of my config files, rather than the "live" version, to make sure everything is checked in.

  9. By Anonymous Coward () on

    I would search for the actual posts and use links, but there is no way to search comments. :|

    "I agree. I even took it a step further. I have a "gold server" where I build releases. Once the "make release" process is complete, I rsync the new install sets to our ftp server. All openbsd servers that I manage have a perl script installed that I wrote that then query /etc/obsd_release, which contains the list of install sets that were used to build the current system i.e. base33.tgz, etc33.tgz, bsd. The perl script fetches all the install sets relevant to the current system, installs, then runs mergemaster to ensure /etc and /var are up to date. I have been doing this for over a year for at least 10 openbsd boxes and it took me ~10 minutes to get them all up to date. I just need a faster "gold server" cause it takes quite a while to build the new install sets. :)"

    ---

    "i usually have some system behind the firewall, generally a faster system with disk to spare, and use it to build releases. see "man release" for that procedure.

    then, you transfer the updated .tgz files you need and unpack them, except for etcXX.tgz. if you see that there are changes in /etc to be made, the thing to do is unpack etcXX.tgz in /tmp or some other place, and merge the changes."

    Comments
    1. By Anonymous Coward () on

      Maybe the owner of the "gold server" will reply. ;)

      Comments
      1. By James () james@unixninja.us on http://www.astrocreep.net

        OK, since people are still interested I will release the scripts soon. :) I have actually been in the process of rewriting said perl script this past week. I need to update the documentation as well.
        Send me an email if you are interested.

        Comments
        1. By James () james@unixninja.us on mailto:james@unixninja.us

          Here it is. It's pretty straightforward, I hope. :)

          https://www.astrocreep.net/scripts/obsdsysup-0.1.tar.gz

          Comments
          1. By James () james@unixninja.us on mailto:james@unixninja.us

            And of course, the gpg sig can be found here:

            https://www.astrocreep.net/scripts/obsdsysup-0.1.tar.gz.asc

            Comments
            1. By Anonymous Coward () on

              btw, thanks. :)

  10. By Jeff () jtharp@smalldark.net on mailto:jtharp@smalldark.net

    As the original poster of this question, I just want to thank everyone who has responded. Lots of good suggestions and links for me to go read.

    I definitely agree with the idea of a single master server to use for distributing updates. It also looks like 'make release' may be helpful. Add in a little PERL or shell script glue and some SSH magic and I can probably get what I was looking for.

    I didn't mention in the original article, but the perfect niche I've found for OpenBSD is as a reverse proxy for exposing internal webservices to our DMZ apps. This works great because the default install includes all I need: Apache, mod_proxy, mod_ssl, and pf. This allows a few layers of security between the DMZ app and our internal enterprise systems. I've talked about using a similar reverse proxy solution as a layer of security in front of our external websites as well. There was a good article on doing this and including the new mod_security. Just not sure if the performance impact would be bad or not.

  11. By Anonymous Coward () on

    I have used webmin to remotely administer multiple systems. There are a number of modules for user admin, system admin and some apps. The drwback is that you have a mini web server running on the platform and another port open. The web server can be set up as an ssl connection and I'm sure you can mitigate most of the security issues by having the webmin port only open on the private nic for firewalls. I do not have first hand experience with this tool on systems in a dmz or public access.

  12. By Anonymous Coward () on

    obsdsysup is now available:
    https://www.astrocreep.net/scripts/obsdsysup-0.1.tar.gz


Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]