OpenBSD Journal

keeping up with errata

Contributed by jose on from the automation dept.

cabbage writes:
"Having recenlty (6 months), switched from linux to OPENBSD for 2 production web servers, I need to know the best way to keep my system in step with the patch branch.

The documentation seems to implay either updateing my source tree and ports against the patch baranch, then compile (kernel/sorces/ports), or get the patches and compile / install manually.

Am I understanding corretcly?, and which method do most admins prefer?"

Has anyone developed a good automation system for keeping -stable up to date on a machine? Would it be as simple as checking against the -stable CVS branch and launching the needed builds?

(Comments are closed)

Comments

  1. By Sean () null@tfh.ca on mailto:null@tfh.ca

    It really depends on what the machine is running (that's not in the source tree) and how much you trust rebooting to patched kernels.

    This can be a problem when you compile a new kernel you must reboot the machine (aka loading the newly built kernel) and then build userland (which may or may not fail). I wouldn't suggest automating this process unless your script to do so is very smart (aka won't reboot if the kernel build failed or will notify you if/when rebooted and building userland failed or not).

    Plus in a perfect world you would be advised to rebuild any third party applications you have from the ports tree (or by hand if not in the ports tree).

    All in all your best bet would be to keep up with stable userland and only rebuild the kernel when there are advisories which affect your services particularly. This is especially true if the machines are not within punting distance.

    I've thought of automating this process myself but there are too many variables that can cause things to go wrong and thereby cause you more pain and trouble. Instead if major updates need to be done (such as kernel security errata) schedule it with your users and live with the downtime.

    Please keep in mind this is all IMHO and based on MY current set of experiences with this.

    Comments

    1. By Anonymous Coward () aschlemm@attbi.com on mailto:aschlemm@attbi.com

      How one goes about getting patches either by download the patches and applying them or by using AnonCVS is well documented on the OpenBSD website (www.openbsd.org). From the steps they list, I've placed the commands into shell scripts and can update my source tree via AnonCVS with a single command. Since I have several critical systems running OpenBSD and I have setup a test/build system which I keep a stable source tree on for the purposes of building kernels and for making my own OpenBSD OpenBSD releases.

      I am able to quickly update the version of OpenBSD on my production systems by downloading my release tarballs from an internal FTP server and un'taring them. I received an update script from an another OpenBSD user and have adapted it for my environment. I put the system I am updating into single-user mode first so it is quiet when I'm doing the updates. I'm lucky that I have physical access to the systems to do this. I know others do this sort of thing in multi-user mode since putting a remote system into single-user modem will drop all of the network connections.

      I have a second test system which I test my updates on to ensure that everything upgrades smoothly and the system comes back up after a reboot. One I'm am sure my compiled release works I then go ahead and update my productions systems.

      When I was trying to figure out how to keep my systems up-to-date I got a very good RTFM response from many OpenBSD users. Check out the manual page for "release(8)". "man release". This has the steps to go through for building your own OpenBSD release.

      Tony

      Comments

      1. By Anonymous Coward () on

        1. CVS tracking is a given.
        2. You have physical access to the machine.
        3. You obviously have some downtime there (whether unanounced or not).

        That last two were just restating the limitations stated in the previous post.

        While installing via the release tarballs is acceptable for some it may not be. It is a decent suggestion none the less.

        Comments

        1. By Anonymous Coward () aschlemm@attbi.com on mailto:aschlemm@attbi.com

          True but the guy I received my "update" script from actually does all this through SSH remotely. Again it's a bit of risk to do things this way but when it comes to updating a running system with a new kernel there is not much of a choice since a reboot is required to use the new kernel.

          When I didn't have a dedicated build system I did manage to update my systems albeit rather slowly. I would get an update-to-date source tree of stable and after having compiled a new kernel I rebooted the system so it was running with the latest kernel, I then set the system off compiling the entire source tree and let it install itself.

          I found this method worked quite well and with the build processes run using "nice" they don't seem to impair the system too much since the build related processes run only when no other processes need to run. This method can take considerable time and you need to have enough disk space on the system being built to perform this operation. This method may be more acceptable since it only requires a reboot to get the system running under a new kernel.

          The rest of the system is built in place which may be less intrusive. The only thing that may still be a problem is that I'm not sure if the old code pages in memory would be released or not since there would be active processes referencing the old code in memory. The only way to ensure that the new code is running may be to reboot the box after the system build has completed its run.

  2. By Anonymous Coward () on

    I kind of miss Solaris for this exact thing. I had some really nice scripts that would just grab a patch (binary patch, more akin to a package than a source code patch) from a central repository on the University's network, try installing it (stopping on failures), and then letting me know when it's done by paging me in case it needs a reboot. I imagine something like this could be done for an BSD box, if things were moved into packages. So the next time apache or ls has a problem, you can just grab the patch-package and install it. The ports/packages infrastructure already exists... it just needs someone to do it.

    Comments

    1. By Anonymous Coward () on

      I kind of miss Solaris for this exact thing.

      most solaris patches dont apply cleanly for me and require a lot of user interaction. i like redhat's system of rpm's, though i wish they were a bit quicker with security updates.

      I imagine something like this could be done for an BSD box,

      that tepatche guy has something like that - http://www.gwolf.cx/soft/tepatche/

      your best bet if you have a lot of systems is to have standardized hardware. makes rolling out patches a lot easier when you have an exact duplicate test machine to work with first, make sure nothing unexpected happens, then allow the other servers to be patched.

      unfortunately only the big players get to have such nice setups and the rest of us are stuck with whatever this year's budget could afford, so organization is key b/c knowing exactly which servers have what hardware and are running what software for those purposes is necessary. that way you know which errata affect you.

      lots of coffee helps too.

      Comments

      1. By Justin () on http://www.gentoo.org

        Gentoo Linux stole the ports system from OpenBSD which stole it from FreeBSD. They renamed it portage and it works extremely well (no surprise) and handles all installations, upgrades/patching, etc. using the command 'emerge'

        Gentoo Linux, for those that don't know, is a source based distro meaning you compile everything during install, and everything new you want to add you compile like using ports that we are familiar with. It is really clean and neat. I think this would be a great idea to re-steal from Gentoo but of course I am not a programmer so I would have to wait for someone else to do it.

      2. By Nicolai () brochard@dolfijn.nl on mailto:brochard@dolfijn.nl

        This is something Redhat does right, and one of the main reasons I've chosen it over any of the BSD's for the primary- and webservers for the ISP i manage technically. Redhat is pretty fast with it's security fixes, though not as fast as OpenBSD. But applying them is so much easier. They're in binary form, all I have to do is rsync my apt-rpm repository with ftp.redhat.com, and have the apt-rpm clients check for updates every night, And applying them automatically. No kernel recompiles, just install an rpm containing the binary (MD5 checksummed). I manage 30 servers alone, takes only a small amount of my time, so there's more left for development. Try that with OpenBSD. (No flame, my bridged firewall at home is OpenBSD CURRENT).

    2. By RC () on

      Why is this so complicated for everyone?


      cd /usr/src
      ftp ftp://ftp.openbsd.org/pub/OpenBSD/patches/3.1/
      mget *patch*

      patch <*patch*
      make && make install

      That wont work without a few changes, but that's the right idea. How long would it really take if one of you would stop complaining and write the script to do it?

      Hell when I complained about the PDF of the FAQ being out of date, and was asked to write a script to do it automatically, it took me less than an hour to make a debug something that:

      creates the directory structure
      fetches the web pages & images
      combines the pages nicely
      and converts it from HTML to PDF

      Of course, that script never got put into use (...), but still, it takes a lot less time to do it than it does to complain about it.

      Comments

      1. By Anonymous Coward () on

        I dunno. I can complain pretty quickly.

      2. By Anonymous Coward () on

        There is a huge difference about doing it and doing it the right way

        Comments

        1. By RC () on

          Yes, but the time difference is nominal...

    3. By OpenBSD Bigot () masterofnone@nulldevnull.org on http://www.openbsd.org

      You cannot begin to compare Solaris with OpenBSD. First of all Solaris is proprietary, and has ONLY support for a limited amount of known hardware configurations. OpenBSD supports many different arch, so that alone makes it difficult to just say this patch fixes problem X. Because, it doesn't work that way. To understand this a little better, you should try some assembler for various architechtures. Create the same programs on many different machine types, and you will begin to grasp the reason that your Solaris experience may have been easier.

  3. By Jedi/Sector One () j@pureftpd.org on http://www.pureftpd.org/

    OpenBSD patches look like that :

    # To apply this patch, type:
    # cd /usr/src/usr.bin/foo
    # patch # make -f Makefile.bsd-wrapper clean all install
    # and reboot your system.
    [patch]

    What if instead of saying what should be done, the whole patch was a script so that typing ./patch-015.sh would apply the patch, compile the stuff, and send a mail to root to say "hello I've applied that patch, please reboot the machine as soon as you can"?

    Not only it would be easier for newbies, but it would also make automatique updates possible.

    Comments

    1. By Anonymous Coward () on

      Good point.

      And don't forget, that patches also should have checksums which should automatically (somehow) be proofed and verified.

    2. By Noryungi () n o r y u n g i @ y a h o o . c o m on mailto:n o r y u n g i @ y a h o o . c o m


      Make patches SCRIPTS???

      Sure...

      For instance, OpenSSH was cracked a few months ago, and some binaries replaced with trojaned versions. The problem was caught quickly, but it was still scary.

      Now, imagine that OpenBSD.org is cracked in the same way, and that every patch includes some 'extra' instructions to install a nice little trojan from www.hackers.to or some such.

      Download patch. Apply patch with nice little script. You just got 0wned!

      Make patches into scripts? Thank you, but ... no, thanks. Newbies be damned.

      I don't run OpenBSD for that.

      Comments

      1. By Anonymous Coward () on

        > Now, imagine that OpenBSD.org is cracked in the
        > same way, and that every patch includes
        > some 'extra' instructions to install a nice
        > little trojan from www.hackers.to or some such.

        Though I see what you mean, it is way out for proportion! As it is now, the patch IS a script! It just needs you to walk the steps, but it no more, no less safer (as in spoofing) than a real script.
        The problem is that it really isn't a no-brainer. Is you /usr/src there? Is it patched by other sources? Does it compile? Does some conf file needs tweaking? 99% of the time it just works, but you need to see it works.
        Talking about security, do you check the diff's when you cvs up the source? If not, you are risking more by cvs than patch scripts. Please keep your perspectives, buddy.

      2. By Anonymous Coward () on

        For instance, OpenSSH was cracked a few months ago, and some binaries replaced with trojaned versions. The problem was caught quickly,

        And how was it detected? Because someone checked the MD5 sigs!! Your average plankton *BSD user rarely does this, I know, but the scripts would be SHA1/MD5 signed for this reason. Oh wait a minute, having to check an MD5 sig takes precious minutes from your day, right?

        Comments

        1. By Noryungi () n o r y u n g i @ y a h o o . c o m on mailto:n o r y u n g i @ y a h o o . c o m

          Oh wait a minute, having to check an MD5 sig takes precious minutes from your day, right?

          Surprise, surprise: I do check the MD5 every time I grab anything from the 'net.

          My point is this: there are no perfect systems. A site can be cracked and patches compromised.

          And you supplied the answer yourself:

          Your average plankton *BSD user rarely does this

          Yes, sir. And that is why auto-download, auto-patching, auto-anything is evil .

          If I wanted auto-update, I'd install Win2K, and not a BSD, thank you.

          Comments

          1. By Anonymous Coward () on

            Go use Win2k, because, as you said, there are no perfect systems, and you seem pretty happy with an imperfect OS after all.

            Yeah, it is a surprise that you had a clue given your previous comments. If you do checked checksums and had known how OpenSSH was trojaned, it's blatently trivial to realize autoscripting would have made the md5s more utilized, not less, and caught the problem in the very example you put forward, debunking your argument before it got posted.

            Instead, you backpeddled and changed your point by redefining it.

            Oh well. You were wrong. I don't care if you realize it or not. Back to the discussion:

            OBSD is highly likely not to make changes to their patch system, if for the simple reason that it is a more raw and pure OS, somewhat traditional if you will. But that does not change the point that they should make changes such that there is simple flag so people can autorun patches into their system or choose not to. Such a solution would not overstep what you want in an OS (being the lack of default autopatching) but still make it somewhat more realistic, if the owner chooses, to automate patches without a 3rd party script.

          2. By Anonymous Coward () on

            MD5 does really nothing to improve protection against trojaned patches. If a server is compromized and patches are horsed, well then, the MD5s will probably be doctored too. It works for the ports tree, since it is distributed with the CDROM set, but how will you have an MD5 for a new patch? The only thing that would work here is a PGP/GPG signed patch.

            And all this is not really affected by scripting or not. I would even say that scripting is better here, since it will check signature by default rather than depending on how paranoid you feel today.

            Sorry that we bug you on this, but to me you sound like a control trip person who is trying to sound sooo L33+ and sooo "security aware". Good for you, but computers are better at doing things persistant. Good scripts works better than good intensions.

        2. By Anonymous Coward () on

          they're not with the patches themselves (on ftp.openbsd.org/pub/OpenBSD/patches/3.1/common)

  4. By Anonymous Coward () anonymous@coward.com on www.coward.com

    I usually just have a release copy of the source tree on a highend development machine. I apply the patches by hand to that source tree and compile whatever is required. I next create a new release build following the "man release" instructions and then copy and tar xvfzp the required *.tar.gz to my other systems.

    I had to do this since most of my firewall systems were 386 and 486 machines. They would take too long to compile on and I'm paranoid to the point of not wanting a compiler on my firewall systems.

    Comments

    1. By Anonymous Coward () on

      I'm used to export of /usr/obj and /usr/src on a developer machine; mount them on the target and issue "make install" there... kernel gets tested and scp'd...

  5. By Mat Charles () mjc42-www@doggi.org on http://www.doggi.org/~mjc42/openbsd/

    There're a few different ways of approaching this, depending on how much control you want over the process. Everyone has their own preference, so what works for me may not be best for you.

    Me, I find it easiest to track the STABLE branch with CVS updates. Applying the patches individually would give you more control and let you scrutinise individual changes beforehand. I prefer to update everything by CVS in one fell swoop, though - that way I can be sure I have all of the updates and a self-consistent source tree.

    You could automate the builds, but I much prefer that a person be involved to make sure that nothing unexpected happens. You also need a person to do the post-build things which require understanding of the context - do any daemons need to be restarted? If you updated the kernel, you'll want to reboot and then rebuild userland. Occationally, some config files in /etc will need to be updated. And so forth. (There's also the slight risk of the build failing for some reason. This shouldn't happen if you have a consistent source tree, but very occasionally the tree will be out of kilter. That's an argument for using hand-applied patches, I guess.)

    One thing I'd strongly recommend is having a non-production box which you use for the updates and builds. That way, your production boxes aren't down if something goes wrong. Also, you won't be thrashing your production boxes during the build. Once you're satisfied, the binaries on the production boxes can be updated. There's more than one way to do that, too - I like to sync the src and obj trees then make install, some people prefer to make release.

    I have a recipie that I use for my own updates at http://www.doggi.org/~mjc42/openbsd/ which you're free to peruse. I don't guarantee it's the best way for you, but it might give you some ideas.

    Cheers,

    Mat.

    Comments

    1. By Anonymous Coward () on

      A couple of how-to's that could be useful to you if you plan to go this route can be located here:

      http://www.screamingelectron.org/phpBB2/viewtopic.php?t=81
      http://www.screamingelectron.org/phpBB2/viewtopic.php?t=82

      They have helped me out in the past.

    2. By Anonymous Coward () on

      I agree; a good solution is two have more than one box. If it's financially possible, use 2 boxes (1 live, 1 standby), and update and build the standby box, verify it, switch it into production. You can always roll back to the old box if there's a problem. Later on you can update the old box, rinse, repeat, etc. Easy peasy (as long as you can afford the extra HW.)

  6. By Anonymous Coward () on

    Also remember that you don't always have to apply errata fixes if they don't affect you. If you just run a mailserver then most of the errata is not of interest to you. Don't just upgrade because there is an upgrade out, check first if it applys to you.

    Comments

    1. By Anonymous Coward () on

      You should _always_ patch your machine, whether you think you understand the full implcations of a race condition/overflow/bug or not. Just because you do not permit untrusted users to have local account, does not mean local exploits cannot be used against your machine in some indirect way.

      Comments

      1. By Anonymous Coward () on

        Why should I patch my machine when there is a security bug in IKE and I am not running that. If you are not sure then of course you should patch. If you know what you are doing you don't want to upset a running system if there is no need. I don't patch the boxes I am administrating just because there is a patch out. System administration is also to know what to do and what implications it will have.

        Comments

        1. By W () on

          I can vouch for that.

  7. By Anonymous Coward () on

    Is there a way to tell which patches you have applied to a system? I imagine i should go rtfm, but i don't have a obsd box handy.

    Comments

    1. By Anonymous Coward () on

      Succinct answer: NO

    2. By Dimitar () vassilev_dimitar@yahoo.com on mailto:vassilev_dimitar@yahoo.com

      Write down what you do by dates and time.Or... create a Perl script that searches for the lines
      in the patches in the target files, containing directions like:
      A)find the target file - can be grepped from the top of the patch
      B)grep the diffs in the patch
      C)search whether the target file has the diffs included
      D) exit 0 if all is ok and provide a small msg like "patch already applied" and a APPLIED/EOF reference.
      E)Unfortunately I am at the very beginning of studying Perl and can't do it myself. Linux's netfilter up to 1.2.5 had a script called `is applied` which parsed kernel code for such references. Check it to seek inspiration.
      F)If someone decides that my idea is worthy, has the resources to do it, feel free to do so under whatever license you wish.Don't forget to send me a brickbat and include me in the README.

    3. By Anonymous Coward () on

      Of course there is a way!

      Read the patch, and read the source. If the source matches the changes called for in the patch, then you've applied that patch. If they don't match, then you haven't applied the patch.

      Reading is good for you and does not hurt. Do it .

      Comments

      1. By Anonymous Coward () on

        The whole point of using computers is to be more productive and make something simpler, not to prove how smart you are. I would think that this is such a basic function.

        I use OpenBSD because I enjoy it, not because I have to, but reading the source to find if a little snippet of code has been changed seems like unnecessary elitism.

        Comments

        1. By Anonymous Coward () on

          Amen. I wholeheartedly agree.

        2. By Anonymous Coward () on

          The truth is, if you are not willing to get your hands at least a little bit dirty, then OpenBSD is not for you. This has nothing to do with elitism. OpenBSD is absolutely designed, executed and maintained as an O/S for those that can and will delve into, at different levels, its various pieces.

          You don't have to be coder of the same caliber as the core team to successfully use and maintain OpenBSD. You just need a willingness to go beyond the point-and-click, 'automate everything for me please' attitude. It really doesn't take much to 'read the source to find if a little snippet of code has changes', and the benefits of doing so are immeasurable in terms of learning. The use of OpenBSD, with any regularity, for any sustained period of time, is evidence of a commitment to, and love of learning, quality and exactitude. Just as OpenBSD's quality, security and stability are evidence of the same motivations for its developers.

          The fact is, OpenBSD's goals are not to make system administration (of which patching, upgrading, etc. are a part) simpler. OpenBSD's goals are high quality source code and security which is decidedly NOT simple. The user experience the developers are aiming for here, is one of security, stability and reliability, not user-friendly interfaces and petty functionality. There are enough examples of operating systems that show us that pretty ain't always nice, and sometimes, the not so pretty and not so easy has a beauty that runs much deeper than pretty.

          Just read the source (even a little), and you'll understand what OpenBSD is really about: Then you'll be able to make an educated decision about whether, and how to best make use of it as a tool. Again, elitism has nothing to do it. There is no room for that here just as there is no room for people who want it all served to them in an easy to swallow pill: you can get that elsewhere -- just be warned, there are some nasty side-effects.

      2. By Dom De Vitto () dom@devitto.com on mailto:dom@devitto.com

        Oh, of course you'll need to be an experienced C programmer, understand the kernel (so read all that code first) and make sure you understand the program and dataflows/structures in the program being patched.

        Learning C may only take a 6 months of evenings, the kernel, maybe only a couple more months and the program maye only a week, so make sure you'll calander has you "busy" until June 2003.

        Meanwhile hackers are running around your systems.....oops.

        Hello? If I wanted to read all the source code, I would have written the OS and support utilities myself (*I* actually am skilled enough), and I'd have a lots better security too, 'cause of security through obscurity...

        The rest of the world needs binary patches.

        Comments

        1. By Anonymous Coward () on

          yer a fuckin' idiot! You are skilled enough for nothing. Security through obscurity. Yeah. OK. Isn't that what the large closed source O/S companies rely on?

          Why _don't_ you go write your own O/S. Clearly, to you, OpenBSD is not safe. Hackers must be running around your systems.

          Applying a patch, reading a patch and a few lines of source code to see if the patch has been applied does not take mastering C, kernel, etc. Read fucking english, dumb fuck!

    4. By Anonymous Coward () on

      I have a directory /usr/src/patches where I leave the patches after applying them. Good memorandum.

  8. By Gustavo Franco () stratus@acm.org on mailto:stratus@acm.org

    You can try the tepatche solution at http://www.gwolf.cx/soft/tepatche/.Previously posted here at: http://www.deadly.org/article.php3?sid=20020627195351

  9. By Dom De Vitto () dom@devitto.com on mailto:dom@devitto.com

    Yea, yea, I renamed all the libc functions so I need to apply patches manually.....blah, blah, blah. Sheesh, some people will always need to do things the "full" way. They are usually the ones who have done something advance or non-standard, and know how to apply patches. The rest of the world read on....

    99% of obsd boxes just want the frikken bad binary(ies) replaced, and a reboot to get everything sane. How do Sun/MS/Apple issue updates? They send you the changes from the "base", they don't tell you to download the OS from CVS and rebuild.

    If the changes are to the kernel, and you're not running GENERIC, you'll need to rebuild. So kernel patches do cd /usr/src/sys/...`uname -v`; make depend && make && echo You need to manually make install and reboot. Thankyou. Luv Theo.

    I think I'm going to setup a binary dist site for this....anyone interested in giving me feedback/advice ?

    Dom

    Comments

    1. By Anonymous Coward () on

      I agree with you totally. Just because you can compile everything from the source code doesn't mean you should HAVE to.

  10. By Anonymous Coward () on

    This isn't related to OpenBSD specifically (so let the flames fly if you want) but it's directed to all the people posting here that they chose to use Redhat instead of a BSD operating system because of rpm updates. Check out binary updates for NetBSD. The cost of the service is pretty reasonable too.

    Comments

    1. By Gerardo Santana Gómez Garrido () santana at openbsd dot org dot mx on http://www.openbsd.org.mx/~santana/

      I've made a modest attempt of a system for building binary patches previously posted here:

      http://www.deadly.org/article.php3?sid=20020708131901

      It's safe and easy to use binary patches. It would be great to see the OpenBSD project adopting a binary patching system.

      Another thought is that there would be no need for those "polling systems" that keep querying the web or the FTP server for new advisories. It would be more efficient to use appropriately the security-announce mailing list.

Latest Articles

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]