OpenBSD Journal

Major system upgrades

Contributed by jose on from the major-leaps dept.

Michael Anuzis writes: "Hey deadly.org. My boss has just asked me to take his old server of OpenBSD2.6 running various services such as DNS, and e-mail/web with virtual hosting for around 30 different companies and with 80 users on the box... I have to admit I've never done anything like this before and I'm wondering if there's any documentation or advice available that would give me pointers to make sure I don't screw this up, especially with the transition from a.out to ELF. Things like transfering the user accounts over without bothering each user, etc, are some of my main concerns. TIA" This does get tricky with such a big leap. Almost every subsystem has been dramatically matured in the intervening three years: IPF to PF, BIND4.9 to BIND9.2, OpenSSH upgrades and configuration changes, Apache is now chrooted by default, current is now ELF, and the list goes on. So, the lessons from the Upgrade Mini-FAQ don't apply so well.

Anyone have any suggestions for Michael to try and make this transition as painless as possible (keeping his configuration and services as intact as he can)?

(Comments are closed)

Comments

  1. By sthen () on

    I think it's easier and safer to treat it as a fresh installation...Ideally use a new hard-drive on spare hardware, build the system, then put the drive into the 'real' machine. <br> <br> If nothing like it already exists, I'd recommend taking the opportunity to write some scripts to build configuration for a domain (add user to passwd files, setup virtual hosting, email forwarding, DNS and so on), then you can use them to ease the transfer and it's a big help when you want to add new sites in the future. Making them modular is helpful and makes maintenance a lot easier... <br> <br> If you keep the same uid/gids you can just merge the passwd and group files and it makes transferring documents for the websites across simpler. Test to make sure things are working, when you want to actually swap across to the new system, disable updates (stop ftp logins etc) and rsync the changes, it'll save a lot of down-time. Switch to the 'real' IP address, move the disk, and fingers crossed.. If you're allowing scripted websites with filesystem access (CGI scripts etc), take special care, you might want to disable them while actually carrying out the transfer, and check they still work OK afterwards. <br> <br> If possible, I would prefer to warn people fairly well in advance of the actual change-over so they can have a quick once-over to make sure their sites are still functional afterwards :-) <br>

    Comments

    1. By Jason () on

      A fresh install is best. I just want to add that when you move user data over, do it twice. Once when you are first building the new system, then another rsync just before you do the change-over (after disabling logins). This way you REALLY minimize your downtime.

    2. By Anonymous Coward () on

      Yeah, that's probably the best approach. I'd give the users at least a couple week's warning and schedule the change over for a weekend when most sites are likely quiescent. If the resources are available, since this machine is probably pretty old, it may be a good time to just fabricate a completely new machine for better performance (three years of hardware revisions are out there too, and price drops on old stuff ;) e.g. nice dell 2450s can be had cheaply on ebay that would have cost $thousands three years ago), install fresh over there, then do the copy-over dance. Keep the old server as a hot spare for a couple weeks to make sure the transition went ok, then reinstall it with current and back-sync periodically to have a (good, updated) hot spare.

  2. By eb0la () on http://exocert.com

    Don't worry about moving from ELF to a.out. <br> 3.3 is the *last* i386 release coming in a.out (afaik). <br> <br> If you can wait 2 more days, upgrade to 3.3. <br> <br> I guess as soon 3.4 will be available (December?) you should be able to run you old a.out binaries without much trouble. <br> <br> Hope this helps, <br> <br> -- I&ntilde;igo

  3. By deekayen () on http://openbsddiary.org

    I say that will be painful no matter what. It's policy for my work to only run supported software, so I had to upgrade to 3.3 from 3.1 when the end-of-life notice came out. Between waiting for upgrades to finish, recompiling libraries, changing stuff in /etc, and modifying configuration like PF, I almost could have done a fresh install.

    My advice: wait for 3.3. Figure out how much trouble it is to do a fresh install, it's worth considering, IMO. If you decide to upgrade instead of fresh install, don't try to jump right from 2.6 to 3.3, do it one upgrade at a time along with the Mini-UpgradeFAQ, just to make sure you catch everything.

  4. By James A. Peltier () james@site-fx.net on http://www.site-fx.net

    Everyone is right. Think of this as a completely new install. Back up the relevant files like your master.passwd file and your zone info etc. Install everthing from scratch and just replace the modified files. Everything should work just the same. Two weeks lead time on the change over should be enough. And make sure you have enough people to handle the possible load after the change-over takes place just in case ;'D

  5. By Darren () on

    Of the "30 different companies and with 80 users on the box", how many are live, and how many are dead? It may be worth siving through them, to make sure unused accounts are locked.

    Also try to find out which sites and users are more important, so you can make sure those work first.

    Comments

    1. By Anonymous Coward () on

      "how many are live, and how many are dead?"

      Are you talking about the users/people...or companies?

      *grin*

      Sorry, couldn't resist.

  6. By Ben Goren () ben@trumpetpower.com on http://www.trumpetpower.com/

    Any chance that any of those sites are dynamic? If so, you'll need to make sure that the latest and greatest versions of Perl, PHP, back-end databases, etc., don't break the hosted sites.

    You'll really have to set this up as a new system. Once you think you've got the new system working as well as the old, you'll have to turn the users loose on it for a couple weeks, so they can be sure you're right. Be prepared, if necessary, to work with them to fix things and to push back the switchover when you discover that somebody needs to re-write that entire e-commerce backend.

    If things get too painful, then just go with two servers. Leave the old one for the incompetent ones who can't get their act together, but with a cut-off date that says, ``the plug will be pulled on such-and-such date.'' Switch everybody else over to the new one. Be sure that ``such-and-such date'' is negotiated and make it clear that, if the client can't fix what's broken, they're responsible for finding somebody else who'll support the old system. Or just charge them out the wazoo.

    Also, be sure to plan for this in the future. Publish a maintenance schedule that just happens to be June 1 / December 1 (or thereabouts). Use the month of lead time to either do dry runs on your development machine, or to prep the replacement computer and swap cables at the right moment.

    You might also consider whether breaking functions into physically separate boxes (one for DNS, one for Web serving, etc.) might make your life easier. At a few hundred bucks for a multi-gigahertz computer, money is no excuse.

    Good luck,

    b&

  7. By Anonymous Coward () on

    Get a different new machine, do not wipe the old one, even after the transition is complete. You never know when you might be scrambling to get something right that worked previously and is now b0rked.

    Nice new hardware is cheap - and can save your bacon if things fail. Worth the investment.

  8. By Michael Anuzis () on

    Thanks to everyone for your help. The advice so far is proving to be very valuable.

    To clarify exactly what's going on, I was definitely planning on performing a clean install and not trying to literally upgrade the same machine from 2.6 to 3.3.

    I've already got a new (but identicle, as far as hardware goes) server set up with a 3.3 snapshot installed (took a snapshot to avoid the ELF catch and allow for easier upgrades in the future). I'm going to be setting it up in the server room within the next couple days and begin transfering over all of the static websites and DNS zone files.

    One thing I'm especially concerned with, however, is the master.passwd file, and the whole issue with the user accounts. How would one go about transfering all the user accounts over from 2.6 to 3.3? Is it simply an issue of copying /etc/master.passwd and /etc/passwd to the new machine with identicle ownership/permission settings? Or is there more to it, ie. dealing with the new _shadow group, and /etc/spwd.db and so on? Is there a process/order for copying these files over to be carried out in? They seem to be a sensitive group of files worth taking the time to transition correctly. Any advice on this topic would be appreciated --Michael

    Comments

    1. By Anonymous Coward () on

      Mergemaster (in the ports tree) will definitely help you out here.

      Copy your current config files to the new machine, install a recent src in /usr/src, then install & run mergemaster, which will automate the process of updating all those files. Newer versions will automatically rebuild your passwd and/or aliases db files.

    2. By Ben Goren () ben@trumpetpower.com on http://www.trumpetpower.com/

      Michael,

      First, you'll be able to get the official version of 3.3 on Thursday if you can beat the crowds at the FTP sites and can wait that long. Then, there'll be no need to go with a snapshot.

      Answers to your questions about the password file can all be found in pwd_mkdb(8) and passwd(5).

      Personally, I'd take the opportunity to do a bit of house-cleaning amongst my users. Let them know that you'll be doing some serious spring-cleaning and that they'll all be assigned new passwords as a security precaution--which is the exact truth. You don't want somebody using the same password for three years! Require them to get their new password from secured means (in person, over the telephone, via FAX, etc., but *NOT* over the 'Net). Also read up on the password aging mechanisms detailed in passwd(5).

      You'll probably not hear back from--or not be able to contact--some of the users. These people must not have accounts on your system. You've got no way to reach them; how can you possibly hold them accountable for anything they do?

      Good luck,

      b&

      Comments

      1. By Brad () brad at comstyle dot com on mailto:brad at comstyle dot com

        >>(took a snapshot to avoid the ELF catch and allow for easier upgrades in the future)

        >First, you'll be able to get the official version of 3.3 on Thursday if you can beat the crowds at the FTP sites and can wait that long. Then, there'll be no need to go with a snapshot.

        Did you not read what he said? He clearly stated why he used a snapshot.

        Comments

        1. By deekayen () on http://openbsddiary.org

          Demanding stability out of an ELF snapshot right now is silly. It severely lacks testing. He went with 2.6 for such a long time, using a.out for a while longer obviously won't hurt.

          Comments

          1. By Michael Anuzis () on

            Hmmm. I have to admit I haven't had any experience with OpenBSD on ELF yet outside of just installing it. Would anyone who's really been able to work with it be able to vouch for its ability (or inability) to work with these basic requirements like web/DNS/e-mail service?

            After asking around on IRC (ef-net #openbsd) I was told I'd be safer going with a snapshot than taking the release because it would let me dodge a difficult transition in the future that'd leave half the files a.out (whatever I compiled on my own, ie. postfix etc) and half ELF.

            Comments

            1. By James A. Peltier () james@site-fx.net on http://www.site-fx.net

              This is completely false. You should _NOT_ be running a snapshot this early on in the -CURRENT development process on a production machine. As stable as the -CURRENT tree is _most_ of the time things are bound to break and when they do... well you figure it out.

            2. By tedu () on

              a couple of ports are broken. mainly boehm-gc, which doesn't work on any ELF platform, and the pseudo-port of mozilla.

              ELF was tested for a long time before the commit. It's not like it was developed one day and committed the next. Hasn't really destabilized anything.

              If you're going to wait another three years before upgrading, I don't think it matters. If you intend to upgrade more often, switching now will save some pain later.

              Comments

              1. By GPS () on

                Sad news about the Boehm garbage collector being broken on all ELF archs. I recently started using the Boehm GC in OpenBSD 3.2 and I love it. I hope that they can find a way of fixing it. I'm even willing to devote a little time to try to fix the problem in -current using my Bochs setup.

            3. By crc () maxentropic at hotmail dot com on mailto:maxentropic at hotmail dot com

              I've used the 4/26 snapshot for a few days now on my home firewall. No issues so far, but I'm not hosting other peoples' sites, either.

              cc

  9. By Michael Anuzis () on

    My sincerest thanks go to Mark Martin for greatly helping me organize a logical structure and methodoligy to the transition. I thought the time and energy he put into writing out a sound method was so well created I re-organized his e-mail into a small website made available at:
    http://www.anuzis.net/openbsd/systemupgrade.html
    I hope other people can find this information useful in similar scenarios in the future.

    --Michael Anuzis

Latest Articles

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]