OpenBSD Journal

Updating from ports... easy!

Contributed by deanna on from the /bin/sh-is-passe dept.

Paul Irofti writes:

I've been following current for quite some time now, and since I always used snapshots and `pkg_add -u`. Although any new application that I wanted to add was through ports.

Last night, after upgrading to the final snapshots I decided to do a full update from ports. I just received an extra dimm of 512DDR and it was time to test the speed changes:)

After browsing through ports(7) I saw that upgrading was easy:

     update      Alternative target to install.  Does not install new pack-
                 ages, but updates existing ones.

The bad news was that as soon as I started
# make update clean
the target attempted to install all the applications from ports. A quick stop and clean session was ahead.
After all that I asked on #openbsd and phaul was good enough to point me to:
/usr/ports/infrastructure/build/out-of-date

Next step--> Ruby script for parsing and updating my wonderful apps!
#!/usr/bin/env ruby

class PBuild
  attr_accessor :uplist
  attr_reader :flavor, :package
  def initialize(uplist = '/tmp/uplist')
    @uplist = uplist
  end
  def parse
    File.open(@uplist) do |file|
      file.each {|line| pkgadd(line)}
    end
  end
  def list
    `/usr/ports/infrastructure/build/out-of-date > #{@uplist}`
  end
  def pkgadd(line)
    line =~ /((\S+)(\/)*)+/
    #port = $&
    flav = $&.split(',')
    @package = flav.first
    flav.delete(flav.first)
    @flavor = "env FLAVOR=\""
    if not flav.empty?
      flav.each do |opt|
        @flavor = @flavor + opt.to_s + ' '
      end
      p "Building package #{@package} with #{@flavor}"
      @flavor = @flavor + "\" make update clean"
    else
      p "Building package #{@package}"
      @flavor = "make update clean"
    end
    `cd /usr/ports/#{package} && #{@flavor}`
  end
end

latest = PBuild.new
latest.list
latest.parse

(Comments are closed)


  1. By Charles Hill (76.16.134.135) on

    Except I tried something like this yesterday and my system crashed, hard.

    I'm running 3.8 on a co-located server a few States away. The machine is a Dell PE1850 w/1 Gb of RAM and a single 3.0 GHz E64T with Hyperthreading disabled and a Soekris VPN 1411 crypto card.

    I did a CVS update to the 3.8 ports tree, which worked fine. I then moved into the security/clamav folder and did "make update". It downloaded the latest version then just disappeared. It took two calls to the co-lo facility for two reboots to get it back alive, and I still have no idea what happened but "last" shows "crashed". No ssh, no ping, no www, no nothing.

    I'm having the unit shipped back to me so I can get a closer look at it, and maybe get an idea as to what the hell happened. Fortunately, this was just a "test" server to see what the issues would be for me running OpenBSD remotely.

    So, in relation to this thread -- any suggestions where to look for things to verify a ports update really goes right. And, if/when it goes wrong, where to dig for evidence. There was nothing of interest in /var/log/*

    Charles

    1. By Paul Irofti (86.106.14.45) bulibuta@gmail.com on

      > I did a CVS update to the 3.8 ports tree, which worked fine. I then moved into the security/clamav folder and did "make update". It downloaded the latest version then just disappeared. It took two calls to the co-lo facility for two reboots to get it back alive, and I still have no idea what happened but "last" shows "crashed". No ssh, no ping, no www, no nothing.


      I might be wrong here, but updating from ports should be accompanied with an update for src too. Thus an updated kernel and userland...

      Did you do that too? And why are you running 3.8? It's going to be obsolete in 2 days or so.

      This is the first time I hear something like this happening from a simple package update.

      1. By Charles Hill (76.16.134.135) on

        > I might be wrong here, but updating from ports should be accompanied with an update for src too. Thus an updated kernel and userland...

        No, I had not. Not recently, anyway. However, it crashed on the first package I was trying to upgrade (clamav), so I find it hard to believe that was the issue.

        > Did you do that too? And why are you running 3.8? It's going to be obsolete in 2 days or so.

        Because the system is at a co-lo facility a couple of States away and the idea of upgrading the entire system without the install media and thru an ssh connection, including replacing the kernel, scares me. :-)

        > This is the first time I hear something like this happening from a simple package update.

        Me, too. I've run OpenBSD systems before without anything like this happening. My first thoughts were hardware, but the system is only 6 months old and I've done ports upgrades during that time. I'll find out when it gets here.

        1. By sthen (85.158.44.146) on

          > Because the system is at a co-lo facility a couple of States away and the idea of upgrading the entire system without the install media and thru an ssh connection, including replacing the kernel, scares me. :-)

          It's not _that_ bad. Having a known-working serial console that you can access as soon as /etc/boot.conf is read can be reassuring and will help fix most things that could possibly happen, but generally there's no problem to do a remote upgrade without even this.

          Better still, having that serial console would have given you any panic message that was displayed. Maybe the colo can help with setting something up, or if you have a few machines and a spare U, a 2509 or 2511 and cables (or a few USB-RS232) wouldn't be terribly expensive.

        2. By Anonymous Coward (67.165.144.217) on

          try checking for bad memory with the software from memtest.org

        3. By Shane J Pearson (202.45.125.5) on

          > > Did you do that too? And why are you running 3.8? It's going to be obsolete in 2 days or so.
          >
          > Because the system is at a co-lo facility a couple of States away and the idea of upgrading the entire system without the install media and thru an ssh connection, including replacing the kernel, scares me. :-)

          I soon will be needing to take care of an OpenBSD machine remotely, although it is only about an hour and a half away. So I've been thinking of ways to make remote patching and upgrades (and backing out) safe, quick and easy. I'm curious to know what people think of this idea, caveats, etc.

          Use two hard disk drives, with an installation of the latest stable OpenBSD and source on both, configured for networking in that environment and both ready to be used. One drive/installation is considered to be the current "production" system and the other one "testing", which can have patches applied and tested. If the patching goes well, then that most up-to-date system can then become the current production system and the other system (the previous production system) can then become your latest known-working backup, which you can fall back to if need be.

          As patches are released, the roles of these two systems will be swapped, as one is patched and the other becomes the latest backup. With the option to back-out at any time, up until the next role change.

          I figured this would be easy to do with a machine with a real console port which allows access down to the firmware. Since the boot disk can be changed as need be. Even over the PSTN with a MODEM.

          A DVDRW or CDRW drive in the machine, with a RW disk could also make things a little more comfortable, allowing a bootable OpenBSD disk with config (and maybe even user data) backups to be kept and used in an emergency.

          Is this feasible?

          Is this feasible on a single disk in such a way that you have two OpenBSD partitions and switch between them as needed?

          I was hoping to do this on sparc64, however it seems that I can only use the whole disk on sparc64. Is that correct?

          I want to feel safe with remote maintenance. How do other people deal with this problem?

          1. By Anonymous Coward (75.132.114.37) on

            > I was hoping to do this on sparc64, however it seems that I can only use the whole disk on sparc64. Is that correct? I want to feel safe with remote maintenance. How do other people deal with this problem?

            The only thing I use Linux for: Opengear serial console servers. The CM4148 has worked like a champ for us. Vendor is responsive, and the price is great.

        4. By Alex Holst (80.160.149.62) on http://a.mongers.org/

          > > Did you do that too? And why are you running 3.8? It's going to be obsolete in 2 days or so.
          >
          > Because the system is at a co-lo facility a couple of States away and the idea of upgrading the entire system without the install media and thru an ssh connection, including replacing the kernel, scares me. :-)

          I've been doing remote upgrades since 2.9/3.0, even skipping a release because I was busy. I've only botched the upgrade once, and it involved getting someone at the data center to reboot the system to bring me back in the game. I accidentally closed my root shell and the new sudo didn't like the old kernel. (yes, I know. Don't try to avoid the first reboot after putting a new kernel in).

          With a bit of practice and planning, it's entirely possible to do. It helps to make sure there is always a valid kernel in /bsd and to have a root shell open with static copies of reboot, etc available.

    2. By Anonymous Coward (64.142.92.182) on

      So your machine crashed. It can happen. With no panic message, trace or other debug info its impossible to help you. See if you can reproduce the crash. Then update to latest snapshot and see if you can still reproduce the crash. If you can reproduce the crash, submit a proper bug report. See the FAQ for how to do that [http://www.openbsd.org/faq/faq2.html#Bugs]

      Really this has nothing to do with ports. Your system just happened to crash during a port upgrade. Thats not supposed to happen, of course. So either you encountered a legit bug in 3.8 (which may be fixed by now, 3.8 is like a year old) or you have dodgy hardware.


      > Except I tried something like this yesterday and my system crashed, hard.
      >
      > I'm running 3.8 on a co-located server a few States away. The machine is a Dell PE1850 w/1 Gb of RAM and a single 3.0 GHz E64T with Hyperthreading disabled and a Soekris VPN 1411 crypto card.
      >
      > I did a CVS update to the 3.8 ports tree, which worked fine. I then moved into the security/clamav folder and did "make update". It downloaded the latest version then just disappeared. It took two calls to the co-lo facility for two reboots to get it back alive, and I still have no idea what happened but "last" shows "crashed". No ssh, no ping, no www, no nothing.
      >
      > I'm having the unit shipped back to me so I can get a closer look at it, and maybe get an idea as to what the hell happened. Fortunately, this was just a "test" server to see what the issues would be for me running OpenBSD remotely.
      >
      > So, in relation to this thread -- any suggestions where to look for things to verify a ports update really goes right. And, if/when it goes wrong, where to dig for evidence. There was nothing of interest in /var/log/*
      >
      > Charles

  2. By Fco. Valladolid (189.130.17.211) ficovh@yahoo.com on http://blog.bsdguy.net

    umm!, Ruby.. !
    What append if only have a simple firewall machine and don't want more softwate like ruby, maybe need install it.

    I believe perl is ok, for this purpose.
    ;-)

    1. By Anonymous Coward (149.9.0.57) on

      > umm!, Ruby.. !
      > What append if only have a simple firewall machine and don't want more softwate like ruby, maybe need install it.
      >
      > I believe perl is ok, for this purpose.
      > ;-)

      Do you`ve a Perl-Version?
      is so please post it because I don`t know Ruby and dislike the idea of installing it on the FW too.

  3. By Anonymous Coward (74.115.21.120) on

    Yes, lets write an ugly script, in ruby no less, to do something the hard way that is already trivial.

    echo FETCH_PACKAGES=yes >> /etc/mk.conf

    1. By Anonymous Coward (216.127.67.122) on

      > Yes, lets write an ugly script, in ruby no less, to do something the hard way that is already trivial.
      >
      > echo FETCH_PACKAGES=yes >> /etc/mk.conf

      Propably peoples do so because they simply have no time to read all the manpages all day long and the OpenBSD-Website is kinda useless if it deals with such stuff.

      Btw: The Manpage does not mention "FETCH_PACKAGES" as Variable so how should peoples do know?

      1. By Otto Moerbeek (87.210.142.234) otto@drijf.net on http://www.drijf.net

        > Propably peoples do so because they simply have no time to read all the manpages all day long and the OpenBSD-Website is kinda useless if it deals with such stuff.
        >
        > Btw: The Manpage does not mention "FETCH_PACKAGES" as Variable so how should peoples do know?

        So people have time to write all kind of scripts but don't have time to read a man page? Strange....

        BTW: It's documented in bsd.port.mk(5).



        1. By Anonymous Coward (85.214.73.63) on

          > > Propably peoples do so because they simply have no time to read all the manpages all day long and the OpenBSD-Website is kinda useless if it deals with such stuff.
          > >
          > > Btw: The Manpage does not mention "FETCH_PACKAGES" as Variable so how should peoples do know?
          >
          > So people have time to write all kind of scripts but don't have time to read a man page? Strange....
          >
          > BTW: It's documented in bsd.port.mk(5).

          THanks! :)

        2. By Ted Walther (24.85.128.203) on http://reactor-core.org/

          > > Propably peoples do so because they simply have no time to read all the manpages all day long and the OpenBSD-Website is kinda useless if it deals with such stuff.
          > > 
          > > Btw: The Manpage does not mention "FETCH_PACKAGES" as Variable so how should peoples do know?
          > 
          > So people have time to write all kind of scripts but don't have time to read a man page? Strange....
          > 
          > BTW: It's documented in bsd.port.mk(5).
          

          FETCH_PACKAGES is documented. And the documentation gives NO CLUE that it will go through the ports tree and only update packages that are already installed. So even if he did "read the manpage" it wouldn't have helped.

          I quote from the man page:

               FETCH_PACKAGES
                             User settings.  If set to `Yes', the package target will
                             download packages missing from the repository from loca-
                             tions in ${PKG_PATH}.  It will only build them if no suit-
                             able packages are found.
          

          There is no indication what constitutes a package "missing" from the repository. How is anyone to know that missing packages only refer to packages that have already been installed? What if the compiled package tarball is still lying around in the tree?

          Also, what is a "suitable" package? How is suitability of a package determined?

          The manpage is as clear as mud, if FETCH_PACKAGES really is the solution to updating your installed ports, but without installing all the other ports too.

          1. By Anonymous Coward (66.11.66.41) on

            Its crystal clear actually. Instead of building the port, it will download the package from $PKG_PATH if available. That's all, very simple. Make upgrade is what decides not to install packages that aren't already installed. The problem is just that I didn't understand what the guy was trying to do or why. FETCH_PACKAGES is not the solution to his problem, running memtest86 is.

    2. By Anonymous Coward (70.48.232.124) on

      > Yes, lets write an ugly script, in ruby no less, to do something the hard way that is already trivial.
      >
      > echo FETCH_PACKAGES=yes >> /etc/mk.conf

      If you'd read the article, you'd see that he just wanted to thrash in some new hardware. It's not uncommon for people to do this, or make build, or whatever.

      Ruby is... er ... why? Agreed. Check the article department. :-)

      The real slashtards are the ones bickering about DNS in an article about OpenCON, nowhatimean?

      1. By Anonymous Coward (74.115.21.120) on

        > If you'd read the article, you'd see that he just wanted to thrash in some new hardware. It's not uncommon for people to do this, or make build, or whatever.

        Its actually hard to understand what he is talking about, what he wanted to do or what happened instead. If he just wanted to test his hardware, what's wrong with building ports?

    3. By Paul Irofti (80.97.12.10) bulibuta@gmail.com on

      > Yes, lets write an ugly script, in ruby no less, to do something the hard way that is already trivial.
      >
      > echo FETCH_PACKAGES=yes >> /etc/mk.conf

      This option fixes the issue with make update regarding installation of non-existant already installed packages. But for that it steps inside every port and checks if it's installed or not. Thus the build process gets a lot bigger, with out-of-date I get a list of exactly the packages I should update and so I can go straight into that port and run make update in there.

      If my script is hard to understand I apologize, I wrote it from work in a small break while I had to wait for something to finish compiling. I think it's quite readable and it'll be my pleasure to explain what is not so clear.

      Regarding the chosen language it's quite simple: I like Ruby. I never comment on the language one picks for doing something. It's his own work and he should be free to choose whatever language he desires. No need to start a religious flamewar for that, specially since you're not forced to use it.

      Please do note that this is just the core of what could be done based on the out-of-date app.

      I didn't mean to offend anyone just wanted to share something I found usefull with the community.

    4. By sthen (85.158.44.146) on

      > Yes, lets write an ugly script, in ruby no less, to do something the hard way that is already trivial.
      >
      > echo FETCH_PACKAGES=yes >> /etc/mk.conf

      "download a whole bunch of packages" doesn't really equate to "build from source all ports which are currently installed and out-of-date".

      "FETCH_PACKAGES=yes" and "make update" cycle through all ports, downloading the *.tgz from a mirror, check the output of pkg_info to see if it's installed, and if so attempt to upgrade from package. If the package isn't already present it will continue to the next one. An inefficient version of "pkg_add -ui".

      1. By Anonymous Coward (66.11.66.41) on

        > "download a whole bunch of packages" doesn't really equate to "build from source all ports which are currently installed and out-of-date".

        Yes, we've established that now. But from the article, it was not clear at all what the goal was, or why he was doing any of the things he was doing. Now that its been made clear that his goal was to do something that makes no sense, I guess writing a script to do it is the way to go.

        1. By Paul Irofti (86.106.14.45) bulibuta@gmail.com on

          > Yes, we've established that now. But from the article, it was not clear at all what the goal was, or why he was doing any of the things he was doing. Now that its been made clear that his goal was to do something that makes no sense, I guess writing a script to do it is the way to go.

          Just so we get clear on things, this is very usefull when you want to test the new patches/versions of what was submitted in ports since your last cvs up.

          You can use this to get a real structured test suite for what is described here:
          http://www.openbsd.org/porttest.html

          Now if you want to play the smartass go ahead, so far you wrote nothing that can be of any use to anyone. I always accept criticism and I've been quite cool with your continued offenses waiting for something productive to come out of your posts.

          So please just shut-up or try contributing with something that someone can use for a change.

          1. By Anonymous Coward (66.11.66.41) on

            > Just so we get clear on things, this is very usefull when you want to test the new patches/versions of what was submitted in ports since your last cvs up.

            Not really, snapshots are built quite frequently, so pkg_add would be useful for this.

            > You can use this to get a real structured test suite for what is described here:
            > http://www.openbsd.org/porttest.html

            No, you can't. If you want to test a port, then test it. Trying to recompile from ports all the packages you currently have installed does not test the ports. Those of us who actually contribute ports appreciate testing. We don't appreciate people doing crap like this and then pretending they've tested.

            > Now if you want to play the smartass go ahead, so far you wrote nothing that can be of any use to anyone.

            I guess we have something in common then.

            > So please just shut-up or try contributing with something that someone can use for a change.

            I contribute for real. Like, actually making ports, not talking about how to write a retarded ruby script to do something that makes no sense at all. And frankly, even me pointing out how dumb your script is has been helpful, as several people are now aware of FETCH_PACKAGES who were not before. Thanks for your wonderful contribution though, we're forever in your debt.

            1. By Anonymous Coward (83.125.32.107) on

              > I contribute for real. Like, actually making ports, not talking about how to write a retarded ruby script to do something that makes no sense at all. And frankly, even me pointing out how dumb your script is has been helpful, as several people are now aware of FETCH_PACKAGES who were not before. Thanks for your wonderful contribution though, we're forever in your debt.

              People like you are why undeadly sucks. Who's going to contribute articles for an audience like this? And you seem to be overlooking the part where he did contribute a story here. That's what this place is for. Where's your contribution?


              1. By Anonymous Coward (74.115.21.120) on

                > People like you are why undeadly sucks. Who's going to contribute articles for an audience like this?

                Lots of people contribute articles. Hopefully an "audience like this" will continue to limit the number of moronic submissions like this so undeadly stays useful, instead of being a pile of slashtards posting retarded shit and then jerking each other off over it. There's plenty of forums that welcome idiots posting worthless crap, and they will even suck your nuts for doing it just like you want. If you want that, go find it. Quit trying to push your pussy ideals here, we don't have to tolerate idiots, much less be nice to them. If you think undeadly sucks, then go away and quit bothering those of us who like it.

                > And you seem to be overlooking the part where he did contribute a story here.

                No, if I were overlooking it we wouldn't be having this discussion would we? I see his "contribution", and acknowledge it for what it is, useless and stupid.

                > That's what this place is for. Where's your contribution?

                Reading is not high on your priority list is it?

                1. By deanna (64.231.233.75) on

                  > Lots of people contribute articles.

                  Clearly, you have no idea what you are talking about. Now please shut the fuck up and stop trolling undeadly.

                  1. By Anonymous Coward (66.11.66.41) on

                    > > Lots of people contribute articles.
                    >
                    > Clearly, you have no idea what you are talking about. Now please shut the fuck up and stop trolling undeadly.

                    Ok, "lots" was an overstatement. But the fact remains contributing useless articles does not make one useful. If you want the site to be a useless pile of circle jerking halfwits that's up to you. But you can't expect those of us who want it to stay useful not to point out how badly the shit stinks when shit gets posted.

  4. By sthen (85.158.44.146) on

    > The bad news was that as soon as I started
    > # make update clean
    > the target attempted to install all the applications from ports.

    Not install, just build. You have to build a package before you install it. If "make update" is an alternative to "make install" which only installs new versions of already-installed packages, it too must build the tgz-file.

    http://www.openbsd.org/faq/faq15.html#PkgVsPorts:
    "The OpenBSD ports team considers packages to be the goal of their porting work, not the ports themselves."

    The tools for the ports tree are mainly there to help people working on it and for building packages. Some of them are useful for non-developers too, but don't expect full package management out of a system that's just there to _produce_ the packages. (for the cases where redistribution of binaries isn't possible - jdk, ngrep, acroread and others - it's the only way to produce the packages, and a more easily-fixed problem is that it's also currently the most convenient package-browser we have, so there are a few cases the 'average user' still needs /usr/ports, but it's not necessary to build software from source all the time - FETCH_PACKAGES=yes bridges the gap there).

    There's no difference between "make install" from a port, and from "pkg_add" an already-built package. As you'll see if you 'make -n install' (or 'make -n update'), ports aren't installed directly: pkg_add is always used.

    If you want to stress-test a machine, picking some of the larger ports (OpenOffice, Java, Firefox, KDE, etc) might be a better choice than just rebuilding the ports you use. "make build" in /usr/src is good, too (you can run sysutils/stress at the same time to make the job harder). Unless you're developing the ports tree it's probably better to leave the job of updating to pkg_add...

  5. By Marc Espie (213.41.185.88) espie@openbsd.org on

    Okay, this is actually bad, for a few reasons.

    People, you do not need to set FLAVOR manually, we've used
    SUBDIR to that effect for a long while now.

    The second point is that it's somewhat stupid to take the output of
    out-of-date and parse it again. That's the bad route that NetBSD has
    taken: output information in a human readable format, reparse it, and
    refeed it to another script. This never works. You always forget special cases.

    In the case at hand, the subdir information is just available in out-of-date, you need about two or three extra lines to extract it.

    Instead of a whole new ruby script...

    We are going to address this. out-of-date is just fine for human readers, but it should also output information you can just reuse.

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]