OpenBSD Journal

New package system features

Contributed by deanna on from the nice tools dept.

Last month we published a writeup from espie@ which talked about the code changes he's been working on in the OpenBSD package system. As a follow up, here's a list of new features that have been recently added.

Marc Espie (espie@) writes:

Lots of commits over the past two weeks, loads of refactorings. Some user-visible changes as well.

  • in case pkg_add cannot find libraries, it's much more specific now. It will tell you if it found libraries that looks vaguely like what it was looking for, and where, so that you can figure out where the dependency chain broke.

  • some error scenarios work better. If you do a ^C in a middle of a package install, the partial package that results shouldn't warn during removal (especially visible on manpages recorded as a temporary names, which used to trigger warnings from makewhatis).

  • shared library handling while doing package replacement is much more general: full information gets copied to .libs-* packages. Also, multiple replacements involving the same libraries should now work flawlessly, you should even be able to go back. The old scenario where you did pkg-1.0 -> pkg-2.0 + .libs-pkg-1.0, and then couldn't go back to pkg-1.0, will now finish in -> pkg-1.0 + .libs-pkg-2.0, as it should.

  • new functionality: pkg_add should now record `manual installations' as opposed to automatic dependency installation (option -a of pkg_add). Anything you install explicitly is recorded with an @option manual-installation. Dependencies do not have that tag. You can later tweak individual installed packages if you want to. Eventually, when that information is correct, you'll be able to check through pkg_info the set of dependencies that are not needed by anything, and you'll be able to remove them.

  • numerous message fixes. pkg_delete now has more consistent -v settings. External commands like ldconfig and mkfontdir now show up consistently in -vv mode.

  • https: urls are recognized, and should hopefully work correctly now.

  • pkg_add now has the same checks that pkg_create has, and will refuse to install packages with fishy file modes.

There are still some ongoing changes... like tweaks to adjust better to font handling schemes in xenocara; auto-formatting of manpages; digital signatures; better accuracy when updating stuff; faster lookups of package names through PKG_PATH (basically, record locations as soon as we can, and not look for the package around); uniform package urls in all commands (still some way off); better tracking of dependencies (adjusting dependencies for stuff already installed, AND for stuff in-progress, not yet installed); better handling of interrupts (being able to interrupt pkg_add more often), and n-to-n simultaneous updates.

(Comments are closed)

Comments

  1. By viq (80.50.239.66) on

    The mention of digital signatures got me interested, does that mean that officialy built packages will be signed so we can verify them?

    Also, there was some mention some time ago of pkg_add being able to grok ports tree, what is the status of that?

    And thank you for the great work, this stuff is just great, and getting better :)

    Comments

    1. By Marc Espie (213.41.185.88) espie@openbsd.org on

      > The mention of digital signatures got me interested, does that mean that officialy built packages will be signed so we can verify them?

      It's still at the design stage. I've got some `proof of concept' that
      works, but I'm not 100% certain what kind of certificates I want to
      use yet.

      > Also, there was some mention some time ago of pkg_add being able to grok ports tree, what is the status of that?

      It is working, somewhat... the only issue being that pkg_add is
      running, and asking the ports tree for more pkg_add -> deadlocks.

      So, we should be more sneaky about it: pkg_add itself can do all
      the dependency work that the ports tree does, and so it needs to
      take that over...


      > And thank you for the great work, this stuff is just great, and getting better :)

      you're welcome

      Comments

      1. By Anonymous Coward (85.178.120.124) on

        > > The mention of digital signatures got me interested, does that mean that officialy built packages will be signed so we can verify them?
        >
        > It's still at the design stage. I've got some `proof of concept' that
        > works, but I'm not 100% certain what kind of certificates I want to
        > use yet.
        >
        > > Also, there was some mention some time ago of pkg_add being able to grok ports tree, what is the status of that?
        >
        > It is working, somewhat... the only issue being that pkg_add is
        > running, and asking the ports tree for more pkg_add -> deadlocks.
        >
        > So, we should be more sneaky about it: pkg_add itself can do all
        > the dependency work that the ports tree does, and so it needs to
        > take that over...
        >
        >
        > > And thank you for the great work, this stuff is just great, and getting better :)
        >
        > you're welcome

        I hope I don`t mix these thigns but:

        The Packages do still use Gzip for the compression (even LZMA would maybe better...) so why not using gzsig wich is already in the base system?!

        Og course you`ve to download the "whole package" to verify the signature but just downloading the half of the content wouldn`t make sense anyway.

        I made this conclusion already last year.. and the year before.....
        So.. isn`t it a option?

        Comments

        1. By Marc Espie (163.5.254.20) espie@openbsd.org on

          > > > The mention of digital signatures got me interested, does that mean that officialy built packages will be signed so we can verify them?
          > >
          > > It's still at the design stage. I've got some `proof of concept' that
          > > works, but I'm not 100% certain what kind of certificates I want to
          > > use yet.
          > >
          > > > Also, there was some mention some time ago of pkg_add being able to grok ports tree, what is the status of that?
          > >
          > > It is working, somewhat... the only issue being that pkg_add is
          > > running, and asking the ports tree for more pkg_add -> deadlocks.
          > >
          > > So, we should be more sneaky about it: pkg_add itself can do all
          > > the dependency work that the ports tree does, and so it needs to
          > > take that over...
          > >
          > >
          > > > And thank you for the great work, this stuff is just great, and getting better :)
          > >
          > > you're welcome
          >
          > I hope I don`t mix these thigns but:
          >
          > The Packages do still use Gzip for the compression (even LZMA would maybe better...) so why not using gzsig wich is already in the base system?!
          >
          > Og course you`ve to download the "whole package" to verify the signature but just downloading the half of the content wouldn`t make sense anyway.
          >
          > I made this conclusion already last year.. and the year before.....
          > So.. isn`t it a option?

          No, we've got more sexy stuff... I should know about gzsig, it's
          basically stuff I did long ago.

          We know should be able to do `just in time' package signatures, because
          all the files inside the tarball have known crypto-hash (md5 for now,
          sha256 in the future), have known meta-information (pkg_add will
          refuse to install setuid binaries if they are not tagged with the right
          owner and mode). So it's enough to sign the packing-list itself...

          which is what this is all about.

  2. By Gonsalu (62.48.171.17) gonsas@gmail.com on

    Some work on improving proxy support would be appreciated (I know it's there, but has quite a few bugs -- check bug tracker for some of them), but it's good to see improvements on OpenBSD in overall! :)

    Comments

    1. By Marc Espie (163.5.254.20) espie@openbsd.org on

      > Some work on improving proxy support would be appreciated (I know it's there, but has quite a few bugs -- check bug tracker for some of them), but it's good to see improvements on OpenBSD in overall! :)

      Most of this is in ftp(1), not pkg_add(1)...

  3. By jirib (195.212.29.187) on

    digitally signed packages? :)

    what about bpg (bsd licence openpgp)

    http://netbsd-soc.sourceforge.net/projects/bpg/

    Comments

    1. By Marc Espie (163.5.254.20) espie@openbsd.org on

      > digitally signed packages? :)
      >
      > what about bpg (bsd licence openpgp)
      >
      > http://netbsd-soc.sourceforge.net/projects/bpg/

      Several points.

      - that project is currently not usable. It's not quite vaporware, but I don't see where importing half-finished code is going to lead me... It still relies on GnuPG for key extraction. It doesn't appear to have a perl interface, a ruby interface doesn't help me at all. I'm more comfortable using the openssl command-line utility, which is already in the core of OpenBSD.
      - I want to have very simple chains of trust, with very specific control over who can sign what. Specifically, in the official openbsd realm, there will only be master keys, that can be used to sign secondary keys ONLY, and secondary keys, that can be used to sign packages ONLY.

      That way, we have a trust structure that anyone can understand, the package tools operate in THAT trust structure by default.

      For that to work sensibly, I need key management to be explicit, and totally unsubtle. I will probably end up with a hard-coded set of master keys (probably 3 or 4), a signed list of secondary keys, and updates to the keys will happen through physical replacement of that list of secondary keys.

      If people want some extra assurance that the keys have not been tampered with, we can provide more external signatures.... note that, in the end, there's always a bootstrap issue. If you really are paranoid enough, you'll want to make sure your whole system has been installed from a reliable media, and that nobody has tampered with it...

      In the past, I've seen enough subtle issues with complex PKI used for stuff it wasn't supposed to. I don't want a complex PKI structure. I don't need stuff that will scale arbitrarily. The basic official OpenBSD trust model is simple, and doesn't have to scale: we know each other, and we meet each other regularly.

      If you want to use another trust structure, you have to explicitly ask for it, and I'm only going to provide an open interface: whatever goes on with another trust realm is going to be perl code I am not responsible for.

      People who provide commercial services based on OpenBSD will be able to roll their own, audit their trust chain, and do whatever they want with it (heck, I wouldn't have any issue signing keys from a lot of these people). But they won't be in the openbsd realm, as far as trust chain is concerned, and whatever kind of pki they want to implement is not my concern.

      Note that, for semi-paranoid people, the main issue signed packages actually solve is that they can use any ftp server to download official packages, and be certain those packages have not been tampered with...

      this doesn't solve the issue of making sure the system that checks the signatures is a trusted system, nor that the packages do not contain any issues the person signing them would be unaware of.

      This is one reason for which I'm reluctant to have signed packages. I'm pretty certain a lot of people will have unreasonable expectations. I see this all the time on https: applications, where a lot of people miss the point and trust anything as soon as it has a signed X509 certificate...

      Let me remind you that Microsoft had the exact same issue with signed ActiveX controls: their signature model doesn't prove anything, except that the developer who signs the control is who they say they are. There was this guy who wrote a control that shut the machine down, to prove the point, and Microsoft had to make a hell of a mess to solve that particular problem...

      Comments

      1. By Anonymous Coward (72.143.102.169) on

        Thanks for the excellent explanation and all the effort to do it right!

      2. By nikns (89.191.97.92) on

        This is suprising how most secure os in the world doesn't make distribution secure as it can be.

        1) Package signatures would be very helpful. However, thats only packages.
        2) Second level mirrors should provide rsync over ssh access.
        3) Third level mirrors should provide sftp/scp access to make use of pkg_add sftp/scp uri handlers.
        4) www.openbsd.org should get valid ssl certificate, so that fingerprints written in ftp.html could be trusted.

        I am ready to provide ssh/https access to my mirror, however I have no guarentee that second level mirrors doesn't get MIM'ed.

        Don't even try to bullshit about "do I trust my compiler" shit.
        The most secure os in the world can be only secure as secure it gets distributed.

        Comments

        1. By Marc Espie (213.41.185.88) espie@openbsd.org on

          > This is suprising how most secure os in the world doesn't make distribution secure as it can be.
          >
          > 1) Package signatures would be very helpful. However, thats only packages.
          > 2) Second level mirrors should provide rsync over ssh access.
          > 3) Third level mirrors should provide sftp/scp access to make use of pkg_add sftp/scp uri handlers.
          > 4) www.openbsd.org should get valid ssl certificate, so that fingerprints written in ftp.html could be trusted.
          >
          > I am ready to provide ssh/https access to my mirror, however I have no guarentee that second level mirrors doesn't get MIM'ed.
          >
          > Don't even try to bullshit about "do I trust my compiler" shit.
          > The most secure os in the world can be only secure as secure it gets distributed.

          You can buy official CDs. That's pretty secure.

          Think about it. What you want costs money, in terms of infrastructure,
          and gives very little to developers. Especially since we *do* have
          secure access to the source.

          Having package signatures is not a critical endeavor for me. I'm mostly working on it because some friends have some interest in it. Those friends have commercial reasons to want so. It's also nice to be able to have some technical stuff that works fast. I don't have a lot of interest in pushing a PKI infrastructure. That's a lot like my day job, and I don't get paid for OpenBSD development.


          Comments

          1. By pachl (65.116.243.26) on

            > > This is suprising how most secure os in the world doesn't make distribution secure as it can be.
            > >
            > > 1) Package signatures would be very helpful. However, thats only packages.
            > > 2) Second level mirrors should provide rsync over ssh access.
            > > 3) Third level mirrors should provide sftp/scp access to make use of pkg_add sftp/scp uri handlers.
            > > 4) www.openbsd.org should get valid ssl certificate, so that fingerprints written in ftp.html could be trusted.
            > >
            > > I am ready to provide ssh/https access to my mirror, however I have no guarentee that second level mirrors doesn't get MIM'ed.
            > >
            > > Don't even try to bullshit about "do I trust my compiler" shit.
            > > The most secure os in the world can be only secure as secure it gets distributed.
            >
            > You can buy official CDs. That's pretty secure.

            What if people do securely buy CDs and run release, then need to update their sources to follow stable? Is this process secure? Also, the pages that list the FTP/rsync/CVSync/etc mirrors are unsecured. Are these issues to be concerned about?

            Comments

            1. By Anonymous Coward (88.198.43.207) on

              > What if people do securely buy CDs and run release, then need to update their sources to follow stable? Is this process secure? Also, the pages that list the FTP/rsync/CVSync/etc mirrors are unsecured. Are these issues to be concerned about?

              You can do things like anoncvs over SSH, so no. And unless you switch mirrors often, someone tampering with the ftp.html page is not going to get anywhere.

              Joachim

            2. By tedu (69.12.168.115) on


              > What if people do securely buy CDs and run release, then need to update their sources to follow stable? Is this process secure? Also, the pages that list the FTP/rsync/CVSync/etc mirrors are unsecured. Are these issues to be concerned about?

              if you pick a mirror that ends in .openbsd.org instead of istealzyourgoodz.org you'll probably be safe.

          2. By Anonymous Coward (89.191.97.92) on

            > You can buy official CDs. That's pretty secure.
            >
            > Think about it. What you want costs money, in terms of infrastructure,
            > and gives very little to developers. Especially since we *do* have
            > secure access to the source.
            >
            > Having package signatures is not a critical endeavor for me. I'm mostly working on it because some friends have some interest in it. Those friends have commercial reasons to want so. It's also nice to be able to have some technical stuff that works fast. I don't have a lot of interest in pushing a PKI infrastructure. That's a lot like my day job, and I don't get paid for OpenBSD development.


            So your excuse is that using crypto ssh/ssl is costly in terms of infrastructure, and actually you don't care about people in community supporting openbsd who doesn't compile every bit from sources.

            I guess that this attitude would change very fast, when some mirrors gets backdoored and openbsd gets a lot of bad press. Who knows, maybe not. gg.

            Comments

            1. By Marc Espie (213.41.185.88) espie@openbsd.org on

              > > You can buy official CDs. That's pretty secure.
              > >
              > > Think about it. What you want costs money, in terms of infrastructure,
              > > and gives very little to developers. Especially since we *do* have
              > > secure access to the source.
              > >
              > > Having package signatures is not a critical endeavor for me. I'm mostly working on it because some friends have some interest in it. Those friends have commercial reasons to want so. It's also nice to be able to have some technical stuff that works fast. I don't have a lot of interest in pushing a PKI infrastructure. That's a lot like my day job, and I don't get paid for OpenBSD development.
              >
              >
              > So your excuse is that using crypto ssh/ssl is costly in terms of infrastructure, and actually you don't care about people in community supporting openbsd who doesn't compile every bit from sources.
              >
              > I guess that this attitude would change very fast, when some mirrors gets backdoored and openbsd gets a lot of bad press. Who knows, maybe not. gg.

              You should learn how to read properly. In my sentence `the source' means `original material'. Not necessarily source stuff. Plus, you're a troll. If you read ports regularly, you would know I'm one of those people who tells users relentlessly to use binary packages and not recompile if they can't avoid it.

              You have no idea how much effort it takes already to have regular snapshots and packages. We are, by and large, not a commercial organization. Unlike redhat or ubuntu.

              Comments

              1. By nikns (89.191.97.92) on

                watch your mouth, braveheart.
                your answer is pointless. good luck avoiding the issue.

                Comments

                1. By Anonymous Coward (71.139.238.137) on

                  > watch your mouth, braveheart.
                  > your answer is pointless. good luck avoiding the issue.

                  I guess someone like me should be thankful you're here, then. Quick, file a couple of bug reports!

                  -The universe exists and is insecure.

                  -Trolls exist and are innately insecure.

        2. By Anonymous Coward (216.68.198.57) on

          I agree. However OpenBSD is by the Developers, for the Developers, and take it as it comes for the rest, because, what else is there really for some environments?
          Trusting a CD in the mail as well, is also a bit loose for high grade worlds.
          I wish OpenBSD would move a bit to a more commerical trustable environment, if it did, it might not have as much as $ problems, and get more market share in special environments, however, that takes time.

          I'm willing to wait for OpenBSD to get more last mile problems solved, however, the IT world is in a weird endgame strategy these days. Thats Negotitation.

          Anyway, a very thankful user to the OpenBSD team and Theo for being top notch, in this nutty world.

          Comments

          1. By Daniel Bolgheroni (201.87.29.100) dbolgheroni@gmail.com on

            > I agree. However OpenBSD is by the Developers, for the Developers, and take it as it comes for the rest, because, what else is there really for some environments?
            > Trusting a CD in the mail as well, is also a bit loose for high grade worlds.
            > I wish OpenBSD would move a bit to a more commerical trustable environment, if it did, it might not have as much as $ problems, and get more market share in special environments, however, that takes time.
            >
            > I'm willing to wait for OpenBSD to get more last mile problems solved, however, the IT world is in a weird endgame strategy these days. Thats Negotitation.
            >
            > Anyway, a very thankful user to the OpenBSD team and Theo for being top notch, in this nutty world.

            This is what I call paranoid.

  4. By Pierre-Yves Ritschard (pyr) pyr@spootnik.org on http://spootnik.org

    Another new feature of the pkg tools is ``pkg_add -t''
    This command lets you show only the packages that are not a dependency of another package, which usually maps to the list of packages you actually use.

    Comments

    1. By Marc Espie (213.41.185.88) espie@openbsd.org on

      > Another new feature of the pkg tools is ``pkg_add -t''
      > This command lets you show only the packages that are not a dependency of another package, which usually maps to the list of packages you actually use.


      Well, first it's pkg_info -t.

      And second, once manual-install works correctly, it will be much
      nicer...

  5. By Anonymous Coward (72.24.124.30) on

    Are you aware of 0install (0install.org) and if so what do you think of their way - are there similarities with your objectives and approaches? Other than the obvious legacy issues, do you see any problems or mistakes with 0install?

    Just curious - 0x1b

    Comments

    1. By Marc Espie (213.41.185.88) espie@openbsd.org on

      > Are you aware of 0install (0install.org) and if so what do you think of their way - are there similarities with your objectives and approaches? Other than the obvious legacy issues, do you see any problems or mistakes with 0install?
      >
      > Just curious - 0x1b

      Totally different goals.

Latest Articles

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]