OpenBSD Journal

g2k16 Hackathon Report: Antoine Jacoutot on binary patches, ports progress and more

Contributed by rueda on from the bin-bin-bin-patches! dept.

Here's yet another g2k16 hackathon report, this one from Antoine Jacoutot, who writes:

This was my first time in Cambridge and I must say I really enjoyed the place. It's a gorgeous town and avsm@ and Gemma's organization of the hackathon was just perfect. So first of all, thank you very much to them and to the OpenBSD Foundation.
The one thing I wanted to work on or rather talk about during that week was having a way to provide binary updates (both for the base system and packages) to ease maintenance of OpenBSD fleets. While it's perfectly possible to build your own stable releases and packages, it's a work that could be done centrally instead of having each and every user come up with his own solution. Plus that would mean we could get official signed updates.

To be honest, I was expecting some reluctance since it would mean a change in our workflow but Theo has been very opened to the idea as long as it's done properly and is up to OpenBSD standards.

So robert@ and I started hacking on it, we kind of knew how we wanted things to be done since we've been doing a similar thing in M:Tier for years outside of the official tree along with jasper@. Robert worked on the build infrastructure (to create the actual binary patches) and my chore was to write the utility that would fetch, verify, install and eventually rollback these patches.

Binary patching is a complex subject so we wanted things to be baby simple. Actually "binary patching" is not what we will really do here (i.e. it will not be a binary diffing/patching utility).

The first bits are in and this is how we would like things to work eventually:

  • fetch the tarballs containing the updated binaries/kernels/files
  • verify them with signify
  • create a rollback tarball containing the installed files that are about to be replaced
  • extract the syspatch tarball to a temporary directory
  • safely install(1) the extracted files on the system
  • if something goes wrong or if a patch introduces a regression, rollback using the tarball we've created above

That's pretty much all. We are not going to implement things like rollbacking a particular patch (but only the last installed one) because the goal is to be synced against stable and not running a Frankenstein system (patches are cumulative and some may depend on others).

Once we have syspatch in place, we can start looking at offering binary packages updates for stable (all the build infrastructure is of course already available, so that's just a matter of integrating it in our stable update workflow).

Besides that, I've worked on a few corner cases that our rc.d(8) does not cope well with. That's not committed yet because it requires a bit more work and beck@ gave me a couple of other ideas that I'd like to implement at the same time. So stay tune for upcoming changes...

Of course, I took a ride under ports/ for a while, adding some USE_WXNEEDED bits here and there (unbreaking gnome-shell and gdm) and jasper@ offered me the huge honor to remove devel/glib (i.e. glib-1.X!) from the tree. I updated a few things as well and sync the login_krb5 port to match some recent changes in base.

One night, fill with boredom^beer, jasper@ and I went to portroach.openbsd.org (which lists outdated ports in the tree) and started an update rampage

All in all it was again an amazing week amongst amazing people!

Thanks for the report, Antoine!

(Comments are closed)


  1. By Anonymous Coward (77.166.151.40) on

    Binary patches are coming!

  2. By Renaud Allard (renaud) renaud@allard.it on

    Wow, official binary patches is something I thought would never happen.
    That's an amazing news. I really want to thank you with this.
    Let's go donate a little bit more :)

    1. By Anonymous Coward (217.84.137.20) on

      > Wow, official binary patches is something I thought would never happen.
      > That's an amazing news. I really want to thank you with this.
      > Let's go donate a little bit more :)

      Indeed. I think there were even fairly recent discussions on misc@ where the consensus was that the team just didn't have the manpower to take care of it.
      This would be an awesome addition, especially for people with not so powerfull machines.

      1. By Anonymous Coward (51.255.202.66) on

        > Indeed. I think there were even fairly recent discussions on misc@ where the consensus was that the team just didn't have the manpower to take care of it.

        That was before they decided to scrap future CD-ROM releases. I guess the time that has been freed up as a result of not having to do anymore CD-ROM releases is now being put to good use.

        > This would be an awesome addition, especially for people with not so powerfull machines.

        Exactly. Especially those with little disk space. Having said that though, as the FAQ suggests, compiling a system puts a lot of stress on the machine, so is a good way to figure out how good the system is. That kind of test is not suited for all machines though.

  3. By journeysquid (Tor) on http://www.openbsd.org/donations.html

    Good work! Hopefully syspatch won't have the same shortcoming that MTier's openup does where it blindly installs patches for the X sets when they are not installed.

    Does syspatch check existing files against checksums before replacing them?

    1. By Anonymous Coward (37.220.35.202) on

      > Good work! Hopefully syspatch won't have the same shortcoming that MTier's openup does where it blindly installs patches for the X sets when they are not installed.
      >
      > Does syspatch check existing files against checksums before replacing them?
      >

      I would like to think they'd consider moving to using a C program instead of a shell script if they want to start introducing that kind of complexity. For one thing, a C program can use pledge(2).

  4. By Gerardo Santana (70.113.112.97) gerardo.santana@gmail.com on https://github.com/santana

    Congratulations to Antoine for this accomplishment.

    As the original author of the binpatch subsystem I remember struggling with convincing the team to publish the errata in a more easy to consume format by a script.

    Antoine, can you push forward an initiative to do that? That would allow anyone to automate binpatch building even further.

    1. By Anonymous Cowboy (213.252.247.231) on

      > Congratulations to Antoine for this accomplishment.
      >
      > As the original author of the binpatch subsystem I remember struggling with convincing the team to publish the errata in a more easy to consume format by a script.
      >
      > Antoine, can you push forward an initiative to do that? That would allow anyone to automate binpatch building even further.

      I would be interested to know what kind of format did you have in mind?

      1. By Gerardo Santana (70.113.112.97) gerardo.santana@gmail.com on https://github.com/santana

        > > Congratulations to Antoine for this accomplishment.
        > >
        > > As the original author of the binpatch subsystem I remember struggling with convincing the team to publish the errata in a more easy to consume format by a script.
        > >
        > > Antoine, can you push forward an initiative to do that? That would allow anyone to automate binpatch building even further.
        >
        > I would be interested to know what kind of format did you have in mind?

        Back then I proposed one based on XML. But anything with structure is ok. What matters is to clearly mark up each part of the errata.

        From there we can automatically generate errata.html as well as the Makefile targets.

  5. By jine jack (kasstri) fgffcvgffhff@gmail.com on http://www.clavier-arabe-pro.com/arabic-keyboard.php

    Indeed. I think there were even fairly recent discussions on misc@ where the consensus was that the team just didn't have the manpower to take care of it.

    That was before they decided to scrap future CD-ROM releases. I guess the time that has been freed up as a result of not having to do anymore CD-ROM releases is now being put to good use.

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]