OpenBSD, the land of flavored ports

OpenBSD, the land of flavored ports

Marc Espie <espie@cvs.openbsd.org>

All these changes were motivated by two main forces:

• restore the infrastructure to something fast, and manageable,
• make package building easier.

Initially, the ports tree concept was brought over from FreeBSD by Ejowi Nuwere a few years back. Then, as usual, the OpenBSD ports tree was partially synch'ed with improvements from sibling projects NetBSD and FreeBSD. There were some indications of major problems brewing, though. When I first timed a ports build on a relatively slow box (amiga, 68040 at 25MHz), it took 7 seconds for a ports build to start ! That's right, 7 seconds between entering `make' and the first indication something was going on. The second major hint was `make -j' falling abysmally: bsd.port.mk was no longer a proper Makefile.

After some major cleaning-up, timing became reasonable again: 2 seconds to start, and make -j is functional again. More importantly, the ports infrastructure has become controllable. A vast number of unused and dubious features were trimmed out, some sound design principles were asserted, and a large number of extraneous invocations of make were removed.

Change of tide, when ports became packages

This is about the time when the Project, after some prompting from Theo, decided to provide many more packages than it used to. The challenge became more interesting: we wanted to provide a nice ports tree still, for those people who want to tinker along with sources (and the basic concept of fetching third-party software automatically, patching stuff for our flavor of BSD, and doing an automatic build is still a very unique and beautiful concept), while also providing safe and useful packages.

The main problem was that packages did not work: even though we asked porters to test them, they didn't. Many packing-lists were broken.

The other half of the problem was the package builder quandary: making packages meant installing lots of ports on one's machine, including potentially unsafe stuff. To add insult to injury, many ports conflicted with each other, installing more or less the same files, which means that a package builder would have to tediously take apart most of his machines to build all possible ports.

The first solution was to build packages into a jail: create a virgin OpenBSD install in a dark corner of the disk, and do a package build chroot'ed into that jail. However, the disk resources and the time needed would be tremendous. Tag that as an inefficient solution. Also, it would not solve the broken packing-list issues.

I don't quite remember where the `fake' idea came from in the first place. The notion that software could configure to install somewhere, and then be redirected elsewhere has been floating around for a long time. It might be that Dug Song mentioned the basic idea to me. All that remained was to check how practical this could be. I spent one week-end trying the basic design, a few ports let themselves be converted without major hastles So we embarked upon the adventure... and managed to fake three quarters of the OpenBSD ports collection right before 2.7 !

Fake ports: turning the world upside-down

The basic notion of `fake' is simple: instead of installing a port, and then creating the package from the port, let's coerce the port into installing elsewhere (a `fake' install, since the result is completely useless without further processing), build the package from that faked installation, and install the package.

The name `fake' is mine. There were some objections, but I believe the name to be fun and evocative. After all, there is no reason why working on the ports tree would have to be boring.

Jackpot: fake solved all problems at once. Packing lists had to be correct, since only the actual package was used. Package builders could stop at the `make package' step, and avoid installing most ports on their machines. Moreover, packing-lists became easier to craft, since the fake'd port ended up being pretty much alone in its fake subdirectory. One very welcome side-effect was that suddenly, it became very practical to build package with conflicting files on the same machine, and so, the sudden increase in ports variations (ports flavors) can be traced back to fake. Also, finishing that boring part of ports building, going from `make install' to a package has become much quicker: make package, fix problems, wipe fake-*, redo make package. I estimate that creating a new port takes about half the time it used to.

The only fun part is: how to trick third party software into installing elsewhere. It turns out that, in many cases, third party software already has provision for that, simply by setting DESTDIR or another such variable to the right prefix. Some other ports can be coerced simply by configuring them with prefix=${DESTDIR}/usr/local: while building, we leave DESTDIR empty, so that the port is instructed to work under /usr/local. Then, the `fake-install' step turns things inside out, and installs the port under ${WRKINST}/usr/local. In all, those two cases cover over 90% of all ports. Most common problems are ports which offer this basic infrastructure, but which has never been used before us, so a few patches are needed to repair it. Also, very few ports actively resist fake'ing, but so far, we have always managed to win.

To see fake work in practice, just choose one port with FAKE=Yes (the list of unconverted ports shrinks almost daily), and type `make package'. After the dust settles down, you should have a skeleton OpenBSD tree under work/fake-* (with lots of empty subdirs, and only the port's files there), and a package under /usr/ports/All/${ARCH}

The future

Where to go from there ? There are still a few salliant issues with the current ports tree. Two major problems are going to be tackled over the following months: handling dependencies more nicely (finally trusting packages like NetBSD does), and creating a mechanism for package update. The dependency handling is mostly figured out, there are just a few lines of code that need writing.

Package updating is a much more interesting problem. The basic idea is to be able to keep package stubs around while other software still depends on it, and while you are compiling a newer package. Basic usage is to keep old versions of shared libraries until they are no longer needed, but more esoteric issues need to be fixed.

All in all, between more flexible dependencies handling and package updates, it is very likely that OpenBSD packaging tools and packing-lists will see a major overhaul over the next few months.

Also, we should see the OpenBSD ports tree resume growing fast. The Project now has quite a few features that make building new ports much easier than it ever was. Ports are getting simpler, there are less and less arcane details to master to make a new port, and more automated tools to help the porter (make update-patches, for instance). I certainly hope many people are going to feel excited about the new OpenBSD ports tree, and are going to want to create new ports. Baring some accident, we should be well beyond the 1000 packages mark in 6 months time.