Contributed by jj on from the sudo-make-me-coffee-in-parallel dept.
In this article, I explore how OpenBSD's clean code and sane defaults recently saved the day. For great science!
It's no secret that OpenBSD is an excellent research platform. From packages(7) for specialised software to out-of-the-box httpd(8), sshd(8), and so on, it's a no-brainer to pop OpenBSD onto a workstation and just get to work.
By way of background, I often [ab]use make(1) for generating visualisations from hundreds of datasets. It usually goes something like this:
- generate a set of parameters, like moments of a distribution;
- generate data files from these parameters;
- generate plotter input by applying parameters to a template;
- plot using the plotter input and data files.
parm1=0.01 p1.png: p1.plot p1.dat gnuplot p1.plot p1.plot: template.plot sed "s!@OUT@!p1.png!;s!@IN@!p1.dat;s!@PARM@!$(parm1)!" $< >$@ p1.dat: longrunning >$@
This trivial example uses gnuplot(1) to generate an image p1.png from a data file, p1.dat, and plot input p1.plot. Assume that the longrunning utility consists of heavy number-crunching. Now imagine hundreds of data files and images... you get the point.
This is fabulously easy on OpenBSD. On a new box I pkg_add(1) the required tools, pull my templates and Makefile from an off-site cvs(1)) repository, then invoke make(1). Time from CD boot? A handful of minutes.
Meanwhile, a recent project was pushing my patience: the parameters needed lots of tweaking, with partial builds taking over ten minutes . I can only drink so many coffees per day (note: conjecture).
Each plain-text data set took from a few seconds up to minutes to generate. It then occurred to me that I could use a nearby backup machine to build dataset targets with ssh(1), since the output format of longrunning was machine-independent.
p1.dat: ssh node1 longrunning ">$@" scp node1:$@ .
(Note: I shell-escape ">$@" for generality. In this snippet, sending directly to $@ is valid.)
This worked, although builds completed unevenly due to longrunning's random execution time. This didn't bug me: speed-up with rough edges is still speed-up, right? But when I tripled my simulations and was stuck guessing remote host load and baby-sitting builds, the hack had outlived its usefulness.
OpenBSD to the rescue!
Unlike the systems of choice for some researchers, where solutions to such problems involve despair and liquor, OpenBSD has a third option: /usr/src. Why not hack make(1) to distribute target builds on-the-fly? No need to hardcode remote hosts and endure load inequality. To the source!
make(1) had always seemed one of those utilities you just run and try not to think about. But it took only a few minutes to walk through usr.bin/make, starting with grep exec, and discover exactly where to play with build targets. Tricky bits, functions, even files themselves were well-documented in the source.
In no time at all, I added a new special source to whitelist distributable targets (.DIST) and some goop to send and receive dependencies and targets. The dispatcher was ready to go!
The bad news: I lost my coffee time.
The good news: distributed
In the end, my research capacity jumped from a small set of simulations to a set proportionate to my build cluster. Adding a new machine? Slap OpenBSD on the disk and add ssh(1) keys. Upgrading the dispatch machine? Re-apply the patch and get back to work.
The moral of the story is that the clean code and low barrier to entry, cd /usr/src/usr.bin/make ---not to mention the excellent default system installation---are invaluable tools. Scientific computing puts a lot of stress on computation, but there's more to it than algorithms and tuning: it's a process. And part of that process is starting dhcpd(8) on a secondary Ethernet device, attaching some boxes, and running them into the ground.
Is this feature useful for the general make(1)? No. Is it useful for me? Absolutely. Can it be improved? Sure. All for another day, and another adventure in /usr/src!
Unofficial Distributed Build Extension of OpenBSD's bsdmake:
(Comments are closed)