OpenBSD Journal

l2k16 hackathon report: LibreSSL manuals now in mdoc(7)

Contributed by pitrh on from the all the crypted pages dept.

The first report out of the just completed l2k16 (LibreSSL focused) hackathon comes from Ingo Schwarze, who writes:

Hackathons are great for starting or finishing tasks. This time, for me, finishing it was.

Back in the spring two years ago, Kristaps Dzonsons started the pod2mdoc(1) conversion utility, and less than a month later, the LibreSSL project began. During the general summer hackathon in the same year, g2k14, Anthony Bentley started using pod2mdoc(1) for converting LibreSSL manuals to mdoc(7).

Back then, doing so still was a pain, because pod2mdoc(1) was still full of bugs and had gaping holes in functionality. For example, Anthony was forced to basically translate the SYNOPSIS sections by hand, and to fix up .Fn and .Xr in the body by hand as well. All the same, he speedily finished all of libssl, and in the autumn of the same year, he mustered the courage to commit his work.

Near the end of the following winter, i improved the pod2mdoc(1) tool to actually become convenient in practice and started work on libcrypto, converting about 50 out of the about 190 manuals. Max Fillinger also helped a bit, converting a handful of pages, but i fear i tarried too much checking and committing his work, so he quickly gave up on the task. After that, almost nothing happened for a full year.

Now i was finally fed up with the messy situation and decided to put an end to it. So i went to Toulouse and finished the conversion of the remaining 130 manual pages in libcrypto, such that you can now view the documentation of all functions taking BIGNUM arguments with

man -lk Fa=BIGNUM

and once inside less(1), you can jump to the description of a particular function by typing
:tECDSA_sign_ex<ENTER>

which are two examples of features requiring mdoc(7) that weren't available with perlpod(1).

You might wonder whether it isn't waste of a valuable hackathon to work on nothing but manuals. It may be, if there are other ways to get the work done. But after two years of stalling, it seemed the right move to me to forcefully push through.

Besides, meeting the LibreSSL crowd in person helped communication, as usual. For example, i wrote the mlinks(1) utility to extract page names from a mandoc.db(5) database. That's needed by portable LibreSSL for installing MLINKS - hard links to manual pages documenting more than one function - on operating systems with a less powerful man(1) implementation that is not based on mandoc(1).

Many functions in libcrypto were documented in more than one manual page, which caused confusion when trying to look up pages by function name, in particular for the new mlinks(1) utility. I cleaned that up completely such that now, each name points to exactly one page in the LibreSSL manuals.

After finishing the main work, i cleaned up the Makefiles and moved the libssl manuals that were misplaced in a doc/ directory to their proper place in man/.

Besides, i implemented one minor improvement to pod2mdoc(1) - simpler formatting of SYNOPSIS prototypes of void functions with .Fn instead of .Fo. The fact that nothing more was needed demonstrates how mature pod2mdoc(1) is by now.

After a hackathon is before a hackathon - there are still huge amounts of work to do on LibreSSL documentation. Many public functions lack manual pages in OpenBSD. Most missing manuals can be imported from OpenSSL, again using pod2mdoc(1), but some aren't documented there either, so some pages need writing from scratch. And many aspects need cleanup and checking in the existing manuals. It is unlikely that i will be able to do all that work, not only because it's too much for one person, but also because much of it requires familiarity with the libraries.

But at least, i have started importing a few missing libcrypto pages and adding the missing license headers. I should at least complete the work on the license headers because having text in the tree that requires consulting version control to figure out who owns the Copyright on a particular file and what the exact license terms are is awkward at best...

Even though i like Le Midi de la France a lot and had visited half a dozen cities within a 200 km radius around Toulouse and another half a dozen within 300 km, this was my first visit to Toulouse itself. But this was a focussed and productive working event, and apart from some casual strolling around the historical city center, i didn't find time for leisure. However, Matthieu@ did a terrific job taking us to a different French restaurant for each dinner. He may have spoilt us, but i really don't hold that against him. Unsurprisingly, the restaurants not specifically advertising vegetarian options were usually a lot better than the those doing so, but surprisingly, almost all had some vegetarian option, and resorting to fish was only required once - and a delight, too.


And for completeness, here is the LibreSSL documentation timeline:

Phase 1: prehistory (0 pages converted)
  • 2014 Mar 20 to Apr 7: Kristaps Dzonsons starts pod2mdoc(1), up to release 0.0.11
  • 2014 Apr 7: Damien Miller: cherrypick fix for CVE-2014-0160 "heartbleed" vulnerability from OpenSSL git; ok sthen@
  • 2014 Apr 8: Ted Unangst's famous exploit mitigation countermeasures mail on tech@
  • 2014 Apr 13: Miod Vallat: Import OpenSSL 1.0.1g
  • 2014 Apr 13: Theo de Raadt: 1st Valhalla commit, s3_lib.c rev. 1.22; Martin Pieuchot and Bob Beck follow within less than 2 hours
  • 2014 Apr 14: Joel Sing: First pass at applying KNF to the OpenSSL code.
  • 2014 Apr 15: Ted Unangst: remove FIPS mode support
  • 2014 Apr 16: first LibreSSL commit by Philip Guenther
  • 2014 May 17: Bob Beck officially announces LibreSSL during BSDCan
Phase 2: antiquity (83 pages converted)
  • 2014 Jul 11 to Jul 19: Ingo Schwarze applies some improvements to pod2mdoc(1), up to release 0.0.13
  • 2014 Jul 11: Anthony Bentley starts using pod2mdoc(1) for libssl
  • 2014 Oct 12: Anthony Bentley commits libssl, 83 pages
  • 2014 Oct 22: Ingo Schwarze implements the Ft/Fo/Fa/Fc formatter for the SYNOPSIS
  • 2014 Nov 17: pod2mdoc(1) release 0.1
Phase 3: middle ages (57 pages converted)
  • 2015 Feb 12 to Feb 23: Ingo Schwarze uses ohash(3) to improve markup
  • 2015 Feb 14: 15 pages converted
  • 2015 Feb 16: 12 pages converted
  • 2015 Feb 23: 18 pages converted
  • 2015 May 19: pod2mdoc(1) release 0.2: fully functional for production
  • 2015 May 24: 1 page converted by Max Fillinger
  • 2015 Nov 11: 5 pages converted by Max Fillinger
  • 2015 Nov 12: 6 pages converted
Phase 4: modern era (130 pages converted)
  • 2016 Nov 2: 34 pages converted
  • 2016 Nov 3: 50 pages converted
  • 2016 Nov 4: 34 pages converted (plus one new function)
  • 2016 Nov 5: 12 pages converted (plus one from scratch)
  • 2016 Nov 10: first post-1.0.1g page imported from OpenSSL

(Comments are closed)


Comments
  1. By Pavan Maddamsetti (68.39.68.205) on

    Thanks Ingo.

  2. By Ryan (184.69.165.214) on

    Awesome work, thank you!

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]