Toronto 2013 Hackathon Report - Ingo Schwarze

During the first Toronto hackathon, I focused on the SQLite database backend for mandocdb(8). Currently, mandocdb is still disabled in OpenBSD-current, but it is intended to become a drop-in replacement for the makewhatis(8) utility, providing enhanced search capabilities for the apropos(1) manual page search utility.

In particular, it is designed to support searches for semantically annotated text in manual pages, for example to specifically search for subsection headers, utility options, file names, include files, function arguments, variable names, preprocessor macros, standard titles, author names, cross references, and much more.

Given that Kristaps Dzonsons (kristaps@) already invested considerable effort into the mandocdb SQLite backend in June 2012, my original hope was to complete his work during t2k13 and get the SQLite-based mandocdb ready for general consumption. Unfortunately, it turned out that a lot of functionality was deliberately dropped or inadvertently broken while replacing the old Berkeley database backend with the new SQLite backend, and that many regressions crept in. Consequently, I spent t2k13 starting rather than finishing SQLite integration work, putting back lots of lost functionality, fixing several bugs and cleaning up the code in multiple respects.

In the OpenBSD development process, we pay attention that each commit to the OpenBSD CVS repository keeps the system in a fully usable state and, in addition to that, makes the system better than it was before the commit. Consequently, I could not yet switch OpenBSD mandocdb to SQLite. Thus, if you enable mandocdb in your personal copy of OpenBSD-current by saying

  # ln /usr/bin/mandoc /usr/bin/mandocdb

that version of mandocdb is still using the Berkeley DB backend.

Given that the mandocdb SQLite backend is not yet ripe for commit to OpenBSD, I had to work mostly in the mdocml.bsd.lv repository. Specifically, I did the following in this area:

Repair functionality broken during the switch to SQLite

• -d/-u: Allow changing existing databases, it didn't work at all.
• Repair parsing of .Nd for both formatted and unformatted manuals.
• In .Xr database entries, mention the manual section again.
• Add back the lost realpath(3) checks at multiple places.
• Restore two sanity checks that were lost in treescan().

Maintenance

• Get rid of variadic macros for stderr output because gcc2-platforms cannot handle them.
• Consistently use the PATH_MAX (POSIX), avoid MAXPATHLEN.
• Rename "mandocdb.db" back to "mandoc.db".
• In dbopen(), check success of remove("mandoc.db~").
• Optimize putkeys() to not call ohash_qlookupi() and ohash_find() twice.

Simplifications

• Drop the unused mdoc_handler flags.
• Drop wordaddbuf() which was identical to putkeys().
• Drop hashget() and integrate its (short) code into the caller.
• Drop stradd() and stradds(), they were pointless.
• Make parse_man() void, the return value was unused.
• No need for strlcpy() and strlcat() when handling constant strings.

As I said, this work is incomplete. There are still a few fixes required in mandocdb, and then I have to look at the new apropos(1) code in the mandoc codebase which, of course, also switches to the SQLite database backend because it has to read the mandocdb databases.

When that will be ready, enabling the code has to be done more carefully than when I first tried: Using the Berkeley database backend, I enabled an enhanced mandocdb/apropos combo hacked up mainly around the p2k11 Budapest hackathon on January 5, 2012, but it caused various regressions and excessive build times and had to be backed out again on January 16, 2012. The plan is to use the lessons learnt back then to proceed more cautiously when retrying with the SQLite backend. It definitely won't be ready for OpenBSD 5.4, but I hope to get it ready for 5.5.

Multibyte character issues

Even though I hate internationalization, localization and multibyte character support in general and try to avoid working on such issues as much as I can, during this hackathon, I temporarily got sucked into that gruesome area:

• I had to fix the mandoc(1) utf8 output mode to work as documented because it frankly didn't do anything at all, at least on OpenBSD.
• In mandoc -mdoc -Tutf8, I fixed .Do .Dq .Ql .So .Sq to generate the correct roff(7) character escape sequences such that output modes like -Tutf8 have a chance to select nice glyphs.
• I had to fix perldoc(1) to only produce UTF-8 output when the user's locale asks for it.
• I had to fix perldoc(1) to not feed UTF-8 input into mandoc or nroff because that resulted in corrupt output.

Besides, I discussed some aspects of Stefan Sperling's (stsp@) new locale(1) utility with him and helped with the review of his code.

Cooperation with many fellow developers was particularly helpful while straying into this bizarre area; thanks to stsp@ matthew@ bentley@ tedu@ naddy@ espie@ millert@ bluhm@ sthen@ and anybody I forgot...

Other topics

Since the spring of 2012, I hacked less on OpenBSD than I used to during the preceding years, and it shows: Bugs got resolved more slowly, bigger projects made little progress, and even some smaller patches got stuck and are still uncommitted. Well, sometimes, that pesky thing called "real life" does get in the way.

For that reason, during this hackathon, I also ticked off quite a few smaller fixes all over the place, not forming any kind of a coherent project. Sorting out such odds and ends during a hackathon is not terribly efficient, but maybe better than letting things rot even longer...

Various bugfixes

• mandoc(1) tbl(7): Do not crash after certain unclosed .T{ macros.
• mandoc(1) input: Reject non-printable characters even when escaped.
• mandoc mdoc(7): Two minor bugfixes related to the .Bk macro.
• makewhatis(8): Do not mishandle "--" at the end of a line as hyphenation.
• sysutils/pstree(1): Substantially improve the manual.

Various mandoc(1) maintenance and code cleanup

• in mdoc(7) .Bk ascii output code
• in tbl(7) parsing and output code
• some code synchronization between mdocml.bsd.lv and OpenBSD
• a new tbl(3) manual documenting the internals of the tbl(7) parser

A global village called Toronto

As I often like to when coming to a new place, I spent a few days in the city after the hackathon, exploring the surroundings. The most impressing point probably is that Toronto is among the most bike-friendly cities I have ever seen. That's certainly surprising in a city of that size, and those few that i might consider even more bike-friendly are less than a percent of its size, mostly located near the border of Gelderland (NL) and Westmuensterland (NRW, D).

The city is publishing a free map of cycling trails and cycling lanes and you can comfortably get from nearly anywhere to anywhere within the city's borders (about 32 by 16 kilometers) using these trails. They are generally in an execellent state and many of them are located in very nice parks and river valleys. I particularly liked the Humber Valley which I biked from its mouth at the shore of Lake Ontario up beyond the city border to Vaughan, but the Black Creek (going down from York University), the Don Valley (except for the noise of the DVP, the Don Valley "Parkinglot") and the Tommy Thompson Park on the man-made Leslie Street Spit in Lake Ontario are also quite nice.

What is even more important than infrastructure is the attitude of motor vehicle drivers. According to my personal experience, almost all automobilists let cyclists pass first - even if they have priority in traffic. And I met nobody who drove a car in an impatient or reckless way, even though I biked well over 100 kilometers all told. Sure, some of the Torontonians I talked to could tell some unpleasant stories, but that's probably the case anywhere.

Oh, the "locals". Half of them literally come from all over the world. Like the guy running the Turkish Restaurant who was born in Sweden or the guy riding his road bike who used to work in Munich. Toronto is still growing, and it's not hard to understand why. Ken, thanks for a very nice hackathon, I'm looking forward to coming back, too, if we get another chance!

Thank you, Ingo, for the detailed report and your continued work on improving mandoc!