OpenBSD Journal

g2k18 hackathon report: Matthieu Herrb on font caches and xenodm

Contributed by Paul 'WEiRD' de Weerd on from the cache-this dept.

Next in from Ljubljana is Matthieu Herrb (matthieu@):

I spent the first day and a half chasing an issue that many people have been experiencing: the first X start after an install or upgrade from a snapshot was slow. The direct cause of this is well-known: it's the fontconfig library (used by all applications that do font rendering) that re-builds its font cache (~/.cache/fontconfig/) because of newly installed fonts.

Except that normally, OpenBSD X sets ship a pre-computed, system-wide cache (/var/cache/fontconfig/) that should avoid this. So what is the problem ?

I found two reasons for this. The first one was a bug in the fontconfig library, introduced with the more recent re-work of the caching code, which would incorrectly compare the nano-seconds part of the time stamp of the installed fonts with the one from the cache (it was comparing it against the integral seconds part instead). This was not enough to fix the original issue, but it gave a clue…

In fact the problem comes from the fact that the fonts extracted from the tarballs get their last modification time stamps restored only with one second precision, since the nanoseconds parts of the mtime field is not stored in the USTAR file format used by OpenBSD distributions sets. But the pre-computed cache uses the full time stamp, including nanoseconds. So after extracting the sets, the cache data doesn't match the actual installed fonts and fontconfig decides to re-build it.

My suggestion to just stop shipping a pre-computed cache and to recompute it as part of the rc.firsttime script that is run on each install/upgrade was rejected by the hack room as too expensive for slower machines. Fixing OpenBSD's pax(1)/tar(1) command to store the extended nanoseconds field in the archives (there are extensions to the USTAR file format for this, they are just not supported) was also considered as too expensive, although some people agreed that OpenBSD should add support for this extension at some point…

So that left us with the solution of get the pre-computed cache to be valid with only one second resolution, ie force the nano-seconds part of the last modification time stamp in the installed fonts to be zero. This solution was suggested remotely by millert@

For this guenther@ and espie@ perl skills helped to stick this in a perl one liner. Curious readers will find it in xenocara/fonts/alias/Makefile.bsd. Et voila, OpenBSD's fonts set now match after extraction what was there when the contents of /var/cache/fontconfig was computed and fontconfig is happy and won't trigger a rebuild.

After this, I got back to my original project of working a bit more on xenodm(1), the stripped down X Display Manager that replaced xdm(1) in Xenocara. I've fixed a couple of bugs cleaned a few more unused code and got a better view on what can be done to eventually better protect it with privilege separation and pledge. But this is not finished yet, so it leaves things to do for a next hackathon.

I've enjoyed Ljubljana for the third time, it's nice food, graffiti and all the interactions with other OpenBSD developers, including some new heads.

Thanks to Mitja and all people who made g2k18 possible.

Many thanks to Matthieu for his report and the work on X in general!

(Comments are closed)


Comments
  1. By Jörg Schilling (schily) joerg@schily.net on

    Your assumptions are wrong.

    Creating or using tar archives that include the POSIX.1-2001 extended headers is not expensive. "star" while archiving extended meta data is e.g. still faster than "gtar" even in case that gtar only archives in the historic POSIX.1-1988 TAR format.

    Checking nanoseconds has become standard since a while and even most "make" implementations check the nanoseconds in the files timestamps.

    So you just need to use the right software ;-)

    Check the latest schilytools source tar archive:

    http://sourceforge.net/projects/schilytools/files/

    BTW: since it is more that 20 years that extended tar archivers exist, it may be time to create POSIX.1-2001 extended archives by default. If you are still using a tar implementation that does not support them, you should upgrade.

    Comments
    1. By Marc Espie (espie) espie@nerim.net on

      Schilly, can you go troll something else ?...

      Making damn sure the whole chain uses nanoseconds is what takes time.
      ...especially without security complications.

      If you want an almost perfect analogy, utf-8 is also quite well-spread,
      but it still has pitfalls.

      Go see Ingo's presentation at eurobsdcon, and come back with actual useful comments.

      Comments
      1. By Jörg Schilling (schily) joerg@schily.net on

        BSD added support for time stamps with sub second granularity in March 1983.

        Most user space tools started with adding related support 20 years ago.

        Welcome in the presence!

    2. By Magic carpet (bodie) bodzart@openbsd.cz on https://www.bodie.cz

      Time for little trip in to the history............

      https://marc.info/?t=110806777500003&r=3

    3. By Marc Espie (espie) espie@nerim.net on

      Also note that I'm responsible for implementing subsecond precision in make, along with the corresponding very real headaches (specifically the fact that NFS does not really play along), and I've looked close enough at tar extensions (specifically useful ones, like long name extensions).

      - adding subsecond precision to tools often finds new issues, because when time appears to go backwards because some tools are clamped at the second level and other ones are not, things go "uh ?"
      - the ustar extensions are a kludge at best. The format is awkward, wasteful of space, somewhat annoying to parse, and desaligns about everything in the archive. It's one case where they definitely should have dumped tar and designed something else.

      Comments
      1. By Jörg Schilling (schily) joerg@schily.net on

        Typical make implementations added support for misaligned clocks with NFS 30 years ago already.

        If your local make implementation correctly deals with that at second granularity, it is a no-brainer to add sub second granularity. It took me approx. 5 hours for smake and I was forced to do this since I made SunPro Make portable (see schilytools since 1.5 years) and that make implementation deals with sub second granularity since 20 years. If however one make implementation does it and the other does not, you get really strange effects.

        The biggest change I was forced to implement was to make sure that "sccs get -o ..." creates a file that is at least a few nanoseconds newer that the related s. file while still using SCCSv4 history files with only a second granularity. Well, of course this requires a tar implementation that deals with sub second granularity. Solaris tar does this since 1997, star does it since 2001. GNU tar does it since a few years and even rsync recently added support for sub second granularity. Keep in mind that filesystems support sub second granularity since more than 30 years and even POSIX added nanosecond support 20 years ago.

        Regarding your statements related to the tar headers, star added a lot of new features to the 1977 tar header in 1985, gtar did something similar in 1991. The POSIX proposal from Sun from 1997 made the tar format a cleanly infinitely extensible archive format. If you believe something else, I recommend you to have a look at the format definitions.

        BTW: supporting really long path names in tar *is* a challenge. But this is not related to the archive format, since the POSIX.1-2001 tar extensions support path names up to 8 GB in length. I am working on really long path name support in star since a month and I am almost done. There are a few remaining static path buffers in star, but "star -c -find" supports to create archives with long path names and "star -x" supports them since last weekend. The rest will most likely ready in a week. If you like to test this, you however first need to add similar support to your shell ;-) My shells support TAB completion, test(1), File name generation and cd/pwd for really long path names...

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]