OpenBSD Journal

p2k17 Hackathon report: Christian Weisgerber on random devices, build failures and gettext

Contributed by Paul 'WEiRD' de Weerd on from the geography dept.

The latest hackathon report to come in today came from Christian Weisgerber (naddy@):

“You're German, you must know Berlin, right?” “Uh, here's the list of European capital cities I live geographically closer to: Amsterdam, Brussels, Luxembourg, Paris, Bern, (Vaduz), Prague.”

Without any splashy projects in the pipeline for p2k17, I went to work on what can be described as janitorial tasks.

Since OpenBSD 4.9, the various /dev/*random devices have all been functionally equivalent. Recently we started paring down the alias devices. Two weeks before the hackathon I removed /dev/srandom. Nothing in the base system had referenced it for a long time, I couldn't even remember its original function, and I was very confident that no port used it. In that I was quickly proven wrong. Shortly before the hackthon I grabbed an amd64 package snapshot, extracted all packages, and ran grep over the contents, binaries and all. It was with this list in hand that I arrived in Berlin and spent the first day working through the respective ports. Many just try opening a bunch of different /dev/*random devices until they succeed, but a few needed fixing. Task #1 accomplished.

Next I looked into a nagging problem that had crossed my annoyance threshold in the previous weeks. I run the official amd64 package bulk builds, a lot of them, and random ports would sporadically error out because they were trying for no good reason to execute some part of the GNU autotools during the build stage. Cleaning up the working directory and retrying would succeed. The overall failure rate was low, but it felt like one port died from this every bulk build. Eventually I started noticing that the affected ports weren't that random after all.

At the hackathon, I picked out audio/schismtracker and, surprisingly, managed to reproduce the problem quickly on my own build machine. Instead of proceeding to the compilation, make(1) first executed autoheader. The Makefile rules generated by automake are clear: autoheader regenerates config.h.in, which depends on configure.ac and aclocal.m4. Somehow one of the source files had to be newer than config.h.in, but that seemed impossible in this port as all the GNU build system files were freshly generated by autoreconf. Absent a bug in make(1), the modification time of a file had to be wrong. Originally, Unix time stamps used to have single-second resolution. As machines became faster, POSIX standardized nanosecond precision, but the old APIs still exist. Now, ls(1) offers no means to see the full modification time of a file, but stat -f %Fm does. This exposed that sometimes config.h.in was indeed older than aclocal.m4, triggering the problem. Revealingly, while the other files had modification times with random-looking fractional seconds, config.h.in always had x.000000000 seconds. Clearly something truncated the time stamp to full seconds, causing the modification time to move back into the past.

In order to cut through the twisty maze of the GNU autotools, I ran the whole autoreconf invocation with ktrace -i, and kdump quickly showed what happens: A perl(1) process opens a temporary file in /tmp, creates config.h.in, reads and writes the contents of the temp file into config.h.in, and at the end copies the modification time without nanoseconds before unlinking the temp file. This happens in autoconf's Autom4te::FileUtils, which performs the file copy with... the move() function of File::Copy, a standard Perl module shipped in base that uses utime() to set the modification time. Who's to blame, autoconf or Perl?

As it turned out, I was not the first one to run into this problem and the autoconf upstream replied that their repository already held a fix; it employs the big sledgehammer of spawning the system's mv(1) command. espie@ pointed out that importing Perl's Time::HiRes module actually provides overrides for stat() and utime() that use nanosecond precision, and our Perl maintainer afresh1@ submitted a fix for File::Copy based on this. One way or the other, the problem of the quasi-random build failures will be fixed. Task #2 accomplished.

Finally I could tackle my main project for this hackathon: further reducing the use of the gettext module. Back in the day, the gettext module made a lot of sense since it abstracted the different requirements of static and dynamic architectures. With static archs a dim memory at this point, the gettext module is more trouble than it is worth, yet another special case that contributes to the byzantine complexity of our ports system, and something I want to eradicate eventually. Getting the dependencies of subpackages right is already difficult enough in MULTI_PACKAGES ports, without having to work around the gettext module trying to add itself everywhere. Grepping the tree for MODGETTEXT found the ports that could be simplified and I spent the rest of the hackathon doing just that. Unsurprisingly, some of the ports proved to have odd dependencies that had probably not been intended like this by their maintainer.

While I was working on this, I noticed a port with an illegal package name. Version numbers must not have a dash, so something like 3.2.0-rc1 is wrong. See packages-specs(7) for details. I prodded the MAINTAINER to fix their port, but this got me thinking. If there was one such port, there were probably more. And yes, a search turned up more instances, easily fixed. I became slightly curious about one of those ports, tried to run it, and it promptly segfaulted. bentley@ looked into it but in the end removed it. It's hard to take three steps in the ports tree without treading on a turd.

As the hackathon wound down, I finished my gettext work. Task #3 accomplished. Some five hundred references to the gettext module are still left, but the remaining cases keep getting simpler. Maybe the tail end will lend itself to a semi-automated solution. But that shall be a task for another hackathon.


  1. By rjc (rjc) on

    The first line should read Weisgerber, not Weisgerer.

  2. By Jorden Verwer (159.46.196.29) on

    Technically Switzerland doesn't have a capital city, although Bern is widely considered its de facto capital city (like Den Haag would have been the de facto capital city of The Netherlands - except that it already has a de jure capital city in the form of Amsterdam, although this change occurred relatively recently, i.e. in the nineteenth century). Based on your list I guess you also live closer to your own country's lovely former de facto capital city, Bonn. ;)

    Anyway, what is the intended final result of the /dev/*random pruning action? I'm guessing removing everything other than /dev/random is a bit too far ahead of the times, but which alternatives should remain and which can go?

    1. By Christian Weisgerber (2003:e5:cbc0:6201:11d9:27b4:12ff:2a3c) naddy@openbsd.org on

      We will keep /dev/random and /dev/urandom, since these are now established across operating system. On OpenBSD, both behave the same, i.e., they “produce high quality pseudo-random output data without ever blocking, even immediately after booting”. To make clear that there's no difference, /dev/random is now a symlink to /dev/urandom.

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]