p2k23 Hackathon Report: Marc Espie (espie@) on a flurry of packages activity

The p2k23 OpenBSD packages hackathon just concluded, and Marc Espie (espie@) wrote in with this report:

Off to Dublin, or almost.

This ports hackathon started with a reminder that real-life bugs do matter: my morning flight was cancelled and I arrived in Dublin late that day.

Turns out that air traffic uses named waypoints, which are supposed to be unique, but there's no central name registry, and two local waypoints ended up with the same name, which caused huge confusion in air traffic to Ireland: planes making large detours. As it turns out, there ARE some safety regulations, so crews can't fly forever, and Aer Lingus had to cancel my flight: no idea whether the plane didn't make it to Paris, or if the crew logged so many hours that they couldn't fly back.
(all of this info courtesy of sthen@ and mlarkin@, one being the thorough guy who always finds out the most obscure details, and the other one having actual flying experience)

As any seasoned OpenBSD developer will tell you, planning a hackathon is pretty much useless. You make all kinds of careful plans and end up getting derailed by other people present during the first day or so.

Also, these days, I have so many things I'd like to move forward, it's difficult to pick and choose.

In reality, this hackathon started a little earlier: thfr@ had made an interesting patch to try to hide most of the gore behind github-hosted projects, and it prompted me into changing other things: we've had a fixed list of MASTER_SITES variables basically forever, and it was starting to get awkward with some modules hijacking MASTER_SITES9, MASTER_SITES8.

I had wanted to make it somewhat open-ended for at least a year, but I thought that having to register the suffixes somewhere would be awkward and inelegant.

So I tried a different approach: why not let make tell me about the variables proper. Creating a mechanism that would list all variable names should be easy, right ? the only potential issue being that this meant adding a new feature, something you generally have to fight for with this crowd.

Luckily, gmake already has the feature, so naming it the same meant that .VARIABLES reached instant acceptance.

I also worked on another feature, but this one didn't make it (yet…): execve has size limitations, and with go ports, some shell fragments overload this limitation.

So instead, of just doing "sh -c somehugecommand", I could have a backup plan that would echo somehugecommand >tmpfile && sh tmpfile.

(Works like a charm, and allows all of the ports framework to run with much larger variables, but it wasn't accepted yet… oh well)

So I had (mostly) done the preparatory work to have DISTFILES.sufx match MASTER_SITES.sufx and have bsd.port.mk auto-discover the names.

A ports hackathon is a great place to move fast on those kind of changes: if anything breaks, you will know about it quickly.

Back to thfr@'s patch: he had a new version using the new mechanism, and it was getting ready to commit. There was an interesting new heuristics to auto set DISTNAME, which I decided to adopt for bsd.port.mk.

Of course, this broke things, until several iterations later, where I have figured out the correct heuristics which is a mix of thfr@'s and mine.

So dist-tuple.mk made it to the tree, so that we could keep working. At this stage it was two times the same code, but with an extra parameter added.

After trying to fold it down using make's dark magic, we settled on removing the extra DIST_TUPLE_MV variable, and just giving one special value to the parameter, to mean "do not move" (.)

Meanwhile, Ireland had definitely unseasonal weather. I didn't see a drop of rain during the week. Our hotel was located in a lovely posh residential area of the city, quite nice scenery and good food, but a bit on the pricey side (more expensive than Paris!!!).

We had food at several somewhat local places, some fancy international place (personal highlight: a ramen place with prawn gyoza and warm sake), went to a few pubs, and sampled the local beverages.

I was at a disadvantage, since I don't drink beer, but I made up for it with the local cider (Bulmer) and whiskys.

(Am I allowed to say that I still prefer scottish whisky ? I understand the irish take on striving for a "lean clean" taste, but I'm more of a heavy peaty/smoky flavor guy)

Very nice hotel, no problems hackroom, and a wonderful host: I can't sing enough praise about Tom. Incredibly generous guy and always smiling.

So, back to hacking. Having everyone together means that consensus is fast.

Testing was a bit more complicated during the second half of the hackathon, thanks to robert's shenanigans in llvm land, but still, we also finally shortened MASTER_SITES into SITES.

This means, of course, that the ports tree is going to see a bit of churn again. But for once, there are no bumps involved since only the name changes.

In order to test, I had to wipe my old distfiles, so I special-cased "make clean=dist" to ignore FETCH_MANUALLY port, since those are always a bitch to get back, and I found out that there are a few errors that were completely unrelated to what I was doing.

So, time to instrument dpb a bit more, so that the "bad" distfiles log gets annotated with more data, specifically making it easier to find distfiles that no longer have any working official sites and where we rely on our own backup sites, and also flagging broken "first" site, so that maintainers and other people can try to find suitable replacement.

As usual, documentation is key, so after making sure DIST_TUPLE was properly documented, I ended up polishing bsd.port.mk(5) a great deal (I lost count of the number of commits!)

My final endeavor was to try and teach portroach about the new stuff: the poor thing gets easily confused, and having several sources of DISTFILES would make it worse.

So there's now official support for ROACH_URL that should point to the main archive of a port (usually automatically deduced, but can be hand-tweaked if need be) and ROACH_SITES, which will be automatically found.

I believe that portroach didn't know about the url replacement scheme from ages ago, so that should already be an improvement. I also figured out that portroach relies on DISTNAME, which isn't always set for us, but using FULLPKGNAME instead is much better for our needs.

Various odds and ends include fixing a stupid counting bugs reported by tb@ that made dpb go to 11 all the time, and interacting with schwarze@ about making man(1) slightly more intuitive.

(we may also kill texinfo in base… more later)

All in all a rather productive week. I was missing a few of my friends who couldn't make it, such as Aja, Ingo, Jasper and Naddy !

You know a good hackathon when you still have things to do at the end !

Again, many thanks to Tom Smyth for making things so good for us, and also for ncsc.gov.ie (maybe we frogs should ask ANSSI) and the OpenBSD Foundation for being our sugar daddies.

And a final thank to my employer, EPITA, for being enlightened about my Open Source activities !

Thanks for the report and the excellent work, Marc!

We hope to bring you more hackathon reports soon!