OpenBSD Journal

A Tale Of Mozilla Ports

Contributed by sean on from the the-long-tail-of-firefox dept.

Landry Breuil (landry@) writes in with a fascinating and sorted tale of porting:

Here's a tale about the Mozilla ports infrastructure, but it could be titled, "I needed the Lightning calendar extension for Thunderbird, and I ended up maintaining all Mozilla ports."

I work in a small public company where we deal with huge amounts of GIS data so I ported geo/qgis, geo/mapserver and geo/tilecache, but that's material for another story. I'm the IT guy at the company and hence we heavily rely on open source software for all our needs including email and calendar stuff. For the latter, I setup an awesome DAViCal server (not yet ported) coupled with Lightning, a Mozilla Thunderbird extension for the client part.

Back in November, after porting most of the GIS software I needed for my daily work, I switched my main workstation from Debian to OpenBSD and only then I realized there was no Lighting port.

I quickly hacked a port for Sunbird 0.9 (the standalone calendar application) heavily based on mail/mozilla-thunderbird, and within a week it landed in the tree as productivity/sunbird. The new port also created a zipped tarball of the files needed for a Thunderbird extension (lightning.xpi). Just extract to /usr/local/mozilla-thunderbird/extensions, and voila, it worked. From there it was quite easy to make it into a MULTI_PACKAGES port, packaging Sunbird and Lightning from the same port.

In the process, I read and learned a lot about how the Thunderbird port worked, and noticed tons of similar constructs and patches in all the Mozilla-based ports. There are TONS of patches in those ports. Food for thought... --The right answer is to consolidate the similar code rather than continue maintaining multiple copies of the same thing.

All was fine, Thunderbird was at version 2.0.0.23. Soon came the new major version with all of its shiny new goodies. This definitely attracted my interest, and I started looking at version 3 in the middle of last January (as says my git log).

Of course, all things are related, so as soon as I got a working Thunderbird 3 port, I realized it required a newer version of Lightning (1.0b1). Back to square one, and then hack, hack and hack on ports, eventually getting a working update for both.

While wading through all Mozilla ports, I noticed we still had the original netscape suite child in www/seamonkey. It was still at version 1.1.x, matching the Firefox and Thunderbird 2.x branches, but these were soon-to-be-deprecated upstream. As I was working on Thunderbird 3, a new major version of SeaMonkey based on Thunderbird 3 and Firefox 3.5 branches was released. Why not give it a shot at the same time? As it turns out, it was not too hard and I soon had a working update for it.

Ok, so we got Thunderbird 3, Lightning 1.0b1 and SeaMonkey 2 working. Then you might notice SeaMonkey can also make use of Lightning, so after -lightning, add a -seamonkey SUBPACKAGE to productivity/sunbird. Similarly, mail/enigmail was built as two separate ports, but could be merged in a single port with two subpackages. Things get really interdependent and complicated, but in the meantime, upstream still releases bugfix versions you have to track.

So time for some user feedback, and hopefully get the whole mess in!

After a whole month of testing, bug hunting (like this nasty one), and sending patches back-and-forth, they finally got committed (Commit 1, Commit 2, Commit 3, Commit 4).

Thanks to those who tested and gave feedback!!

Now at that point, I was deep inside those ports and gaining insight in how those monsters are built. It was time to tackle the other big players namely Firefox and xulrunner. At that time, we had www/mozilla-firefox shipping 3.0.x, www/firefox35 shipping 3.5.x (imported in another directory by martynas@ as a regularly updated work-in-progress with Java plugin support) and www/firefox36 shipping 3.6.x but not linked to the build. Quite a mess.

I read, hacked, fixed, and merged all of those. After some discussion and fixing some bugs there like this one on sparc64 (thanks to martin@netbsd for the hint), www/firefox36 was merged back into www/mozilla-firefox directory where the most recent stable Firefox version is supposed to be. It was committed at the end of April.

I also looked at devel/xulrunner/1.9. It was shipping 1.9.0.x versions corresponding to the to-be-deprecated Firefox 3.0.x branch. A few ports were using it (like www/gecko-mediaplayer), so it was time to update to a maintained branch. The new 1.9.1.x version was committed shortly thereafter.

Then, some cleanup went in such as making it use the system wide devel/nspr and security/nss libraries, instead of the bundled ones permitting us to remove lots of duplicate patches:

To continue the cleanup, one of my goals was to get rid of the old unmaintained xulrunner 1.8. I started converting ports use the new 1.9 version. Some were easy, like x11/gnome/yelp and x11/vlc done by David Coppa (dcoppa@). Others were harder like www/kazehakase and x11/gnome/py-gnome-extras. The conversion effort uncovered problems in the new 1.9 xulrunner port itself. We still have two ports (gtk-vnc-plugin and devel/eclipse) using the old xulrunner 1.8 branch but all others are now using the new 1.9 branch.

I went with Jasper Adriaanse (jasper@) to pkgsrccon and meet some porters from NetBSD and FreeBSD. Some good ideas were exchanged about our different infrastructures and workflows. While talking with tnn@netbsd who maintains all Mozilla ports for pkgsrc, he hinted at looking into how xulrunner could be built from the Firefox source tarball. It was a really good hint. Building xulrunner from Firefox gave us:

  • Synchronisation between firefox and xulrunner.
  • Share source tarball, one less 50mb+ tarball to mirror.
  • Share patches between ports, currently a real maintenance burden.

Quickly after pkgsrccon, xulrunner was updated to the 1.9.2.x branch, and all duplicated patches were removed (commit). Now we merged together all those crazy similar constructs across Mozilla ports, and put everything together into a MODULE (a nice feature provided by our infrastructure) to share common things. It only took me a few hours on a train to have something looking good enough to be commited as the new www/mozilla/mozilla.port.mk module.

I converted all the Mozilla ports to use the new module:

The size of the Makefiles were reduced by a huge factor, and now contain only relevant things. My mental health is also much better.

Of course, you try to track the new versions from upstream, so you eventually get goodies like the new Thunderbird 3.1 update bringing some nice user interface improvements. I have to admit, this update was easier than I thought due to all the previous Mozilla infrastructure improvements we had made. Time to relax.

While working on this update and looking at how tnn@netbsd did things in pkgsrc, I realized the Lightning extension could be built directly from the Thunderbird (and SeaMonkey) source tarball using the --enable-calendar switch. Building the extension directly from the host application tarball instead of building it from a slightly different source tree worked better. I had to move sub-packages around but it was easy since @pkgpath and @conflict markers in PLIST allows for seamless upgrades. The Sunbird port lost its two sub-packages (specifically) so it is now only the stand alone application, and the SeaMonkey port gained a Lightning sub-package.

Since we now have a sane infrastructure for Mozilla ports, what's next? A few items from my 'to do' list:

  • Fix some printing bugs. As of now, there are crashes when you print within Firefox or Thunderbird. It is yet to be found if CUPS is the guilty part (see this bug report,and this one).
  • Build Sunbird from the Thunderbird tarball. Upstream lacks manpower and won't do formal releases of Sunbird, but it also appears it can be built this way. That would again get rid of yet another 50mb+ tarball to mirror.
  • Firefox 4 is around the corner, already at beta2, but of course this needs an update to security/nss and devel/nspr.
  • Find someone to update our JDK allowing us to use the latest and greatest Java plugin from icedtea (as does pkgsrc). We could then get rid of www/firefox35, since it is only still around due to the Java plugin we have in our current Java port not working in Firefox 3.6 and later.
  • Now that infrastructure is sane, find a brave soul to take over maintenance of all those ports?

So here it is, I hope now you have a better global view of what our ports tree ships coming from Mozilla, and how it is built. :) Remember, tests and feedback when updates are posted to ports@ are ALWAYS welcome, but bug reports saying, "It crashed," without a back trace or debug symbols are totally worthless.

Landry

(Comments are closed)

Comments

  1. By Noryungi (noryungi) noryungi@yahoo.com on

    Landry,

    From a very happy OpenBSD user and administrator, I'd like to thank you for all the hard work that went into these updates!

    I am looking forward to using these updated versions pretty soon...

  2. By John D (JohnD) john.danks@gmail.com on

    Thank you for all the hard work on this.

    I would be very interested in testing a DAViCal port.

  3. By Timmy (mpty) on

    DAViCal link has extraneous 't' at this time.

    Comments

    1. By J.C. Roberts (jcr) on http://www.designtools.org

      > DAViCal link has extraneous 't' at this time.

      Yep, Fixed. Thanks. I'm not sure how that slipped past me when editing.

      And thanks to Landry for all his efforts on Mozilla ports!

  4. By Martin (martinp) on

    Really nice work, Landry!

    I can see there's a lot of patches in the Mozilla ports. Isn't it possible to get theses things fixed upstream, so it would build right out of the box? Or is the Mozilla stuff just "Linux software"?

    Another thing...it was announced last fall that the -stable ports tree would receive security updates and fixes, but this initiative seems to have stopped again?

    Comments

    1. By Matthias Kilian (kili) on

      > Another thing...it was announced last fall that the -stable ports tree would receive security updates and fixes, but this initiative seems to have stopped again?

      There have been 35 commits to the OPENBSD_4_7 branch of the ports tree, all made by william@, who takes care of stable ports. Please note that doing stable ports is not just updating some stuff on occasion (think about dependencies).

      Comments

      1. By Martin (martinp) on

        > > Another thing...it was announced last fall that the -stable ports tree would receive security updates and fixes, but this initiative seems to have stopped again?
        >
        > There have been 35 commits to the OPENBSD_4_7 branch of the ports tree, all made by william@, who takes care of stable ports. Please note that doing stable ports is not just updating some stuff on occasion (think about dependencies).

        Oops, you're right. Sorry for not noticing the great work from william@.

Latest Articles

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]