OpenBSD Journal

Marc Espie talks about ports internals

Contributed by mk/reverse on from the interrogation dept.

Marc Espie is an interesting sort of chap living in Paris, France. When tired, he watches a lot of DVDs. He enjoys juggling and riding through Paris on rollerblades.

Sometimes he hacks on the OpenBSD operating system. We tricked him into talking about time machines - and the internals of OpenBSD ports.

(Comments are closed)

Comments

  1. By uriel (82.182.149.44) lost.goblin@gmail.com on

    When is he rewriting it in sh/awk so we can remove Perl from the base system?

    Comments

    1. By Nate (65.95.127.113) on

      He probably wouldn't, Marc seems anti-shell script.

      The better question is when are you going to submit the patches?

      Comments

      1. By Anonymous Coward (85.224.154.90) on

        If he doesn't like shell scripts maybe he should not be using Unix.

        And I will send the patches when rc becomes the default OpenBSD shell ;)

        Now seriously, to be honest I think the current ports system is over designed and over engineered; something much simpler is enough for me, all I want is to see Perl go, FreeBSD got rid of it a while ago, and it would be sad to see OpenBSD falling behind.

        Comments

        1. By Marc Espie (62.212.102.210) espie@openbsd.org on

          If you are to write trolls, you should at least try to write believable ones. Not liking shell-scripts ? Hah! I'm actually one of the developers in OpenBSD that hacks on shell-scripts. What do you think bsd.port.mk is anyways ? What do you think is the underlying language of Makefiles ? I don't often go around putting my weight in arguments, but if you have a simple look at the CVS commits in OpenBSD, you'll find hundreds of commits I did in shell-scripts. I'll even point you at the old cpp script, or lorder, or /etc/rc to give you an idea about the kind of stuff I hack. Heck, I also simplified the memory system of ksh so that it could cope with libtool. And I did write the update-patches script from scratch, and about ALL the shells scripts you'll find under /usr/ports/infrastructure if you care to look. That said, the shell has some limitations. Not being structured is one of them. Not coping well with weird names is another. And once you start needing tools like awk, it's simpler to bite the bullet and go perl. About the ports system being over-engineered, what's funny is that bsd.port.mk in OpenBSD is roughly 2500 lines long, whereas the corresponding files in both NetBSD and FreeBSD are over 5000 lines these days. And the pkg_tools they have are growing each day, with more tweaks, and knobs, and checks, and new commands. Based on really shaky fundations, because the underlying C tools are downright buggy. Like, they don't handle things right, so they add a layer on TOP of that to try to fix the issues. Well, you've got one thing right: you don't like perl, and I do.

          Comments

          1. By Anonymous Coward (66.179.38.239) on

            Don't let the trolls get to you, man. You (and all the other hard-working devs out there) rock! :)

          2. By Chris (144.178.103.79) chriswareham@chriswareham.demon.co.uk on

            About the ports system being over-engineered, what's funny is that bsd.port.mk in OpenBSD is roughly 2500 lines long, whereas the corresponding files in both NetBSD and FreeBSD are over 5000 lines these days. And the pkg_tools they have are growing each day, with more tweaks, and knobs, and checks, and new commands. Based on really shaky fundations, because the underlying C tools are downright buggy.

            Is it fair to answer a troll with another (partial) troll? The reason the NetBSD mk files are much larger than the OpenBSD ones is mostly due to the excellent cross platform support of pkgsrc. This includes support for OpenBSD, and judging by the reasonable number of OpenBSD related posts on the pkgsrc mailing list there are people who use pkgsrc in preference to OpenBSD's ports. I'd be shocked if there weren't some warts in the pkgsrc code, but your assertion that it's "based on shaky foundations" doesn't fit with my experience, which is that the tools are becoming increasingly sophisticated as new features are added - not because of the addition of dubious workarounds.

            Chris

            Comments

            1. By Marc Espie (62.212.102.210) espie@openbsd.org on

              Yes and no.
              There is some useful stuff in NetBSD, but there is also a lot of cruft we did streamline in OpenBSD.

              One of my regular activities involves monitoring NetBSD and FreeBSD to see what I should grab from their infrastructure.

              As for people using pkgsrc on OpenBSD, I'm pretty certain those are the people with cross-platforms needs who don't want to have differing behavior when they change systems.

              To give you instances of stuff we removed: NetBSD always defines
              pre-*/do-*/post-* targets, whereas we only use these if they exist.
              This shaves about 10 targets from each Makefile.

              I've also finished sorting through variables and targets, so that all targets now appear after all variables have been defined.

              There's also the very useful trick of storing shell script fragments in variables for specializing stuff that's becoming pervasive in OpenBSD.

              We have removed stuff. I can see it still in NetBSD. In all cases, it was stuff we could do without (and that we no longer use).

              Comments

              1. By Chris (144.178.103.79) chriswareham@chriswareham.demon.co.uk on

                Hi Marc. I use pkgsrc on NetBSD and Solaris, so your comment about people using pkgsrc across various platforms to keep the behaviour the same rings true. It's good to see from your other comments that there's cross-pollination of code between the various BSD package and ports frameworks. I also noticed your recent post to the NetBSD pkgsrc list, which will hopefully lead to further cross-pollination of ideas and code.

                Chris

          3. By uriel (82.182.149.44) on

            It was not me who said that you were "anti-shell script". I commented that _if_ that was the case, then Unix might not be the right OS for you.

            What I will say is that if you like Perl, you probably don't understand Unix. Specially if you think that is shell scripts are "not structured". Structuring shell scripts in small programs that do one thing well and work well together is one of the most fundamental Unix principles.

            I recommend that you read: The Unix Programming Environment by Brian Kernighan and Rob Pike and The AWK Programming Language by Aho, Kernighan, and Weinberger,

            Perl and Perl programmers that didn't understand the Unix philosophy were one of the main causes of the death of Unix.

            Comments

            1. By Marc Espie (62.212.102.210) espie@openbsd.org on

              Thanks, uriel. You're really the funniest little troll there is on
              undeadly.

              Your messages are so ludicrous they make me laugh.

              Tell me to learn Unix and shell programming ? hehehe. BWAHAHAHA.

        2. By Michael Knudsen (217.157.199.114) on

          It's not exactly like removing Perl from FreeBSD base has made their life any easier.

          Comments

          1. By Anonymous Coward (69.197.92.181) on

            Hasn't it? It allows people to use whatever version of perl they want cleanly and easily just like they can with any other piece of software. Not everyone wants their OS bundled with apache and bind and perl every other random thing that belongs in ports.

            Comments

            1. By Michael Knudsen (217.157.199.114) on

              It hasn't made it easier for the developers who now have to maintain two versions of perl along with extra complexity to handle it.

              Regarding users then I've only heard complaints about having issues with different versions being installed. While this may be issues that stem from uneducated users, having no choice would prevent this. Also, now users have to keep track of which version of perl they've installed.

              This is why I say their move to remove perl from base hasn't made their lives easier.

              Comments

              1. By Anonymous Coward (69.197.92.181) on

                What about "devs" maintaining two versions is so bad exactly? How is that any different from offering the choice of apache 1 and apache 2? It lets users run the software they want, instead of the software the developers want.

                Comments

                1. By tmclaugh (24.60.174.16) on

                  Port Maintainer 1 maintains foo and tests it against Perl 5.8. Port maintainer 2 maintains bar and tests it against 5.6. As a user I use foo and bar with Perl 5.8. When an update comes down the CVS pipe at least one of the ports I use is untested against my Perl version and I get to be the guinea pig. Will one package break? I don't know but I'll find out. Forcing one Perl version forces users and developers to be on the same page. If you like a Perl version _that_ much then pull the last version of the port that you like, install it, and use it at your own risk.

                  Comments

                  1. By Anonymous Coward (69.197.92.181) on

                    Are you kidding me? Neither breaks, install both versions of perl and have a nice day. Do you see python in the openbsd ports tree right now? You are trying to make a problem that doesn't exist just to put down freebsd because they don't agree with your "include every stupid thing on the planet in the base system" attitude.

                    Comments

                    1. By Anonymous Coward (69.243.46.8) on

                      Submit a port.

                      I'd rather the devs work on something useful instead of reinventing the perl applications already on the system.

                    2. By tmclaugh (192.216.27.32) on http://www.mono-project.com/Mono:FreeBSD

                      I'm a FreeBSD port maintainer. I believe in making sure that my ports work. I could maintain two Mono ports, one for each line. I don't because it doubles my testing and I really don't have time for that. It's not about "user X wants this version but user Y preffers this version". I don't care. What I do care about is that user X and user Y have working and tested ports at all times. If someone is that anal about a version, they can pull an older port and use that but they're on their own from there.

                      Having two installed versions of Perl also tends to be a major reason for user fuckups.

                2. By henning (80.86.183.227) on

                  you imply choice is good.
                  the opposite is true, choice is bad.

                3. By Marc Espie (163.5.254.20) espie@openbsd.org on

                  People have a limited amount of time.

                  Your reply implies that a maintainer caring for two versions of the same port will spend twice the time maintaining the two versions.

                  This is usually not true. It most often means the maintainer will split the same amount of time between the two ports. This is critical for testing, and also for bug-reports. Especially stuff that is hard to reproduce because it depends on versions used.

                  There's also some combinatorial complexity involved. If there are several programs involved, then each version choice *multiplies* the number of possible configurations, and the number of factors that may trigger bugs.

                  There's a large number of system integration bugs where diagnosing the exact problem is the hardest issue. Once you've found that this issue occurs in package foo, because you installed bar, and because there's a tiny little issue in frob that only shows up if you install version 2 of frob.

                  Some of these issues do not get fixed, but filed instead as `not a bug' because they can't get reproduced.

                  All other things being equal (number of people working on a project and time they can spend on that project), more choices means less quality.

            2. By Anonymous Coward (82.155.157.74) on

              If you don't want all these tools choose another OS and go away, duhh!

              Comments

              1. By m0rf (68.104.57.241) on

                he also doesn't want citrus or acpi support, but he hasn't gone away yet.

                still hanging on for openbsd to become plan9 i guess.

        3. By Anonymous Coward (67.34.129.203) on

          Perl has been around so long, and it's so useful, that it's now an essential part of Unix, just as much as /bin/sh, sed, awk and friends. And for that matter, I'm willing to bet there's lots of folks who would feel lost without it on Win32 boxes (even though it's not as powerful on that platform, but it does what it can with what's available...)

        4. By henning (80.86.183.129) on

          all I want is to see Perl go, FreeBSD got rid of it a while ago, and it would be sad to see OpenBSD falling behind.

          falling behind? I call keeping perl staying ahead.

        5. By Gary (221.126.242.205) on

          all I want is to see Perl go

          who cares what you want anyway??

    2. By Brad (216.138.195.228) brad at comstyle dot com on

      Perl will not be removed from the base system.

      Comments

      1. By xhrl/thomasw (24.80.39.250) on

        a smashing interview. what marc stated about the 'cold reality' of coding pkg_add is so true: simplicity in design and execution is a virtue in many tools. i enjoyed marc's thoughful responses, and the care taken to ask him thought provoking questions in the first place.

      2. By Anonymous Coward (131.202.10.5) on

        I am curious what you fellas think about how Perl's "artistic" license (man perlartistic) fits in to OBSD's license philosophy. Obviously, it is considered acceptable ... I'm not sure to what extent. It sounds somewhere between BSD and GPL to me (but that is just my take on it). It isn't mentioned on http://www.openbsd.org/policy.html explicitly. Perhaps you guys will decide to do that, since it is being (in a fashion) more deeply integrated with the base system.

        Comments

        1. By tedu (64.173.147.27) on

          perl lives in the gnu part of the tree.

        2. By Marc Espie (62.212.102.210) espie@openbsd.org on

          Perl lives in gnu because gnu denotes `everything non bsd' and cvs sucks enough that no-one would seriously consider renaming that.

          That said, the artistic licence is much less obnoxious as the GPL, from a BSD point of view. It's not quite BSD, in particular, it's a bit too long for its own good.

          But if I read it right, it says that you can do whatever you want with the software, as long as you don't attempt to make lots of money from the software itself. You can more or less sell anything you ADD for an outrageous sum, be it support, additional routines, or scripts, and you're not obligated to distribute any kind of source. The only thing this more or less forbids is to distribute extended perl binaries without also providing a basic version, and distributing mutant versions of perl without renaming it.

          Pretty free if you ask me...

    3. By Lars Hansson (203.65.245.7) lars@unet.net.ph on

      The question is not when he will do it but when *you* will do it.

    4. By pdemb (217.98.20.20) on

      AFAIK sh and awk are not intended to be used for making package management systems. IMO Perl is one of the best languages for that task, the other one is Python :>

    5. By lis (193.194.84.198) iulian@create.ro on

      nice interview

  2. By Chris (144.178.103.79) single.white.male@gmail.com on

    He enjoys juggling and riding through Paris on rollerblades.

    What, at the same time?!?

    Comments

    1. By ViPER (212.123.197.34) viper@dmrt.net on http://www.dmrt.net

      Now thats multi tasking.

  3. By Jim (63.225.96.149) on

    I just want to say that the best thing aout it is it works. For a long time I thought you updated your system, downloaded ports.tar.gz, unzipped it and went through using 'make install'. The dependencies and all just got handled behine the scenes.

    Then, to my surprise, I read somewhere that pkg_add was the way to go, and wow! was that ever easier! But it is only because of unsung heros quietly working away, making it work and building the packages.

    A great interview, giving well deserved recognition.

    My thanks to you Sir Espie!

Latest Articles

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]