OpenBSD Journal

Binary Patching

Contributed by jose on from the easy-upgrades dept.

Matt writes: "With all the buzz about patching our systems for the latest OpenSSH bug, my first thought is that I now have to recompile the source on a test box, then mount the drive via nfs so I can patch my firewall box, which only has an 800mb hard drive and is ungodly slow.

Many deadly regulars will recognize homebrew script names like tepatche, easybakeoven88, binpatch, etc. My question is does the core development team still see these as untested, not useful, not needed, or most likely not a priority? Has there been any recent tallks to bring more advanced patching functionality into the core system in v3.5?"

Personally, I doubt we'll see binary patches coming soon from the official project. However, will these other, third-party upgrade systems make sufficient headway? Comments, anyone?

(Comments are closed)


Comments
  1. By Peter Hessler () spambox@theapt.org on http://www.theapt.org

    I've never had good luck with binary patching, however, I do think it would be a nice feature for the project to have.

    Which kind of binary patching is desired? The kind that would need `diff` and `patch` to work with binary files? Or are we talking about replacing binaries, and libraries, when needed?

    Comments
    1. By tedu () on

      you can make binary patches by uuencoding before and after binaries, and diffing that. then uuencode on the target machine, patch, and uudecode.

      Comments
      1. By mad () on

        But gzipped diff of uuencoded binaries (before and after) is bigger than gzipped tar archive with binaries!

        Here is example of recent OpenSSH 3.3-stable patches:

        openssh-3.6.1$ du
        625 ./usr/bin
        77 ./usr/share/man/cat1
        52 ./usr/share/man/cat5
        36 ./usr/share/man/cat8
        166 ./usr/share/man
        167 ./usr/share
        2 ./usr/libdata/ssh
        3 ./usr/libdata
        1220 ./usr
        1221 .

        openssh-3.7.1$ du
        641 ./usr/bin
        80 ./usr/share/man/cat1
        52 ./usr/share/man/cat5
        36 ./usr/share/man/cat8
        169 ./usr/share/man
        170 ./usr/share
        2 ./usr/libdata/ssh
        3 ./usr/libdata
        1239 ./usr
        1240 .

        $ diff -u 3_6_1.uuencoded 3_7_1.uuencoded | wc
        62028 62304 3900073
        $ diff 3_6_1.uuencoded 3_7_1.uuencoded | wc
        61730 123075 3922706

        $ diff -u 3_6_1.uuencoded 3_7_1.uuencoded | gzip -9 > diff-u.gz
        $ diff 3_6_1.uuencoded 3_7_1.uuencoded | gzip -9 > diff.gz

        Result:
        diff-u.gz (gzipped patch) = 1677636 bytes
        diff.gz (gzipped patch) = 1669087 bytes
        openssh-3.6.1.tar.gz (tar-gzipped binaries) = 509923 bytes
        openssh-3.7.1.tar.gz (tar-gzipped binaries) = 516547 bytes

        Comments
        1. By tedu () on

          3.6 -> 3.7 is more than a small patch. compare with just the patch.

  2. By Anonymous Coward () on

    Why not just create a binary package that replace the program ? This is the solution used in many Linux distribution.

    Comments
    1. By tedu () on

      patches are smaller and easier to disribute

      Comments
      1. By Anonymous Coward () on

        OK. How about an option to build binary packages using the already existing ports/packages system (with some modifications)? This way, you would still only have to distribute the source patch. Users responsible for many machines could easily generate their own packages. Plus having patches installed like a package would also allow one to sit down a machine you don't (always) maintain yourself, and instantly ascertain it's patch status.

        Yes, I know it isn't going to happen, but a guy's got to have a dream right? =)

        Comments
        1. By tedu () on

          google: "openbsd binary patches" :)

  3. By Anonymous Coward () on

    An automatic update facility would make it easier to sell OpenBSD in medium to large companies where system maintenance eventually becomes the responsibility of the operations staff.

    Microsoft has Windows Update, Debian has APT, Red Hat has up2date, etc. OpenBSD is close to having this because...

    Developers in the Debian project had ported most of the dpkg and apt suite to OpenBSD, but then then became distracted with porting glibc.

    (Why port glibc to OpenBSD? Isn't that missing the point?)

    OpenBSD could probably be deployed with APT when somebody creates Debian control files for the base system, and writes a conversion script for the way that the ports system describes dependencies.

    I investigated this, and it would probably be four weeks of work. Anybody that actually gets around to finishing the job would probably become famous and/or worshipped by administrators.

    Comments
    1. By Peter Hessler () spambox@theapt.org on http://www.theapt.org

      It's called "trivial shell script". I wrote one that cvs's the source, builds kernel/world, and installs everything. All you have to do is (1) start it (2) wait a few hours (you can still work in the background) (3) run mergemaster after it cvs'd (4) reboot whenever it is convienent (to activate new kernel.

      Comments
      1. By Anonymous Coward () on

        If the script is trivial, then why is something like it not part of the base system?

        Companies revolve around policy and procedure, and "automatic update" is now a feature on the purchasing checklist.

        Any person that doesn't cringe when they hear "all you have to do..." is probably competent and thus has more important things to be doing. Applying patches to an operating system is something that should be delegated to the highschool intern.

        I hear the begging on misc@ for CD-ROM purchases, but it is harder to make a business case for a software product without a 'standard' feature like automatic updates.

        Comments
        1. By Brett Jones () brett@5foot2.com on mailto:brett@5foot2.com

          "Applying patches to an operating system is something that should be delegated to the highschool intern."

          Yeah, trust the weakest link in the chain to do security/stablility updates, that's a great idea. They should do backups of critical data also, I mean shit, it's only backups.

      2. By Anonymous Coward () on

        I think some miss the point here. CVS is not considered too complicated by anyone, I think. It is just that it is tedious repetition of steps along with too long compile time.

        Take old hardware for example. Disk too small, CPU too slow, not necessarily x86, no source tree handy, now upgrade this one without leaving a remote root open for more than, lets say 10 minutes.

        Yes, I think signing patches and making them applicable in seconds is doable and of benefit, though not trivial if done right.

        Comments
        1. By Ben Johnson () on

          now upgrade this one without leaving a remote root open for more than, lets say 10 minutes. This one wasen't too hard - just use pf (or ipf) to block port 22 (ssh) and poke in a small hole for port 22 for the machine that your ssh'ing from. Then you have all the time in the world to fix the problem.

          Comments
          1. By Giacomo Cariello () jwk@bug.it on mailto:jwk@bug.it

            The keyword you're missing is "production".

        2. By krh () on

          > CVS is not considered too complicated by anyone, I think.

          *raises hand*

          I have to admit that I don't have the slightest clue of how to use CVS. I follow the recipes in the FAQ and sometimes it works. Other times it doesn't. Whether or not it works feels very random to me, almost as if I am using Windows. This is why I've given up on following -current and only use CVS if I feel like I have no other choice.

        3. By Anonymous Coward () on

          No, I think that also misses a big part of the problem. Most large companies I'm aware of use firewalls and http/ftp proxies to provide access to the internet. Any sensitive server will reside behind this firewall. CVS/cvsup is then not an option at this point.

          It really bit me this week with my FreeBSD and NetBSD servers...patches had not been released and source tree synchronization was required.

    2. By Anonymous Coward () on

      Patching OpenBSD isn't hard, it's not complex, but it is a royal pain in the ass to do on more than 1 box. An "up2date" would be the best (though, that wont happen without a commercial entity in the middle), and a downloadble "official" OpenBSD binary patch (i.e. Microsoft hotfix) would be good.

      What I'm getting at, the average person doesn't want to cvs or patch source, then compile. They want to install a new binary.

      The best solution? Use what the OpenBSD project already has:

      1) An AWESOME CVS, Ports, Packages system
      2) Dedicated Contributors

      With those, the best idea I can think of is to have OpenBSD project members (with CVS check-in access) write ports (and therefore packages) for updates.

      This is great for a few reasons:

      1) It is easy to apply (pkg_add file.tgz)
      2) It is easy to track (pkg_info)
      3) It works as expected (just as Microsoft has Add/Remove programs for listing/managing hotfixes, we would have pkg_add)

  4. By Anonymous Coward () on

    I'd be happy with a new make target, such as "make package" or something similar, in the base source tree. Something that worked similar to make install, except created a .tgz file that could then be used on the target machine like this:

    cd /
    tar pxfvz foo.tgz

    Does anyone know if there is a (relatively) simple way to do this now?

    Comments
    1. By Anonymous Coward () on

      Just to be clear, I'll put it in some context - the recent ssh bug:

      cd /usr/src
      patch -p0 <017_sshbuffer.patch
      cd usr.bin/ssh
      make obj
      make cleandir
      make depend
      make && make install

      Followed by:

      make package

      Which would then create a tarball (perhaps in /usr/src/packages or something) called ssh.tgz, which I could copy to another machine and type:

      cd /
      tar pxfvz /path/to/ssh.tgz


      Comments
      1. By Stefan () stefanjo@telia.com on mailto:stefanjo@telia.com

        Maybe like chekinstall? (http://checkinstall.izto.org/)

        I dont know if this works "out of the box" (probably not) but I dont think it would be to hard to adapt.

  5. By djm () on

    This is really, really easy to do.

    Just make a release "man release" and copy the resultant tarballs to the hosts that you wish to upgrade. Once there, unpack them from the root with "tar zxvpf /path/xxxyy.tgz". Building a release will also build a GENERIC kernel which you can copy out too.

    Comments
    1. By James F. Wilkus () tflat+deadly@astrocreep.net on mailto:tflat+deadly@astrocreep.net

      I agree. I even took it a step further. I have a "gold server" where I build releases. Once the "make release" process is complete, I rsync the new install sets to our ftp server. All openbsd servers that I manage have a perl script installed that I wrote that then query /etc/obsd_release, which contains the list of install sets that were used to build the current system i.e. base33.tgz, etc33.tgz, bsd. The perl script fetches all the install sets relevant to the current system, installs, then runs mergemaster to ensure /etc and /var are up to date. I have been doing this for over a year for at least 10 openbsd boxes and it took me ~10 minutes to get them all up to date. I just need a faster "gold server" cause it takes quite a while to build the new install sets. :)

      Comments
      1. By Michiel van Baak () michiel@vanbaak.info on mailto:michiel@vanbaak.info

        I like the make release too.
        Now that I read about your little nifty perl script, is it possible I can have a copy ?
        I'm not that good in scripting so I cannot make my own.

        Thnx in advance

  6. By RC () on

    Hmm, I wounder if people know that snapshots exist?

    That's right, current, updated, binaries. All you'd really need to do is upgrade to the snapshot version.

    If building from source is really that big of a burden, as people continue to claim, you always have that binary-only option.

    Comments
    1. By Anonymous Coward () on

      Sure, on production machines. Don't worry about the downtime etc. What are you smoking?

      Comments
      1. By RC () on

        No downtime required.

        Comments
        1. By Anonymous Coward () on

          The snapshots are off off CURRENT, not STABLE. So it's very possible to end up with some downtime.

          Comments
          1. By RC () on

            Yes, I realize they are off -CURRENT

            What you are saying is that you never test software before you install it on a production system?

            Personally, I was just pointing out that, those that need utmost stablity aren't going to have a problem compiling their OS, while those that consider it to much of a hassle, can just use the snapshots.

            Even if you had binary-snapshots from OpenBSD, you're likely going to end-up causing some downtime with that attitude anyhow (not testing anything).

        2. By Anonymous Coward () on

          Then you don't know what you're talking about, sorry.

  7. By Colin Percival () cperciva@daemonology.net on http://www.daemonology.net

    At BSDCon last week, I presented a paper about FreeBSD Update. FreeBSD Update is a tool which automatically determines which binaries are affected by a given source code patch, generates an update index, signs it, and then (when the update client is run on each machine) determines which files need to be updated and updates them. Using FreeBSD Update, the latest openssh and sendmail fixes were a two-minute job requiring about 100kB of bandwidth:
    1. `freebsd-update fetch`
    2. [optional: look at the list of files for which updates were downloaded]
    3. `freebsd-update install`
    4. restart sshd.

    This is much better than any of the existing update tools for OpenBSD; it's also more secure, since it uses 2048-bit RSA to sign the updates. (CVSup, by contrast, is completely insecure.)

    The only requirements for FreeBSD Update to be used is that the system must have started with a binary install of the official release image, and the base system must not have been recompiled. (Any files which are recompiled might not be updated -- in particular, if you recompile the kernel, FreeBSD Update will thereafter not update the kernel.)

    Nobody at BSDCon seemed interested in porting FreeBSD Update to OpenBSD (I expect this to be a one-day job -- there's really nothing FreeBSD-specific in FreeBSD Update). Maybe someone here will be interested?

    The FreeBSD Update web site is http://www.daemonology.net/freebsd-update/

    Comments
    1. By Anonymous Coward () on

      Sounds great. I have time so I'll look at it and consider the port.

    2. By Anonymous Coward () on

      There is also APT on FreeBSD (from the debian project), in ports/emulators/linux_base-debian. Maybe we could use it to upgrade binaries on OpenBSD.

    3. By Anonymous Coward () on

      That sounds really great.
      But the main problem remains: who'll trust non openbsd patch ?
      This solution will be viable only if patch are signed by some openbsd's staff trusted/agreed key.

      Comments
      1. By Colin Percival () cperciva@daemonology.net on http://www.daemonology.net

        Absolutely; someone associated with OpenBSD should be building and signing the updates. But FreeBSD Update is the right tool to use for that purpose.

  8. By Anonymous Coward () on

    binary or source, I don't care which. I need some kind of solution to update OpenSSH on my openbsd2.6 box. I am unable to update the box itself, but if it is vulnerable I need to get openssh patched on it.

    I tried compiling the source straight and I get a compile error, I tried compiling with the pre-3.3 patch and I still get a compile error.

    Is there anything I can do to get openssh protected on openbsd2.6?

    Comments
    1. By Jeff Flowers () jeff@goo.cc on mailto:jeff@goo.cc

      I don't know the answer to your question but I would ask it on either a OpenBSD mailing list or the OpenSSH mailing list (after doing a search of course).

    2. By Johan () on

      From openssh.org: "Note: This above source fails to build on earlier OpenBSD, because IPV6 support in libc was incomplete."

      Where "This source" is openssh 3.7.1. With "earlier OpenBSD", I assume they mean pre-3.x.

    3. By djm () on

      Lots of people use portable OpenSSH on older OpenBSD platforms. Old OpenBSD isn't half a broken or lacking in support for various things as many of the other Unices that we support :)

    4. By blahblahblahblahblahblahblahblahblahblahblahblahbl () on

      install the portable version, you have to add a single include somewhere. search the archives.

  9. By Anonymous Coward () on

    I really miss a simple way to update OpenBSD with necessary patches.
    E.g.: It's quite simple to run OpenBSD from a CF card, but neither gcc nor /usr/src will fit onto this, and I don't want to run another OpenBSD box just for the purpose of compiling stuff.
    A mechanism like APT or even RPM would be really helpful.
    Also, virtual servers (very popular in DE), are way to slow to make fast updates.

    Comments
    1. By Thomas W. Horna () on

      I am in process of learning how to sysadmin and to use unix, expecially openbsd and freebsd, but i never saw any 'un-simplicity' in normal way of patching, i did this so: i cvs-ed www.openbsd.org,
      patched 2-3 times using faq, and now i know that thingies by heart. Where is the Problem at all?
      I love to retain freedom of compiling my binaries by myself.

  10. By chuck () on

    There are projects in various states of progress for binary patching of openbsd (many of them have already been mentioned on deadly.org).

    what I think most people would like to see is the core openbsd group embrace one of these projects and "officialize" it. of all OS fans, I believe that openbsd users would be the hardest to convince to grab a binary patch from any old joe as opposed to compile it.

    That said, straight up source patches definitely have their strong points. Example: I use saslauth, and need to compile support for that into sendmail. Using Source RPMS, i would have no idea where to begin. On the downside, I do agree with a lot of people that I dont normally have the HD space for all the source.

    Now that I think about it.....
    Perhaps instructions included with the patch stating how to grab the portions of source from CVS that you need to compile would work best?

    Comments
    1. By Anonymous Coward () on

      That's a pretty good idea. Actually I get from CVS only the part of code to compile an with only 100 Mbs is enogh.

      But is not easy... sendmail took me some time to realize how much source tree was needed to compile.

      After that anyone can make bin updates if he/she/it needs to update more than one system.

      Comments
      1. By chuck () on

        I am by no means a CVS master (i've used other source tracking software alot, but for cvs it tends to be a one-way deal).

        CVS has got to support labels, right?

        So, with some volunteers, it might be able to come up with the correct CVS labels of different programs on Open. Perhaps do it the "evolutionary" openbsd way, where when a patch is given, thats when the label is created for that software, preventing a massive amount of work requiring to be done all at once.

        Or maybe I'm crazy.
        What I am imaging is something like:
        cvs get gnu/usr.sbin/sendmail but not quite like that, instead using labels.
        maybe....
        cvs get -rOPENBSD_3_3_sendmail

        please tell me, am I off my rocker?

  11. By Mark Beihoffer () mark att dragonfly-7 daught com on http://www.dragonfly-networks.com

    I am a big fan of apt, and Debian is one of the only Linux projects that I admire. I don't know if apt could be adapted to OpenBSD, but something must be done to make remote administration of large quantities of machines more feasible.

    I don't know which type of system would work best, although others have made good points about disk space for source compilation (especially for Flash devices).

    One thought that did occur to me as a Perl fan would perhaps to adapt the CPAN model, and have a shell that would allow you to search for patches and apply them interactively?

    That for me would be easier and more comprehensible than some of the other suggestions. CVS is certainly an option, but it would appear to be a somewhat bloated, time/cpu/disk intensive one at best.

    Comments
    1. By Anonymous Coward () on

      There is an APT port for netbsd/freebsd from the Debian GNU/BSD project. It will not be an hard job to create an OpenBSD one.

  12. By Anonymous Coward () on

    Wow, everyone - even there on deadly were trolls aren't rare - seems to be agree with the necessity of a binary update system for openbsd. Even ideas about how to proceed seems to converge.

    For the first time, we see a divergence beetween the choices in the dev community and popular demand of the users community.

    Will they hear us ?

    Comments
    1. By Martin Reindl () wildweasel@bsdcow.net on mailto:wildweasel@bsdcow.net

      I'm sure, if you pay some dev to do it and it fits the project goals, it will be implemented. After all, most of them code and research during their free time, for fun, and because they want to see the project move forward.

      Of course this doesn't mean input from users is useless or unwanted, but it will happen more likely if you provide code to start with that fits OpenBSD instead of crying out loud "I want that!"

      Regards,
      Martin

      Comments
      1. By ben () mlben@zouh.org on mailto:mlben@zouh.org

        Hello

        User community has already done the necessary code (see other posts, particularly the one about freebsd-update).

        The only need at that point is to be aknowledged by the open staff (ie: could be just a cert key signed by openbsd CA, to let us sign patches ...).

        No projects could envolve without this because users won't install untrusted third party binaries.

        Comments
        1. By Colin Percival () cperciva@daemonology.net on http://www.daemonology.net

          You'd be surprised what people will install. FreeBSD Update has been used by hundreds and hundreds of people, even though I had (until now) no recognition in the FreeBSD community and no official sanction from the project.

          Ok, maybe people running OpenBSD are more paranoid than those using FreeBSD; but I think you'd still see quite a few people taking advantage of binary updates a la FreeBSD Update if they were provided for OpenBSD, regardless of who provides them.

          Comments
          1. By ben () mlben@zouh.org on mailto:mlben@zouh.org

            You'de convinced me.
            It could become, at least, a group of people who'll endorse responsability for bin patches.

            I can make i386 and sparc updates.
            Also will see what can be done with your code under openbsd.
            Anyone to associate (especially with alpha, vax ...) ?

            my email is provided.

        2. By Martin Reindl () wildweasel@bsdcow.net on mailto:wildweasel@bsdcow.net

          I know users already provided code, but just not code that really *fits* OpenBSD. I am also aware of the work the guys from openbsd.org.mx do with binpatch (http://www.openbsd.org.mx/en/projects/binpatch.html). Maybe some form of freebsd-update or binpatch or whatever makes it into openbsd one day.

          Until then i will continue to follow the good old way of upgrading and compiling on my boxen with standard Unix tools that OpenBSD provide and support. I don't feel there is something wrong with it, but maybe that's just me :)

          Comments
          1. By Anonymous Coward () on

            I agree. IMO OpenBSD doesn't need a fancy 'one-command updates my whole system' piece of software. One should implement his/her own way of updating OpenBSD boxes, according to the situation and infrastructure.

            'make release' is a good start, one other solution I've been using is making a list of files from the output of 'tar tvzf base.tgz' that I then pass to tar as a list of files to include. On one machine (the build machine) I 'make build', then tar all the files that were initially in base.tgz, scp/ftp it over to the other boxes, bring them down to single user if necessary and then just untar the base.tgz in /. One can do the same for other sets.
            Updating /bsd is even easier. One of the machines I update this way has a 480MB HD (base.tgz && etc.tgz && bsd), so I've been forced to find a solution with a dedicated build machine.

          2. By ben () on


            I'm mostly agree with you: the better, the one - if there must be only one - way of upgrading is the plain old way (patch|cvs, make etc). Nothing "wrong" with this, indeed ;)

            Furthermore, I think a tool to upgrade continuously all the binaries could deserve openbsd (see the dselect dependency nightmare, and all linux "bazaar" ;).

            The need is different: only providing binary updates for security concerns, and admin will patch they're net fastly (or at least, they will do it;).

            See for instance: I've a bunch of soekris VPN gateways to maintain with differents versions of openbsd (w/ compact flash drives). Also, we've a politic of "the less is the best" for our firewalls (implies no gcc or source trees). Patching sendmail then openssh (then openssh again) took me 3 plain days. I did idle all my current tasks (was hard to convince my managers ...). Not good.
            I know a lot of admins around there that didn't upgrade (or that waited too long) for similar reasons: lazyness, lack of time ... you may think it's they're concern, but I don't: a trojaned box is a problem for all neihgbours.

            See the poing ? binay updates for security problems is a real plus for large networks, for semi-emmebeded systems and for lazy (or busy) admins.

            Give us reactivity to honour dev's proactivity ;)

  13. By Randy Poznan () randy@hackuser.com on mailto:randy@hackuser.com

    I dont think binary patching will be efficient or worthwhile. These days since size is less of a problem whats wrong with a MD5 compare of binary programs? Of course this is only of use to patch updates.

    After cvs update. A make file will only build whats new in the src. Compare program could md5 compare (apps, libs, etc) and output a db of new binaries. Patch build program would read the db and create archive of new files, log changes, sign it, and name it with the date.

    I dont exactly know if this project does unit testing? Binary patches probably would need testing to build user trust. A automated suite of tests for every application or library native to the distro. This could execute tests on programs based on old bugs or defined functionality via a single command.

    Comments
    1. By Matt () on

      A makefile will only compile 'the new stuff' if you've compiled the source once before. What happens if you're setting up a virgin box that never compiled anything? you either have to compile evverything, or use your home brewed release script to make an up to date set of tarballs.

      Comments
      1. By Randy Poznan () randy@hackuser.com on mailto:randy@hackuser.com

        This system would only work if run from a specific dev box. This box would be built and kept up to date via cvs on a regular basis. As long as you dont make clean, only new src files are compiled and linked during the make process. The compare program and patch builder would run on it. The output would be new patches in a specific format. In addition I think a test server that can run a suite of tests on new patches and their install process would be a necessity.

        New systems and deployed systems could install patches over the base install.

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]