OpenBSD Journal

Efforts to maintain OpenBSD 3.0

Contributed by jose on from the oldland-security dept.

Arrigo Triulzi writes:
"We've had this discussion over and over again and the camp splits quite evenly into:
  • Your problem: upgrade,
  • Just keep going and patch as things come out.
This doesn't just hold for OpenBSD 3.0 but all the previous versions.

I've started following two separate techniques: the first, and obvious, is to install over OpenBSD native versions whenever something "necessary" appears (for example: apache, openssl, sendmail...) but the new method I have been following is to apply OpenBSD 3.1 patches where possible.

I thought this was never going to work but, for example, the recent LPD patch works just fine because the code is from 1999. So, with a grain of salt you can happily patch your OpenBSD 3.0 source tree with OpenBSD 3.1 patches. Another patch which worked was the openssl patch - I had tried compiling the latest OpenSSL from source and was unable to reproduce the exact setup (including dynamic libraries) of the OpenBSD source tree.

Would it make sense to start having an archive of patches which apply clean to previous versions of OpenBSD and fix vulnerabilities? This need not be an "official" page but it would definitely help."

This is not uncommon to find situations where upgrading quite yet is an option, and it gets put off almost indefinitely. While the newest software always does have the known fixes in it, it may also introduce more complications than are acceptable.

Its sometimes not possible to just apply the new code to an old tree, as APIs change (as has happened in OpenSSL, for example), breaking software. However, with some effort, new patches can be rolled for the older versions which preserve the compatability while fixing known issues. The real question is are any of you doing this?

(Comments are closed)

Comments

  1. By Anonymous Coward () on

    That's exactly my problem, I considered using OpenBSD for a shell server (50,000+ users), but then I've decided to go with a commercial unix, since it's a configuration almost impossible to upgrade without reinstalling the whole thing, downtime is not acceptable, my client will never allow that.

    Comments

    1. By Anonymous Coward () on

      So, untarring a few tarballs and rebooting once was too much for your? :|

      Comments

      1. By Anonymous Coward () on

        reboot hurts
        new custom kernel takes just another reboot and some broken features...
        always nice to have few boxes for important purpose

        Comments

        1. By pravus () on

          build your custom kernel on another "build" machine and copy it over before you reboot. one reboot, minimal down-time.

          if you can't even have down-time for maintenance, you really should re-think your strategy for resource allocation.

      2. By Anonymous Coward () on

        If you think you can just reboot a production server just like that, you have absolutely no sysadmin experience in the big corporate world.

        And if you think one can just replicate a configuration that is spread on 50+ computers to make it work as it should, then you may have unlimited funds or something.

    2. By gwolf () gwolf@gwolf.cx on http://www.gwolf.cx

      Use Debian in any server that needs to be 24/7.
      I have upgraded many machines from 2.2 to 3.0, 2.2 to unstable, 3.0 to unstable, etc. - Never even had to reboot for this. And if Linux is not your stuff, you can install Debian/NetBSD, although it is still quite experimental, and help the project :)

      Comments

      2. By tedu () on

        what's debian's trick to upgrade the kernel without rebooting?

        Comments

        1. By Anonymous Coward () on

          I'm curious as well.

        2. By Anonymous Coward () on

          Debian doesn't upgrade the kernel symlinks by default. You have to do it yourself.

          You can't upgrade the kernel without rebooting.

          Apparently he is running an ancient and probably highly vulnerable Linux kernel.

      3. By djm () on

        Debian may be a nice OS, but the users really need to drop their "holier than thou" attitude. Having a few smart packaging tools doesn't nearly make up for a clueful admin and a consistent OS.

        Unless you run Debian Unstable, you are likely using 2+ year old software (no nice 6-monthly release cycle). The usual response of Debian advocates to this is "use unstable, it is really stable", but so is OpenBSD -current for me.

        Their security integration isn't all it is cracked up to be either: http://lwn.net/Articles/22624/

        You can also say goodbye to all the other wonderful things that OpenBSD's release model brings you: whole-of-tree audits, patches via CVS, the ability to easily rebuild the entire OS from source, etc.

    3. By elmore () on www.screamingelectron.org

      I can understand your position, while I've not offerred a shell service to 50,000+ users, my company runs the majority of its servers and services on OpenBSD. I spoke to my supervisor about getting backup servers or computers. While this is a significant investment it has paid off in the long run, and I have found it rather easy to justify.

      So what I typically do is track stable, 1 release behind on the production box and track stable for the current release on the backup computer. When end of life happens for the production box I merely turn it off and the other picks up for it. No downtime, no one ever even knows.

      Obviously in your situation people would know but, you could cut downtime however to an absolute minimum, no more than a minute or two this way.

      The only problem I see with this is if data is stored on the OBSD box itself I.E. peoples home's. In my environment homes and all data are mounted via NFS I don't really have a problem with that. Anyways just an idea. Maybe it'll help you out.

      Comments

      1. By Matt () on

        Not to be a smartass, but how exactly do you just turn it off and have the other one pick up for it? I'd like to know what you did to accomplish this feat and try it myself on some test boxen.

        I would think you need to do at least a few things like syncing user accounts, change ip, change hostname... what else would be required to make a drop-in replacement?

        Comments

        1. By Bas () beetjebrak78@hotmail.com on mailto:beetjebrak78@hotmail.com

          Just a wild idea: mount anything variant (like usr homes, things like that) remotely and sync /etc and /usr/local/etc before the switch. You should know what you're doing, and know what config files you need to transport to the backup computer. But generally /etc doesn't change much after it's been set up (mine doesnt). Just before you sync, you could disable access to any commands the user could use to change their password. That's not actual downtime, however with 50,000+ users I'd hope you have some DB backend or LDAP for authentication.

        2. By elmore () on

          The boxes I do this on are on my local LAN, I accomplish this with my own local DNS servers. I'd be more than happy to write up a how-to if you're really interested.

  2. By cycloon () cycoon at is-root dot org on mailto:cycoon at is-root dot org

    Wow. That was exactly the problem i thought about the last few weeks. Our Router/Firewall at home was a openbsd 3.0 and I worried about updating to 3.2 stable.
    After i read about the new altq-merges and pf I additionally thought about updating to -current. The first thing i tried was cvs, but i got a lot of problems there (most standing in the update-mini-faq).
    So last week i decided to upgrade with those binary snapshots. I did this like described in the faq. First, simply unpacking the base33.tgz, comp33.tgz and so on. Afer that I rebooted and of course got some config problems (since i didnt upgrade /etc). So i updated those things by hand (mainly merging nat.conf in pf.conf and updating things like ppp.linkup). Another reboot and everything worked, with some errors, but it worked.
    So now I decided to update /etc. Like described I unpacked everything to ~/new_etc and manually looked over the changes. most configs weren't changed by me so i could simply copy them over to /etc. Ater that another reboot and everything was fine.
    Great:
    OpenBSD 3.3 (GENERIC) #19: Thu Mar 6 18:43:26 MST 2003

    Conclusion: dont worry about updates at all.

    Comments

    1. By Anonymous Coward () on

      The OS itself is quite easy to update but then you come to the packages. It is usually much more pain to upgrade those. I just upgraded my system from 3.2 to the 3.3-beta and it took half a day. But then of course I compiled all the packages myself, could probably have saved a couple of hours by downloading the precompiled ones.

    2. By Anonymous Coward () on

      For a home user, sure. I update my home -current boxes a couple times a week, no big deal. In a business environment things are different. I have a couple of 2.9 machines that I simply can not upgrade. Any downtime is unacceptable. I can patch and restart services (with advanced warning) but other than that my hands are tied. I've managed to keep them fairly up to date anyway, but that part of my life would be easier if I had a supported 2.9-stable CVS tree.

      Comments

      1. By Ben Johnson () on

        Sometime's you can just throw some hardware at the problem -

        I had two 2.9 fiewalls/database servers with the OpenSSH vunerability - and being lazy and stupid, instead of patching OpenSSH and actually figguring things out, I just ploped in a new 3.2 firewall infront of the old firewall.

        Just a thought.


        Comments

        1. By RC () on

          I don't see the point. It's very easy to upgrade OpenSSH.

          The only thing people are compalining about is that they are forced to reboot (eg. 2 seconds of down-time) to upgrade to a new release, and hence, a new kernel.

        2. By Anonymous Coward () on

          As the original Anonymous Coward on this particular thread... Actually the 2.9 OpenSSH update is not a problem at all. Go the OpenSSH website and they've got a patch for the latest release and install instructions. I've done it many times and there have been no issues.

      2. By click46 () click46@genmay.net on mailto:click46@genmay.net

        what would you pay, if anything, for a supported 2.9-stable CVS tree?

        Comments

        1. By Noob () on

          I would not be interested in paying for a supported tree for 2.9-stable

          I would rather invest money in the purchase of the latest release CDROM and help future developments.

          The upgrade proceedures which already exist work for me just fine. I'm actively using OpenBSD 3.2-stable on production systems and running OpenBSD 3.3-current on my test systems.

          I have upgraded in the past without problems or any critical downtime.

        2. By Anonymous Coward () on

          Obviously nothing. I buy all the new releases to support (one of) my favorite OSes. I know these two boxes are unsupported, I accept that. It takes a little more time to keep them updated than say a 3.2 box would. I only sweat the remote problems, stuff like the recent file won't get patched. The customer is willing to pay for the time I spend, and not willing to let me do a version upgrade. The boxes are well firewalled from the public. I was just pointing out that comparing a home firewall upgrade to a production server upgrade is rather silly.

    3. By Anonymous Coward () on

      You really shouldn't use -current or snapshots here. They're for developers. cvs and use -stable. Here's what openbsd.org says about snapshots:

      .../OpenBSD/snapshots/
      For our major architectures, we tend to build mini releases of unknown stability and quality about every month or so. This is where we place those test releases.

      Comments

      1. By cycloon () on

        since were relatively close to the next release the stability shouldnt be so bad. Additionally there were some comments on that on the -misc ml.
        And as i wrote, i wanted to test some features i will use at work in future.

      2. By tedu () on

        they're for more people than just developers. install a snapshot on a testbox, if it seems to be working, it should be good on a real system. remember, developers have to run current, so it's not likely to be broken often.

        you don't have to upgrade every week, of course. but if there's some feature you want in current, don't be afraid to try it out.

  3. By click46 () click46@genmay.net on www.genmay.net

    Does anyone else smell a nice, niche business opportunity here? Not a bad way to employ some OpenBSD developers...

    Comments

    1. By pravus () on

      i don't neccessarily believe that i would want the developers working on this. they should remain focused on improving upon what's already been laid out. seems like a good way for other businesses to make some extra cash on the side though. as OpenBSD gains in popularity, i'm sure we'll see this sort of thing pop-up here and there.

      Comments

      1. By click46 () click46@genmay.net on mailto:click46@genmay.net

        I used the term developers lightly. I too would not want Theo and the like working on old code. What I meant was the general community of OpenBSD developers, or, basically, OpenBSD programmers. I know there are quite a few out there that would love nothing more than to be paid to work on OpenBSD; even if half or three quarters of their time is spent working on and mainting old branches.

  4. By grey () on

    While I hope someone can answer your query of a decent way to go about doing this (though it sounds like your own efforts are already pretty good) - I think other comments about the OpenBSD team supporting older releases really needs to deal with reality. How many developers are there for OpenBSD? Very very few (especially as compared with the likes of say, FreeBSD or Linux). From what I can tell, financial support is also pretty low. I mean sure there have been a few large grants recently; but if those covered everything that the developers needed, do you really think that there would be posts about how we need to encourage more CD sales due to declining purchases? Heck, look at http://www.openbsd.org/want.html or even the CVS diffs - it's stayed mostly the same for a long time (or well, things get added, but not much is getting removed from the list). OpenBSD just doesn't have the resources to devote to things unnecessarily.

    Until that situation changes, we can't expect developers to maintain stable branches of trees older than already outlined (and again, I realize that Arrigo Triulzi wasn't asking for that - so I hope some folks have some other suggestions for him). That said, there are methods by which to minimize the impact that upgrades have in production environments.

    Separation of duties is a good first - break off your httpd, smtpd, DNS server, user files (this is aimed at the guy with the 50+k shell users) et al into separate -dedicated- machines, that way upgrades can be done incrementally. If your resources allow, hopefully at least one rollover machine can be purchased (this is less of a challenge if you're dealing with x86 hardware) such that you can make a preparation machine - bring it up to the level (3.3-current/whatever) you need, get the service you need (say, named) all working hunky dorey, and then have a cutover. The machine that is removed from service can then be the next rollover machine [that's if you can only afford to have one extra box].

    The ability to have planned maintenance windows really should not be sacrificed - and with proper resources, they can be pretty transparent to the user (even shell servers - look at the old netcomxx.netcom.com shell situation as an example, where a group would be cycled out periodically for upgrades, allowing users to round robin to still up machines all in turn [of course, this meant that their user home directories were handled differently than just a pure local fs, think about it]).

    If maintenance windows are still a real challenge, maybe some of the newer load balancing features of pf can assist in redirecting traffic to machines during transition, but that is going to be pretty case specific.

    If the challenges in upgrading are due to breaking other packages (whether in house or external) well, it's hard to say (especially since admins often inherit systems) but if possible, try to use alternatives that are maintained themselves - that or find a way to hide the old (possibly vulnerable) machine from the rest of the world by putting something in between it and everything else.

    These are all primarily administrative methods of approach to reducing downtime while still tracking patches/new features. Asking OpenBSD developers to devote their resources to dealing with something which is primarily a user methodology issue isn't all that wise.

    Feel free to donate more money to OpenBSD in hopes that they can maintain older trees - but I don't really hear a groundswell of support for that [binary patches would yield much bigger support IMO]. And ask yourself, do you really _want_ support for older releases and the tree system that tends to entail (just look at the horror of the FreeBSD tree system). Even superficially it looks complicated - I can only imagine the pains of insuring that patches really fix the right problems across several different supported areas.

    Comments

    1. By Anonymous Coward () on

      > we can't expect developers to maintain stable > branches of trees older than already outlined

      I agree, and anyone willing to prove otherwise, within reason, is welcome.

  6. By Chris Cappuccio () chris@nmedia.net on mailto:chris@nmedia.net

    I have been upgrading my production boxes since 2.1.

    Make sure nobody is logged in and there is no crap in
    /tmp.

    I generally start by doing this in /bin/sh:


    # cd /tmp
    # ftp ftp.openbsd.org
    ftp> cd /pub/OpenBSD/snapshots/i386
    ftp> prompt
    prompt off
    ftp> mget bsd *gz
    ftp> quit

    # tar xpzf etc33.tgz
    # rm etc33.tgz
    # cd /

    # rm -r /usr/include /usr/share /usr/libexec

    # for i in /tmp/*gz; do
    > tar xzpf $i
    > done

    # mv /bsd /obsd

    # mv /tmp/bsd /bsd

    start like this:
    # cat /tmp/etc/group
    # vi /etc/group

    # vipw

    # cd /tmp/etc
    # cp rc login.conf netstart daily weekly monthly security fbtab moduli /etc

    # cat /etc/rc.conf

    # cp /tmp/etc/rc.conf /etc

    # vi /etc/rc.conf

    # cat /etc/rc.securelevel

    # cp /tmp/etc/rc.securelevel /etc

    # vi /etc/rc.securelevel

    # cat /etc/rc.shutdown

    # cp /tmp/etc/rc.shutdown /etc

    # vi /etc/rc.shutdown

    # cat /tmp/var/cron/tabs/root

    # crontab -e root

    # cd /dev

    # rm -r altq

    # ./MAKEDEV all

    Of course, you may be upgrading from an ealier version
    than 2.8. Also, you may be upgrading a system like sparc64 where the binary format changed. In this
    case, you may want to preserve gzip and tar for
    the extraction before you do it. So, do this in place
    of the 'for' loop above:


    # cp /usr/bin/tar /tar
    # cp /bin/gzip /gzip

    # mv /bsd /obsd
    # mv /tmp/bsd /bsd

    # for i in /tmp/*gz; do
    > /gzip -dc $i | /tar xvpf -
    > done

    Of course, you have to be a lot more careful when
    you are upgrading like this. You will have to reboot after the for loop and before you do anything else! Because you are most likely going to have to reinstall EVERY package and anything else that you compiled, this is obviously a more complicated method. For people not familiar with how the system is organized, you may just be better off installing from scratch if you are starting from a very old version of OpenBSD.

    There are always little things that will linger after the upgrade that you have to take care of. You will want to recompile major programs/ports that you depend on, like database servers, MTAs or apache modules. You may need to wipe out your whole site_perl directory and recompile the modules if you are moving from non-propolice to a propolice version of Perl.

    On my home machines I always delete all the ports and recompile everything since they can be down for a little bit and KDE is always more reliable when every library and everything else is current.

    Finally, some people may have custom kernels and/or other things to pay attention to. Don't forget to use config to update the new kernel before you reboot.

    If you are using bind4, bind9 is going to require that you re-do your config in the new format. And, the config goes in /var/named/etc now.

    Having just upgraded about 20 machines in the past week, I can say that this only takes about 10-30 minutes per machine, depending on how customized /etc is and if the kernel is custom and so on. That's why I say it's EASY. I do it remotely all the time. Once you've practiced a bit, it is all downhill from there. The only sticky points are when you have a lot of apps that depend on ports and such. Recompiling ports, mysql, etc can take quite a while.

    FYI, I normally download the release/snapshot that i want to upgrade with to a local FTP server and then install all the other machines from that FTP server rather than use ftp.openbsd.org every time.

    Upgrading OpenBSD involves quite a bit of work if you are going to recompile the software/packages, but you can always get away with running the same old crappy stuff if both releases are binary compatible. So, if you are upgrading i386 from 2.8 ro 3.3, no biggie. Upgrade the packages when you have time. Perl and Apache modules are the only problem points that I've had with this lazy method. Of course, anything compiled under 3.3 will get ProPolice protection, and you will probably be getting a newer, (possibly) more secure version of the software/port/package that you are installing since this is 2.5 years after the obsd 2.8 release!!!

    Expect any platform switch to ELF to be more complicated, of course. i386 will do this eventually. You
    will have no choice but to upgrade all your Perl and Apache modules, and probably all your ports at this point. For people who maintain lots of machines, remote machines, and/or machines that have lots of software/data that you don't want to fuck with, this method is vastly superior to installing from scratch and copying things back over.

    Another method is to format only your / and /var and /usr
    partitions but keep your data partitions when you do the reinstall. Obviously, this cannot be done remotely and requires a bit more reconfiguration.

    The first method outlined here is pretty close to what you'd have to do using the (U)pgrade option on the boot floppy, only you are unpacking the distribution and making devices yourself.

    Hope this helps people, like I said I have machines that have been upgraded this way since 2.1. If you aren't comfortable with the system then you may get lost upgrading like this because too many things will bite you in the ass. Reaing the upgrade-minifaq can give you hints about other things that have changed between releases that I don't talk about here.

    Have fun

    Comments

    1. By Anonymous Coward () on

      Thanks man, that's some good stuff :-)

    2. By Anonymous Coward () on

      BAD!
      -# mv /bsd /obsd
      +# cp /bsd /obsd
      # mv /tmp/bsd /bsd


      much better!

      Comments

      1. By tedu () on

        small note. the bootloader will find obsd if bsd is missing.

    3. By jose () on http://monkey.org/~jose/

      this is actually how i maintain my laptop, with the following changes:

      1.uninstall all packages
      2. download bsd.rd and boot that (ramdisk)
      3. delete all system files outside of /etc, /var, /dev/, and /home ... ie everything being upgraded. no stale libs, binaries, etc ...
      4. unpack tgz's as chris does (tar -zxvpf), install new kernel
      5. if needed (ie ELF upgrade on sparc) install new boot blocks
      6. reboot new kernel.
      7. mergemaster on /etc files
      8. new packages get installed (i should make a metaport of my stuff to smooth the process).

      hope that helps. its definitely a very hackish way to do it.

    4. By jose () on http://monkey.org/~jose/

      this is actually how i maintain my laptop, with the following changes:

      1.uninstall all packages
      2. download bsd.rd and boot that (ramdisk)
      3. delete all system files outside of /etc, /var, /dev/, and /home ... ie everything being upgraded. no stale libs, binaries, etc ...
      4. unpack tgz's as chris does (tar -zxvpf), install new kernel
      5. if needed (ie ELF upgrade on sparc) install new boot blocks
      6. reboot new kernel.
      7. mergemaster on /etc files
      8. new packages get installed (i should make a metaport of my stuff to smooth the process).

      hope that helps. its definitely a very hackish way to do it.

      Comments

      1. By Chris Cappuccio () chris@nmedia.net on mailto:chris@nmedia.net

        It's nice to preserve your old libs if you want to run old binaries.

        But, you do want to at least rm /usr/lib/*c_r* to get rid of the old threaded crap.

  7. By Chris Cappuccio () chris@nmedia.net on mailto:chris@nmedia.net

    Sorry for the duplicate post. The php weblog on here chewed up my last post because i wasn't using the ampersand thingies.

    I have been upgrading my production boxes since 2.1.

    Make sure nobody is logged in and there is no crap in
    /tmp.

    I generally start by doing this in /bin/sh:

    =download openbsd 3.3 into /tmp=
    # cd /tmp
    # ftp ftp.openbsd.org
    ftp> cd /pub/OpenBSD/snapshots/i386
    ftp> prompt
    prompt off
    ftp> mget bsd *gz
    ftp> quit
    =at this point you should still be in /tmp=
    # tar xpzf etc33.tgz
    # rm etc33.tgz
    # cd /
    =get rid of directories which will have old crap that can interfere with your compiles/ports/stuff=
    # rm -r /usr/include /usr/share /usr/libexec
    =here is a for loop that actually does the unpacking of the new system. the P flag is very important, don't forget it=
    # for i in /tmp/*gz; do
    > tar xzpf $i
    > done
    =preserve your old kernel in case the new one is bogus on your hardware=
    # mv /bsd /obsd
    =copy over the new kernel=
    # mv /tmp/bsd /bsd
    =Now you must edit /etc/group and add in all the
    groups that start with _underscore from /tmp/etc/group=
    start like this:
    # cat /tmp/etc/group =Take notes=
    # vi /etc/group =edit the group file=
    =Now you must edit /etc/master.passwd and add in all
    the users that start with _underscore from /tmp/etc/master.passwd=
    # vipw
    =Now you should copy over all the standard scripts from
    /tmp/etc to /etc=
    # cd /tmp/etc
    # cp rc login.conf netstart daily weekly monthly security fbtab moduli /etc
    =Take note of your /etc/rc.conf=
    # cat /etc/rc.conf
    =Copy over the 3.3 rc.conf in place of your old one=
    # cp /tmp/etc/rc.conf /etc
    =Add your old changes to the stock 3.3 rc.conf=
    # vi /etc/rc.conf
    =Take note of your /etc/rc.securelevel=
    # cat /etc/rc.securelevel
    =Copy over the 3.3 rc.securelevel in place of your old one=
    # cp /tmp/etc/rc.securelevel /etc
    =Add your old changes to the stock 3.3 rc.securelevel=
    # vi /etc/rc.securelevel
    =Take note of your /etc/rc.shutdown=
    # cat /etc/rc.shutdown
    =Copy over the stock 3.3 rc.shutdown in place of your old one=
    # cp /tmp/etc/rc.shutdown /etc
    =Add your old changes to the stock 3.3 rc.shutdown=
    # vi /etc/rc.shutdown
    =Now you must edit root's crontab and bring it up to sync
    with the new stuff in /tmp/var/cron/tabs/root=
    # cat /tmp/var/cron/tabs/root
    =Add sendmail and whatever else is new from there to your old root crontab=
    # crontab -e root
    =make sure you have new devices, not old ones=
    # cd /dev
    =altq is gone now, get rid of the directory=
    # rm -r altq
    =make youre new devices=
    # ./MAKEDEV all

    Of course, you may be upgrading from an ealier version
    than 2.8. Also, you may be upgrading a system like sparc64 where the binary format changed. In this
    case, you may want to preserve gzip and tar for
    the extraction before you do it. So, do this in place
    of the 'for' loop above:

    =preserve the old tar and gzip so you can keep running them over and over in the loop=
    # cp /usr/bin/tar /tar
    # cp /bin/gzip /gzip
    =copy over the new kernel ahead of time=
    # mv /bsd /obsd
    # mv /tmp/bsd /bsd
    =here is the loop=
    # for i in /tmp/*gz; do
    > /gzip -dc $i | /tar xvpf -
    > done

    Of course, you have to be a lot more careful when
    you are upgrading like this. You will have to reboot after the for loop and before you do anything else! Because you are most likely going to have to reinstall EVERY package and anything else that you compiled, this is obviously a more complicated method. For people not familiar with how the system is organized, you may just be better off installing from scratch if you are starting from a very old version of OpenBSD.

    There are always little things that will linger after the upgrade that you have to take care of. You will want to recompile major programs/ports that you depend on, like database servers, MTAs or apache modules. You may need to wipe out your whole site_perl directory and recompile the modules if you are moving from non-propolice to a propolice version of Perl.

    On my home machines I always delete all the ports and recompile everything since they can be down for a little bit and KDE is always more reliable when every library and everything else is current.

    Finally, some people may have custom kernels and/or other things to pay attention to. Don't forget to use config to update the new kernel before you reboot.

    If you are using bind4, bind9 is going to require that you re-do your config in the new format. And, the config goes in /var/named/etc now.

    Having just upgraded about 20 machines in the past week, I can say that this only takes about 10-30 minutes per machine, depending on how customized /etc is and if the kernel is custom and so on. That's why I say it's EASY. I do it remotely all the time. Once you've practiced a bit, it is all downhill from there. The only sticky points are when you have a lot of apps that depend on ports and such. Recompiling ports, mysql, etc can take quite a while.

    FYI, I normally download the release/snapshot that i want to upgrade with to a local FTP server and then install all the other machines from that FTP server rather than use ftp.openbsd.org every time.

    Upgrading OpenBSD involves quite a bit of work if you are going to recompile the software/packages, but you can always get away with running the same old crappy stuff if both releases are binary compatible. So, if you are upgrading i386 from 2.8 ro 3.3, no biggie. Upgrade the packages when you have time. Perl and Apache modules are the only problem points that I've had with this lazy method. Of course, anything compiled under 3.3 will get ProPolice protection, and you will probably be getting a newer, (possibly) more secure version of the software/port/package that you are installing since this is 2.5 years after the obsd 2.8 release!!!

    Expect any platform switch to ELF to be more complicated, of course. i386 will do this eventually. You
    will have no choice but to upgrade all your Perl and Apache modules, and probably all your ports at this point. For people who maintain lots of machines, remote machines, and/or machines that have lots of software/data that you don't want to fuck with, this method is vastly superior to installing from scratch and copying things back over.

    Another method is to format only your / and /var and /usr
    partitions but keep your data partitions when you do the reinstall. Obviously, this cannot be done remotely and requires a bit more reconfiguration.

    The first method outlined here is pretty close to what you'd have to do using the (U)pgrade option on the boot floppy, only you are unpacking the distribution and making devices yourself.

    Hope this helps people, like I said I have machines that have been upgraded this way since 2.1. If you aren't comfortable with the system then you may get lost upgrading like this because too many things will bite you in the ass. Reaing the upgrade-minifaq can give you hints about other things that have changed between releases that I don't talk about here.

    Have fun

    Comments

    1. By Arrigo Triulzi () on http://www.alchemistowl.org/arrigo

      This is very interesting. I need to try this out before I do it on a remote machine to which I haven't got easy physical access.

      Thanks for posting it!

      Comments

      1. By jose () on http://monkey.org/~jose/

        make sure you're careful to run a spare sshd process which doesn't need the sshd user, or test it out before you reboot if that's your only way to log in. a spare pc beside it with a serial console to the openbsd box (with /etc/ttys spawning a getty on console) will be useful here.

  8. By Arrigo Triulzi () on http://www.alchemistowl.org/arrigo

    What the posting does not reflect is that behind it is a long discussion with Jose and quite a bit more thought than what might transpire.

    The key points for those who took this posting as a troll:

    1) it is clear, accepted (at least by me) and eminently understandable that the OpenBSD project cannot support all the releases,

    2) there are some situations in which upgrading is not possible: production systems in the middle of nowhere come to mind.

    3) "remote production systems" is my problem, and it is exacerbated by the fact that PCs are hopeless in headless configurations. I have done zillions of installs and upgrades over serial terminal concentrators on systems which support this. The PC was simply not designed for this.

    4) I am aware of a number of pretty upgrade options, some better than others. This is *not* being disputed.

    5) The point I am trying to make is that, if necessary, it is possible to keep older systems alive without excessively compromising on security.

    6) "without excessively compromising on security" means making an educated decision as to whether the limitations of the current system (e.g. no propolice) are acceptable or not.

    Other than that if anyone knows what happened to those people who sold PC cards which gave you a serial interface into the BIOS (ie. you could intercept output from BIOS startup onwards, a real serial console setup) I would love to know.

    Arrigo

    P.S. Just for reference: it is trivial to upgrade a remote OpenBSD Sun box or indeed Alpha via the serial console in comparison to a PC...

    Comments

    1. By Anonymous Coward () on

      http://www.realweasel.com/

      Comments

      1. By Arrigo Triulzi () on http://www.alchemistowl.org/arrigo

        Thank you! The first "anonymous coward" saying something useful!!

        Do you have any experience with them? Do they really work?

        Comments

        1. By grey () on

          To accompany the realweasel, you will also want to take a look at some of the following sorts of products (there needs to be a way to access it):

          http://www.cyclades.com/products/index.php?id=4
          (approximately $300)

          Or, for something on the cheap (main downside being no sshd), where you have more rackspace:

          http://unixsurplus.zoovy.com/product/TERM_TANDEM_8PORT

          The DIY cyclades would be doable with something like a soekris board.

          The realweasel appears to be pretty useful in general, though keep in mind you can obviate the need for one if you don't care about BIOS options, by simply editing your boot.conf to redirect to a serial port on your x86.

          Ideally, if you have the rack space and the finances, you could do it all DIY with each unit (serial console device & actual server) having two serial ports, and cross connect them to each other, so that you can use one to upgrade the other if necessary. A little crazy, but it would be pretty flexible.

          And well, realweasel could also be paired well with just a modem in the event that things are really horked (e.g. whole network down, so you couldn't access a cyclades if you wanted). Some folks combine these features into their remote serial console devices (I believe even cyclades does, though it'll probably cost a bit more).

          ymmv

        2. By Anonymous Coward () on

          No experience. Can never seem to get a $350 PCI card past purchasing unfortunately, but this entry from their testimonial page is enough to convince me.

          "I love my Weasel." -- Paul Vixie

    2. By Piotr Kapczuk () Piotr.Kapczuk@hoop.pl on mailto:Piotr.Kapczuk@hoop.pl

      HP solds something that might interest you http://h18004.www1.hp.com/products/servers/management/riloe2/index.html

      There are also servers with integrated remote management http://h18004.www1.hp.com/products/servers/management/ilo/index.html

    3. By jose () on http://monkey.org/~jose/

      dell and compaq and ibm all make rackmount systems which support serial terminals. not a whole lot more expensive that regular rackmount pcs.

      uprading hardware may not be an option tho, either ... but something to keep in mind for future stuff.

      Comments

      1. By Marco Peereboom () slash@peereboom.us on www.peereboom.us

        I have a few headless dell powerapp 100's. No problems at all; just change the install floppy to use serial instead of kvb. After that it is a walk in the park.

  9. By Anonymous Coward () on

    go with debian ... it's more suited to that kind of task and is less of a headache to upgrade. Some of the packages are a little "old" but they are patched from security.debian.org to iron out any security kinks.

    openbsd has too many core changes to become a "stable trusted environment". It excells at network security and little else. It's too volatile to trust in a server environment (sorry if that pisses anyone off but I appear to be the only one being realistic).

    Comments

    1. By Anonymous Coward () on

      > It's too volatile to trust in a server environment (sorry if that pisses anyone off but I appear to be the only one being realistic).

      No, it just appears that your definition of "server environment" is different from mine.

      For instance, I don't mind wiping out everything on every server once every year or two, reinstalling new version and restoring user's data - it only takes one night to do it and couple more nights to tweak and finalize. May be I am just lucky that our servers need to be operational only for 14-16 hours a day...

    2. By Anonymous Coward () on

      I'm sorry to disappoint you, but in my corporate environment, our downtime should not exceed 5mn per year. We have some mainframes perfectly working, uptime: 5+ years.

    3. By Anonymous Coward () on

      oh good idea, lets convert our openbsd 3.0
      servers to debian, great solution, very
      realistic.

      why are you on deadly.org anyway? please
      return to debian-trolls posthaste.

      Comments

      1. By Stefan () - on mailto:-

        I'd really like to see an OS which can update it's kernel w/o rebooting, and even debian can't do that ;) ... and I've updated quite a few debian boxes - if you have lots of packages and changed their configuration, updating debian can be a lot of work too. However, if you changed only a few files in /etc/ it's a peace of cake - same with OpenBSD, as a few posters have already stated here.

        If you want no downtime at all, you would need a standby/failover-solution - so you can reboot one system while the other acts as a "standby".

        However, if you want to use some kind of "shared storage" and you want to upgrade
        the cluster-filesystem, then there's no way you can get away w/o downtime
        or maybe at least a three-or-more-system-cluster?

        So far, I haven't seen a "free" (as in coffee) solution for this ...

        Maybe someone has an idea?

        cheers
        stefan

Latest Articles

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]