OpenBSD Journal

A Puffy in the corporate aquarium

Contributed by weerd on from the puffy-puppeteering dept.

Antoine Jacoutot (ajacoutot@) writes in with a story about how M:tier, the company he works for, uses OpenBSD.

For several years now I've been working for M:tier, a very OpenBSD-friendly company, and I think it's about time that we share about "how" we use OpenBSD in some aspects.

As for "what" we use it for, it's simple: everything.

As a company we are very dedicated to what we do because we are "forced" to use our operating system of choice and we want our customers to be as happy as we are at using it :-)

So our paid job is hacking on and deploying, maintaining, supporting... OpenBSD installations. We are also required to hack on things that can be merged back into OpenBSD itself and when it's not possible, then we change what we did so that it can be. Of course some developments are very specific to what we do and have no place in the project's CVS tree.

So, amongst other services, we set up and maintain several 100% OpenBSD-based infrastructures (going from the entry site firewall to the secretary's workstation) and this is what I'm going to talk about here.

As a side note, it is important to know that we are working exclusively for Fortune 500 companies (each operating in totally different and unrelated sectors).

What it means is that:

  • We are not setting up systems for small geek-friendly-only companies but for huge ones with a long IT history (some of them are present in >100 countries worldwide). While I cannot reveal any names, it is important to know that OpenBSD can fit in the Big Ones.
  • We have to comply to very large and complex technical and legal specifications.

While most people will see it as a useless effort, we think it is very interesting to make a non-mainstream operating system comply with the corporate rules.

The Big Picture

We are currently managing over 600 users in several locations around the world (expecting a large increase before the end of the year).

All these locations are fully running under OpenBSD, that is:

  • the firewalls: PF, IPSEC, CARP...
  • the infrastructure servers: DNS, DHCP, TFTP, FTP, HTTP, NFS, LDAP, puppetmaster, Kerberos, proxy, print server...
  • the desktops (workstations and laptops): The GNOME Desktop and plethora of graphical applications.

95% of the services are redundant (including our puppet distribution setup). Shutting down one server has zero impact on production. Everything is monitored by Zabbix: there is a Zabbix proxy running on each site that sends its data over an IPSEC VPN to our central Zabbix server.

We are also developing a Zabbix mobile client that runs on the iPhone, Android, Blackberry and Windows Mobile 6 so that the sites managers can see the status of their OpenBSD machines from anywhere ;-)

There have been countless articles about how OpenBSD can be used as a <insert your infrastructure service>, this is why in this article, I will focus on two usually less covered aspects: how we use puppet to manage these installations and the use of OpenBSD as a Joe User Desktop.

Another aspect of our solution is that we exclusively use signed packages. This has nothing to do with security but we want to make sure the packages used on the machines are all coming from us. There is a nice howto about how to build and use signed package in the OpenBSD FAQ.

Joe User meets OpenBSD

As a regular user, when the IT staff starts migrating your Windows XP workstation to Windows 7, it is very traumatic! But, you accept it because everyone else in the world is more or less doing the same. But imagine if you had to migrate from Windows XP to Mac OS X! Ok you can also accept it because, well you own an iPod like everyone else, so Apple is known to you.

But the real challenge comes from migrating users from a >10 years habit of using Windows and MS Office to an OpenBSD GNOME Desktop with Libre/Openoffice without impacting their daily work, aka production, aka company revenue.

The important aspect of such a migration even before installing the new system is obviously information and ... information. A user always feels somewhat punished when his computer work environment changes, so it is very important to explain why it is happening and how it will in the end serve and benefit him or her. We have a tendency to be on the user's side here, people want to get work done and not learn how computers work.

On the technical aspect, a typical Joe User Desktop is made of the following parts:

  • a pre-configured GNOME (mandatory screenshots 1 and 2) environment for the user interface
  • Libre/OpenOffice suite for enterprise editing
  • OpenNX for accessing Windows Remote Desktop or Terminal services (that is needed because companies use internally developed applications running on Windows only)
  • the rest of the graphical applications needed for daily work (web browser, ftp client, multimedia player, picture editor...)

The Desktop configuration in itself is done using a script containing gconf commands (this script is run by the puppet daemon).

We can totally transform the Desktop by running gconftool-2 with correct arguments.

We provide some (really not many) scripts to ease daily usage of the Desktop. After several years of using Windows, people are used to a certain behavior of their computer and we try to bring that to them (e.g. pluging a USB key will automatically mount it on the Desktop thanks to an hotplugd(8) script and open a nautilus window with a right-click menu to unmount the device).

We also wrote some graphical interfaces for commonly used tasks like encrypting a USB drive, restarting the network, setting up ppp/pppoe/WiFi connections, connecting to the office IPSEC VPN, changing the local and Kerberos passwords...

At last, it is good to know that all users' home directories on the laptop are encrypted using standard OpenBSD tools and with a passphrase that is different for each user.

Everything a user needs is just a click away, he/she is by no means required to use the terminal.

Joe User meets tiny OpenBSD

In some situations, a full-blown Desktop is not required. For this particular case, we have developed an OpenBSD-based thin client solution.

The thin client is a small OpenBSD installation with base and x* sets which runs under a read-only "/" partition with others that are needed for normal operation as mfs filesystems in read-write mode (these are /tmp, /home, /var/log and /var/db).

When the boot process is done, X.org and the thin client software get started right away from /etc/ttys.

The thin client uses a central update server running puppet on the M:tier side, and on the client puppet remounts "/" read-write, does its run then remounts it back to read-only mode.

Depending on the configuration IPSEC may be used. IPSEC and LAN status are all indicated on the client itself.

Note that besides a modified /etc/rc (stripped down to speed up boot times) the thin client is running a standard OpenBSD installation.

Installation, maintenance and configuration

Here we will talk in a bit more details about some technical sides of managing all the different locations.

Installation

The installation process for desktops and servers is the same. On site we have a redundant pxeboot server and puppetmaster that holds all the necessary pieces for an unattended installation.

Installing a machine is just a matter of PXE booting on a modified bsd.rd kernel. This bsd.rd automatically partitions the hard drive and installs the required sets from the local ftp according to configuration files held on the pxeboot server.

At the end of the installation, the ruby-puppet package is installed and puppetd is started. Puppet will then finish the installation using different manifests and recipes whether we are installing a laptop, workstation, server...

Puppet takes care of everything from installing the packages to creating daemon users and configuring the whole setup.

Once the installation is finished, the machine automatically reboots and we use rc.firsttime to do some interactive post-install administrative tasks (like setting up a root password, registering the machine into the Kerberos realm...).

Note that the puppetmaster is running with --servertype=mongrel and we use nginx to load balance the requests which is required to have decent concurrent connections support.

Configuration deployment

For configuration purposes we exclusively use puppet. Amongst the obvious benefit of having a central location for configuration files, it also allows us to setup generic configuration files that will be shared amongst all customers but that we can still make specific by only changing some variables.

The puppet recipes and manifests are actually contained into a package. When a configuration change is requested, we edit the recipe, create the new package which then gets pulled by the requesting site and the puppetmaster does the rest.

Obviously, most of the tasks are automated with sanity checks but an interactive manual approval is always needed before deploying a new configuration.

Being written in Ruby, puppet is easily extensible and can be made to do pretty much anything. We are abusing this feature quite a lot :-)

Maintenance

Surprisingly, we are also using puppet for that ;-) The puppet daemon running on all machines will automatically check for packages updates, new antivirus definitions... Yes we use an antivirus because we are interacting with Windows users and the private and public shares that are available to the users over NFS are also available via FTP (encapsulated into the IPSEC VPN) from the remote Windows Terminal server.

For base system updates, we use "binpatches" built using the binpatch-ng framework.

A binpatch is just a regular OpenBSD package containing the updated binaries that also creates a backup of the replaced files at pkg_add(1) time so that the update can be reverted in case something goes wrong. Binpatches are handled using the regular OpenBSD package tools.

Backups are handled in two different ways. On the Desktops we use "backintime" which takes an incremental snapshot every hour into a hidden private share on the NFS server. Backintime is really handy for users because it provides an easy to use graphical interface to restore files and or directories from a particular point in time. For enterprise compliant backup, we use bacula to backup all the servers to a central point (that includes the NFS server, so in fact user data has a double backup). For bare metal recovery, all that is needed is to bootstrap a server from scratch then running bacula to restore everything. Critical files are of course saved off site as well. This dual setup allows for easy access to backup recovery mode while maintaining compliancy with corporate specifications.

Obviously not being physically on site, we need to have local system administrators. Almost none of them ever heard of OpenBSD before, some exclusively worked on Windows and know nothing else. To deal with that, we provide them with several days of theoretical and practical trainings which are of HUGE interest for us because we get questions and feedback that we would never get from people used to using Unix systems.

Conclusion

It usually takes several months for a complete office to finally be accustomed to its new solution but in the end users are very happy with their new work environment.

It is a great pride for us when entering an office and seeing all these users running OpenBSD on their machine. While from a first glance it just looks like another office like we've all seen countless times, when you actually pay attention at what is on the screen it is quite different ;-)

While we are facing several social, legal and technological challenges, one of the most interesting one is the use of OpenBSD on the desktop. Most people use OpenBSD as a server, router, firewall or embedded device but for a desktop it is pretty rare (I'm not accounting OpenBSD enthusiasts that are the only ones running an OpenBSD workstation at their workplace). That means we are building our experience from scratch because there is not much feedback from anywhere else.

The best of all is that there is no clear line between our hobby and our work!

I hope you enjoyed reading this article that I hope uncovered some ideas about how OpenBSD can be used in the enterprise world in a way that is not obviously expected.

Thanks go to Antoine for this peek into the kitchen of M:Tier!

(Comments are closed)


  1. By phessler (phessler) spambox@theapt.org on http://theapt.org

    I love hearing about how OpenBSD is being used in "non-traditional" applications. So many people think OpenBSD is only good for firewalls and servers, when it is really good for many other projects. Sure, I'm an enthusiast/developer, but I do use OpenBSD on my desktop. I'm interested in fixing bugs or "unwanted" behavior, to make my chosen OS even better.

  2. By Nicolas P. M. Legrand (nlegrand) nlegrand@ethelred.fr on http://www.ethelred.fr/

    Nice story. I didn't believe it was really possible and just thought there were only "OpenBSD enthusiasts that are the only ones running an OpenBSD workstation at their workplace" like me :). Congrats !

  3. By Marc Espie (espie) espie@nerim.net on

    Now that Antoine has written this article, I can complete it.
    The infrastructure for signing packages was written in a generic way, but there were several roadblocks, and generally, deployment issues, that prevent it happening on the main openbsd packages. Most specifically, it's a question of trust: signed packages only guarantee that you get exactly what got compiled.

    Then Antoine approached me with his needs, and I polished the initial prototype so that he could use in production.

    The cool thing is that this forced me to think in a generic way, how to sign and distribute packages for anyone, with any kind of certificate.

    This also the fact that the "deployment" issue is very simple in his case. It's more a question of his company wanting to make sure people are using the packages they're providing, and not something else.

    In this case, they are using the basic package tools that are also in generic OpenBSD (they got one single patch: their installation refuses to install non-signed packages. That's a two-lines diff...)

    1. By Alvaro Mantilla (alvmangi) on http://www.alvaromantilla.com/

      How do you handle the "Flash" issue on Desktops? Most of the companies I know had a lot of training courses and things like that Flash based. Also, you mentioned about to run windows applications remotely: that means the customer run windows browsers to have access to that Flash based information?

      1. By Adam P (adamrt) on

        > How do you handle the "Flash" issue on Desktops? Most of the companies I know had a lot of training courses and things like that Flash based. Also, you mentioned about to run windows applications remotely: that means the customer run windows browsers to have access to that Flash based information?
        >

        Awesome article! I'm also interested in the 'flash' issue on the desktops. This is the only thing that keeps me from running OpenBSD on my desktop.

        Thanks again for the article! Great read.

        1. By Sam Taufa (samt) on http://www.nomoa.com/bsd/

          > > How do you handle the "Flash" issue on Desktops? Most of the companies I know had a lot of training courses and things like that Flash based. Also, you mentioned about to run windows applications remotely: that means the customer run windows browsers to have access to that Flash based information?
          > >
          >
          > Awesome article! I'm also interested in the 'flash' issue on the desktops. This is the only thing that keeps me from running OpenBSD on my desktop.
          >

          Also loved the article, and motivated.

          For our Flash restricted environment, we have a Windows Terminal Server (firewalled away, local accounts only, rdp access from the LAN) that users who need Flash can use. No-one else in the 1,000+ Windows workstations are allowed to install Flash on their workstations. If you want to avoid MSFT, then I guess that box could be a Linux box with Flash installed.


          Sam T.

          1. By Steven Rosenberg (steven) steven@stevenrosenberg.org on http://blogs.dailynews.com/click

            > For our Flash restricted environment, we have a Windows Terminal Server (firewalled away, local accounts only, rdp access from the LAN) that users who need Flash can use. No-one else in the 1,000+ Windows workstations are allowed to install Flash on their workstations. If you want to avoid MSFT, then I guess that box could be a Linux box with Flash installed.
            >

            That is a very compelling solution to the Flash "problem."

            There are so many good tips and things to think about in this article for OpenBSD users - thanks for posting it.


    2. By Joachim Schipper (Joachim) on http://www.joachimschipper.nl/About me

      > The infrastructure for signing packages was written in a generic way, but there were several roadblocks, and generally, deployment issues, that prevent it happening on the main openbsd packages. Most specifically, it's a question of trust: signed packages only guarantee that you get exactly what got compiled.

      Maybe I just don't get it, but that seems like an improvement. Sure, I'd still have to trust someone, but it's not like I audit every commit, anyway.

      Is there something someone (like me?) could do to help?

      1. By Marc Espie (espie) on

        > The infrastructure for signing packages was written in a generic way, but there were several roadblocks, and generally, deployment issues, that prevent it happening on the main openbsd packages. Most specifically, it's a question of trust: signed packages only guarantee that you get exactly what got compiled.
        >
        > Maybe I just don't get it, but that seems like an improvement. Sure, I'd still have to trust someone, but it's not like I audit every commit, anyway.
        >
        > Is there something someone (like me?) could do to help?

        It's complicated.

        Current package signatures do exactly what they're supposed to do: you put root authority certificate(s) where they belong, and signed packages get installed get checked for corruption.

        The problem is that it does not solve any issue with Theo:
        - he wants to control who can sign packages all the time. This requires a signing server, which is significantly more complicated to do.
        - this would also be a kind-of privilege separation process so that if the package building machine/bulk builder person gets hacked, the key stays safe.
        - people will tend to trust signed packages... whereas, in reality, this only insures one step of the process: making sure the package does not get tampered with once it's been signed...

        note also that actual certificate revocation issues are a nightmare.

        In short, the mechanism to sign packages is there because commercial outfits, like Antoine's, may be perfectly happy with this specific guarantee... especially since you can check afterwards that installed packages have not been tampered with, and this allows more use of OpenBSD. But there's nothing more to it.

        It would be nice to be able to deploy signed packages, especially since it would give you guys an incentive to buy at least one official release CD (guess what's the simplest way to distribute CA certs), but hey, lots of other more important (and funnier) things to do.



  4. By Magic carpet (bodie) on http://www.openbsd.cz

    Antoine for president ;-)

    It will be nice to have this story in BSD magazine or on informit.com or any other similar server.

  5. By Denis (Denis) openbsd@ledeuns.net on

    Amazing article ! Thank you for the insight.

  6. By Gabriel Guzman (gabe) gabe@guzman-nunez.com on http://guzman-nunez.com/

    The real question is: Are you buying an official CDROM for each user and server?

    1. By brakeb (brakeb) on

      > The real question is: Are you buying an official CDROM for each user and server?

      Even if he bought 100 servers and workstations copies of OpenBSD every 6 months for 3 years, it's probably still cheaper than a 100 site license for Windows or 100 basic support Redhat copies.

    2. By Antoine Jacoutot (ajacoutot) on http://www.bsdfrog.org/

      > The real question is: Are you buying an official CDROM for each user and server?

      No the real question is: how much code and money did you contribute to CentOS to have it running on your website?

      1. By Bret Lambert (tbert) on

        > > The real question is: Are you buying an official CDROM for each user and server?
        >
        > No the real question is: how much code and money did you contribute to CentOS to have it running on your website?

        ZING!!!

    3. By phessler (phessler) on http://theapt.org

      > The real question is: Are you buying an official CDROM for each user and server?

      I think you missed an important detail. Antoine is a developer with the OpenBSD project, and his ports and contributions have gone into OpenBSD proper. Are you using GNOME? Are you using sysmerge? There are dozens of other examples of things that he has written specifically for OpenBSD because his company needed them. He contributes to OpenBSD in extremely valuable ways.

  7. By Jeff (bifeo3) bifeo3@yahoo.com on

    This is the article I've been waiting for. The only thing holding me back from going 100% OpenBSD in the work place is email. We do use Exchange, but my work is 99% *NIX, command line, and Linux. Any ideas on how to get over the Outlook hurdle aside of fighting with IT to open IMAP? Thank you again for this great article.

    -Jeff

    1. By Norman Golisz (ngolisz) on

      > Any ideas on how to get over the Outlook hurdle aside of fighting with IT to open IMAP?

      Outlook Web Access over HTTPS? I'm using it daily and works fine for me.

      Norman

  8. By Jeff Quast (dingo) af.dingo@gmail.com on

    Very much would love that zabbix mobile app,

    as with any apps like these, the userbase would be low, but highly appreciated. I'd pay $20 for it.

  9. By jirib (jirib) jirib@mailinator.com on

    * will some autoinstallation framework be incorporated into openbsd?

    * how do you make those encrypted home dirs?

    after reading this article it explained me how is it possible to see all those great commits by some guys on ports@ during whole day :-) good work!

    jirib

    1. By Joachim Schipper (Joachim) on http://www.joachimschipper.nl/About me

      > * will some autoinstallation framework be incorporated into openbsd?

      I'm not sure, but in the meantime there's yaifo.

      > * how do you make those encrypted home dirs?

      softraid(4). Look at the EXAMPLES section of bioctl(8).

      1. By jirib (jirib) on

        > * how do you make those encrypted home dirs?
        >
        > softraid(4). Look at the EXAMPLES section of

        i'm not english native speaker so... i understood it that encrypted home dir means encrypted per user when logging in... it is not clear what does it mean in the article.

        1. By Joachim Schipper (Joachim) on http://www.joachimschipper.nl/About me

          You'll have to do some custom hacking to make *that* work.

  10. By SzymonN (alazif) alazif@windowslive.com on techcore.microcreative.pl

    bleble where is the migration of Outlook, calendars, ERP system, an additional soft, network printers, scanners, for multimedia terminal can migrate each because there is nothing on it is not

    1. By Predrag Punosevac (Oko) on

      What about network printers and scanners? They work fine on OpenBSD. Do you need a how to?






Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]