OpenBSD Journal
Home : : Add Story : : Archives : : About : : Create Account : : Login :
Developer blog: reyk: hoststated(8) preinspection
Contributed by dlg on Thu Feb 22 04:33:27 2007 (GMT)
from the schnell-schnell-schnell dept.

Some month ago I was looking for a solution to run OpenBSD as a fully-featured loadbalancer. It was bothering me for a long time that I had to run some systems coupled to the Alteons, F5s, Linux virtual servers, or even Cizzz-coeees of the world. So I was very happy when I found pyr's work. I helped to improve his daemon, imported it into OpenBSD as hoststated(8) and convinced Theo to get him into the project as a new developer. hoststated is amazing and I'm already using it in production, but one major thing was missing: Layer 7 Loadbalancing.

Loadbalancing in OpenBSD is not very new; you can do it for a long time now using the pf(4) redirection features, typically using layer 3 round-robin distribution to a table of internal hosts. Besides pf, it is also possible to handle a bigger load by using the layer 2 loadbalancing functionality which I implemented in the trunk(4) driver. You can also use the ARP balancing functionality of carp(4) but this requires to run OpenBSD on all the backend servers (this is nice but unfortunately not the typical case).

hoststated implements one missing link, the ability to do keep alive checks of the internal servers. If you run a simple pf redirection in front of a webserver pool, you would never know if all of the internal hosts are really available and the users would worry about failed connections. But the daemon will dynamically alter the redirection table of active hosts and is even able to check various protocols, like HTTP, for an expected reply.

So what about Layer 7 Loadbalancing? I was very happy that I could offer OpenBSD-based loadbalancers now, when I suddenly got asked: "Do you support SSL Acceleration?". Sure, we do, but there are many ways to achieve this functionality. And I did not really like the existing solutions. These "reverse-proxies" were either very complex, bloated, or non-free. And the preferred solutions like pound, squid or even OpenBSD's version of the Apache mod_proxy fall into the categories of beeing bloated and complex. And I did not like the licenses...

First of all, I was concerned about the security of the existing solutions. Security's worst enemy is complexity (thanks Bruce). I looked at pound, the most promising candidate, but stumbled over things like pthreads, regular expressions, and weird string functions like strcpy with dynamic input buffers. This doesn't necessarily mean that pound is insecure but it doesn't match our idea about software and the principle of proactive security. I don't like pthreads in networking daemons, because of potential risks in wrong use and the complexity of the API itself. You can implement high performance networking daemons using a single-threaded and non-forking event model with non-blocking I/O (this is how most of our networking daemons like bgpd(8), ospfd(8), or even hoststated work). Furthermore, the regular expressions are a powerful way to match input strings and to split the result into smaller pieces. Even though the regex(3) and regexp(3) APIs are very popular and probably the key feature of scripting languages like Perl, the implementation is very complex and a big enigma to us developers; it is a potential risk. And hence, I replaced the initial use of regex in the hoststated send/expect code with the very simple fnmatch(3) interface (fnmatch is also used for shell globbing, the advanced directory "wildcards").

So I decided to implement something on my own, something simple but powerful. I already had some code as part of my company's preinspection functionality to do generic TCP relaying with some additional protocol inspection. In fact, preinspection is meant to be an answer to the general UTM (Unified Threat Management) and deep inspection/intrusion prevention hype. I hate the idea of parsing the contents of every single packet payload, it is normally very slow, you have an incredible amount of false positives/negatives, and the protocol parsers are typically very buggy and a security nightmare on the firewall itself. So preinspection is more like behavior based blocking as we do in OpenBSD with pf overload tables, connection rates, max source connections, greylisting and greytrapping with spamd(8), and many other smart features.

Prevent an instrusion before looking into the packet details, leave the fancy UTM work to other systems behind the firewall. Who cares about the name of every single virus in the networks, as long as it's kept out of it!

The relay code was a good starting point for my planned SSL accelerator. In theory, such an SSL accelerator is nothing more than a TCP relay which listens for SSL connections on the one side and opens "plain" connections to the target system on the other side. In contrast to a protocol proxy, a basic TCP relay or SSL accelerator doesn't need to know any details about the transmitted layer 7 protocol, it just forwards any traffic between both sides. But you end up in one major problem if you use a relay as a SSL accelerator or reverse proxy: the target system doesn't know about the real client's IP address, it just sees incoming connections from the relay itself.

I needed to extend the relay to do a little bit of HTTP processing. The trick was to add an additional HTTP header to forwarded requests, the "X-Forwarded-For" header as used by some other HTTP proxies and non-free loadbalancers. But instead of hardcoding the X-Forwarded-For header, I implemented a generic way to add, remove, change, expect or filter HTTP headers. It is also possible to append values to existing headers, because it's very common to append the client's IP address to a comma-separated list in the existing header (a HTTP request may pass several proxies on its way to the target host).

protocol myhttp {
        protocol http
        append "$REMOTE_ADDR" to "X-Forwarded-For"
        append "$SERVER_ADDR:$SERVER_PORT" to "X-Forwarded-By"
        change "Connection" to "close"
        remove "Keep-Alive"

        # Various TCP performance options                                       
        tcp { nodelay, socket buffer 65536 }
}                                                                               

A perfect place to add the relay code was the existing hoststated. The existing framework of the daemon and the multi-process privsep model allowed to add independent processes handling the relay functionality while using the state information provided by the existing host check engine. And while I was already working with pyr on improving the hoststated for the upcoming 4.1 OpenBSD release, I was able to extend my TCP relay and SSL accelerator to become a real layer 7 loadbalancer. I simply had to implement some loadbalancing protocols like roundrobin, loadbalance, and hash (indeed, roundrobin and loadbalance got inspired from my existing work in trunk(4)) to be used independently from the pf redirection. The loadbalance protocol uses a hash over the source and destination IP addresses to select the target host and hash allows to feed specified HTTP header values into the hash instead.

table web {
        real port 80
        check http '/' code 200
        host $webserver1
        host $webserver2
}                                                                               

relay sslaccel {
        listen on www.myhost.com port 443 ssl

        # Relay protocol definition, see above                                  
        protocol myhttp

        # Layer 7 loadbalancing using src/dst hashing                           
        table web loadbalance
}                                                                               

And I went further than I expected... Some web applications use session IDs in the URL to track the users. For example, you may have a session ID like http://www.myhost.com/shop.php?page=foo&sessid=387874683268264826482&shop=bar I extended my relay code to be able to parse the variables in an URL, the GET-Vars, and to feed specified values into the hash. It allows to grab the sessid variable and to feed it into the computed hash for the loadbalance and hash protocols. This allows to do session-aware loadbalancing to a pool of HTTP servers, if your web application doesn't support any kind of internal clustering. And of course, you can also use the filter or expect rules on the GET-Vars.

The relay has been tested to be very efficient and I initially installed it as a SSL accelerator in front of a high volume web shop. Our systems are running on dual core Opteron servers with PCI express em(4) gigabit NICs, high end crypto accelerators, and using OpenBSD's amd64 multiprocessor kernel. We tested the SSL accelerator with up to 110.000 pf states, about 5.000 new SSL connections per minute using layer 7 loadbalancing and HTTP header rewriting, and nearly the same volume of concurrent HTTP connections using layer 3 pf redirection. After about six hours of testing we decided that the implementation is good enough for production use ;-)... We had to stop the test when we reached the limits of the backend webservers (pf's state and source connection limits will protected the servers in production). Amazingly, the loadbalancer was only up to 20% to 40% busy with a peak load of 4.0 and below 40MB user memory usage (SSL session cache disabled). While the relay code is designed to be non-blocking and single-process based as mentioned before, I use a configurable count of preforked relay processes to prevent any delays that can happen in the SSL handshake.

A drawback of layer 7 TCP relays is that you cannot provide stateful failover like we do with carp and pfsync(4). To improve the situation, I implemented support for the carp demotion counters as used by bgpd to force a quicker failover on error conditions. You can enable carp demotion of a specified interface group, typically the group called carp, to increase the demotion counter if all hosts in a table are down. An increased carp demotion counter will force the local system into BACKUP state to allow a graceful failover to a backup loadbalancer. Furthermore, a global carp demotion option allows to force the system into BACKUP state if hoststated exits or is beeing killed to reload the configuration. It will clear the specified group's demotion counter on startup and set it to 128 on exit.

A different thing, but a very good addition to the high performance loadbalancers, is mpf's implementation of IP-based loadsharing for CARP. Instead of running in the typical BACKUP/MASTER configuration, the redundant OpenBSD systems will compute a source/destination hash on the IP addresses to decide if they accept the incoming connections. Both systems will run in BACKUP and MASTER state at the same time and the load will be equally shared. I considered it as an option for the SSL accelerators, and tested the functionality with positive results, but the good performance of the primary relay didn't require to run a CARP "cluster" (But I hope that this new carp mode will be available after the 4.1 release).

Anyway, relay is more than just a loadbalancer! It also supports transparent proxying using NAT lookups, generic TCP redirection, and we're thinking about many other smart features in the future. A very simple patch to add URL white/blacklisting is on the way and you can do some nice tricks with TCP-based connections. And here we end up in a new trap - hoststated is a host state monitoring daemon, partially a loadbalancer but now it's getting a general-purpose preinspection thing? Should we change the name again? Probably not.

Nevertheless, the relay functionality and hoststated will be available in OpenBSD 4.1. It is still considered to be experimental, and I have some work on my TODO list, including additional cleanup, code review and improvement. The latest discussions on our mailing lists about possible reverse proxy solutions showed me that it was the right time to release something new, to carefully enter layer 7 in OpenBSD.

More to follow...

[topicblog]

<< Call for OpenOffice.Org testing! | Reply | Flattened | Expanded | Developer blog: deanna - systat sensors >>

Threshold: Help

Related Links
more by dlg


  Freak! (mod 15/45)
by marklar (203.11.81.235) (marklar_@hotmail.com) on Thu Feb 22 05:03:38 2007 (GMT)
  This is insane, you're about to put F5 out of business if you keep this up... good job!!
Nice to see the solid platform that OpenBSD has built makes this kind of development possible, I look forward to a HowTo because I'm itching to try it.
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]
      Re: Freak! (9/31) by Nate on Thu Feb 22 05:17:22 2007 (GMT)
        Re: Freak! (2/28) by Anonymous Coward on Thu Feb 22 05:40:52 2007 (GMT)
        Re: Freak! (11/37) by reyk on Thu Feb 22 05:52:37 2007 (GMT)
      Re: Freak! (-1/29) by Pierre-Yves Ritschard on Thu Feb 22 15:45:29 2007 (GMT)

  session ids (mod -1/37)
by Jeremy Huiskamp (kamper) on Thu Feb 22 07:09:03 2007 (GMT)
  Sounds like cool stuff. While some apps do use session ids in the url, is it not more common to put them in cookies? That's been my experience anyways.
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]
      Re: session ids (-1/27) by Anonymous Coward on Sat Feb 24 18:53:07 2007 (GMT)

  Re: Developer blog: reyk: hoststated(8) preinspection (mod 4/38)
by Henrik Kramshřj (90.184.69.92) (hlk@kramse.dk) on Thu Feb 22 07:55:54 2007 (GMT)
www.kramse.dk
  Nice article!

I just want to point your attention to Varnish high-performance HTTP accelerator which also help a couple of big sites with accellerating.

The project is not very old, and some of the readers that needs
these technologies might not know about it.
http://varnish.projects.linpro.no/

Actually I haven't tried it on OpenBSD yet, but it is on the ToDo list
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: Developer blog: reyk: hoststated(8) preinspection (mod -3/41)
by Pete (80.203.236.21) on Thu Feb 22 10:09:00 2007 (GMT)
  hi reyk,

nice work :-) I'm a long time pound user, but this new stuff looks very promising

is it possible to make sessions sticky based on cookie contents ? e.g. a pretty common cookie is JSESSIONID, based upon which I'd love to be able to tie to a particular backend.

also, what is the behaviour when a client is tied to a backend which is suddenly determined as unavailable ? is the session reset or is it transparently remapped to an alternative avaiable backend ?

thanks

/pete
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: Developer blog: reyk: hoststated(8) preinspection (mod 0/36)
by Matt Van Mater (69.255.1.181) on Thu Feb 22 13:28:13 2007 (GMT)
  This is great, I have been looking for a non-Apache, non-Pound, non-squid way of doing this natively on openbsd for some time now. Keep up the good work, i appreciate it.
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  A rose by any other name .... (mod 0/30)
by Anonymous Coward (128.171.90.200) on Thu Feb 22 19:29:38 2007 (GMT)
  "Should we change the name again? Probably not."

oh go on ...
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Multiple ports on same host (mod 5/43)
by Curis (199.6.38.58) on Thu Feb 22 20:06:19 2007 (GMT)
  This is a great feature but looking into the hoststated.conf(5), hoststated(8) and hoststatedctl(8) man pages I didn't see a way to test multiple ports on one host.

I'm thinking of my Rails project where I run a Mongrel cluster on ports 8000-8005 and would like to be able to check that they are still alive and also loadbalance between those 5 (as well as another machine that also runs the same setup).

Is this capable, or on the TODO list? Either way this is looking great and I can always just keep the two front facing Apache servers on port 80 I have now with proxy balancing between the Mongrel clusters.
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: Developer blog: reyk: hoststated(8) preinspection (mod -1/37)
by Anonymous Coward (68.104.220.48) on Fri Feb 23 04:18:28 2007 (GMT)
  Brilliant! What a development. Frankly I've been impressed with the load balancing capabilities of a simple rdr server pool but this L7 stuff is cool, especially at the performance level. Another ++ for the solid OpenBSD architecture and adherence to doing things the best way possible.

I hadn't realized that this was the case:

> You can also use the ARP balancing functionality of carp(4) but this requires to run OpenBSD on all the backend servers (this is nice but unfortunately not the typical case).

What's with the dependency on the arpbalance on the backend OS?

And what is the required way of enabling the hardware load balancer for use in the SSL acceleration? I have a "Hifn 7955/7954" and haven't been clear on how to use it for acceleration.
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: Developer blog: reyk: hoststated(8) preinspection (mod -3/31)
by Ed White (151.41.8.53) on Fri Feb 23 19:02:53 2007 (GMT)
  I would prefer to have 2 separated software to handle "host state monitoring" and "stuff".

  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: Developer blog: reyk: hoststated(8) preinspection (mod 1/33)
by reyk (212.202.20.246) on Sat Feb 24 01:00:55 2007 (GMT)
http://team.vantronix.net/~reyk/
  Update: The hoststated relay SSL mode was tested by a PCI DSS (Payment Card Industry Data Security Standard) audit with the result that I had to add two options to disable the old SSLv2 protocol and to reject any medium or low grade cipher suites. Good thing; now it seems to comply, you can run it as a SSL accelerator for e-commerce applications, and put you mind at rest...

        ssl { no sslv2, ciphers "HIGH" }

I also tweaked some other options, added the log action to track header/url values, and implemented a configurable listen backlog option for an improved performance with many concurrent connections.

  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: Developer blog: reyk: hoststated(8) preinspection (mod 2/34)
by David Steinberg (71.232.117.76) (dave@redterror.net) on Mon Feb 26 02:48:38 2007 (GMT)
http://www.geekisp.com/
  Awesome work! I was curious if there was any feature planned to enable actions to be conditional based on certain headers? Specifically with Pound I often do special backend routing based on the Host header - for instance with the following (rough) pound.conf snippit:
Service "railstest"
     HeadRequire "Host: railstest"
     Backend
        Address int01
        Port 8080
     End
End

Service "default"
     Backend
        Address int01
        Port 80
     End
End
The first chunk would pick up a request for http://railstest/ and the latter would serve everything else. I use this on a single IP, which lets me avoid DNS changes for backend handling changes. I didn't see anything in the man page about this, so please do apply the clue stick if I'm missed it. Again, great work, I'd dump pound in a moment if I could use hoststated (not that pound is bad, just OpenBSD rules)!
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: Developer blog: reyk: hoststated(8) preinspection (mod -3/29)
by Motley Fool (MotleyFool) (motleyfool@dieselrepower.org) on Mon Feb 26 20:37:15 2007 (GMT)
  Do you have any thoughts towards manipulating HTML on the fly? I have a video streaming server I'd like to provide controlled access to the outside world. It's web server only talks http. I initially setup squid reverse proxy to get to it, unfortunately the embedded server hard codes the server IP address using hidden values, so the ActiveX control can access the video stream. Rewriting HTML content on the fly could replace the internal IP address with the OpenBSD system running hoststated.

g.day
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  No Subject Given (mod -1/11)
by Harper (45.116.233.53) (zgtsqylc@eelmail.com) on Sun Jan 15 11:42:52 2017 (GMT)
Manning
  Moc se mi líbí ty moderní Dřevostavba, rekreační objekty, které se objevily asi v noblesních prostorách. Ty většinou podobají boxy ačkoli oni vypadají příliš dobré, aby to byla pravda, a já jsem zjistil, levnější řešení pro ně stejně.
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: Developer blog: reyk: hoststated(8) preinspection (mod 1/11)
by Blake Green (182.182.3.250) (ztyijnmv@tafmail.com) on Tue Jan 17 07:44:03 2017 (GMT)
  Finally a post sharing the solution to run OpenBSD as a fully-featured loadbalancer. I will share about it at students essay writing service reviews too. I always had issues with HTTP processing and the Open BSD, informative post.
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  No Subject Given (mod 0/12)
by Harper (128.199.33.160) (zgtsqylc@eelmail.com) on Sun Jan 22 15:55:49 2017 (GMT)
Manning
  Good write-up i must claim in addition to appreciate it with the facts. Knowledge is really a sticky theme. Even so, is one of many foremost issues your time period. When i get pleasure from ones write-up in addition to count on far more. hcg drops reviews
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  No Subject Given (mod 0/12)
by max (42.201.251.156) (davidwilcox09@gmail.com) on Thu Mar 2 12:23:19 2017 (GMT)
  Speedily this great site will certainly irrefutably always be renowned amid most writing a blog men and women, automobile careful content as well as testimonials. writer
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: Developer blog: reyk: hoststated(8) preinspection (mod 0/0)
by rozi (45.116.232.41) on Sat Apr 29 12:51:58 2017 (GMT)
  Amazing, this is great as you want to learn more, I invite to This is my page. หารายได้เสริม
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: Developer blog: reyk: hoststated(8) preinspection (mod 0/0)
by rozi (45.116.232.41) on Sat Apr 29 12:53:32 2017 (GMT)
  I like to recommend exclusively fine plus efficient information and facts, hence notice it: Send flowers to brazil
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: Developer blog: reyk: hoststated(8) preinspection (mod 0/0)
by rozi (45.116.232.41) on Sat Apr 29 12:54:59 2017 (GMT)
  In this particular article, you will see a summary, satisfy browse this post. gomovies
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: Developer blog: reyk: hoststated(8) preinspection (mod 0/0)
by rozi (209.197.30.86) on Mon May 1 20:11:40 2017 (GMT)
  Within this webpage, you'll see the page, you need to understand this data. his secret obsession
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

[ Home | Add Story | Archives | Polls | About ]

Copyright © 2004-2008 Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to April 2nd 2004 as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. Some icons from slashdot.org used with permission from Kathleen. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. Search engine is ht://Dig. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]