OpenBSD Journal

OpenBSD Benchmarked... results: poor.

Contributed by jose on from the read-this-very-closely dept.

James Herbert writes: "This popped up on Slashdot:

http://bulk.fefe.de/scalability/

Possibly biased against OpenBSD in style, but not fact...

The important quote is

"OpenBSD 3.4 was a real stinker in these tests. The installation routine sucks, the disk performance sucks, the kernel was unstable, and in the network scalability department it was even outperformed by it's father, NetBSD. OpenBSD also gets points deducted for the sabotage they did to their IPv6 stack. If you are using OpenBSD, you should move away now. "

Folks, read these results very closely and very critically. Performance testing is, in fact, considerably more scientific than this page makes it seem (graphs don't make it scientific, the approach is what matters). Note the following:

  • the installation routine is a matter of personal preference. the 1024 cylinder limit can be gotten around using GRUB, for example.
  • the kernel stability comment comes from a crash in one test, which doesn't indicate overall stability. (Though I have crashed several of my own systems with too many processes.)
  • the IPv6 stack comment has little, if any, foundation in this report.
All of that said, it does indicate what people have known and acknowledged for quite some time, that OpenBSD's performance isn't top notch because the project has focused on other goals. However, the material is clearly presented with a strong bias and agenda, so I encourage you to review it independently and learn about performance testing in the process. But this is also a great time to look into improving OpenBSD's performance with things like zero-copy TCP and improved memory management performance (to get back some of the overhead from memory protection schemes).

(Comments are closed)

Comments

  1. By jose () on http://monkey.org/~jose/

    as another person who sent this in said:
    [this is] part of work presented to Linux Kongress 2003 (http://www.linux-kongress.org/2003/index.html). Unfortunately the results are very bad for OpenBSD - but the testers methodology has to be put at question: comparisons between unstable development branches, and lack of resolving anomalous issues as part of the process.
    i couldn't agree more, and suggests a bias in the test design (was there even any review of the test design to ensure its validity? is this person qualified to "tune" these operating systems? do repeated runs of a program really qualify as testing?).

    Comments

    1. By Sacha () on

      This report bashes OpenBSD to bits. I wonder which tweeks he did to Linux and *BSD.

      Personally: I think he's a Linux lover.

      Comments

      1. By Anonymous Coward () on

        Common settings

        On all of the operating systems, I took the default settings. If I had to turn a knob to get PCI IDE DMA enabled, I did so. If the OS had power management support, I did not disable that. I enabled (and used) IPv6 support on all operating systems.

        Linux 2.4

        I benchmarked a stock Linux 2.4.22 kernel.

        Linux 2.6

        I benchmarked a stock Linux 2.6.0-test7 kernel.

      2. By David Gerard () on http://velvet.net/~fun/

        He does provide the code he used, so presumably people will be running these tests themselves on other systems and alerting the world to notable discrepancies.

        Comments

        1. By Anonymous Coward () on

          He provides the code for the web server and some IO libraries.

          He does not provide the code for the benchmark procedure itself.

          Comments

          1. By ajax () on

            He does not provide the code for the benchmark procedure itself.

            and i quote:

            ---
            The Code

            I used my experimental web server gatling to measure these numbers. All the benchmark programs are also part of the gatling package.
            ---

            if you're upset by these numbers, run the benchmarks, find the hot spots, and fix them.

            to all: please stop complaining about the tester or his biases. ad hominem attacks, while convenient, are a cop-out.

            the only really surprising result to me is the O(n) kqueue performance in OpenBSD for connect. it's not a huge problem but it's certainly strange. it would be interesting to see if switching to poll gives the same results as NetBSD.

            the other "interesting" thing is the general non-linearity of OpenBSD's performance (bind, mmap, request latency). this to me suggests that the OpenBSD code has not been written with fast paths in mind.

            as for the "Even Windows" comment: biases aside, 4 million cycles to read the first byte on a freshly mmap'd page is atrocious. it would be interesting to see how different OpenBSD releases compare here; i wouldn't be surprised if the W^X changes made a difference.

            Comments

            1. By Dom De Vitto () on

              No.
              If you're unhappy with these benchmarks, write other benchmarks,on OpenBSD and fry the competetion.

              That would be a better, more productive effort than screwing with someone elses's linux-oriented codebase.

            2. By tedu () on

              "biases aside, 4 million cycles to read the first byte on a freshly mmap'd page is atrocious."

              that's because he doesn't know what he's measuring. he measured time, and converted to cycles. the cpu did not actually spend 4 million cycles trying to handle a page fault.
              amazingly, if you look a little closer, you notice that converting cycles back into times (which they should have been reported it in, cycles is totally misleading) they are right about what a disk access costs. so he's reading from disk on openbsd because of a contrived benchmark that artificially enforces wrost case performance on openbsd.

              Comments

              1. By ajax () on

                if i understand this correctly, you're saying that openbsd defers pulling the page in from disk until it's actually read, basically punting the hard work to the page fault handler. (see the graphs: nearly instant map setup but long access times on first read, to me that looks like lazy mapping.)

                that makes more sense, but it doesn't explain how linux-2.6 manages to get O(1) performance for both maps and reads. in theory it's the same amount of work whether you do lazy or immediate mapping, lazy mapping just trades higher initial latency for lower memory pressure. somehow linux-2.6 beat the system.

                this is probably the explanation:
                ---
                [...] the benchmark starts by reading every of those pages once, so they are in the buffer cache. Then this benchmark takes the time it takes to mmap each page, and the time it takes to read the first byte of each page.
                ---

                linux is probably being less aggressive about evicting the file from the cache, where it appears openbsd is being extremely aggressive. again, maybe this is something to learn from. disk access is slow . if i'm right about the cache eviction behaviour, openbsd is forcing itself to do at least twice the amount of disk i/o for the same task. i know there exist sysctls for the vm thresholds in netbsd, they probably exist in openbsd too.

                again, look at the results, not the tester. the results seem to say that openbsd spends too much time talking to the hard disk.

                now to find out why bind() and connect() latencies suck. anyone know the answer to this?

                Comments

                1. By tedu () on

                  mmap'd pages do not go in the buffer cache. his whole premise that mapping twice guarantees a page is in the buffer cache is wrong on openbsd. ubc would fix this, but in the real world, it's not as staggering as the graph appears.

                  what is more aggravating is that instead of even attempting to ask "why?" he immediately draws inaccurate conclusions. when you run a test and notice that one result is orders and orders of magnitude out of alignment, you take a minute to recheck your test and assumptions instead of just saying "wow, that sucked."

                  do you think anybody could use openbsd as a webserver if it really were 10000 times slower? NO. so if your test indicates such, you're missing the larger picture.

      3. By Anonymous Hero () on

        It is no secret Felix von Leitner is a ''Linux lover". It is no secret he doesn't like OpenBSD's IPv6 implementation either. Just search a bit on several mailinglists, for example Qmail mailinglists where he had discussions with Henning Brauer.

        Why can't the official IPv6 behaviour be enabled in OpenBSD? What's wrong with that? What's wrong with choice and adoring standards?

        I had the problem with too many processes and crashes too on my previous OpenBSD installations. No problems with -current as of yet.

        As for FreeBSD 5.1: this isn't known as 'stable' yet he should have tested FreeBSD 4.8 as well, because of that.

        To say he doesn't like OpenBSD and state that's the reason OpenBSD performs bad according to this test is a fallacy. Just ignore the flames.

        For example: Even Windows would probably outperform OpenBSD. This is not necessary. Agreed.

        I hope the OpenBSD people will not feel attacked or been emotionally put to ''he's a Linux lover therefore thid doesn't count''. If you don't believe his analysis then please do your own. Else, see this is positive criticism and try to improve OpenBSD. We all benefit from this.

        Peace.

        Comments

        1. By Anonymous Coward () on

          Very good reply, you have my full agreement.

        2. By Anonymous Coward () on

          I agree with you on most of what you said; however, I have to take a bit of an issue with the "positive criticism" bit.

          Although it doesn't mean that the performance benchmarks were invalid, the methodology (e.g., testing both stable and development branches of all the OSes in question [as he did with the Linux kernels, so he can't be accused of being entirely ignorant] would have made it a much better test) seems designed to give the result that he wanted beforehand.

          Also, mingling a potentially valid criticism with hyperbolic vitriol isn't going to improve the reception one gets.

          Lastly, I have to wonder who gave the go-ahead for this minimally-useful propaganda to be presented at a major Linux forum; I mean, I'd think that they'd want a little more from presenters there. Of course, I've never been to the Linux Kongress, so take my rant with a truckload of salt.

        3. By Lincoln Rutledge () lrutledge@lnlattorneys.com on lnlattorneys.com

          I have to say that anyone that attacks OpenBSD instead of just showing results should not consider being taken seriously. If this guy runs his own 'Leanux' tree, obviously his kernels are going to work better with his user space software.

          For example, I could post my own benchmark that says that Red Hat Linux performs awful on my laptop because the installer requires 64MB of RAM and my machine has 24MB. And even the RULE installer dies because I have a wierdo chipset. This has nothing to do with performance once the machine is running and is irrelevant.

          When this guy knows how to use UKC and the booter he might change his tune. I use floppy32 all the time as a rescue disk; I recently installed NT4 on a laptop with no CD-ROM using it. The genius could have bootstrapped his machine many different ways without having to use an undersized partition. Although I would like to see a calculator function added to disklabel so I don't have to scribble long addition on the edges of used post-it notes whenever I install...

          What is the point in this exercise? If he has wanted to prove that his tree performs better with his test software, he has done it.

          I have *never* had an OpenBSD kernel die on me, which I cannot say for FreeBSD or Linux (I don't think NetBSD ever fell down either...). Obviously not everyone in the world has had this experience, so what? It is irrelevant.

          Besides, does Leanux run on 68K Macs? Sparc? Fffft.

          Comments

          1. By Anonymous Coward () on

            Besides, does Leanux run on 68K Macs? Sparc? Fffft.

            Linux runs on more architectures than OpenBSD.
            http://www.linuxdevices.com/articles/AT8728350077.html
            http://www.linuxdevices.com/articles/AT8498487406.html
            http://www.tldp.org/HOWTO/User-Group-HOWTO-1.html
            I believe Debian is the mainstream Linux distribution that supports the most architectures: http://www.debian.org/ports/

      4. By Anonymous Coward () on

        I think you're a troll. Did you actually read the article?

    2. By Anonymous Coward () on

      is this person qualified to "tune" these operating systems?

      If he found the installation of OBSD difficult (I find OBSD's installation to be the easiest of all OSes), I'd say "no."

    3. By Security First () on

      Dunno about a bias, but as far as I see it, there are at least three very important aspects to consider when choosing OS. One of them is security, another is stability, and the third is performance. Of course, it would be very nice if an OS would excel in all of these three areas, but, unfortunately, one often has to make choises. For desktop use I would probably choose something that excels in performance, but for server use I place my preference first on security, second on stability, and performance comes only as my third consideration. This benchmark test doesn't change at all my opinion that OpenBSD is the most secure server operating system available at the moment and that it is also one of the most stable - simply because it is a BSD. ;-)

      Comments

      1. By aliquis () on

        "Of course, it would be very nice if an OS would excel in all of these three areas, but, unfortunately, one often has to make choises."

        Have you forgot about Microsoft Windows? =D

  2. By Clint () e@mail on www.betterthanyou.com

    I would just like to point out that I could care less if OpenBSD is "slower" than Linux or any other operating system. Clearly the design goal of OpenBSD is to make a usable yet secure OS. If there were to be some sort of "Security Benchmark" OpenBSD would favor higher than others. So should it really be a surprise that OpenBSD doesn't fare as high on a "Performance Benchmark" considering there are other flavors that focus on that instead of security?

    You could show me a test where Win98 was 200 times faster than OpenBSD, and would I switch to it on my firewall and web server? um.. no.

    One more comment. the "the installer sucks" part of this is what possibly takes the most credibility away from the author. There is no better installer than the Openbsd installer, its faster (*gasp*) and easier than any linux or *bsd or windows, or hell even netware or commercial unixes that i've ever seen. Fast, easy, efficient, and gives you only what you need. Perfect.

    The closing remarks of the quoted paragraph of "if you are using OpenBSD, you should move away now" is obvious flame bait. If you where to look at security "features" and track records, and security was your goal, you might say "if you are using anything but OpenBSD you should move away now".

    If performance is the most important thing, would you really be considering an OS that doesn't have SMP support in the first place?

    Comments

    1. By Anonymous Coward () on

      One more comment. the "the installer sucks" part of this is what possibly takes the most credibility away from the author.

      It's so hard to make a little curses based installer. Let's say it does just the same actual installer does... but a bit more eye friendly.

      This would hurt OpenBSD? Dunno.

      Comments

      1. By Joris () nimadeus@pandora.be on http://users.pandora.be/nimadeus/

        OpenBSD isn't about fancy stuff, which is why it is so great. It's plain and simple just like it should be.

      2. By Clint () e@mail on www.betterthanyou.com

        Would it "hurt"? only in that it adds things that are not needed. Would it help? only if you're an idiot that can't follow a shell script prompting you with questions. (Especially since the faq and cd liner notes walk you through each question and what to do).

        And if you want candy...

        http://www.gobie.net/

        Comments

        1. By Anonymous Coward () on


          So, let's see if I understand you correctly. You say if anyone has trouble with installing OpenBSD the person is an idiot?

          (can't stand criticism, check...arrogant, check...hmm either an Apple fanboy or an OpenBSD fanboy, letsee...no mention of Jobs...believes one must be born with knowledge, check)

          Congratulations, you are a certified OpenBSD fanboy. Congratulations, now go back to being irrelevant.

      3. By zoc () on

        I like eye friendly girls bodies, not installers ;-)
        Actually i want to say it would be a waste of floppy space. Really there is no need to have gui installer, and it's well documented how to use it, so all people need to know is to read.

        Comments

        1. By .jon () on

          Heck, I do not care about graphical installers or not.
          OpenBSD still is not installed on my 120GB hda since the label/fdsk utility does not understand my disk-layout. I tried all available options, auto, manual, useing different BIOS presets and so on. Not what you would expect in late 2003. I would really hate in being forced to install Linux on that server, just because OpenBSD 3.3 does not handle modern (and basic!) hardware.

          But then, Linux ain't that insecure either, with the grsecurity pathces applied...

          Comments

          1. By aliquis () aliquis@link-net.org on mailto:aliquis@link-net.org

            Don't go there! =D

            (short version in the end)

            I, like most people in here, like the OpenBSD installer, it's the fastest I've ever used and so simple aswell. To bad it can't handle that 1024 cylinders thing thought.

            Anyway, for servers and machines I want to get up quickly I prefer openbsd, because I also think it's easiest to upgrade and configure. (good work with http://www.openbsd.org/stable.html guys, and the faq, or, well, good work with all the documentation for openbsd, it rocks =D).

            However, I've tried NetBSD aswell and think it's the BSD I like most, it seems so stable, well documented and "thought thru".

            For the moment I run FreeBSD 5.x thought since neither of netbsd or openbsd want to use my promise "fake raid" controller (fasttrak 133 bla bla integrated on my motherboard). OpenBSD can use it as a normal ide controller.

            Anyway, that wasn't the important things, the important things was that if openbsd doesn't work good for you install free- or netbsd since they are just as great aswell. Back in the days I used debian which are way to outdated now adays, recently I tried gentoo but came back to *BSD since it really wasn't my cup of tea, only good thing with it is the USE flags but you kind of have those in BSD aswell. And we shouldn't talked about RPM-based distributions like RedHat which I have to use "at work". It's so frustrating then some small tool like configure-this-and-that-program requires gtk and a lot of other packages, or then this-and-that-library aren't compiled with that-function which you wanted so you have to compile it yourself anyway but package-xyz requireds the rpm version of that library so you have to keep it and have both installed anyway.
            What really makes the difference between *BSD and Linux is the documentation thought, they don't come close here (probably due to all documentation written by some user or coder for an application instead of bye some package maintainer for the current distribution).

            short version:
            1) *BSD owns Linux
            2) If OpenBSD doesn't install, use another *BSD.

      4. By Anonymous Coward () on

        And I'll be able to use this fancy installer without problems
        on serial console based systems like headless UltraSPARC systems?

        Besides, there is no room to squeze more stuff onto the floppy disks...

        Comments

        1. By Anonymous Hero () on

          So, because you want to use this on your UltraSPARC system, other users who do not use a SPARC cannot have another choice?

          I'd say such installers can coexist together. Better yet, provide one library with different frontends. Eventually with 2 floppies. Wow, big deal.

          Then again, i'm not a coder.

          Comments

          1. By Wim Vandeputte () wim@kd85.com on http://overworked-3.4-packageshipping.com

            I'm honoured you consider me to be the trendsetter in project goals, but let's be realistic ;-)

            If you look at the platforms OpenBSD supports, UltraSPARC is not the only one to benefit from flawless serial console support. hp300, alpha, mvme68k, vax, ... we don't live in a PC i386 world. Heck, half of my Intel boxes don't have a VGA monitored hooked up anymore, especially now that the better BIOSes support serial console redirect, even in the pre-OS stage.

            I'd say having two coexisting installers is going to be a real burden for two reasons:

            1. it duplication of work and a waste of time, if you take in account what our installer does:
            a. partition disk
            b. ask for network topology
            c. determin which of the major system parts need to be installed
            d. answer a few multiple choice questions
            Who is going to do the work? You? Sure not me. Or krw.

            2. there is no room on the installdisk to add more cruft, unless we give up some drivers. You want to sacrifice functionality for eye candy? Oh no, wait, let's get two floppies. Sorry, I've always considered the single boot floppy OpenBSD has as a very elegant solution. Multiple boot floppies is going have certain machines run into resource issues like not enough memory to load a fat kernel + bloated RAMdisk.

            Going from 1 elegant boot floppy to 2 floppies IS a big deal. If you can't see the elegance and the practical issues, you defenitly are not a coder.

            Best regards,

            Wim. (back to licking envellopes)

            Comments

            1. By Anonymous Coward () on

              While on the one hand I wasn't enthusiastic about OpenBSD's installer (I tried it some years ago and failed miserably;) and on the other hand I don't ask for a graphical installer I think s/he meant not two necessary boot disks but one "install without gui" and the other one (or perhaps two - which shouldn't concern you anyway:) "install with gui".

              Comments

              1. By Anonymous Hero () on

                That's indeed what i meant. Personally i don't ask for it either.

                If you allow me to draw this analogy with NetBSD, it's been a while
                since i've used NetBSD, but when i last used it had an ncurses
                installation GUI. They provide several installation disks it seems:
                ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-1.6.1/i386/installation/floppy

                I do not see why it is impossible to allow the current users to stay
                with this 1 floppy solution while you also allow multiple floppies for
                people who'd like to have a GUI installation, without harming the first
                group who can only use the first solution on non-x86 hardware.

                Comments

                1. By tedu () on

                  who do you suggest build and test all these multiple floppy images? how will you keep them in sync? who will be responsible for making sure that they always fit? who will check each one for room after a new driver is added?

                  Comments

                  1. By Anonymous Hero () on

                    Who i suggest to build and test all these (2, perhaps) floppy images? Whoever wishes to do so. I'm not anyones boss.

                    It could be a 3rd party who then posts his/her work op Deadly and/or mailinglists. After s/he tested it him/herself, Deadly users can do so. Seems to be a nice moment, taken OpenBSD 3.4 is almost out. It can then remain a 3rd party alternative, eventually linked from at the FAQ, or it can become an official part of OpenBSD which means problem #2/#3/#4 are a problem when there's no common interest in this.

                    He/she could also be an official developer too, if anyone of the OpenBSD developers is interested. Again, it requires time and problem #2/#3/#4 remain.

                    Taking into account OpenBSD and NetBSD are quite similair, i think the NetBSD floppies are a good start.

                    #2: Do you really think that costs a lot of time? I don't know how the current floppies are made. I think a simple shell script could copy the correct drivers over to each image, leaving both install scripts mostly alone.
                    #3/#4: Making sure they always fit isn't according to my experience with creating my own floppies using Dietlinux hard nor time consuming. I've just tested boot1.fs and boot2.fs from NetBSD 1.6.1 i386. It loads a full 1.44MB floppy first. After that it loads the second, which is almost 1MB big. After those 2 are loaded, it boots up. There's still room to add more drivers or extend it otherwise on the second floppy. Therefore one only needs to check floppy number 2 for room. If that one's full, move to a new one.

            2. By Adam () on

              What motherboards are you using that let you redirect the bios to serial? I want some.

              Comments

              1. By anomdebus () on

                I think most Serverworks bios mainboards support this feature.
                I have a Tyan Thunder Le-T that does this:
                http://www.tyan.com/products/html/thunderlet.html

              2. By Wim () on

                Wel, it's been a while since I've bought any i386 machines, the last batch we bought were plain Intel manufactured L440GX+ server boards. I think the L440GX+ can hardly be considered to be top of the line these days (built year 2000), but I guess any serious Intel server board will have them now.

                I think I've bought an athlon machine that has the same feature, but I can't remember the exact type number. So there must be others out there.

                Since then I've only been buying Sun equipment (and Soekris, that counts for serial only too ;-)

      5. By Charles Hill () chill@herber-hill.com on mailto:chill@herber-hill.com

        Everyone is missing/ignoring the point on the installer.

        His main gripe wasn't a pretty GUI, but the inability of the OpenBSD installer to load /bsd when located above the first 8G or 1024 cylinder limit.

        His problem was multibooting his system with half-a-dozen OSes and OpenBSD not wanting to boot because its partition was above the 8Gb mark.

        Like it or not, GNU GRUB handles partitions that start above 1024 cylinder, and higher than 8G as long as your BIOS understands LBA mode.

        According to the FAQ, the OpenBSD bootloader does not. http://www.openbsd.org/faq/faq14.html#LargeDrive

        Comments

        1. By mirabile () mirabile@bsdcow.net on https://MirBSD.BSDadvocacy.org:8890/

          Like if GNU GRUB would care about supporting the
          BSDs, or vice versa.

          Like if the OpenBSD biosboot(8) would support LBA.

      6. By Anonymous Coward () on

        Then write one, stick it on a floppy and see if anyone wants to use it.

        // hdw

      7. By Aaron () on

        When I first used the curses-based installer in FreeBSD and NetBSD, I found them to be very unstable and much harder to use, especially since the key bindings are not entirely intuitive. Once I learned how to use the disklabel utility, I've never found a better installer than OpenBSD's. It always works right, and does exactly what I expect and nothing more.

        One thing about the OpenBSD installer that *REALLY* impresses me, and that I can't get enough of is that, for every i386 system I have ever set up, I was able to accomplish the entire process using ONLY 1 FLOPPY. No other *complete* OS the size of OpenBSD has this. Net and Free both take 2, and most linuxes take 2 as well (other than distros that fit on one floppy themselves...)

        Also, because of the way the installer on OpenBSD is designed, it is [er, I'm sure it is, though I won't try it until my 3.4 disc arrives] very easy to modify the startup scripts (since there's no curses app to rip apart) to customize the installation per your site. So you could easily create a floppy that gets an IP address by DHCP, loads some values, like the hostname and maybe disklabel, and the installation sets, from a central FTP location, and automates most of the install. So you could unattended bootstrap installs for hundreds of systems with one floppy.

        Really, I think the 1.44MB floppy is a pretty good measure of the installer. If it won't fit on one floppy, I would feel that perhaps you have too much fancy stuff included.

        Please don't flame me for being such and OpenBSD-installer-zealot. I just like the installer very much, and I feel that if OpenBSD broke down and went curses-based (or, even worse, X-based! :-P) like everybody else, the entire community would lose out on this wonderful option. Don't blindly follow the majority just because they're the majority. Stick to what YOU believe in.

    2. By Anonymous Coward () on

      I would guess he means that people who uses OpenBSD for things it isn't designed for (High scalability) should switch.

    3. By Chris () on

      I think network performance in OpenBSD should be a big concern.

      Yes, if I want an extremely high performance machine I may lean towards Linux or FreeBSD, but it's the OpenBSD firewall/router in front of it that I'd expect to protect it.

      If OpenBSD falls over as easily as the author of these benchmarks suggests, then it makes one of OpenBSD's strength kinda moot. I think a good/reliable/fast network stack is pretty important for something you use as a firewall.


      Personally, I've not run into any problems yet though.

    4. By mirabile () mirabile@bsdcow.net on https://MirBSD.BSDadvocacy.org:8890/

      The installer is really okay (having tried to get
      a mere _shell_ on a FreeBSD 5.1 CD, and being unable
      to install NetBSD from the 1200K floppy because of
      an error in openpty(), I think I can say that).

      The boot loader just sucks.

    5. By Randy Poznan () on www.hackuser.com

      I like the HTTP server install, that is awesome! I leave it on apache or tomcat, set up a path and im installing via the network in minutes with no ftp setup or boot server setup!

      BSD disk setup is simple, it makes it's sub partitions in one physical partition. On x86 you 3 leftover physical partitons to use for other OS's.

      I use OpenBSD for its security and highly innovative features like the pf interface and secure/encrypted filesystems, and buffer overrun protection. Great work guys.

  3. By Felix von Leitner () felix-benchmark@fefe.de on http://www.fefe.de/

    Funny you mention grub, because that was my first idea as well. I couldn't for the life of me get grub to boot the system. grub did it's part and loaded the kernel, but the kernel immediately dropped into the kernel debugger and said the boot loader was too old. I was using the current CVS version from grub.

    There is some more detail about the IPv6 issue in the PDF slides. It did not adversely affect the benchmarks, but it forced me to implement workarounds in the benchmark code. If you want testers to look favourable at your code, don't make them waste their time to work around issues like this that are motivated purely politically.

    The OpenBSD kernel actually crashed more than once, but all the crashes were in the fork benchmark. There actually is one more benchmark that didn't make it into the graphs: pthread. OpenBSD has pthread support, which lifts it above NetBSD and FreeBSD (which is said to have pthread support in 5.1, but the library is not called libpthread and my test program core dumped when linking to it). So I'm not saying that OpenBSD is not innovative. I'm saying that it fails to convey a feeling of technical superiority to me as a user, not even over Windows 2000. And user happiness (and a feeling of "snappiness") is even more important than actual scalability. Most people will never run into situations that force them to have 10000 open connections, but all users will notice if their system feels snappy. To me, OpenBSD doesn't.

    Comments

    1. By Alejandro Belluscio () baldusi_NOTWORKING@hotmail.com on mailto:baldusi_NOTWORKING@hotmail.com

      I guess the patch just stayed a patch. Or hasn't been implemented yet. I've sent you a mail to you stating the difference in OpenBSD world between RELEASE, STABLE and CURRENT. The last one is usable the two months previous to RELEASE. And crazy to use the three monthes post RELEASE. Which is exactly the one you used. But anyway, I guess the 1024 cylinder limit patch wasn't applied yet. I'll just try to ask to misc.
      The other point that I would rise, is that you actually tested mmap which in OpenBSD is heavely checked due to the use of ProPolice, WxR, randomization of pages and so many other things. So yep, it will suck, heavily. In fact seeing your graph you see a certain stocastic component in the latency of the pages accesses. Which would correspond to doing some random work there. I might be wrong.
      Regards

      Comments

      1. By Marc Espie () espie@openbsd.org on mailto:espie@openbsd.org

        >The other point that I would rise, is that you
        >actually tested mmap which in OpenBSD is heavely
        >checked due to the use of ProPolice, WxR,
        >randomization of pages and so many other things.
        >So yep, it will suck, heavily. In fact seeing
        >your graph you see a certain stocastic component
        >in the latency of the pages accesses. Which
        >would correspond to doing some random work
        >there. I might be wrong.

        Oh yes, you are completely wrong.

        Comments

        1. By Alejandro G. Belluscio () baldusi_NOTWORKING@hotmail.com on mailto:baldusi_NOTWORKING@hotmail.com

          So the problem with the performance has nothing to do with ProPolice or W^X? Has this to do simply with unoptimized code? Do you percieve this as a problem?

          Regards

          Comments

          1. By Anonymous Coward () on

            Some of the 'problems' (cases where you have a O(1) implementation on FreeBSD/Linux and OBSD does not) are not likely to be caused by ProPolice or W^X. Yes, this is a supposition on my part (i.e. I don't fully know what I'm talking about), but I can't imagine the OBSD developers putting in place code that would actually increase the computational complexity of operations in fundamental subsystems. ProPolice etc. probably just extract some constant penalty.

            More likely the differences are due to the choice of algorithms/datastructures used in or by the functions in question. Linux/FreeBSD are trading memory usage for performance -- and OpenBSD is still using more memory conservative code (perhaps a design choices that may now dated, perhaps they are sub-optimal algorithms, or perhaps just a different design choice!). You have to keep in mind that Linux and FreeBSD people both spend a great deal of effort working at such improvements, and I'm sure some of the changes are major undertakings for benefits most people will never see the full potential of outside of a piece of paper (i.e. how often do you have 10000 connections open).

            Comments

            2. By Anonymous Coward () on

              I, myself, applaud OpenBSD for offering the option of a more memory-conservative approach, even considering all the memory and performance cost additional security could imply. 3.2 runs on my 486 notebook with 8MB of ram, and that's just fine for me.

          2. By tedu () on

            mmap sucks because the benchmark sucks. try it again with anonymous mappings and see what happens.

    2. By mirabile () mirabile@bsdcow.net on https://MirBSD.BSDadvocacy.org:8890/

      I think the person mentioning GNU GRUB was trying
      to say that /boot is LBA capable, and you could
      have used grub to load it.

      I just wonder how you would be able to do it with
      the first-stage boot loader (partition boot record,
      a.k.a. /usr/mdec/biosboot (or /usr/mdec/pbr_ffs in
      MirBSD)) is not LBA capable?

      Note that you can use the MirBSD pbr_ffs and
      installboot for OpenBSD, too.

    3. By Todd T. Fries () todd@fries.net on http://FreeDaemonConsulting.com

      You complain about IPv6 features you want, but it is truly not that hard to bind to a v4 and then a v6 socket. sshd, ftpd, bind, sendmail, apache, .. the list goes on and on of daemons that correctly deal with this situation.

      However, the above misses the real point entirely.

      There is a word known well to OpenBSD camp'ers. Pro-active. In OpenBSD, this means one does not require an existing exploit to believe an idea is bad and act accordingly.

      I encourage anyone even thinking IPv6 needs help in OpenBSD to realize that the split stack is a proactive security concern. I personally fully support Itojun and his stance with regards to this issue.

      For further reading consult:

      ... and perhaps other links from Itojun's list of technical papers/drafts he's written: http://www.itojun.org/paper/techpaper.html .

      Comments

      1. By mirabile () mirabile@bsdcow.net on https://MirBSD.BSDadvocacy.org:8890/

        I respect itojun-san (and even more since I learned
        what he does for Free/NetBSD too), but I still don't
        like this socket idea not being available as a
        sysctl-enablable option.

        You mention bind and apache (which is IPv6 for me),
        but you forget about e.g. djbdns (it was hard).

        Not all software authors don't lack the cluefulness
        to IPv6-enable their software.

        Greets

    4. By Norbert Koch () on

      Just one thing about pthreads:
      FreeBSD has userland pthreads since years.
      A look into FreeBSD's online man pages
      would have helped here.
      In 5.1 you can alternatively link to
      the experimental libkse or libthr libraries
      allowing kernel based m:n and 1:n
      scheduling.
      And - if you really like to - you
      could also install a port of the
      linuxthreads system.
      That means you have 2 thread packages
      under FreeBSD up to 4.X and 4 packages
      under 5.X.

  4. By Anonymous Coward () on

    I thought it was pretty funny that the unstable branch of FreeBSD schooled OpenBSD, NetBSD, and Linux in most tests. It's a shame they didn't include 4.8 or 4.9rc2 in there as well to compare against Linux 2.4.

  5. By netchan () deadly@NO.SPAM.netchan.cotse.net on mailto:deadly@NO.SPAM.netchan.cotse.net

    In the end I had so little space that I had to copy the logs over to my desktop box after each benchmark, otherwise the filesystem would be full. vs. disk performance sucks

    Aren't these two related?

    netchan

  6. By Anthony () on

    OpenBSD lost most of the benchmarks, but I really don't see this as news. Given that it doesn't even have SMP support, I'm not surprised that other optimizations haven't been incorporated yet. That accounts for the cases where it appears that an algorithm is at fault.

    The other pretty obvious thing is that OpenBSD does everything safely, and that basically means testing a lot of things that the other OSes don't test. That does bad things to the pipline, especially if the results of previous branches have been dropped from the cache.

    That being said, in my experience OpenBSD easily outperforms Linux with NAT stuff. That's based on throughput and latency on my XP and OS X machines behind Slackware 9.0 (I think it was kernel 2.4.18?) and OpenBSD 3.3, running on the same hardware. I'm not very knowledgable about these things, but if I had to guess I'd have to say it's because PF rocks.

    Apart from a few ssh connections per day where time spent on authentication overwhelm the effects of the OS, the only thing the machine does is NAT. It's also easier to maintian.

    You use OSes for what theyre they're good at. Linux and FreeBSD are famous for being extremely good at server tasks. These benchmarks don't surprise me.

    Comments

    1. By Anonymous Coward () on

      If you think SMP support is the solution to these lackings, you really need to go back and take a basic OS course.

      Comments

      1. By Chris () on

        That's not what he said. He used the lack of SMP support as an example to indicate that OpenBSD isn't very performance/scalability oriented.

      2. By Anthony () on

        "Given that it doesn't even have SMP support, I'm not surprised that other optimizations haven't been incorporated yet. That accounts for the cases where it appears that an algorithm is at fault."

        In case I wasn't clear, I said that I'm not surprised other optimizations haven't been done when the big one (SMP) hasn't been done.

        I was wrong on one thing, SMP support isn't an optimization, it's a feature. If you want to insult me for a mistake I actually made, be my guest.

    2. By root () on

      The other pretty obvious thing is that OpenBSD does everything safely, and that basically means testing a lot of things that the other OSes don't test. That does bad things to the pipline, especially if the results of previous branches have been dropped from the cache.
      What the fuck are talking about?!?!?!?!

      Comments

      1. By root () on

        What the fuck are talking about?!?!?!?!


        Oops. Excuse me for bad language.

  7. By Anonymous Coward () on


    Despite the flaws in the exercise, it does seem clear that obsd has been given notice that there are areas for improvement. While obsd is focused on security, it can't ignore providing the right sort of balance with improvements in other areas. It just seems that as part of deciding on work to be carried out in future releases, that this exercise may be useful to know where to focus some of the effort.

    Comments

    1. By Anonymous Coward () on

      Well the author was clearly prejudice, no one ever promised OpenBSD was the fastest. OpenBSD is the first OS to promote security and I still remember when everybody was happy transmitting clear text passwords to Linux in Telnet and FTP.

      But hopefully the advocates won't get their panties in a bunch. Instead, use a negative as a positive. Improve performance if it is ideal. Better yet, be a leader and push high availablility technologies like AIO. Remind the Linux advocates why we put "Open" in front of everything!

  8. By FenderQ () on

    I think to get the best out of OpenBSD you really have to know what your doing. I think this person was just frustrated because they actually had to read and/or learn something new.

    I personally love the OpenBSD install. I have had great kernel stability throughout my history in using OpenBSD. I also wanted to note that this person also checked out -current, which is always in the middle of development and they were testing for stability? lol.

    I do not hold any value in a scientific review which rates things using "sucks" as a classification.

    This review had no impact on my views towards OpenBSD. OpenBSD does what I need it to do and it does it perfectly.

    That is all I have to say about that :-P

    Comments

    1. By Anonymous Coward () on

      *groan* Yeah, because OpenBSD couldn't POSSIBLY have any shortcomings. Attitudes like this will make sure that OpenBSD performance stays complacent and unimproved.

      Comments

      1. By Deadmeat () on

        OpenBSD has it's shortcomings, lack of SMP support being the biggest one in my view. SMT and multicore chips are going to become commonplace pretty soon and no SMP is going to bite but that is another discussion. In my experience of running OpenBSD (2.9 onwards, always using the most recent release -stable branch) The installer has been simple and trouble free and the kernel has been very stable. Perhaps the report author would like to submit bug reports to get any installer/kernel problems fixed instead of spouting unscientific results as 'research'. That applies to all the systems tested not just OpenBSD.

        Comments

        1. By FenderQ () on

          I agree.

    2. By Anonymous Coward () on

      Oviously the guy who did it is a complete moron! I've never seen anything so stupid.

      Besides, the OpenBSD installer rocks! I wouldn't want it changed for anything! Once you realise how simple it is, that's really what it is! Simple, nice and quick to do...

      Comments

      1. By FenderQ () on

        Exactly my thinking as well. ;-)

  9. By Ashcrow () ashcrow@phreaker.nosp44m.net on mailto:ashcrow@phreaker.nosp44m.net

    I hate to say it but this study will turn CTO's away from OpenBSD. We all know OpenBSD does a kick ass job in security and stability, but this report makes it sound like a piss-poor OS. If I would have read it years ago I might not have even given OpenBSD a try.

    Comments

    1. By Anonymous Coward () on

      well, who is this "Felix von Leitner" any way?

      Comments

      1. By Anonymous Coward () on

        He isn't that unkown at all; if i remember exactly he has some projects on freshmeat as well (dietlibc?). At least I knew his name before already..

        Comments

        1. By Anonymous Coward () on

          But hey, this is 'experimental' code, can we realy trust the benchmarks ? and there are two different versions of the libs, did he use different one on linux?


          ---
          The Code

          I used my experimental web server gatling to measure these numbers. All the benchmark programs are also part of the gatling package.

          You can download gatling via anonymous cvs from:

          % cvs -d :pserver:cvs@cvs.fefe.de:/cvs -z9 co libowfat
          % cvs -d :pserver:cvs@cvs.fefe.de:/cvs -z9 co gatling

          libowfat contains my implementation of the IO API, gatling is the webserver. You need to build libowfat first. If you are using Linux, also check out

          % cvs -d :pserver:cvs@cvs.fefe.de:/cvs -z9 co dietlibc


    2. By llk () antoine@uklinux.net on mailto:antoine@uklinux.net

      Luckily for me, I tried it just a few days ago, but the lack of hardware support stopped me from trying again after the install crashed a few times.
      Oh btw, I am enough of a techie to find my way around non-gui installs, fdisk and others, but I am sorry to say that I wasn't impressed. I really wanted to give OpenBSD a try, I though FreeBSD's install was quite bad too, but it was nothing compared to this.
      I guess I'll try again when 3.5 comes out...
      Please don't flame!

      Comments

      1. By Adam () on

        The whole entire installation process is documented on the website. It holds your hand and walks you through it step by step. You don't need to wait for 3.5.

        Comments

        1. By Anonymous Coward () on

          Convenient when you are installing the machine that is going to be used to get the internet connection.
          Sorry, not convinced.
          I believe text installers can be better/faster/easier than GUI ones, but OpenBSD does not cut it.

          Comments

          1. By Not an Anonymous Coward () lrutledge@lnlattorneys.com on lnlattorneys.com

            The install 'crashed'? Did it lock up your machine? Did it not boot up all the way?

            Sometimes setting up a machine with any OS is a chore.

            Either you want to do it, or you don't. The quality of the installer has nothing to do with your level of determination to run an OS.

            Also the mailing lists are very helpful for such issues.

          2. By Adam () on

            OpenBSD and NetBSD have the best installers I have seen of any OS. I'd give OpenBSD's a slight edge because it fits in a single floppy. If you can't manage to do the install, then there is something broken, and you should report the problem, or you are dumb.

          3. By Deadmeat () on

            Is it such a intellectual leap to just RTFAQ and even print a few bits out for reference before you do this to your internet connecting machine? Sheesh.

        2. By Mike () on

          Totally agree... I switched from Win world to

          OBSD 3.3 and have no problem with install

          procedure.. none.. really.. so When I (totall

          stupid win user) was successfull, there

          must be a mistake in mind...

    3. By Anonymous Coward () on

      If CTO's are impressed by wordings such as 'OpenBSD sucks', they shouldn't be CTO's in the first place.

    4. By Anonymous Coward () on

      good

      who wants them?

    5. By djm () on

      If your CTO is so reactive as to base development or deployment decisions on a single, flawed benchmark paper then you should consider changing jobs.

      Comments

      1. By Ashcrow () ashcrow@phreaker.nosp44m.net on mailto:ashcrow@phreaker.nosp44m.net

        I tend to agree, but generally CTO's are just way out of touch techies, sysadmins, and engineers. All it takes is on paper or a bad review on CNET to make them 'know' it's crap.

      2. By squid () on http://www.simulacra.us

        i agree, but that would eliminate 90% of all jobs. it's a shame that most businesses allow CTO's with such little technical background to achieve the positions they have.

        so the problem really is more of a shortcoming in the corporate world, but it's a practical matter that should be considered. i personally wouldn't mind helping with an effort to "re-benchmark" obsd with the other OS's. i know obsd is a performance power-house, but it ("it" meaning the -stable branch) clearly performs better than those results (the "lack-of-stability" remarks particularly annoy me).

        Comments

        1. By squid () on http://www.simulacra.us

          whoops, that was supposed to say
          "i know obsd is *not* a performace power-house"

          sorry

  10. By Anonymous Coward () on

    So.. how big was that partition?

    In that case obsd had the same space linux has only for swapping! Did obsd have a chance to swap? I doubt it :)

  11. By Anonymous Coward () on

    The study is correct in many points. I'm not flaming here, and I'm not even suggesting these things are something the project should focus on, but plain and simple - OpenBSD is not as fast and stable as other systems.

    Sure, it can run fine for long periods of time, with no hiccups whatsoever. Sometimes.

    Anyone who has run a REALLY high-traffic website (10M+ hits per day) on OpenBSD knows - it crashes way too easily (or, much more easily than FreeBSD).

    Another issue, is that the IDE disk performance is piss poor. Try timing a copy of a 100mb file from and to the same partition on OpenBSD, then do it on FreeBSD. The difference?

    OpenBSD> time dd if=/dev/zero of=x bs=1m count=100
    real 0m5.369s
    user 0m0.000s
    sys 0m1.147s

    FreeBSD> time dd if=/dev/zero of=x bs=1m count=100
    real 0m2.639s
    user 0m0.000s
    sys 0m1.235s

    (identical hardware, both machines idle, no optimizations on either)

    With that said, I still run OpenBSD for everything that isnt performance critical.

    However, I trust FreeBSD more when it comes to the "i dont want to wake up at 5am to reboot a fsckin server" stuff.

    Of course, for the stuff that performance is key (database server,etc), OpenBSD isn't an option due to lack of SMP support.

    Linux? Blah! :)

    Comments

    1. By Brian () on

      Im fairly certain that freebsd enables softupdates by default and I know that openbsd does not. This could explain a substantial difference in write speeds.

      See:
      http://openbsd.org/faq/faq14.html#SoftUpdates

      Comments

      1. By tedu () on

        nothing to do with single file writes.

      2. By Per-Olov Sjöholm () on

        I tried:
        time (dd if=/dev/zero of=x bs=1m count=100; sync; sync)
        on a default installed OpenBSD 3.3 on a Celeron 2GHz, 256MB RAM, Seagate 160GB IDE HDD hardware.

        The partition was 5GB in size. It took about 1.7 sec to finnish the operation. After that I enabled soft updates and rebooted.... With soft updates the operation took about one more second to finnish (2.7-3.1 seconds).

        I experimented a little bit with the disk cache buffer size (cachepct). With that the result bacame better... but not much.

        I am not an expert in this area but have followed this interesting discussion. I hope this discussion ends up with some more tuning tips for my favorite server OpenBSD.

        Thanks
        Per-Olov

        Comments

        1. By mirabile () mirabile@bsdcow.net on https://MirBSD.BSDadvocacy.org:8890/

          This is because softupdates actually caches the
          data, instead of writing them straight to disc
          (sync mount, standard in BSD).

          Leenuxen use async mounts, but these cannot ever
          guarantee data consistency in case of a sudden
          power loss (not even metadata), in contrast to
          softdep.

          In general, softdep is a speed-up, because it
          is not speeding up the writing process itself.
          Plus, you can skip fsck safely.

      3. By Per-Olov Sjöholm () on

        :O
        Ooops typo...
        It actually took 1.7 secs to finish the operation, not to finnish it :))

        /Per-Olov

    2. By tedu () on

      can you run that test again with

      time (dd if=/dev/zero of=x bs=1m count=100; sync; sync)

      ?

      Comments

      1. By rintek () on

        wuuk:~# time (dd if=/dev/zero of=x bs=1M count=100; sync; sync)
        100+0 records in
        100+0 records out
        104857600 bytes transferred in 1.679819 seconds (62421962 bytes/sec)

        linux 2.6.0.-test7 :-)

        groetjes,

        Rintek

        Comments

        1. By Anonymous Coward () on

          Ehm, and? We do not know that your disks+controller+CPU+friends are the same that the
          other ones used. Here is the NetBSD benchmark:

          104857600 bytes transferred in 9.124 secs (11492503 bytes/sec)

          Slow eh? Pentium-133, older 30Gb disk.

          104857600 bytes transferred in 3.563 secs (29429581 bytes/sec)

          Athlon XP-1.6Ghz, 80Gb disk

          You fail to see why we can weigh FreeBSD to OpenBSD. They use a filesystem with the exact same semantics and we know these semantics to be stable. I do not even know which mount options you
          used for your disks and thus we only have numbers but no reference. So dont go implying that the above numbers means anything but compares an P133 to a Athlon on the same operating system. Thanks.

      2. By Anonymous Coward () on

        (i am original poster)

        OpenBSD> time (dd if=/dev/zero of=x bs=1m count=100; sync; sync)

        100+0 records in
        100+0 records out
        104857600 bytes transferred in 4.717 secs (22226622 bytes/sec)

        FreeBSD> time (dd if=/dev/zero of=x bs=1m count=100; sync; sync)
        100+0 records in
        100+0 records out
        104857600 bytes transferred in 3.138960 secs (33405204 bytes/sec)

        Comments

        1. By tedu () on

          erm, none of you guys are showing me the results of time. the whole point of running sync is to see how long it takes to sync. :)

          Comments

          1. By Anonymous Coward () on

            100+0 records in
            100+0 records out
            104857600 bytes transferred in 2.576 secs (40694500 bytes/sec)
            3.56s real 0.00s user 0.49s system

            Comments

            1. By Janek Richter () janek@bsd-daemon.org on http://www.bsd-daemon.org/

              infinity% time (dd if=/dev/zero of=x bs=1m count=100; sync; sync)
              100+0 records in
              100+0 records out
              104857600 bytes transferred in 1.350 secs (77652912 bytes/sec)
              (; dd if=/dev/zero of=x bs=1m count=100; sync; sync; ) 0.00s user 0.26s system 12% cpu 2.041 total

      3. By Anonymous Coward () on

        100+0 records in
        100+0 records out
        104857600 bytes transferred in 2.817 secs (37220939 bytes/sec)

  12. By Anonymous Coward () on

    A very intereting comment by Bert de Jong at openbsd-misc .

    I think is worth reading.

    Comments

    1. By Anonymous Coward () on

      Thank you for this pointer, it's a nice read indeed!

  13. By Steve () on

    This benchmark is about as biased as the fox news channel.

    Comments

    1. By Anonymous Coward () on

      Fox isn't biased! you must be some pinko leftist terrorist-lover.

      Comments

      1. By Anonymous Coward () on


        well, well, well, another patriot using the lies of his gouvernement to protest again..

        http://opengov.media.mit.edu/

        this can be educated you about your gouvernement...

        Comments

        1. By djm () on

          I think you need to adjust your sarcasm detector, it's set a bit too low.

          Comments

          1. By Anonymous Coward () on


            Oh, like that would be a real useful device! (Sarcasm detector blows up.)

      2. By Steve () on

        That's it.

    2. By Anonymous Coward () on

      *waah waah*

  14. By Anonymous Coward () on

    It's good to see that the general response in the OpenBSD community isn't one of calling conspiracy and such, as what usually happens in the Linux community, when they instantly scream "FUD" any time they don't like something.

    The OpenBSD community distinguishes itself by taking the facts to heart and resolving to better things.

    Comments

    1. By Anonymous Coward () on

      I still see the OpenBSD community is taking pop shots at the Linux community.

      Comments

      1. By Anonymous Coward () on

        Mod parent up!

        Come on guys, I made a couple of posts, trying to be positive and get the OpenBSD community to help improve things so that more of us (yes I do run other BSDs and Linux) can have ago.

        And most of the responses (to me and others) sound like 'you are not good enough', 'rtfm', it is slow because it is secure...

        Come on! I felt like I was reading comments from Linux zealots talking to a windowsUser-trying-to-convert about 5 years ago...

        Go on, flame me and prove that you don't understand.

    2. By Anthony () on

      When someone says "OpenBSD is slower than FreeBSD and Linux.", that's not news, that's an obvious side effect of the project's stated goal. No ones egos were bruised, so there's no reason to flame.

      There's a bit of irritation about the wording and obvious bias of the tests, but apart from that it just doesn't matter.

      Watch what happens if someone claims Linux is more secure than OpenBSD though.

    3. By Anonymous Coward () on

      Is this a joke? Have you read any of the comments on this site? Nothing of the sort is happening.

      Comments

      1. By krh () on

        It depends a lot on the issue. This benchmark has generated a lot of comment but few flames. I think this is because most of us are either mature enough to recognize that the testing methodology is severely flawed or are comfortable enough with OpenBSD's focus on security that we don't feel attacked when someone says that OpenBSD is slow. Indeed, most of the complaints I have read in this thread have either stated that OpenBSD needs performance improvements, the test was flawed, or that the poster doesn't think performance is a relevant question. All three are true.

        Where you will find us angry is when you attack OpenBSD's development team or OpenBSD's security. Security, is, of course, the reason why most people switch to OpenBSD, and the community is rightly concerned with the public perception of OpenBSD's security. The development team is the heart of OpenBSD, and while not everyone agrees with everything the development team does (and some of us openly disagree), we believe that they do good work and produce a good product. We like their work, and we want them to keep doing it, so we are, again rightly, concerned with public attacks on and the public perception of the development team.

        It's easier than a lot of people realize to have cordial relations with the OpenBSD community:

        1) Don't attack us and don't attack the project; that makes us emotional.

        2) Do make sure you are fair when comparing OpenBSD to other operating systems; we realize that OpenBSD is not a perfect operating system, but we think it has its advantages, and we would like you to politely acknowledge our beliefs.

        That's it. I don't think it's unreasonable for us to ask anyone to do that. I realize that we're not always capable of it ourselves, and I realize that even when you are impartial you may be unjustly criticized. Nevertheless if you treat us as a respected colleague we will do our best to treat you as the same.

    4. By Anonymous Coward () on

      To be fair, though, Linux does get FUDded a lot by (for example) Microsoft, so it's a conditioned response in the community to see criticism as FUD. Though I imagine their reaction to a report like this would probably be more along the lines of "that's stupid and unscientific".

  15. By Anonymous Coward () on

    One thing I forgot to mention in my original comment, is that indeed OpenBSD's mmap() is horrible (worse than it seems). In fact, along with HPUX, it's actually quite broken.

    A couple of years back, OpenBSD wouldn't even run on OpenBSD, due to its use of mmap() to store its TDB files. A workaround inside the Samba code had to be used to not use it... and it worked on every other OS.

    IMO, things like bad mmap support, very bad (getting better!) pthread support, etc. really hurt OpenBSD. Team members might not see them as high priority, but they do disqualify a large user base.

    Comments

    1. By Anonymous Coward () on

      Wow, I am tired. I meant:

      (worse than it was described)

      SAMBA wouldn't even run on OpenBSD

      Time for sleep!

    2. By tedu () on

      can someone explain to me the nature of this breakage?

      Comments

      1. By Anonymous Coward () on

        I never looked into it further, but here's what would happen (with Samba using mmap):

        1) Samba opens it's .TDB files
        2) Create one connection to Samba, everything is fine, the .TDB file is updated with connection information (think utmp/wtmp)
        3) Create another connection to Samba, and the calls to mmap() spiral out of control, and keep writing to the TDB file until the disk is full.

        My guess is that something isn't up to spec, or theres a bug that makes it incoherent.

  16. By mirabile () mirabile@bsdcow.net on https://MirBSD.BSDadvocacy.org:8890/

    He's been using BSD for a long time and I know
    someone who still knows him as a BSD liker.

    He was basically annoyed by some of the stones the
    OpenBSD developers throw in their ways.

    I don't really like Fefe, but the points
    he noted are true:

    - MirBSD has no 1024 cylinder boundaries since ages,
    and the OpenBSD person wouldn't just import it
    "because it uses the wrong assembly syntax".

    True is: OpenBSD as(1) has .intel_syntax pseudo-op
    since a fairly long time now.


    - The IPv6 breakage is a mess. Having ported djbdns,
    _with_ IPv6 support, I can only tell so.

    If someone had a patch to re-enable the NetBSD-style
    behaviour, I'd gladly accept it. (They say that both
    NetBSD and FreeBSD will adopt the itojun-draft in some
    (near?) future, but I haven't seen anything like that
    happen yet.)


    - OpenBSD performance sucks. True.
    (Not that performance would be a design goal - but hey,
    avg. 1.2 MB/sec copying from filesystem to filesystem
    on the same notebook HDD is, uhm...)

    But softdep is okay, and I can live with that.

  17. By Anonymous Coward () on

    from fefe's pages concerning OpenBSD/IPv6:

    "I find this behaviour of pissing on internet standards despicable and unworthy of free operating systems."

    right, that's why the world got OpenSSH for example???

    When you want to do a technical/scientific review of something, start by keeping your comments free of opinionated blubber then go on to comparing what is comparable, perhaps using RELEASE versions of the OS'es? Then again who cares that the zealots call us zealots?

    People who use OpenBSD don't care because they know better. People who are considering using OpenBSD should run their own tests. Think for yourself, it's free.

    Comments

    1. By Anonymous Coward () on



      When you want to do a technical/scientific review of something, start by keeping your comments free of opinionated blubber then go on to comparing what is comparable, perhaps using RELEASE versions of the OS'es? Then again who cares that the zealots call us zealots?


      I don't know about that. If a person has a bias, I prefer to see it expressed up front, so I can apply a filter to the rest if the material.

      It is unfortunate that emotional comments in the study elicit emotional responses, but that goes with the territory.

    2. By Anonymous Coward () on

      OpenSSH is great!
      And what is your point again?
      btw, I have no idea about IPv6 standards compliance in OpenBSD, but OpenSSH has got absolutely nothing to do with it?

      It's like saying: "I haven'got driving license, but I can cook"

  18. By monkey.org is where all the elite developers hang () on

    Can someone from the OpenBSD developership give an example of how security makes you choose an O(n) algorithm over O(1)? Do you not know how to abstract architecture from implementation? Or is it simply easier to just cvs update from NetBSD? LOOKING FORWARD TO YOUR REPLY!!! Clearly if security is your reason for having crappy performance, you should have some examples to back this up. A constant performance hit is understandable, yet these benchmarks had nothing to do with that at all. There's no reason for it, just excuses and more excuses. Please, it's clear that performance isn't a tradeoff for security, so instead of trying to spin the issue, just tell your users you could care less about performance. That way OpenBSD can further slip into obscurity because it will run like a 386 on new hardware (the kind people never run).

    Comments

    1. By gwyllion () on

      The lack of performance can be explained by the lack of developers working in this area (which more or less the consequence of the project's focus on security).

      Please provide us with patches to increase OpenBSD's performance, instead of trolling.

      Comments

      1. By Stephen Gutknecht () Stephen@RoundSparrow.com on mailto:Stephen@RoundSparrow.com

        I agree with you gwyllion.

        If anything, OpenBSD has less users and less "real world testing".

        Smaller development team means less is being done. It can't be competitive on every front.

        Linux is a swiss army knife with tons of developers focused on every aspect of operating systems.

        OpenBSD does what the people who use it want it to do. The "team" has said that clearly. They work on something as they want to / have a use for it.

        Linux community seems more focused on displacing other products. Heck, even "Linux" isn't "Linux" - there are dozens of common distributions in use.

  19. By Boris () postmaster@localhost on http://openbsd.org/

    After using almost only openbsd for years, I was impressed by this article. Then I started looking (again) for freebsd, evaluating a migration path for a few loaded ISP production servers, full of in-house patches and mods.

    After 10 minutes, I decided that those measurements were lies by incompetence, and not worth a cent of the experience I ever had with openbsd. I Then migrated back to friendly openbsd. the cost of migrating servers back and forth
    was then greatly reduced.

    people buy dual-procs because they understand
    so little about it.
    I'd rather buy 2 servers than a big dumb dual proc. 1 dual proc = 1.5 times the price, 1.2
    actual performance gain. 2 servers = 2 times the
    price, 2 times actual performance gain.
    So I usually wait until I get a bit more cash, and I gainvmore worth hardware.

    The openbsd installer line mode is intentional, not because developpers are lazy with ncurses
    (as if they're working free for you). if you don't know why it is line mode, then go back to school. study harder.

    OpenBSD does not need more users. users need
    more OpenBSD. They choosed OpenBSD because they understood long ago OS comparisons reports are worth their ink in used toilet paper, and took the time to try and test on their own rather than Lazily Relying on anybody poping up saying anything.
    Compared to oneself's self testing, All those reports are worth /dev/urandom. Wait, make that /dev/srandom.

    Comments

    1. By Peter Ross () prossini@firemail.de on mailto:prossini@firemail.de

      > Then I started looking (again) for freebsd, evaluating a migration path for a few loaded ISP production servers, full of in-house patches and mods.

      > After 10 minutes, I decided that those measurements were lies by incompetence

      It would be nice to give the hints why you stopped the migration. What happened?

      Regards
      Peter

  20. By Danny MacMillan () on

    The bulk of the comments I've read here call into question the character and the agenda of the tester instead of addressing the tests. I'm a relative newcomer to OpenBSD. I use OpenBSD. I like OpenBSD. But these tests look very, very bad for OpenBSD. Calling the tester's character and agenda into question will not help at all. Calling the validity of the tests into question will hardly help at all. The only thing that will really rebut these results is to devise and execute a test that is valid and publish the results. I am not technically competent to do this, but it seems clear to me that it needs to be done.

    Comments

    1. By Anonymous Coward () on

      The bulk of the comments I've read here call into question the character and the agenda of the tester instead of addressing the tests.

      The nature of the tests is important in determining the validity of the results.

      THe validity of some results are easy to determine, such as a kernel panic with too many procs running, and so a bug report has already been submitted about this.

      The validity of other results are very difficult to determine due to the character of the test and the bias of the author. For example, why does the NetBSD fork test stop at 3000 procs? This is a discrepancy that really needs an explanation.

      I can make pretty graphs of any data, put them up, and declare that XYZ OS sucks but ABC OS is great because of those. But no one shoudl listen to me unless i've also made sure my tests are accurate and reproducible. Otherwise, it's just FUD.

    2. By Can Erkin Acar () on

      But these tests look very, very bad for OpenBSD

      How? these tests are only (maybe) relevant when you are using OpenBSD for serving thousands of simultaneous connections. And in these loads, you are possibly better off balancing the load among multiple servers.

      These benchmarks give little information about the actual server performance and experience in 'normal' loads, ie. you would not notice a thing.

      Calling the tester's character and agenda into question will not help at all.

      Well, the tone and language of the test is obviously biased. You dont start praising/bashing your test subjects at the first paragraph in an objective research. You wait until the conclusion to do that.

      Calling the validity of the tests into question will hardly help at all.

      It will, but not much. I do not claim or even believe that OpenBSD will be at the top in a fair performance benchmark. Performance is simply _not_ the priority of developers.

      Performance is also not (always) the priority of administrators too. Security, consistency, stability and documentation is usually more important.

      However, when a performance problem is put to the attention of the developers, some of them may decide to improve things a little. Submitting patches also help.

      The only thing that will really rebut these results is to devise and execute a test that is valid and publish the results.

      There is no need to rebut the results. If a performance test brings OpenBSD to the top, I would call it unfair too.

      What needs to be done is to understand the problem and fix what needs fixing.

  21. By James Herbert () jamesherbert@gmx.net on mailto:jamesherbert@gmx.net

    From: "Felix von Leitner"
    To:
    Sent: Friday, October 24, 2003 3:19 PM
    Subject: Please help me tuning the OpenBSD kernel for bulk.fefe.de/scalability


    > Hi!
    >
    > Several people have asked me to re-run my benchmarks after the kernels
    > have been properly tuned.
    >
    > To ensure a fair test, I will ask each kernel team to send me a list of
    > things to do to a GENERIC kernel to optimize it for optimal performance
    > on my benchmarks.
    >
    > Please also tell me if you have suggestions for other benchmarks that
    > say something about the scalability of an operating system, that I could
    > (and should) include in my benchmark suite.
    >
    > Felix

    Nice of him...

  22. By Bo () on

    Nagging, nagging and more nagging.

    I just don't believe some of the reactions to the "benchmarking" test, that I have read. If someone finds something strange, that could indicate a problem, be professionel about it and download the tests, write some tests yourself and then hammer the kernel to see if something is wrong. Run the test with 10 times the load. Do whatever it takes to prove or disprove that there is a problem, not for his sake but for OpenBSD's.

    Good god, OpenBSD is your/our baby, if something could indicates that there is a problem, or that it's not behaving 100% as expected, don't waste time arguing about it, find out what is going on.

    Grow up, don't take it personally.

    Bo

Latest Articles

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]