OpenBSD Journal

New Ruby benchmarks show OpenBSD improvements

Contributed by jason on from the benchmarks-are-meaningless-but-we-cant-help-ourselves dept.

NetBSD developer Jaime Fournier has published a new round of Ruby benchmarks comparing the various BSD projects and Ubuntu Linux. Some readers will remember he performed a similar test a couple of years ago. A quick review of the data reveals improvements by OpenBSD. The following is a snippet from the full report available on his website.

Benchmarks from ruby-benchmark-suite run on ruby-1.9.2-p180 with
rubygems-1.5.3 

Benchmarks found at https://github.com/acangiano/ruby-benchmark-suite

Dragonfly 2.9 - With Hammer root filesystem
FreeBSD 8.2 - Default disk layout
NetBSD 5.1 - Default disk layout
OpenBSD 4.8 - Default disk layout
Linux - Ubuntu 10.10 Server Edition
The hardware which is an intel i7 with a dedicated 40GB SSD
used for each os with default installs.

FreeBSD dmesg here http://linbsd.org/fbsd.dmesg
OpenBSD dmesg here http://linbsd.org/obsd.dmesg

These are the mean values taken from each benchmark.
The name format is benchmark_parameters_iterations
Charts might come as more OS's are added.

Warning: 
No attempt to ensure the same compiler was used on each os
as this would be unrealistic given these systems come with compilers
and typically would be used to build ruby.

Email any questions or rants to jaimef@linbsd.org


Totals Wins by OS
 190 NetBSD 5.1
 100 Dragonfly 2.9
  88 Linux - Ubuntu 10.10
  65 OpenBSD 4.8
  46 FreeBSD 8.2

Editor's Note: Some readers may point out that OpenBSD performance is near the bottom of the list. I would remind those users that if web application performance is your key requirement, then Ruby probably shouldn't be your language of choice anyways.  :)

(Comments are closed)


Comments
  1. By Tom Van Looy (tvlooy) tom@ctors.net on twitter.com/tvlooy

    OpenBSD performance is near the bottom of the list but the report says:
    "Lower is better! Smaller number wins..." so that's a good thing not?

    Comments
    1. By Tom Van Looy (tvlooy) on twitter.com/tvlooy

      > OpenBSD performance is near the bottom of the list but the report says:
      > "Lower is better! Smaller number wins..." so that's a good thing not?

      Shit, I get it now.

    2. By David Trotter (fossala) on

      > OpenBSD performance is near the bottom of the list but the report says:
      > "Lower is better! Smaller number wins..." so that's a good thing not?

      The lower the number the better applies to the tests. But the numbers where OpenBSD is second to last is the results (who won how many times) thus higher the better.

  2. By Dave Steinberg (redterror) on

    His choice of metric, 'wins' seems pretty kooky. The results seem to indicate that netbsd is a lot "better", but there's no indication of what the margins are. If the actual difference between 1st and 5th place were hypothetically something like an aggregate 10% performance difference, I would find that to be a lot more informative.

    It seems like he punted on the hard part here.

    Comments
    1. By John L (JohnL) j@bitminer.ca on bitminer.ca

      Many of these benchmarks have nothing to do with web hosting, e.g. hilbert matrices. Does anyone truly use ruby for these?

      The missing measurements (where the platform does not complete the test) would be more concerning.

      Picking a few "web-like" tests at random, I see OpenBSD within 1 standard deviation most of the time and slow only on a file IO and much better on a threads test. A true believer in benchmarks would select only these micro and macro tests of concern and evaluate those.

      Too bad there isn't a numerical security test that lets OpenBSD win based on, say, number of remotely exploitable holes in the default install.

      Comments
      1. By Jaime Fournier (jaimef) on http://mauthesis.com

        > Many of these benchmarks have nothing to do with web hosting, e.g. hilbert matrices. Does anyone truly use ruby for these?
        >
        > The missing measurements (where the platform does not complete the test) would be more concerning.
        >
        > Picking a few "web-like" tests at random, I see OpenBSD within 1 standard deviation most of the time and slow only on a file IO and much better on a threads test. A true believer in benchmarks would select only these micro and macro tests of concern and evaluate those.
        >
        > Too bad there isn't a numerical security test that lets OpenBSD win based on, say, number of remotely exploitable holes in the default install.
        >
        >

        There are many problems with these microbenchmarks.
        These initial results are for baseline purposes as these tests
        are used quite a bit within the ruby community.

        The next tests are more web specific with RubyOnRails and a full
        stack including database.

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]