OpenBSD Journal

Ask Undeadly: OpenBSD and SCSI RAID adapters?

Contributed by grey on from the hardware recommendations dept.

We received the following anonymous query, hopefully our readers will share their experiences and recommendations:

Hello my fellow OpenBSD Journal readers. I need your collective opinions and experience regarding a specific inquiry. At work, a couple of Linux advocates are trying to convince management to replace aging proprietary file servers with SUSE or Debian ones. I would like to use OpenBSD instead. We will either build our own servers or order them from a third party. I have found documentation on several OpenBSD RAID drivers but I would like to read about your first hand experience with these cards.

Which SCSI or IDE RAID cards have you had the best experience with? Any horrible issues or scenarios? Direct support for OpenBSD from the hardware vendor is a big plus for me. I would like to support the vendors who offer technical specifications and documentation to the OpenBSD development team.

Also, I have read horror stories online about people complaining that having a RAID array on a proprietary controller can lead to headaches later on. If a vendor stops supporting that card or goes out of business and your card needs to be replaced for whatever reason then your data is locked in. Have any of you experienced that?

I thank you all for sharing your knowledge.

(Comments are closed)

Comments

  1. By Breeno (70.64.188.156) on

    There are several vendors on the market - Accusys comes to mind as one - that offer RAID controllers which emulate single drives. You install the RAID box in a 5.25" bay, connect your drives to the RAID box, and then connect the RAID box to the SCSI controller. If your goal is RAID 1, then it meets all your needs: no drivers to worry about and any single drive may be removed and connected directly to a SCSI controller to retrieve the data. For RAID 5, I believe that if replacement hardware is unavailable you would be in trouble no matter what hardware solution you choose. The only way to cover your back 99.9% would be to purchase spare RAID hardware just in case your setup fails and the vendor disappears.

    Of course, you aren't using the RAID as a backup solution, right? It isn't a backup solution, it is a redundancy solution. A backup solution is something you can take off-site in case the building burns down, or floods, or gets picked up by a tornado, or the building is broken into and the disks are stolen, or... well, you should get the picture by now. A redundancy solution only provides additional insurance against downtime.

    They have a number of products at their website (www.accusys.com.tw) that you may want to look into. I've heard only good things about the ACS-7500. Granted, it's not SCSI, but it is an indicator that their SCSI stuff is probably good as well.

    Comments

    1. By Michael Joncic (195.70.110.145) on

      Or you could try one of the wonderful controllers from ARECA (www.areca.com.tw), which support RAID 6 in Hardware (I used the ARC-6060 once for a client). They can be managed via the serial port or the integrated WEB-Server!

      brgds
      Michael

  2. By Anonymous Coward (69.197.92.181) on

    Direct support from the vendor is non-existant, you would have to use freebsd or linux to get that. OpenBSD isn't supported by any RAID vendor, and there are no management utilities for any RAID controllers on OpenBSD.

    I believe the aac driver for adaptec raid controllers is able to tell you when a drive in your array fails, and the array is in degraded mode, at least it can for some models. These controllers aren't the greatest performance wise, but they are probably your best bet for being able to know what's going on with the array at least. Performance wise, the icp controllers are very good.

  3. By dafros (83.31.155.11) admin@chem.uw.edu.pl on

    gdt driver (Intel RAID) works fine under non stop heavy load. 10 SCSI 36G 10K discs OpenBSD 3.5 RAID 5 + 1 drive HOT-FIX

    Comments

    1. By Anonymous Coward (69.197.92.181) on

      Totally off topic, but if you are really using that setup under heavy load, then at some point you are going to be in for some serious hurt. Once a drive dies, RAID 5 becomes very slow, as requests need to calculate the missing data. Although it wastes more space, RAID 10 or RAID 1+0 is pretty much required if you want to maintain performance while the array is degraded.

      Comments

      1. By Anonymous Coward (143.166.255.16) on

        Bullshit.

        All relevant RAID cards have XOR engines. Calculating or not calculating parity runs at virtually the same speed. You would never notice the difference.

        Gosh the amount of ignorance on RAID is astonishing.

        RAID 0: The fastest one out there but no redundancy RAID 1: Slow on writes because you are going as slow as the slowest drive; fast reads but not as fast as RAID 5 or 0. RAID 5: Almost as fast as RAID 0; add a spindle and it'll be as fast. RAID 10: depending on the implementation or either real good or real bad. Don't forget the RAID 1 part is as slow as the slowest member. RAID 50: Fast and very redundant; bus speed becomes the issue.

        This is very general and can easily be disproved; all RAID cards have different behavior and performance sweet spots. Choose your RAID set wisely for your application.

        No one is right here...

        Comments

        1. By Anonymous Coward (69.197.92.181) on

          You are mistaken if you think having hardware XOR is enough to make the array perform the same degraded as healthy. I am not talking about your generic explanation of RAID levels, I am talking about actual real world performance. Using a very fast ICP controller, RAID 5 drops to half performance when degraded. Using a not very fast adaptec controller, I am not sure how bad the drop is, but its certainly noticable and the load average sky rockets as sendmail processes get slowed down and there are suddenly hundreds of them running. Its not like this is hard for you to test yourself and see, so try it.

          And as to your random slander of RAID 10, all RAID levels can be bad "depending on implimentation", that has nothing to do with anything. The fact is, in worst case scenario, RAID 10 is just as redundant as RAID 5, faster, faster when degraded, and wastes more disk space. You may even be able to survive multiple disk failures if you do 1+0 instead of 10.

          Comments

          1. By Anonymous Coward (67.64.89.177) on

            *yawn*

        2. By ansible (69.243.11.47) ansible@xnet.com on

          All relevant RAID cards have XOR engines. Calculating or not calculating parity runs at virtually the same speed. You would never notice the difference.

          No. In a healthy RAID-5 if you want to read a small chunk of data, you just read it. In a sick RAID-5 array, you have to read bits from the entire array, because you have to reconstruct the data. So you have increased the amount of reading you have to do.

          RAID 1: Slow on writes because you are going as slow as the slowest drive;

          This is just as true of RAID-5, isn't it? Besides, if you're building a RAID array with stuff you've dug out of your extra parts bin, you've got other problems. If you have any sense, you build a new server with all the same drives purchased at the same time (with spares).

          So in general, you're wrong about RAID-1 being slow. In fact RAID-1 will often give you better performance on reads than RAID-5. Write performance can vary vs. RAID-5 depending on the size of the writes.

          These days, with the cost per gigabyte of drives having fallen so far, it doesn't make sense to do RAID-5 anymore, in my opinion. RAID-1 can also be much more robust as well (you can have a three-drive mirror, for example).

          Comments

          1. By sthen (81.168.66.229) on

            If you have any sense, you build a new server with all the same drives purchased at the same time (with spares). Is this really a good idea? My preference has been to use drives purchased at different times, to try and avoid problems which might affect a whole batch...

      2. By dafros (83.31.129.104) on

        You missed something - there is HOTFIX (HOTSPARE) drive, if drive fail there is no need to recalculate data. If you need maximum redundancy RAID 5 should be your goal.

        Comments

        1. By Anonymous Coward (69.197.92.181) on

          The hotspare drive will still need to be filled with data. The RAID controller cannot guess in advance what drive will fail, and maintain the hot spare full of that same data. When a drive fails, the array will have to be rebuilt with the hot spare drive taking the place of the failed drive, this will be very slow.

        2. By Anonymous Coward (69.197.92.181) on

          Oh, and RAID 5 does not give you maximum redundancy, RAID 10 is equally redundant when you are unlucky, and possibly more redundant if you lose drives from seperate mirrors. Since its also faster, if you can afford the wasted drive space its definately the way to go.

  4. By cruel (195.39.211.10) on

    we've using Adaptec gear (2110 and 2120) with no problem:

    ...
    iop0 at pci1 dev 1 function 0 "DPT SmartRAID (I2O)" rev 0x01: I2O adapter <ADAPTEC 2110S>
    iop0: interrupting at irq 5
    ppb0 at pci1 dev 1 function 1 "DPT PCI-PCI" rev 0x01
    pci2 at ppb0 bus 2
    ...
    iop0: configuring...
    ioprbs0 at iop0 tid 521: <ADAPTEC, RAID-5, 380E> direct access, fixed
    scsibus0 at ioprbs0: 1 targets
    sd0 at scsibus0 targ 0 lun 0: <I2O, Container #00, > SCSI2 0/direct fixed
    sd0: 52503MB, 6693 cyl, 255 head, 63 sec, 512 bytes/sec, 107526144 sec total
    ...

  5. By Simon (217.157.132.75) on

    I was wondering, people recommend Adaptec and Intel RAID adapters, do they both provide feedback to the operating system... well I know they do, but is it supported by OpenBSD?

    Not must fun in have hardware RAID if you aren't told when a drive fails.

    Comments

    1. By Nonesuch (163.192.21.46) on

      Many of the Adaptec RAID controllers will provide "health" information (status) to OpenBSD 3.6 TMK, there are not RAID management tools, but the controller will report the status of an array to the driver, which logs events in Syslog.

      This includes many of the SCSI-based "PERC" controllers in mid-range Dell rackmount servers.

  6. By Shane (144.136.76.4) on

    Since configuration tools and health reporting is not supported under OpenBSD by hardware RAID card vendors. The best bet may be to use a generic SCSI card with an external RAID cage (with built in RAID engine) which provides it's own LCD screen, buttons and alarms for status, config and health.

  7. By Charles Hill (216.229.170.65) on

    Do you build your own machines, or buy equipment from a major vendor like IBM, HP or Dell?

    If it is the latter, you'll save yourself tons of headaches by getting a supported OS (Linux).

    For example, Dell just bumped their line of PowerEdge servers from 17xx to 18xx. The changes included Intel E64T Xeons, Double DDR RAM and an update to the RAID controller (optional).

    The lovely update came with drivers for Red Hat, Dell's supported version of Linux. It also came with source code, fortunately. We use Debian where I work and it took 3 1/2 *days* to get everything configured properly with a kernel that supported all of our equipment. ("All" being about 1/2-dozen models of Dell servers, of which only the PERC 4e/Si RAID was the problem.)

    After this little fiasco, I'd vote for SUSE Linux as being the best compromise between not-Windows, yet still getting some vendor support.

    OTOH, if you're able to select specific RAID cards or are building your own equipment, then go for it. Being able to control the hardware means you can happily use OpenBSD with your RAID.

    -Charles

    Comments

    1. By Marco Peereboom (143.166.226.18) slash@peereboom.us on www.peereboom.us

      I use a whole bunch of Dell machines with OpenBSD without major issues. I use both ami(4) and aac(4) extensively. I even run ami(4) on amd64 (not available yet to the general public, mail me if you want a diff). There is one nagging ami(4) issue but that'll be fixed soon.

      From a SCSI perspective I run ahc, ahd, mpt, isp & siop (I am sure I'm forgetting one) and they all do their job. I particularly like mpt.

      FWIW,
      /marco

  8. By ansible (69.243.11.47) ansible@xnet.com on

    Because there is so little support for hardware
    RAID on OpenBSD, I don't recommend trying to use it.

    I've had decent sucess with software RAID on OpenBSD, and
    it can report the health of the drives, which is very important.

    I don't think there is as much of an issue as far as speed goes
    if you're running RAID-1.

    It is not an optimal solution, of course. The system doesn't respond
    as quickly to a drive failure as a hardware solution would. You
    should definitely test it with your application before putting it
    into production.

    Comments

    1. By Anonymous Coward (67.34.129.203) on

      "The system doesn't respond as quickly to a drive failure as a hardware solution would."

      Doesn't RAIDframe just immediately fail the disk if it gets an unrecoverable error? (and then kick in the hot spare if there is one)

      Comments

      1. By ansible (69.243.11.47) ansible@xnet.com on

        Immediately? No. You get a few minutes of disk error messages
        spewing on the console first. Then, after the OS has decided
        that this is a permanent failure, the upper levels get notified
        and the rebuild on the hot spare starts.

        I've seen hardware RAID solutions notice a disk failure within a few seconds.

        There ought to be (or maybe already is) a kernel tunable parameter which controls how long/often temp failures turn into a permanent failure.

        I haven't researched the issue myself. It's not a big deal to me, as
        hardware failures aren't that frequent. But it is something to be aware of. You ought to be testing any kind of RAID system before
        putting it into production anyway. This is kind of hard on non-hot-pluggable hardware, but you need to see if you've done things right. I usually just yank the power cable from a drive to fail it
        manually.

        Comments

        1. By Anonymous Coward (69.197.92.181) on

          The problem is failures that aren't really failures. On a very busy raidframe array, it will occasionally mark a drive as failed because it couldn't keep up. It will even do this to both drives in a mirror, and then panic because the raid device no longer has any drivers. Raidframe works fine for redundancy on low volume machines, but for high load you really do need hardware raid.

          Comments

          1. By ansible (69.243.11.47) ansible@xnet.com on

            Interesting.

            Maybe you've already done this... I would want to fully investigate the hardware setup to make sure there aren't any other problems.

            I've had issues on some systems where the cabling wasn't quite up to snuff, and there were problems running the drives with the higher levels of DMA on the controllers.

            At any rate, I haven't really pushed a RAIDFrame system like you apparently have. I mostly used OBSD for firewall and router systems. My fileservers are still all Linux.

            Still, it would be interesting to set up a new system with SATA drives, and see what that does with RAIDFrame.

            Comments

            1. By Anonymous Coward (69.197.92.181) on

              No, I have had this experience with multiple machines, and they were all SCSI, so the wrong DMA mode was definately not an issue. There is no weird problem to look for, its very simple. Raidframe will disable a drive and consider it failed if it hasn't responded quick enough. That timeout is a bit on the short side, so on overloaded servers, you can get raidframe "I have no drives" panics.

  9. By Anonymous Coward (66.189.124.7) on

    We just bought 2 Dell machines with their PERC 4/Di cards, and they work fantastically. But this makes sense-- all of the RAID software in in the card's BIOS. You configure the array at boot time. We're running RAID-5 with 3 disks each. These cards are rebranded LSI cards, IIRC.

    If you can't afford/don't want the Dell PERC cards, go with a true hardware RAID solution, since the software is built-in to the card. We're also using some old PERC cards with OpenBSD, but I can't recall their model names. In any case, check out the OpenBSD supported hardware list.

  10. By Anonymous Coward (216.162.177.130) treyg@griffinsystems.com on

    I use a Supermicro SuperServer 6013P-T with an LSI MegaRAID SATA 150-4 and it kicks as a samba server giving you 700+ gigs of raid5 when used with 250gig drives. I love it and LSI cards are very nice to work with.

  11. By doxavg (4.17.250.5) on

    I've seen comments that some driver/card combos support notification of drive failures in RAID arrays. Can anyone elaborate on which driver/cards? I've got a whole slew of boxen with Adaptec 2100S cards in them - they work, but don't notify on drive failure. This "question" came at a good time, one of my drives, somewhere in two racks of 20+ machines has died. I know this because I hear the card screaming at me for help. The iop driver is useless for this (not a driver issue, but a protocol issue from what I understand). The Freebsd driver does in theory support all sorts of drive querying and configuring, but it's closed source *sigh*. Soooo any recommendations on cards that can notify OpenBSD that drives have failed? The 2100S is a single channel U160 card...something in the same class would be preferable.

    Comments

    1. By Anonymous Coward (69.197.92.181) on

      One of the adaptec cards listed as supported by the aac driver.

    2. By Anonymous Coward (138.217.52.28) on

      not the 2410SA (serial ata, 4 port). I've used this card, which is listed on i386.html but provides no sw monitoring in obsd whatsoever, the card does emit a loud beep when something is wrong tho...

  12. By Brad (211.29.226.142) on

    OpenBSD isn't designed to run tasks that need the speed of SCSI raid.

    If you intend to run OpenBSD as a heavily used server you'll find GENERIC won't do, you need to recompile and crash a number of times before you get the kernel variables correct, each time you will be the one looking bad.

    My experience is with Dell's and Perc 3's.

    Comments

    1. By Anonymous Coward (69.197.92.181) on

      You are on drugs.

  13. By Bob (202.37.106.8) eletter@rawl.co.nz on

    Good question. We've been using raidctl on obsd since somewhere around
    3.2 with very good results. Would ideally prefer hardware RAID but the
    big downsides have always been (a) lack of RAID status feedback (b) lack
    of RAID control from the OS level and (c) the proprietary data formats
    used to support HW RAID.

    Item (a) is the biggest drawback by far - RAID buys time (not backups)
    and diminished status information equates to diminished time to manage
    disk replacements. This is a big issue for us.

    Item (c) is addressed by backups - although it is often good to know
    disk data can be restored if so required.

    We throw CPU and SCSI cards at raidctl in order to negate the SW versus
    HW resource issues.

    Biggest single gripe with raidctl (in comparison to HW RAID used on many
    UNIX variants) is the downtime while parity rebuilds after an
    unsolicited shutdown. Not a insurmountable issue as obsd seldom if
    ever crashes - esp. as we are very picky on hardware/OS compatibity.

    Would be interested to hear feedback from others.

    Bob

Latest Articles

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]