OpenBSD Journal

Support for Logical Domains on Sun's CoolThreads servers

Contributed by dwc on from the coolness dept.

Mark Kettenis writes:

Over the last couple of weeks, I've been working on support for Logical Domains (LDoms) on Sun's CoolThreads servers, that use the UltraSPARC T1 and T2 (Niagara) processors. These processors have very strong support for virtualization, offering much better seperation between domains than the virtualization available on x86. The LDoms software allows you to split up your server in up to 128 domains, giving each domain dedicated resources such as CPU threads, crypto units, memory and physical I/O. For more informtion on LDoms, see http://www.sun.com/servers/coolthreads/ldoms/index.jsp.

Read on for the good stuff...

Although OpenBSD already did run in a logical domain on those systems, it only had access to "real hardware" made available to the domain. This meant you'd only have network or disk access if you're running in an I/O domain, and the number of possible I/O domains is fairly limited on Sun's current hardware offerings.

The new LDoms support consists of two new drivers to support virtual I/O (VIO). The first one is vnet(4), which is a virtual network driver that allows you to talk to a virtual switch running in another domain that runs Solaris. Since this virtual switch can be associated with a real network device on the Solaris side, this gives you a network path out of the machine. But of course it is also possible to talk to other domains that have a virtual network device connected to that same switch.

The second new device driver is vdsk(4), which is a virtual disk driver. This driver talks to a virtual disk server running in another domain, giving you access to a physical disk, a disk slice/partition or a disk image on a filesystem. The driver emulates SCSI, so you'll see an sd(4) device showing up on your vdsk(4) "controllers".

The new code offers some interesting possibilities. For examples it is possible to run a pf firewall in a dedicated domain that protects a couple of other domains running Solaris, all in a single box. On machines such as the T1000 and T2000 that have a split PCIe bus, you can give the pf firewall direct access to a physical network interface, such that "untrusted" packets don't even have to enter the control domain.

On UltraSPARC T2 systems, there is now also support for the on-chip random number generator through the vrng(4) driver. The entropy data gathered from the processor is added to the kernel entropy pool.

Here's a dmesg of a domain with 12 virtual CPUs (threads), 4GB of memory, a virtual network interface and two virtual disks:

console is /virtual-devices@100/console@1
Copyright (c) 1982, 1986, 1989, 1991, 1993
        The Regents of the University of California.  All rights reserved.
Copyright (c) 1995-2009 OpenBSD. All rights reserved.  http://www.OpenBSD.org

OpenBSD 4.4-current (GENERIC.MP) #7: Sat Jan 17 23:33:12 CET 2009
    kettenis@sparc64.openbsd.org:/usr/src/sys/arch/sparc64/compile/GENERIC.MP
real mem = 4294967296 (4096MB)
avail mem = 4138582016 (3946MB)
mainbus0 at root: SPARC Enterprise T5120
cpu0 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu1 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu2 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu3 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu4 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu5 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu6 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu7 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu8 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu9 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu10 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
cpu11 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz
vbus0 at mainbus0
"flashprom" at vbus0 not configured
"n2cp" at vbus0 not configured
"ncp" at vbus0 not configured
vrng0 at vbus0
vcons0 at vbus0: ivec 0x111
cbus0 at vbus0
vnet0 at cbus0 chan 0x0: ivec 0x200, 0x201, address 00:14:4f:f8:38:e7
vdsk0 at cbus0 chan 0x2: ivec 0x204, 0x205
scsibus0 at vdsk0: 2 targets, initiator 2
sd0 at scsibus0 targ 0 lun 0:  SCSI3 0/direct fixed
sd0: 9216MB, 512 bytes/sec, 18874368 sec total
vdsk1 at cbus0 chan 0x3: ivec 0x206, 0x207
scsibus1 at vdsk1: 2 targets, initiator 2
sd1 at scsibus1 targ 0 lun 0:  SCSI3 0/direct fixed
sd1: 2MB, 512 bytes/sec, 5120 sec total
vrtc0 at vbus0
softraid0 at root
bootpath: /virtual-devices@100,0/channel-devices@200,0/disk@0,0
root on sd0a swap on sd0b dump on sd0b

So far the code has been tested on an UltraSPARC T1 system with LDoms 1.0.1 and an UltraSPARC T2 system with LDoms 1.0.3. However it should work fine with LDoms 1.1 which was released by Sun last december. There is a good chance that OpenBSD will also run fine in a Guest Domain on UltraSPARC T2+ systems. If you have such a system, please try!

An easy way to try OpenBSD on these systems is to download the miniroot44.fs file from a recent snapshot and add a virtual disk to a domain that is backed by this disk image. Simply boot your domain from this virtual disk and start the installation.

(Comments are closed)

Comments

  2. By Anonymous Coward (67.159.44.138) on

    /me drools at the dmesg porn

  3. By Mayuresh Kathe (59.182.225.78) kathe.mayuresh@gmail.com on http://mayuresh.kathe.in/

    This is really cool stuff.
    With OpenBSD, I've learned one thing, it takes longer than others, but when it gets there its better than the rest.

  4. By Nick (63.118.86.10) on

    Great work! Do you have physical access to these monsters, or do you work remotely?

    Comments

    1. By Brad (2001:470:b01e:3:216:41ff:fe17:6933) brad at comstyle dot com on

      > Great work! Do you have physical access to these monsters, or do you work remotely?

      A 1U system is far from a monster.

      Comments

      1. By Anonymous Coward (204.8.156.142) on

        > > Great work! Do you have physical access to these monsters, or do you work remotely?
        >
        > A 1U system is far from a monster.

        Monsters don't have to be big, ugly and hungry.
        With the possibilities opening up i'd call the system a monster, too.

        Comments

        1. By Brad (2001:470:b01e:3:216:41ff:fe17:6933) brad at comstyle dot com on

          > Monsters don't have to be big, ugly and hungry.
          > With the possibilities opening up i'd call the system a monster, too.

          You're buying into the hype from Sun.

          Comments

          1. By Anonymous Coward (91.185.203.253) on

            > > Monsters don't have to be big, ugly and hungry.
            > > With the possibilities opening up i'd call the system a monster, too.
            >
            > You're buying into the hype from Sun.

            Hm, consolidating servers and getting some new features sounds intresting to me. Don't know where the hype fits in there.
            I dont like virtualisation, but the hardware approach taken by Sun or IBM on their bigger systems looks sane to me.
            Sun hardware is ok, at least as good as the usual whiteboxes.
            The cheap T1 has 4 cores. Lets assume it has the split PCIe. Not sure it has.

            - OpenBSD Firewall
            - OpenBSD appserver 1
            - OpenBSD appserver 2 (or folded together with two cores)
            - Solaris 10 for hosting the filesystems for the above on ZFS

            With the snapshot features of ZFS and stuff like iSCSI i can now get into the OpenBSD sessions that makes me feel a little bit tingly inside.

            Will have to ask around in my luug if someone has a T1 system i could try that out on or ask sun for a that trial they offer.

        2. By Miod Vallat (miod) on

          > > A 1U system is far from a monster.
          >
          > Monsters don't have to be big, ugly and hungry.

          Especially those who lurk under your bed.

          Comments

          1. By Anonymous Coward (85.25.152.185) on

            > > > A 1U system is far from a monster.
            > >
            > > Monsters don't have to be big, ugly and hungry.
            >
            > Especially those who lurk under your bed.

            No no no, none under the bed.
            They didn't like the water when cleaning the floor and their tails got in the way too often. Also they made too much noise.
            They now are living in the basement. Some in the attic too, but those are doing some kind of wintersleep.

            But there might be dragons?

      2. By Anonymous Coward (128.6.239.95) on

        > > Great work! Do you have physical access to these monsters, or do you work remotely?
        >
        > A 1U system is far from a monster.

        This one is a LOUD monster in my book. It's louder than virtually all of the other servers in my machine room.

        Thank you for the great work! I'll be trying it soon.

    2. By Anonymous Coward (195.72.48.12) on

      Which is the cheapest sun server that can use this technology?

      Comments

      1. By Anonymous Coward (217.20.112.132) on

        > Which is the cheapest sun server that can use this technology?

        Sun {Sparc Enterprise,Fire} T1000 Server = 3395 US$
        But you might have to put some more $ in for upgrades.
        Personally i am looking at the Fishworks stuff.

        Btw great work Mark. This rocks!
        Now i wont get a new car. Damn you! :)

        Comments

        1. By Anonymous Coward (66.230.230.230) on

          > Personally i am looking at the Fishworks stuff.

          The 7000 series are AMD systems, not T1/2.

        2. By Anonymous Coward (85.25.152.185) on

          > Sun {Sparc Enterprise,Fire} T1000 Server = 3395 US$

          try'n'buy 40% off.

  5. By Adrian Close (203.11.81.235) adrian-undeadly@close.wattle.id.au on

    Works fine with LDoms 1.1 on a T6300 blade (similar to T1000).

Latest Articles

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]