OpenBSD Journal

12 Cpus, 96 GB of RAM, OpenBSD

Contributed by paul on from the Enterprise OpenBSD dept.

A recent commit from Mark Kettenis outlines support for yet another system that can now run OpenBSD, a Sun Fire V1280. Much of the work for this was done at the hackathon, with the remainder done remotely working on a machine clear across the globe.
Mark writes:
This one was a bit of a bugger. I started hacking on it during the c2k8 hackathon. Paul Greidanus had set me up with access to the system controller, a nice fast v210 to compile my kernels, and a private network between them such that I could netboot my kernels. However the machine didn't even like our bootloader. This was quite worrying since in the past the bootloader was the part that always just worked. Even for supporting the new sun4v architecture I didn't have to make changes to it.
I noticed that after resetting the machine (which takes quite a while on such a big box) it greeted me with: 
Sun Fire V1280
OpenFirmware version 5.17.0 (03/05/04 11:28)
Copyright 2001-2004 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
SmartFirmware, Copyright (C) 1996-2001.  All rights reserved.
98304 MB memory installed, Serial #54879917.
Ethernet address 0:3:ba:45:66:ad, Host ID: 834566ad.
WTF? SmartFirmware? OpenFirmware 5.x? Even Sun's latest offerings use 4.x. This machine has seriously weird firmware!
After fixing some issues in the bootloader I had it load a kernel, and it actually printed a first message on the console. After that, I hit a brick wall. As soon as I tried taking over control from the firmware completely, the machine would just reset. With a few cheats I got it to print a copyright message and after some more hacking I even got it to probe all devices. But it wouldn't run any userland code. Instead it crashed ending up in a seemingly impossible state. When I left Edmonton it was still in this rather sorry state.
I spent most of the three weeks after the hackathon with travel, hiking and recovering from jetlag. But last week I decided to have another go at the v1280. Looking at the problem again with a fresh mind helped, because I soon realized that the stupid firmware was playing tricks behind my back. After changing the register window handling to be "compatible" with Solaris the machine was much happier, and booted multi-user soon. Meanwhile I had accumulated quite a number of changes to placate the firmware. Testing these changes on other machines took a bit of time, but I finally got them all in the tree last weekend. 
console is /sgcn
Copyright (c) 1982, 1986, 1989, 1991, 1993
	The Regents of the University of California.  All rights reserved.
Copyright (c) 1995-2008 OpenBSD. All rights reserved.  http://www.OpenBSD.org

OpenBSD 4.4-beta (GENERIC.MP) #27: Sun Jul  6 01:44:32 MDT 2008
   root@v1280_head.cein.ualberta.ca:/usr/src/sys/arch/sparc64/compile/GENERIC.MP
real mem = 103079215104 (98304MB)
avail mem = 101204992000 (96516MB)
mainbus0 at root: Sun Fire V1280
ssm0 at mainbus0
cpu0 at ssm0: SUNW,UltraSPARC-III+ (rev 2.3) @ 900 MHz
cpu0: physical 32K instruction (32 b/l), 64K data (32 b/l), 8192K external (512 b/l)
"memory-controller" at ssm0 not configured
cpu1 at ssm0: SUNW,UltraSPARC-III+ (rev 2.3) @ 900 MHz
cpu1: physical 32K instruction (32 b/l), 64K data (32 b/l), 8192K external (512 b/l)
"memory-controller" at ssm0 not configured
cpu2 at ssm0: SUNW,UltraSPARC-III+ (rev 2.3) @ 900 MHz
cpu2: physical 32K instruction (32 b/l), 64K data (32 b/l), 8192K external (512 b/l)
"memory-controller" at ssm0 not configured
cpu3 at ssm0: SUNW,UltraSPARC-III+ (rev 2.3) @ 900 MHz
cpu3: physical 32K instruction (32 b/l), 64K data (32 b/l), 8192K external (512 b/l)
"memory-controller" at ssm0 not configured
cpu4 at ssm0: SUNW,UltraSPARC-III+ (rev 2.3) @ 900 MHz
cpu4: physical 32K instruction (32 b/l), 64K data (32 b/l), 8192K external (512 b/l)
"memory-controller" at ssm0 not configured
cpu5 at ssm0: SUNW,UltraSPARC-III+ (rev 2.3) @ 900 MHz
cpu5: physical 32K instruction (32 b/l), 64K data (32 b/l), 8192K external (512 b/l)
"memory-controller" at ssm0 not configured
cpu6 at ssm0: SUNW,UltraSPARC-III+ (rev 2.3) @ 900 MHz
cpu6: physical 32K instruction (32 b/l), 64K data (32 b/l), 8192K external (512 b/l)
"memory-controller" at ssm0 not configured
cpu7 at ssm0: SUNW,UltraSPARC-III+ (rev 2.3) @ 900 MHz
cpu7: physical 32K instruction (32 b/l), 64K data (32 b/l), 8192K external (512 b/l)
"memory-controller" at ssm0 not configured
cpu8 at ssm0: SUNW,UltraSPARC-III+ (rev 2.3) @ 900 MHz
cpu8: physical 32K instruction (32 b/l), 64K data (32 b/l), 8192K external (512 b/l)
"memory-controller" at ssm0 not configured
cpu9 at ssm0: SUNW,UltraSPARC-III+ (rev 2.3) @ 900 MHz
cpu9: physical 32K instruction (32 b/l), 64K data (32 b/l), 8192K external (512 b/l)
"memory-controller" at ssm0 not configured
cpu10 at ssm0: SUNW,UltraSPARC-III+ (rev 2.3) @ 900 MHz
cpu10: physical 32K instruction (32 b/l), 64K data (32 b/l), 8192K external (512 b/l)
"memory-controller" at ssm0 not configured
cpu11 at ssm0: SUNW,UltraSPARC-III+ (rev 2.3) @ 900 MHz
cpu11: physical 32K instruction (32 b/l), 64K data (32 b/l), 8192K external (512 b/l)
"memory-controller" at ssm0 not configured
schizo0 at ssm0: "Schizo", version 4, ign 600, bus B 0 to 0
schizo0: dvma map c0000000-ffffffff, iotdb 17080000-17180000
pci0 at schizo0
sbbc0 at pci0 dev 4 function 0 "Sun SBBC" rev 0x02
pciide0 at pci0 dev 3 function 0 "CMD Technology PCI0646" rev 0x07: DMA, channel 0
configured to native-PCI, channel 1 configured to native-PCI
pciide0: using ivec 0x608 for native-PCI interrupt
atapiscsi0 at pciide0 channel 0 drive 0
scsibus0 at atapiscsi0: 2 targets, initiator 7
cd0 at scsibus0 targ 0 lun 0:  ATAPI 5/cdrom removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 ignored (disabled)
schizo1 at ssm0: "Schizo", version 4, ign 600, bus A 0 to 0
schizo1: dvma map c0000000-ffffffff, iotdb 17184000-17284000
pci1 at schizo1
siop0 at pci1 dev 2 function 0 "Symbios Logic 53c1010-66" rev 0x01: ivec 0x61c, using
8K of on-board RAM
scsibus1 at siop0: 16 targets, initiator 7
sd0 at scsibus1 targ 0 lun 0:  SCSI3 0/direct fixed
sd0: 34732MB, 24622 cyl, 27 head, 107 sec, 512 bytes/sec, 71132959 sec total
sd1 at scsibus1 targ 1 lun 0:  SCSI3 0/direct fixed
sd1: 34732MB, 24622 cyl, 27 head, 107 sec, 512 bytes/sec, 71132959 sec total
siop1 at pci1 dev 2 function 1 "Symbios Logic 53c1010-66" rev 0x01: ivec 0x61d, using
8K of on-board RAM
scsibus2 at siop
There is still a bit of work to do. These machines don't have a traditional console. Instead they communicate with a system controller through a bit of shared memory. Writing on the console is done by writing the output into shared memory and pinging the system controller. When there is console input, the system controller pings us and we read it from a different bit of shared memory. This doesn't work yet, but for now the firmware console works fine.
Now that the Sun Fire V1280 is supported, OpenBSD will almost certainly run on the Sun Fire 3800/4800/4810/6800 series. There's a fair chance it will also run on the Sun Fire 12K/15K and Sun Fire E2900/E4900/E6900 series. We may even run on the Sun Fire E20K/E25K now! Most of these machines can be partitioned into multiple domains, I can imagine that people may be interested in running an OpenBSD firewall on a small partition in the same box. If you have one of these boxes and a partition that you can play with, it would be nice if you could give a new snapshot a spin on it and tell me about the results. But even if you're running Solaris, you could help me out by sending me the output of "prtconf -pv" for the box.
Great work Mark, keep it up!

(Comments are closed)

Comments

  1. By David Chisnall (82.7.199.50) on

    Is there any benefit yet from running OpenBSD on a machine with so many CPUs, or does lock contention in system calls kill any potential performance benefits?

    Comments

    1. By mk (193.128.72.68) on

      > Is there any benefit yet from running OpenBSD on a machine with so many CPUs, or does lock contention in system calls kill any potential performance benefits?

      It depends on what you're doing on it. If you don't do a lot of IO etc., it'll probably work fairly well for you.

      Comments

      1. By Anonymous Coward (96.21.15.58) on

        > > Is there any benefit yet from running OpenBSD on a machine with so many CPUs, or does lock contention in system calls kill any potential performance benefits?
        >
        > It depends on what you're doing on it. If you don't do a lot of IO etc., it'll probably work fairly well for you.

        So, just surfing the web, with lynx is ok? ;-)

    2. By Anonymous Coward (72.14.20.144) on

      > Is there any benefit yet from running OpenBSD on a machine with so many CPUs, or does lock contention in system calls kill any potential performance benefits?

      Unfortunately, there is almost no serious benefit unless it's strictly userland and not heavily I/O dependent, at least not until rthreads is stable. =(

      Comments

      1. By Anonymous Coward (198.175.14.193) on

        > > Is there any benefit yet from running OpenBSD on a machine with so many CPUs, or does lock contention in system calls kill any potential performance benefits?
        >
        > Unfortunately, there is almost no serious benefit unless it's strictly userland and not heavily I/O dependent, at least not until rthreads is stable. =(
        >

        That's only true if MySQL is your big app... For instance, PostgreSQL already scales across multiple CPUs under the current model, so does gcc with make -j, apache with multiple processes, etc.

        As the kernel moves to get partitioned out with fine-grained locking and starts to run on multiple CPUs, that will be the biggest benefit for some users. For others who aren't yet maxing out the first CPU with the kernel, they can take advantage today. For MySQL, or other heavily threaded apps, rthreads is important.

        FreeBSD's version 7 benchmarkers found that MySQL's thread model is poorly constructed and can't deal with locking contention beyond 16 CPUs. They found that PostgreSQL, which uses no threads at all, just traditional message passing between processes, scales far beyond 16 CPUs. Libthr, their current threading library, is very similar in concept to rthreads (mapping one process per thread.) Rthreads isn't the solution to everything...but it will be nice for some users.

        Comments

        1. By Anonymous Coward (72.14.20.144) on


          > That's only true if MySQL is your big app... For instance, PostgreSQL already scales across multiple CPUs under the current model, so does gcc with make -j, apache with multiple processes, etc.

          <snip the blabber about MySQL vs. PostgreSQL>

          So why even initiate the changes proposed by rthreads? The bottom line is that you can make a few valid points in explicit cases, but that's the extent of the benefit. I can name just as many shortcomings of the current model, not the least of which have forced me to use other platforms for specific applications.

          IMHO, the bottom line is that more users need to enable rthreads on their own platforms to push feedback to the development team so that the giant [Lock] monkey can be cast off our collective backs.

          Comments

          1. By Brad (206.51.28.2) brad at comstyle dot com on

            > IMHO, the bottom line is that more users need to enable rthreads on their own platforms to push feedback to the development team so that the giant [Lock] monkey can be cast off our collective backs.

            rthreads is not at the point where it needs feedback yet. It is still very experimental and needs a lot of work which hopefully should change quite a bit after the 4.4 release. At the moment there are way too many bugs and unresolved issues to realistically use rthreads.

          2. By Damien Miller (djm) on http://www.mindrot.org/~djm/

            > IMHO, the bottom line is that more users need to enable
            > rthreads on their own platforms to push feedback to the
            > development team so that the giant [Lock] monkey can be
            > cast off our collective backs.

            rthreads and biglock are almost completely orthogonal. Finishing rthreads won't magically make the kernel reentrant, and making the kernel reentrant won't fix userland threads.

          3. By Anonymous Coward (66.39.160.90) on

            >
            > So why even initiate the changes proposed by rthreads? The bottom line is that you can make a few valid points in explicit cases, but that's the extent of the benefit. I can name just as many shortcomings of the current model, not the least of which have forced me to use other platforms for specific applications.
            >
            > IMHO, the bottom line is that more users need to enable rthreads on their own platforms to push feedback to the development team so that the giant [Lock] monkey can be cast off our collective backs.

            Rthreads has absolutely nothing what-so-ever to do with the kernel giant lock.

            How can you see shortcomings in the current model when you have no idea what you're talking about?

            Rthreads isn't in a working state yet. Once it works and fully implements a POSIX thread environment with working signals and other common unix features, then we can talk about shortcomings. Right now the shortcoming is that you can't even fucking use it. And, people are working on that.

            Comments

            1. By Gimlet (66.167.69.18) on

              > Right now the shortcoming is that you can't even fucking use it.

              Oh great, it's a GNU project.

      2. By Anonymous Coward (24.91.217.63) on

        What is it with this website and moderating any slightly harmful quote towards OpenBSD to oblivion? Moderation is for trolls, not arguments. Take your ideology wars back to slashdot.

        Comments

        1. By Anonymous Coward (66.39.160.90) on

          > What is it with this website and moderating any slightly harmful quote towards OpenBSD to oblivion? Moderation is for trolls, not arguments. Take your ideology wars back to slashdot.
          >

          Too many morons who don't have the background to actually understand technical issues or make coherent arguments are the ones who spend their worthless time moderating. I suspect a majority who actually have some clue will, at a minimum, not spend their time moderating. So you end up with jingoistic ideology working at a base level against Al Qaeda's anti-OpenBSD messages. Fight on, soldiers.

        2. By Anonymous Coward (89.77.162.243) on

          > What is it with this website and moderating any slightly harmful quote towards OpenBSD to oblivion? Moderation is for trolls, not arguments. Take your ideology wars back to slashdot.
          >

          So -1/3 is oblivion eh?

          OMG... it's full of... moderated comments! Repent now!

          ps. will you chill? most people here have lives and better shit to do than modding...

  2. By Motley Fool (MotleyFool) motleyfool@dieselrepower.org on

    I know where there are a few E6900 at work, unfortunately they're in use and the sys admins won't let me boot a snapshot kernel. :-(

  3. By 1337 (75.216.199.145) on

    What's funny is that in 3 years from now everybody will have a 128 cores cpus anyway.

    You may think I'm kidding, but I'm really not. It's good to know, however that OpenBSD is a little ahead of the curve and maybe the current big iron hardware can help tune scheduling algorithms accordingly, so when these hit our machine rooms or desks, we'll have some serious computing power... to run bittorent and nethack :)

    No, seriously, this is a great accomplishment, while I highly doubt lots of companies will be running OpenBSD on this $300k monster, it's still very neat. I'd love to see some benchmarks on this box, especially vs. Solaris (and sun cc).

    A very nice box with a very nice operating system, yet a little noisy for my apartment.


    Mia

    Comments

    1. By Anonymous Coward (2a01:348:108:155:216:41ff:fe53:6a45) on

      > No, seriously, this is a great accomplishment, while I highly doubt lots of companies will be running OpenBSD on this $300k monster, it's still very neat.

      Read the article about "Most of these machines can be partitioned..." for one possible use.

Latest Articles

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]