OpenBSD Journal

Developer Blog: kettenis@: Porting OpenBSD to UltraSPARC T1

Contributed by merdely on from the 32-virtual-cpus-on-the-wall,-32-virtual-cpus,-take-one-down dept.

Mark Kettenis (kettenis@) has been hard at work making OpenBSD work on yet another processor type: the UltraSPARC T1. Below, Mark talks about the new processor type and his experience making it work.

Last month, Sun graciously donated a SPARC Enterprise T1000 to make it possible to port OpenBSD to their UltraSPARC T1 processor. These processors have up to 8 cores, and each core runs 4 threads, presenting the operating systems with 32 virtual CPUs. Now that OpenBSD/sparc64 has SMP support, getting it to run on the T1 actually makes some sense.

Mark continues below.

The UltraSPARC T1 (also know as Niagara) is radically different from earlier UltraSPARC CPUs, and includes support for virtualization through a hypervisor that runs in hyperpriviliged mode. The new architecture around the UltraSPARC T1 (and the new UltraSPARC T2) is called sun4v (the old UltraSPARC architecture was sun4u).

All this mean meant that getting OpenBSD to run on these new machines wasn't trivial. That fact that the instruction set is still mostly the same as on the older UltraSPARC CPUs and the fact that the new architecture still uses Open Firmware made it easy to bootstrap a kernel. And the fact that the hypervisor provides a simple and reliable way to print stuff on the console helps a lot with debugging problems. On the other hand, the fact that Sun reduced the number of trap levels from 5 to 2 gives you very little room to make mistakes. As soon as you trap three times in a row, it's game over!

Anyway, I've made quite a bit of progress over the last three weeks, and as of today, the machine runs multiuser. It's only running on a single CPU yet, but getting GENERIC.MP to work is my next major goal. And GENERIC.MP will be dearly needed on these because running on a single thread doesn't give you the speed you'd expect from a 1GHz CPU.

The last major bug that kept me from going multiuser was an interesting one, since it is probably fixes one of the last major bugs that showed up on multiprocesser kernels on the older UltraSPARC machines. I can probably kill three open PR's now [5617, 5637, 5657]. Looking back it is amazing how OpenBSD/sparc64 did manage to stay up at all, and it must have affected uniprocessor machines as well. A fine example that shows that adding support for new hardware helps fixing bugs.

You'll probably see some bits and pieces of the sun4v work hitting the tree in the coming days. But the majority of the stuff probably won't hit the tree before OpenBSD 4.3 is released.

Here is the dmesg:

console is /virtual-devices@100/console@1
Copyright (c) 1982, 1986, 1989, 1991, 1993
        The Regents of the University of California.  All rights reserved.
Copyright (c) 1995-2007 OpenBSD. All rights reserved.  http://www.OpenBSD.org

OpenBSD 4.2-current (GENERIC) #857: Thu Jan  3 21:22:46 CET 2008
    kettenis@bruckner.sibelius.xs4all.nl:/home/kettenis/sun4v/sys/arch/sparc64/compile/GENERIC
real mem = 17171480576 (16376MB)
avail mem = 16806535168 (16027MB)
mainbus0 at root: SPARC Enterprise T1000
cpu0 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
"SUNW,UltraSPARC-T1" at mainbus0 not configured
vbus0 at mainbus0
"nvram" at vbus0 not configured
vcons0 at vbus0
"ncp" at vbus0 not configured
vrtc0 at vbus0
"loop" at vbus0 not configured
"loop" at vbus0 not configured
"echo" at vbus0 not configured
"fma" at vbus0 not configured
"sunvts" at vbus0 not configured
"sunmc" at vbus0 not configured
"explorer" at vbus0 not configured
"led" at vbus0 not configured
"flashupdate" at vbus0 not configured
"flashprom" at vbus0 not configured
"system-management" at vbus0 not configured
vpci0 at mainbus0: bus 2 to 2, dvma map 80000000-ffffffff
pci0 at vpci0
vpci1 at mainbus0: bus 2 to 4, dvma map 80000000-ffffffff
pci1 at vpci1
ppb0 at pci1 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xb3
pci2 at ppb0 bus 3
bge0 at pci2 dev 4 function 0 "Broadcom BCM5714" rev 0xa2, BCM5715 A1 (0x9001): ivec 0x7d4, address 00:14:4f:83:47:7a
brgphy0 at bge0 phy 1: BCM5714 10/100/1000baseT PHY, rev. 0
bge1 at pci2 dev 4 function 1 "Broadcom BCM5714" rev 0xa2, BCM5715 A1 (0x9001): ivec 0x7d5, address 00:14:4f:83:47:7b
brgphy1 at bge1 phy 1: BCM5714 10/100/1000baseT PHY, rev. 0
ppb1 at pci2 dev 8 function 0 "ServerWorks HT-1000 PCIX" rev 0xb3
pci3 at ppb1 bus 4
bge2 at pci3 dev 1 function 0 "Broadcom BCM5704C" rev 0x10, BCM5704 B0 (0x2100): ivec 0x7c2, address 00:14:4f:83:47:7c
brgphy2 at bge2 phy 1: BCM5704 10/100/1000baseT PHY, rev. 0
bge3 at pci3 dev 1 function 1 "Broadcom BCM5704C" rev 0x10, BCM5704 B0 (0x2100): ivec 0x7c1, address 00:14:4f:83:47:7d
brgphy3 at bge3 phy 1: BCM5704 10/100/1000baseT PHY, rev. 0
mpi0 at pci3 dev 2 function 0 "Symbios Logic SAS1064" rev 0x02: ivec 0x7c0
scsibus0 at mpi0: 63 targets
sd0 at scsibus0 targ 0 lun 0:  SCSI2 0/direct fixed
sd0: 70007MB, 14100 cyl, 24 head, 423 sec, 512 bytes/sec, 143374738 sec total
sd1 at scsibus0 targ 1 lun 0:  SCSI2 0/direct fixed
sd1: 70007MB, 14100 cyl, 24 head, 423 sec, 512 bytes/sec, 143374738 sec total
ebus0 at mainbus0
com0 at ebus0 addr c2c000-c2c007 ipl 10: st16650, 32 byte fifo
"pcons" at mainbus0 not configured
softraid0 at root
bootpath: /pci@7c0,0/pci@0,0/pci@8,0/scsi@2,0/disk@1,0
root on sd1a swap on sd1b dump on sd1b

To continue to facilitate great improvements in OpenBSD like this, be sure to buy an extra CD set (Worldwide, Europe) and make a donation. And, continue to test and send in bug reports.

Thanks to Mark Kettenis for taking the time to share his experience porting the T1 with us.

(Comments are closed)


Comments
  1. By Anonymous Coward (24.37.242.64) on

    Sun graciously donated...? WOW! That's both a surprise and great news too, what made them do such a thing all of a sudden? Did they provide the proper and needed documentation too?

    PS: Thank you Sun!

    Comments
    1. By Anonymous Coward (82.243.242.13) on

      > Sun graciously donated...? WOW! That's both a surprise and great news too, what made them do such a thing all of a sudden? Did they provide the proper and needed documentation too?
      >


      This CPU is opensource : http://www.sun.com/processors/opensparc/index.jsp

      Comments
      1. By Anonymous Coward (24.37.242.64) on

        > > Sun graciously donated...? WOW! That's both a surprise and great news too, what made them do such a thing all of a sudden? Did they provide the proper and needed documentation too?
        > >
        >
        >
        > This CPU is opensource : http://www.sun.com/processors/opensparc/index.jsp

        Ohhh, I didn't know that. Should I take back my thanks? kidding... ;-)

        Thanks for letting me know.

    2. By Anonymous Coward (74.13.31.156) on

      > Sun graciously donated...? WOW! That's both a surprise and great news too, what made them do such a thing all of a sudden? Did they provide the proper and needed documentation too?
      >
      > PS: Thank you Sun!

      They have a webpage where they're slowly opening up documents from the machines they've done. They've even consulted OpenBSD developers on it.

      http://wikis.sun.com/display/FOSSdocs/Home

      Comments
      1. By Anonymous Coward (24.37.242.64) on

        > > Sun graciously donated...? WOW! That's both a surprise and great news too, what made them do such a thing all of a sudden? Did they provide the proper and needed documentation too?
        > >
        > > PS: Thank you Sun!
        >
        > They have a webpage where they're slowly opening up documents from the machines they've done. They've even consulted OpenBSD developers on it.
        >
        > http://wikis.sun.com/display/FOSSdocs/Home

        That's awesome, it's nice to see people and companies alike start realizing the amazing potential of OpenBSD and it's developers involved.

        In many respects and aspects, nothing else compares - in terms of closed or FOSS to what's possible from these developers and OpenBSD's potential.

        My $0.02.

        Thanks for the reply!

        Comments
        1. By Anonymous Coward (62.75.149.224) on

          > In many respects and aspects, nothing else compares - in terms of closed or FOSS to what's possible from these developers and OpenBSD's potential.
          >
          > My $0.02.
          >
          > Thanks for the reply!

          That's true but every potential is useless if nobody uses it...
          OpenBSD is just partly good enought for "real" things.
          The IPv6 Stack is ugly and 500Mbit via 10GBit Cards is not that much either but things hopefully progress.

          So lets see what comes out at the end. :-)

          Comments
          1. By Terrell Prude' Jr. (151.188.247.104) tprude@cmosnetworks.com (this is a spamtrap address) on http://www.cmosnetworks.com/

            > That's true but every potential is useless if nobody uses it...
            > OpenBSD is just partly good enought for "real" things.
            > The IPv6 Stack is ugly and 500Mbit via 10GBit Cards is not that much either but things hopefully progress.
            >
            > So lets see what comes out at the end. :-)

            Tell that to anybody (like me) who uses it as a production anti-spam gateway or firewall. I work on Cisco PIXes every day, and OpenBSD has at least feature parity with the PIX. And the failover is superior to the PIX. On top of that, it's about what, US$20,000 less money per box, with no silly "activation key" hassles?

            I know of at least one major ISP that uses OpenBSD firewalls, and they *love* it.

            So if you're going to say something like what you said, you'd do well to back it up.

            --TP

  2. By xender (xender) ventejuy@gmail.com on www.tod-os.com

    This is a opportunity to improve SMP and multithread, not the best part in scalability technologies on OpenBSD.

  3. By Brian Poole (65.117.234.99) on

    Awesome job Mark and good to know about the bug fix in current. I'll be sure to pull that down and recompile on the ole Ultra 2 this evening.

    Is the T2 (Niagara) processor also supported by the same code or are there enough differences to make it a different beast?

    Comments
    1. By Mark Kettenis (82.92.89.47) on

      > Awesome job Mark and good to know about the bug fix in current. I'll be sure to pull that down and recompile on the ole Ultra 2 this evening.
      >
      > Is the T2 (Niagara) processor also supported by the same code or are there enough differences to make it a different beast?

      This gets us a long way towards supporting the T2 too. The CPU is very
      similar, as is the hypervisor interface. But the T2 really is a System-on-a-Chip (SoC), with integrated 10GigE and crypto support. OpenBSD has no drivers for those parts.

      Comments
      1. By Anonymous Coward (24.37.242.64) on

        > > Awesome job Mark and good to know about the bug fix in current. I'll be sure to pull that down and recompile on the ole Ultra 2 this evening.
        > >
        > > Is the T2 (Niagara) processor also supported by the same code or are there enough differences to make it a different beast?
        >
        > This gets us a long way towards supporting the T2 too. The CPU is very
        > similar, as is the hypervisor interface. But the T2 really is a System-on-a-Chip (SoC), with integrated 10GigE and crypto support. OpenBSD has no drivers for those parts.

        Are there plans to get driver support for that eventually too?

        Comments
        1. By David Gwynne (203.213.7.131) loki@animata.net on

          > > > Awesome job Mark and good to know about the bug fix in current. I'll be sure to pull that down and recompile on the ole Ultra 2 this evening.
          > > >
          > > > Is the T2 (Niagara) processor also supported by the same code or are there enough differences to make it a different beast?
          > >
          > > This gets us a long way towards supporting the T2 too. The CPU is very
          > > similar, as is the hypervisor interface. But the T2 really is a System-on-a-Chip (SoC), with integrated 10GigE and crypto support. OpenBSD has no drivers for those parts.
          >
          > Are there plans to get driver support for that eventually too?

          I have a couple of Neptune controllers (Suns 10Gb chips), but they don't want to work in i386 or amd64 machines. I'd like to work on them, but I'd need a sparc64 with pci-express to do it.

          Comments
          1. By Anonymous Coward (199.42.80.225) on

            > > > This gets us a long way towards supporting the T2 too. The CPU is very
            > > > similar, as is the hypervisor interface. But the T2 really is a System-on-a-Chip (SoC), with integrated 10GigE and crypto support. OpenBSD has no drivers for those parts.
            > >
            > > Are there plans to get driver support for that eventually too?
            >
            > I have a couple of Neptune controllers (Suns 10Gb chips), but they don't want to work in i386 or amd64 machines. I'd like to work on them, but I'd need a sparc64 with pci-express to do it.
            >

            Is this time for another fundraiser to get David another box? I'll pitch in a few $CDN, like I did for the Itanic.

  4. By Magnus (24.136.247.63) on http://unixbeard.blogspot.com

    "Now that OpenBSD/sparc64 has SMP support (...)"

    I must have been sleeping; when did that happen?

    I still don't see any reference to it on the sparc64 arch page. Is it not yet stable? Or just missing from the arch notes? Or am I missing it?

    Comments
    1. By Brad (2001:4978:104:3:216:41ff:fe17:6933) brad at comstyle dot com on

      > "Now that OpenBSD/sparc64 has SMP support (...)"
      > 
      > I must have been sleeping; when did that happen?
      > 
      > I still don't see any reference to it on the sparc64 arch page.  Is it not yet stable?  Or just missing from the arch notes?  Or am I missing it?
      

      You were sleeping.

      The MP code has been around for two and a half months now.

      You missed the announcement on undeadly.

      There were a few fixes after the initial check in of code to ensure proper operation and by now should be fairly darn stable. The sparc64 snapshots have been built on a MP system since check in without any issues.

      The arch notes will be updated once a release has gone out.

  5. By Anonymous Coward (85.178.81.252) on

    Are there Plans to rewrite the SMP Code to improve the SMP ability?
    Porting OpenBSD to a T1 is nice, running it with the current SMP code maybe too but still it's not the "best" way.

    Also if you get a 1▀GBit NIC (wich was offered here too) do you may improve the situation related in this area?
    On one talk it was pointed out that OpenBSD can do just 500Mbit via 10Gbit cards and that's kinda useless. So OpenBSD (f.e. if I may have a real network) becomes more and more unattractive.
    It wont be useable as real Firewall nor as Fileserver or DNS or whatever in a 10GBit enviroment (like DeCIX in germany) if it doesn't perform good enought. :-/

    Maybe these are the issues wich could get adressed during the pure porting because a T1 will show the disadvantages of the SMP implementation and a sponsored 10Gbit Card allos further improvements there (or you might just convience the people to use Offloading..))

    Comments
    1. By Anonymous Coward (219.90.152.122) on

      10GBit cards are hard to feed. Performance is secondary to getting them working at all.

      Comments
      1. By Anonymous Coward (128.197.11.30) on

        > 10GBit cards are hard to feed. Performance is secondary to getting them working at all.

        Well if a 10GBit card provides me 500Mbit and a 1Gbit Card nearly 1Gbit... why using a 10Gbit Card then?

        Of course a working driver is great. Please don't realy get me wrong!
        But ISPs and Datacenters (where oBSD may could have fans... f.e. for OSPF or BGP) upgrade heavily their infrastructure to handle the permanent growing amount of data (wich means I personaly don't know any "bigger" center wich is not using 10Gbit anymore).

        So sometimes "performence" isn't secondary.
        I'm not bitching about a 500Mbit differnce or so.
        Lets face it: 500MBit of 10Gbit are 1/20 (with tweaks!) and not a little "performence" glitch. :(

        As I said I won't fight and I'm sure some developers in the OpenBSD community work for such companies as well (ISPs or so) so they're for sure aware of this and think about a good solution. :)

        And hopefully the money from the companies (cross-related to the oBSD Foundation) helps to get this done right. :)

        And if they may work on the kernel they may improve the SMP code as well.

        Comments
        1. By henning (80.171.30.176) henning@ on

          > So sometimes "performence" isn't secondary.
          > I'm not bitching about a 500Mbit differnce or so.
          > Lets face it: 500MBit of 10Gbit are 1/20 (with tweaks!) and not a little "performence" glitch. :(

          you just don't get it do you?
          these were numbers with an early driver and a specific test - if memory serves, it was even a single tcp stream. hardly relevant.
          these numbers only serve one purpose - you can compare after changing code.

          > As I said I won't fight and I'm sure some developers in the OpenBSD community work for such companies as well (ISPs or so) so they're for sure aware of this and think about a good solution. :)
          >
          > And hopefully the money from the companies (cross-related to the oBSD Foundation) helps to get this done right. :)
          >
          > And if they may work on the kernel they may improve the SMP code as well.

          whine whine whine whine

          where's your donation?
          where's your code?

          btw, I have OpenBSD routers at customers that handle over 800 MBit/s of real world traffic on older hardware without even remotely maxing it out.

          but hey, why test if you can whine, right?

    2. By David Gwynne (203.213.7.131) loki@animata.net on

      > Are there Plans to rewrite the SMP Code to improve the SMP ability?
      > Porting OpenBSD to a T1 is nice, running it with the current SMP code maybe too but still it's not the "best" way.
      >
      > Also if you get a 1▀GBit NIC (wich was offered here too) do you may improve the situation related in this area?

      of course.

      > On one talk it was pointed out that OpenBSD can do just 500Mbit via 10Gbit cards and that's kinda useless. So OpenBSD (f.e. if I may have a real network) becomes more and more unattractive.

      that talk said we started at 500Mb before c2k7, but we managed to make a lot of significant improvements. did you read all the slides in that talk?

      > It wont be useable as real Firewall nor as Fileserver or DNS or whatever in a 10GBit enviroment (like DeCIX in germany) if it doesn't perform good enought. :-/
      >
      > Maybe these are the issues wich could get adressed during the pure porting because a T1 will show the disadvantages of the SMP implementation and a sponsored 10Gbit Card allos further improvements there (or you might just convience the people to use Offloading..))

      Comments
      1. By Anonymous Coward (85.178.81.252) on

        > > Are there Plans to rewrite the SMP Code to improve the SMP ability?
        > > Porting OpenBSD to a T1 is nice, running it with the current SMP code maybe too but still it's not the "best" way.
        > >
        > > Also if you get a 1▀GBit NIC (wich was offered here too) do you may improve the situation related in this area?
        >
        > of course.
        >
        > > On one talk it was pointed out that OpenBSD can do just 500Mbit via 10Gbit cards and that's kinda useless. So OpenBSD (f.e. if I may have a real network) becomes more and more unattractive.
        >
        > that talk said we started at 500Mb before c2k7, but we managed to make a lot of significant improvements. did you read all the slides in that talk?

        Well I hopefully understood it too...

        With "tweaks" 500MBit:
        http://www.openbsd.org/papers/cuug2007/mgp00008.html

        Others can do 2.5Gbit (well still JUST 1/4! Sounds like using just 8 our of 32 CPU cores or so...) but is that realy a "great" goal? :-/

        Well there some bottlenecks pointed out but I'm unsure if you could improve the speed that much (like 8-9Gbit) without using Offloading.

        http://www.openbsd.org/papers/cuug2007/mgp00009.html

        I am so far not aware of more precise details or I couldn't find the slides to any more accurate slides.

        If you've updated informations I would be happy to read about it! :-)

        > > It wont be useable as real Firewall nor as Fileserver or DNS or whatever in a 10GBit enviroment (like DeCIX in germany) if it doesn't perform good enought. :-/
        > >
        > > Maybe these are the issues wich could get adressed during the pure porting because a T1 will show the disadvantages of the SMP implementation and a sponsored 10Gbit Card allos further improvements there (or you might just convience the people to use Offloading..))

        Comments
        1. By tedu (204.14.154.8) on

          > Well I hopefully understood it too...
          >
          > With "tweaks" 500MBit:
          > http://www.openbsd.org/papers/cuug2007/mgp00008.html

          uhm, that slide says with *no* tweaks.

    3. By Dan Farrell (thedanno) on

      > It wont be useable as real Firewall nor as Fileserver or DNS or whatever in a 10GBit enviroment (like DeCIX in germany) if it doesn't perform good enought. :-/
      >

      That's just plain BS. A real DNS server doesn't need to operate at 10Gig speeds. Almost no firewalls operate at that speed unless you shelling out some serious money. And if you are truly saturating the 1Gig links on your current fileserver, well, bravo.



      Comments
      1. By Anonymous Coward (68.104.80.54) on

        I gotta agree here. OpenBSD makes a terrific firewall, so I dunno what the other poster who said that it doesn't was smokin'. I don't know of any enterprises (I don't mean Tier 1 ISP's, I mean *enterprises*) that actually run 10Gb links to the Internet. Maybe some 10Gb links internally, but not *to the Internet*. I work on a 200,000-user network with a 1Gb link to the Internet, and we see 400Mb on it all the time, but never above 500Mb. And remember, we're a large enterprise.

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]