OpenBSD Journal

Developer blog: dlg: tht(4)

Contributed by dlg on from the 10-gigabit-is-fast dept.

About a month and a half ago Reyk and I emailed a few vendors who produce 10Gb ethernet products, one of which was Tehuti Networks Ltd. We offered to write drivers in OpenBSD for their 10Gb ethernet controllers if they would supply hardware and documentation. After some discussion they decided to send us the gear, and a fortnight ago a couple of controllers arrived. 10 days and 80ish commits later the tht(4) driver was enabled on i386 and amd64.

I have to be honest though, Tehuti originally was happy to provide us equipment, but only under an NDA and only for a certain period of time. Obviously this meant that we (OpenBSD) would be unhappy so we declined the offer, but we explained why as well. Our contact in Tehuti understood where we were coming from and took the matter to his CEO, who in turn approved of a real donation of the cards and documentation without the need to sign anything.

As for the NIC itself, it is amazingly simple and clean compared to some other network controllers I've looked at.

The only "strange" thing about it to me is the way it presents each port. Each port is controlled by completely independant sets of registers, with completely independant control for the interrupts and IO paths and everything. Most devices that have this level of separation appear as two different PCI devices (well, functions), but the Tehuti controller appears as a single PCI device, with a single interrupt line and a single register map. Inside this register map are two windows, one for each port.

So my tht(4) driver is actually two drivers in one. There is a thin chunk of driver code that attaches a device called thtc(4) to the PCI bus. It is responsible for mapping the PCI interrupt and registers, but then it iterates over each port of the controller and calls the attach routine for tht(4) itself. tht(4) then creates a register window looking at the ports registers, and establishes its own interrupt handler. So on a dual port card you'll have two interrupt handlers on the same interrupt line.

After splitting each port into its own device, the actual tht(4) driver itself was really easy. You don't have to worry about stepping on the other port since they've been isolated from each other. Each port has a set of four FIFO's (or circular rings of commands) for moving packets on and off the controller. The driver posts commands to the controller via a pair of these FIFOs, and the controller responds with some descriptors of its own in the other pair. The reading and writing of these FIFOs is fairly consistent, so a lot of the code for handling them is abstracted away. Handling the specifics of each FIFO then is very easy, and is surprisingly short.

The chip simply generates an interrupt when it writes something to its pair of FIFOs. We just have to process them, and perhaps take the opportunity to write something more to our own FIFOs for the chip to deal with. I haven't had a chance to try running the NIC with real traffic yet (I'm still waiting on a CX4 cable), but I expect the interrupt load to be relatively low.

Once I do get the cable I can then work on tuning the performance of the driver. Unfortunately due to some deep technical limitations with the OpenBSD kernel, we can't take advantage of some of the features the hardware provides. We're going to do our best to make these nics run well.

Here is the dmesg pr0n for those of you into that sort of thing:

thtc0 at pci1 dev 0 function 0 "Tehuti Networks TN3014" rev 0x00: irq 11
tht0 at thtc0 port 0: address 00:e0:ed:03:40:0e
tht1 at thtc0 port 1: address 00:e0:ed:03:50:0e

I would like to thank Tehuti Networks for being so supportive of this effort. They even put me into contact with their software engineers, who fixed a bug of mine I just couldn't see. They waived the NDA, donated the cards, put a free license on their firmware for us, provided documentation, and provided me with access to their software engineers. Right now they're my favourite vendor. If you guys would like to email a good vendor and say thanks for once, rather than emailing a closed vendor asking for doco, then please feel free to do so.

More specifically I would like to thank Nick Bhavsar, Nadav Shemer, and Alexander Indenbaum at Tehuti. I'd also like to thank Reyk Floeter and Theo de Raadt for working with me on talking to these vendors.

I hate to finish by asking for donations, but Reyk, Claudio Jeker and I are working on supporting all the 10Gb chipsets we can get our hands on. OpenBSD already has support for xge(4), ixgb(4), now tht(4), and soon nx(4), but there is still a list of chips we don't have yet to work on the want page along with some other gear we need to do the work. Just look for the 10Gb ethernet section.

(Comments are closed)


Comments
  1. By Leonardo Rodrigues (201.34.47.150) on

    Hey, great news. I'll be looking for some >=1gb controllers soon for a new set of computers at work. Seems that tht stuff is hot stuff =)

  2. By Karl Sjödahl (Dunceor) dunceor@gmail.com on

    Its nice to see some succesful stories and not only crap vendors that want NDAs.

    It souunds like a interesting driver with the two drivers in one. I need to check out the code later to see what you did.

    Nice work.

    Comments
    1. By Jason L. Wright (134.20.32.102) jason@openbsd.org on http://www.thought.net/jason


      > It souunds like a interesting driver with the two drivers in one. I need to check out the code later to see what you did.
      >
      sk(4) has a similiar arrangement with a PCI DMA glue chip in front of a pair of MACs. In the case of sk/skc there are three different chips involved, but independent control of each MAC behind the glue chip is cool.

      --Jason L. Wright

  3. By rene (202.63.60.49) on

    dlg great work once again.
    could you describe in more detail the said limitations of the kernel?

    Comments
    1. By David Gwynne (dlg) on

      > dlg great work once again.
      > could you describe in more detail the said limitations of the kernel?

      that would be telling ;)

      tht has some nice features that would make it perform better if multiple cpus were able to push packets onto the hardware at once. because openbsd uses the big giant lock we can't take advantage of that (only one cpu will be running in the kernel at a time).

      thats the big reason.

      Comments
      1. By Cobalt (70.162.93.223) on


        > tht has some nice features that would make it perform better if multiple cpus were able to push packets onto the hardware at once. because openbsd uses the big giant lock we can't take advantage of that (only one cpu will be running in the kernel at a time).

        Are rthreads the solution to this? I know that's project lurking in the wings, but tedu never makes a pretty writeup here at undeadly to satisfy the curious.

        Comments
        1. By Bret Lambert (tbert) on

          >
          > > tht has some nice features that would make it perform better if multiple cpus were able to push packets onto the hardware at once. because openbsd uses the big giant lock we can't take advantage of that (only one cpu will be running in the kernel at a time).
          >
          > Are rthreads the solution to this? I know that's project lurking in the wings, but tedu never makes a pretty writeup here at undeadly to satisfy the curious.

          No, rthreads isn't going to allow a second (or third, or fourth...) process to enter the kernel; basically, every system call/interrupt causes a CPU to try to acquire a spinlock to begin execution in the kernel. Until this is changed (and making several million lines of code thread-safe is, obviously, a major undertaking), the one CPU executing in the kernel at any one time will remain hard and fast.

          As for an rthreads writeup, tedu's short paper should answer most user questions:

          http://www.openbsd.org/papers/eurobsd2005/tedu-rthreads.pdf

        2. By rene (202.63.60.49) on

          >
          > > tht has some nice features that would make it perform better if multiple cpus were able to push packets onto the hardware at once. because openbsd uses the big giant lock we can't take advantage of that (only one cpu will be running in the kernel at a time).
          >
          > Are rthreads the solution to this? I know that's project lurking in the wings, but tedu never makes a pretty writeup here at undeadly to satisfy the curious.

          http://www.onlamp.com/pub/a/bsd/2005/10/20/openbsd_3_8.html?page=last
          See Niklas' response in the interview

  4. By ahafey (82.69.184.245) on

    dmesg pr0n *ROFL*

    really, really, really good news all around!

    I hope that in cases like this the hardware drives a redesign on the limiting factors in the kernel because many people (unfairly) give OpenBSD a hard time in the performance stakes.

    Cheers,
    Alex.

  5. By rgouveia (84.18.240.246) on

    "We offered to write drivers in OpenBSD for their 10Gb ethernet controllers if they would supply hardware and documentation"
    Free OpenBSD Driver Development :-D

  6. By Anonymous Coward (88.191.51.214) on

    "Unfortunately due to some deep technical limitations with the OpenBSD kernel, we can't take advantage of some of the features the hardware provides."

    Can someone elaborate on what the limitations are and what hardware features will not be used? I'm really curious to find out what they are.

    Comments
    1. By Brynet (Brynet) on

      > "Unfortunately due to some deep technical limitations with the OpenBSD kernel, we can't take advantage of some of the features the hardware provides."
      >
      > Can someone elaborate on what the limitations are and what hardware features will not be used? I'm really curious to find out what they are.

      Can't you read?
      http://undeadly.org/cgi?action=article&sid=20070430050134&pid=3

      Jheez..

  7. By garethc (196.25.255.210) garethc@bsdhosting.co.za on

    Dear Linux Devs

    This is how it should be done.

    Secret negotiation with hardware vendors for backroom, linux only deals is not how its done.

    Let this be a lesson to all the people who'd settle for anything less. There should be no compromise.

    The lesson is that not compromising works.

  8. By tmib (tmib) t m i b AT x s 4 a l l DOT n l on

    Awesome, wish more vendors were as cool at Tehuti Networks. Hope this will gain them a lot more business, by being so Open Source/Free Software-friendly.
    Sent a "thank you" email to them and sent a headsup email to the purchasing dept at my work.

    Comments
    1. By Anonymous Coward (207.59.237.99) on

      Not to belabor that which should be obvious, but purchase orders which have a "thank you for being open source friendly" probably pull a whole lot more weight than an email.

      Comments
      1. By tmib (tmib) t m i b AT x s 4 a l l DOT n l on

        > Not to belabor that which should be obvious, but purchase orders which have a "thank you for being open source friendly" probably pull a whole lot more weight than an email.

        Yeah, I know. However, I do not make the decisions regarding purchases, I can only point the team in the right direction and ask them to consider said vendor. Everything after that is beyond my control.

        Comments
        1. By Anonymous Coward (207.59.237.99) on

          > > Not to belabor that which should be obvious, but purchase orders which have a "thank you for being open source friendly" probably pull a whole lot more weight than an email.
          >
          > Yeah, I know. However, I do not make the decisions regarding purchases, I can only point the team in the right direction and ask them to consider said vendor. Everything after that is beyond my control.

          This was by no means a dig as to why you hadn't made purchases; I pretty much figured you didn't have the power to do that when you mentioned 'the guys in purchasing.'

          I was just making a point to the general audience that the better way to improve future vendor response was to reward a well-behaved vendor with sales, and to remind them that that was a factor in gaining your business.

          Then their sales people will move to other companies and remember how it all went down...

  9. By Thank you Reyk and Tehuti Networks Ltd. (82.224.188.215) on http://forum.manucure.info

    That's very good news.

    Thank you Tehuti Networks for stepping in, hopefully other vendors learn the lesson. Finally a vendor who understands that it can only be a good thing for everyone to *help* developpers instead of hiding everything behind trade secrets.
    And thank you Reyk for working hard on the driver.

    Where is it possible to order Tehuti Networks adapters? And besides Cisco and HP, can anyone recommend 24 ports 10 Gb switches that would properly work with these adapters?

    Comments
    1. By Motley Fool (MotleyFool) on

      > That's very good news.
      >
      > Thank you Tehuti Networks for stepping in, hopefully other vendors learn the lesson. Finally a vendor who understands that it can only be a good thing for everyone to *help* developpers instead of hiding everything behind trade secrets.

      Other vendors have stepped up. Over a year ago Metanetworks/Force10 donated qty 2 10G h/w based snort boxes. I'm not sure what the status of those boxes are, but I worked pretty hard to get them donated.

      > And thank you Reyk for working hard on the driver.

      yes, thanks around to all the developers who get 10G support up and running.

  10. By Krunch (213.219.185.12) a_kunysz@y!.com on http://krunch.be/

    I was going to add them to the Vendor Watch wiki but it seems the website is down. Anyone know something about it ? If you don't understand what I'm talking about : Vendor Watch

  11. By David Gwynne (dlg) dlg@openbsd.org on

    I asked Tehuti where to buy controllers from. Here's what they said:

    > The best bet is to refer people to our website where they can find a local
    > sales rep:
    >
    > http://www.tehutinetworks.net/contact/contact.htm

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]