Contributed by dlg on from the 10-gigabit-is-fast dept.
About a month and a half ago Reyk and I emailed a few vendors who produce 10Gb ethernet products, one of which was Tehuti Networks Ltd. We offered to write drivers in OpenBSD for their 10Gb ethernet controllers if they would supply hardware and documentation. After some discussion they decided to send us the gear, and a fortnight ago a couple of controllers arrived. 10 days and 80ish commits later the tht(4) driver was enabled on i386 and amd64.
I have to be honest though, Tehuti originally was happy to provide us equipment, but only under an NDA and only for a certain period of time. Obviously this meant that we (OpenBSD) would be unhappy so we declined the offer, but we explained why as well. Our contact in Tehuti understood where we were coming from and took the matter to his CEO, who in turn approved of a real donation of the cards and documentation without the need to sign anything.
As for the NIC itself, it is amazingly simple and clean compared to some other network controllers I've looked at.
The only "strange" thing about it to me is the way it presents each port. Each port is controlled by completely independant sets of registers, with completely independant control for the interrupts and IO paths and everything. Most devices that have this level of separation appear as two different PCI devices (well, functions), but the Tehuti controller appears as a single PCI device, with a single interrupt line and a single register map. Inside this register map are two windows, one for each port.
So my tht(4) driver is actually two drivers in one. There is a thin chunk of driver code that attaches a device called thtc(4) to the PCI bus. It is responsible for mapping the PCI interrupt and registers, but then it iterates over each port of the controller and calls the attach routine for tht(4) itself. tht(4) then creates a register window looking at the ports registers, and establishes its own interrupt handler. So on a dual port card you'll have two interrupt handlers on the same interrupt line.
After splitting each port into its own device, the actual tht(4) driver itself was really easy. You don't have to worry about stepping on the other port since they've been isolated from each other. Each port has a set of four FIFO's (or circular rings of commands) for moving packets on and off the controller. The driver posts commands to the controller via a pair of these FIFOs, and the controller responds with some descriptors of its own in the other pair. The reading and writing of these FIFOs is fairly consistent, so a lot of the code for handling them is abstracted away. Handling the specifics of each FIFO then is very easy, and is surprisingly short.
The chip simply generates an interrupt when it writes something to its pair of FIFOs. We just have to process them, and perhaps take the opportunity to write something more to our own FIFOs for the chip to deal with. I haven't had a chance to try running the NIC with real traffic yet (I'm still waiting on a CX4 cable), but I expect the interrupt load to be relatively low.
Once I do get the cable I can then work on tuning the performance of the driver. Unfortunately due to some deep technical limitations with the OpenBSD kernel, we can't take advantage of some of the features the hardware provides. We're going to do our best to make these nics run well.
Here is the dmesg pr0n for those of you into that sort of thing:
thtc0 at pci1 dev 0 function 0 "Tehuti Networks TN3014" rev 0x00: irq 11
tht0 at thtc0 port 0: address 00:e0:ed:03:40:0e
tht1 at thtc0 port 1: address 00:e0:ed:03:50:0e
I would like to thank Tehuti Networks for being so supportive of this effort. They even put me into contact with their software engineers, who fixed a bug of mine I just couldn't see. They waived the NDA, donated the cards, put a free license on their firmware for us, provided documentation, and provided me with access to their software engineers. Right now they're my favourite vendor. If you guys would like to email a good vendor and say thanks for once, rather than emailing a closed vendor asking for doco, then please feel free to do so.
More specifically I would like to thank Nick Bhavsar, Nadav Shemer, and Alexander Indenbaum at Tehuti. I'd also like to thank Reyk Floeter and Theo de Raadt for working with me on talking to these vendors.
I hate to finish by asking for donations, but Reyk, Claudio Jeker and I are working on supporting all the 10Gb chipsets we can get our hands on. OpenBSD already has support for xge(4), ixgb(4), now tht(4), and soon nx(4), but there is still a list of chips we don't have yet to work on the want page along with some other gear we need to do the work. Just look for the 10Gb ethernet section.
(Comments are closed)