Developer blog: marco

Contributed by marco on 2006-05-19 from the let-us-all-raid dept.

Today we are going to talk about mfi(4). I just finished the initial version of mfi(4) which adds support for the next generation MegaRAID cards from LSI and Dell. The hardware is physically different for both companies because LSI and Dell both made their own boards however Dell licensed the firmware from LSI so the driver is the same.

I started writing it a few weeks ago after being inspired by the fact that FreeBSD beat me to it. This part was sort of nice because the FreeBSD guys did a good job on the driver. I used some ideas and (re)used their mfi structure and command headers. The Linux driver is not too bad but it really is a Linux driver and there is no fixing that. There wasn't much inspiration to be had there.

This driver is about the cleanest and simplest I have ever seen. All the magic is done in the firmware and the only thing the driver does is simply fill out a frame and tell the firmware to do DMA it. Whenever ready the firmware will raise an interrupt and the driver completes the IO to the upper layers. Nothing magical at all here. What is really cool is that the driver doesn't need to run in splbio() in the strategy (minus queue manipulation). Another cool thing is the amount of code in the IO path. It is in the order of 50-odd lines of code in the strategy and 25 or so in the completion code. There are no busy-waits or any other stuff that slows down the driver to compensate for hardware issues. In fact this driver is so simple that there isn't much I can tell you about it. The biggest hurdle I faced was a PERC 5/e that wasn't plugged in right and would therefore hang during boot. Re-seating it fixed that PEBKAC bug. The other issue I had was simply a wrong variable name used when calculating the number of frames the firmware had to DMA. All in all this driver was a real joy to write without too much hassle. Give it a go and let me know how it goes.

If you'd ever want to study a storage driver this is a good candidate. It does everything and is very small with simple understandable code.

These are simple single drive RAID 0 SAS disks:

mfi0 at pci8 dev 14 function 0 "Dell PERC SAS" rev 0x00, Dell PERC 5/i: irq 5 mfi0: logical drives 2 version 5.0.1-0030 RAM 256MB scsibus3 at mfi0: 2 targets sd1 at scsibus3 targ 0 lun 0: SCSI5 0/direct fixed sd1: 34432MB, 34432 cyl, 64 head, 32 sec, 512 bytes/sec, 70516736 sec total sd2 at scsibus3 targ 1 lun 0: SCSI5 0/direct fixed sd2: 16896MB, 16896 cyl, 64 head, 32 sec, 512 bytes/sec, 34603008 sec total

This is a RAID 1 using SATA2 drives on a Dell PowerVault MD-1000 enclosure:

mfi1 at pci14 dev 14 function 0 "Dell PERC SAS" rev 0x00, Dell PERC 5/e: irq 5 mfi1: logical drives 1 version 5.0.1-0026 RAM 256MB scsibus4 at mfi1: 1 targets sd3 at scsibus4 targ 0 lun 0: SCSI5 0/direct fixed sd3: 475648MB, 475648 cyl, 64 head, 32 sec, 512 bytes/sec, 974127104 sec total

The test machine:

OpenBSD 3.9-current (GENERIC) #0: Thu May 18 17:17:00 CDT 2006 root@moo64.marcolab.com:/usr/src/sys/arch/amd64/compile/GENERIC real mem = 3489005568 (3407232K) avail mem = 2991525888 (2921412K) using 22937 buffers containing 349110272 bytes (340928K) of memory mainbus0 (root) bios0 at mainbus0: SMBIOS rev. 2.3 @ 0xf9bb0 (87 entries) bios0: Dell Computer Corporation PowerEdge 2800 ipmi0 at mainbus0: version 1.5 interface KCS iobase 0xca8/8 spacing 4 cpu0 at mainbus0: (uniprocessor) cpu0: Intel(R) Xeon(TM) CPU 2.80GHz, 2793.36 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,LONG cpu0: 1MB 64b/line 8-way L2 cache pci0 at mainbus0 bus 0: configuration mode 1 pchb0 at pci0 dev 0 function 0 "Intel E7520 MCH" rev 0x09 ppb0 at pci0 dev 2 function 0 "Intel MCH PCIE" rev 0x09 pci1 at ppb0 bus 1 ppb1 at pci1 dev 0 function 0 "Intel IOP331 Channel 0" rev 0x06 pci2 at ppb1 bus 2 ami0 at pci2 dev 14 function 0 "Dell PERC 4e/Di" rev 0x06: irq 5 ami0: Dell 16e, 32b, FW 513O, BIOS vH418, 256MB RAM ami0: 2 channels, 0 FC loops, 1 logical drives scsibus0 at ami0: 40 targets sd0 at scsibus0 targ 0 lun 0: <AMI, Host drive #00, > SCSI2 0/direct fixed sd0: 139900MB, 139900 cyl, 64 head, 32 sec, 512 bytes/sec, 286515200 sec total scsibus1 at ami0: 16 targets safte0 at scsibus1 targ 6 lun 0: <PE/PV, 1x4 SCSI BP, 1.0> SCSI2 3/processor fixed scsibus2 at ami0: 16 targets safte1 at scsibus2 targ 6 lun 0: <PE/PV, 1x4 SCSI BP, 1.0> SCSI2 3/processor fixed ppb2 at pci1 dev 0 function 2 "Intel IOP331 Channel 1" rev 0x06 pci3 at ppb2 bus 3 ppb3 at pci0 dev 3 function 0 "Intel MCH PCIE" rev 0x09 pci4 at ppb3 bus 4 ppb4 at pci4 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09 pci5 at ppb4 bus 5 fxp0 at pci5 dev 4 function 0 "Intel 8255x" rev 0x0d, i82550: irq 5, address 00:0e:0c:63:e0:f7 inphy0 at fxp0 phy 1: i82555 10/100 PHY, rev. 4 ppb5 at pci4 dev 0 function 2 "Intel PCIE-PCIE" rev 0x09 pci6 at ppb5 bus 6 ppb6 at pci0 dev 4 function 0 "Intel MCH PCIE" rev 0x09 pci7 at ppb6 bus 7 ppb7 at pci7 dev 0 function 0 vendor "Intel", unknown product 0x0370 rev 0x00 pci8 at ppb7 bus 8 mfi0 at pci8 dev 14 function 0 "Dell PERC SAS" rev 0x00, Dell PERC 5/i: irq 5 mfi0: logical drives 2 version 5.0.1-0030 RAM 256MB scsibus3 at mfi0: 2 targets sd1 at scsibus3 targ 0 lun 0: <DELL, PERC 5/i Adapter, 1.00> SCSI5 0/direct fixed sd1: 34432MB, 34432 cyl, 64 head, 32 sec, 512 bytes/sec, 70516736 sec total sd2 at scsibus3 targ 1 lun 0: <DELL, PERC 5/i Adapter, 1.00> SCSI5 0/direct fixed sd2: 16896MB, 16896 cyl, 64 head, 32 sec, 512 bytes/sec, 34603008 sec total ppb8 at pci7 dev 0 function 2 vendor "Intel", unknown product 0x0372 rev 0x00 pci9 at ppb8 bus 9 ppb9 at pci0 dev 5 function 0 "Intel MCH PCIE" rev 0x09 pci10 at ppb9 bus 12 ppb10 at pci10 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09 pci11 at ppb10 bus 13 em0 at pci11 dev 7 function 0 "Intel PRO/1000MT (82541GI)" rev 0x05: irq 11, address 00:11:43:d2:15:17 ppb11 at pci10 dev 0 function 2 "Intel PCIE-PCIE" rev 0x09 pci12 at ppb11 bus 14 ppb12 at pci0 dev 6 function 0 "Intel MCH PCIE" rev 0x09 pci13 at ppb12 bus 15 ppb13 at pci13 dev 0 function 0 vendor "Intel", unknown product 0x0370 rev 0x00 pci14 at ppb13 bus 16 mfi1 at pci14 dev 14 function 0 "Dell PERC SAS" rev 0x00, Dell PERC 5/e: irq 5 mfi1: logical drives 1 version 5.0.1-0026 RAM 256MB scsibus4 at mfi1: 1 targets sd3 at scsibus4 targ 0 lun 0: <DELL, PERC 5/E Adapter, 1.00> SCSI5 0/direct fixed sd3: 475648MB, 475648 cyl, 64 head, 32 sec, 512 bytes/sec, 974127104 sec total ppb14 at pci13 dev 0 function 2 vendor "Intel", unknown product 0x0372 rev 0x00 pci15 at ppb14 bus 17 uhci0 at pci0 dev 29 function 0 "Intel 82801EB/ER USB" rev 0x02: irq 11 usb0 at uhci0: USB revision 1.0 uhub0 at usb0 uhub0: Intel UHCI root hub, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered uhci1 at pci0 dev 29 function 1 "Intel 82801EB/ER USB" rev 0x02: irq 10 usb1 at uhci1: USB revision 1.0 uhub1 at usb1 uhub1: Intel UHCI root hub, rev 1.00/1.00, addr 1 uhub1: 2 ports with 2 removable, self powered uhci2 at pci0 dev 29 function 2 "Intel 82801EB/ER USB" rev 0x02: irq 5 usb2 at uhci2: USB revision 1.0 uhub2 at usb2 uhub2: Intel UHCI root hub, rev 1.00/1.00, addr 1 uhub2: 2 ports with 2 removable, self powered ehci0 at pci0 dev 29 function 7 "Intel 82801EB/ER USB2" rev 0x02: irq 3 usb3 at ehci0: USB revision 2.0 uhub3 at usb3 uhub3: Intel EHCI root hub, rev 2.00/1.00, addr 1 uhub3: 6 ports with 6 removable, self powered ppb15 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0xc2 pci16 at ppb15 bus 20 vga1 at pci16 dev 13 function 0 "ATI Radeon VE QY" rev 0x00 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) pcib0 at pci0 dev 31 function 0 "Intel 82801EB/ER LPC" rev 0x02 pciide0 at pci0 dev 31 function 1 "Intel 82801EB/ER IDE" rev 0x02: DMA, channel 0 configured to compatibility, channel 1 configured to compatibility atapiscsi0 at pciide0 channel 0 drive 0 scsibus5 at atapiscsi0: 2 targets cd0 at scsibus5 targ 0 lun 0: <TEAC, CD-224E, K.9A> SCSI0 5/cdrom removable cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2 pciide0: channel 1 disabled (no drives) isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 pms0 at pckbc0 (aux slot) pckbc0: using irq 12 for aux slot wsmouse0 at pms0 mux 0 pcppi0 at isa0 port 0x61 midi0 at pcppi0: <PC speaker> spkr0 at pcppi0 lpt0 at isa0 port 0x378/4 irq 7 fdc0 at isa0 port 0x3f0/6 irq 6 drq 2 fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec uhub4 at uhub3 port 3 uhub4: Dell product 0xa001, rev 2.00/0.00, addr 2 uhub4: 2 ports with 2 removable, self powered, multiple transaction translators dkcsum: sd0 matches BIOS drive 0x80 dkcsum: sd1 matches BIOS drive 0x82 dkcsum: sd2 matches BIOS drive 0x81 dkcsum: sd3 matches BIOS drive 0x83 root on sd0a rootdev=0x400 rrootdev=0xd00 rawdev=0xd02

(Comments are closed)

Comments

By Anonymous Coward (69.70.207.240) on 2006-05-20 00:32

The Linux driver is not too bad but it really is a Linux driver and there is no fixing that.

Is there a hidden meaning to this? ;)
Comments
1. By Anonymous Coward (67.64.89.177) on 2006-05-20 04:14
  
  Linux drivers have many (mostly useless and dumb) entry points from the kernel. Too much granularity is not always good.
By Anonymous Hippy (203.206.113.116) on 2006-05-20 02:12

Which new LSI cards are these? I was about to go LSI but the MegaRAID SCSI 320-0 offered appears, old.
Comments
1. By Anonymous Coward (67.64.89.177) on 2006-05-20 04:13
  
  It's all in the man pages and on the web page. If you need something now an ami is a very good idea.
2. By Amir Mesry (66.23.227.241) on 2006-05-20 06:50
  
  > Which new LSI cards are these? I was about to go LSI but the MegaRAID SCSI 320-0 offered appears, old.
  
  It's not the 320 series, but the newer Serial Attached SCSI/Sata/Storage series which required a change in architecture to get better throughput and speed as well as some other features that are planned, and they mostlikely will not change Arches for a while, as they like being able to do a universal driver.
3. By Berk D. Demir (85.103.204.99) on 2006-05-20 11:34
  
  > Which new LSI cards are these? I was about to go LSI but the MegaRAID SCSI 320-0 offered appears, old.
  
  On the other hand if you opt for a ready box, all Dell PowerEdge series featuring SAS drives uses this particular card.
  
  Dell used to be a loyal customer of LSI Logic (AMI formerly) for years. PERC (which stands for PowerEdge RAID Controller) is supported by ami(4) and now mfi(4) drivers.
  
  On the other hand, Dell decided to serve alternative AMD Opteron processor powered servers in its PowerEdge product line.
  Officially announced in Dell's Q1 report at http://biz.yahoo.com/bw/060518/20060518005789.html?.v=1
  After this announcement, Intel CEO Otellini was in to tears.
  
  Dell PEs was generally head ache free with OpenBSD. Now it seems they will become perfect configurations.
  
  World is changing...
By Anonymous Coward (24.34.57.27) on 2006-05-20 16:49

This driver is about the cleanest and simplest I have ever seen. All the magic is done in the firmware and the only thing the driver does is simply fill out a frame and tell the firmware to do DMA it.

Is that really such a good thing? I assume this firmware is closed source?
Comments
1. By Antonios Anastasiadis (147.102.223.163) on 2006-05-20 18:17
  
  > Is that really such a good thing? I assume this firmware is closed source?
  
  Yes, it is a good thing. Firmwares run on the hardware they are supposed to support, so simplyfing the driver via them is ok. The problem is that they are not always allowed to be distributed with the os.
  Do not confuse them with binary blobs that run on the kernel.

Latest Articles

Thu, Apr 18
- 05:05 Coming soon to a -current system near you: parallel raw IP input (0)
Wed, Apr 17
- 05:33 In -current, default write format for tar(1) changed to "pax" (10)
Wed, Apr 10
- 18:50 OpenSMTPD 7.5.0p0 Released (2)
Tue, Apr 09
- 04:49 20 years since "and we're just starting": undeadly.org turns 20 (2024-04-09) (13)
Fri, Apr 05
- 06:16 OpenBSD 7.5 released (3)
Thu, Mar 28
- 18:18 LibreSSL 3.8.4 and 3.9.1 released (0)
Tue, Mar 12
- 06:53 OpenSSH 9.7/9.7p1 released! (0)
Mon, Mar 11
- 11:28 Game of Trees 0.97 released (0)
Sun, Mar 10
- 09:06 LibreSSL versions 3.8.3 and 3.9.0 released (0)

Credits

Copyright © 2004-2008 Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to April 2nd 2004 as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]