OpenBSD Journal

on iommu

Contributed by mbalmer on from the good-bits-come-in-groups-of-eight dept.

By Jason Wright

AMD64 and SPARC64, two of the 64-bit architectures supported by OpenBSD, contain a device called the IOMMU (Input/Output Memory Management Unit). The IOMMU allows 32-bit devices to see all of the (64-bit addressed) main memory although with a 32-bit address bus you can only address a 32-bit address space. It is a MMU that translates DMA virtual addresses to real physical addresses.

Sounds weird? Well, here is some background information by OpenBSD developer Jason Wright (jason@)

In an i386 machine with less than 4GB of memory, a 32bit device can address all of main memory with no translation assistance. The problem comes when there is more than 4GB of memory. Two solutions exist for this problem.

The more simplistic approach is to use bounce buffers. This is precisely what is done for isadma devices. isadma has a maximum address space of 16MB in the lowest physical addresses. When a buffer needs to be transferred to one of these devices, a bounce buffer is allocated from the low 16MB, the original buffer is copied into the bounce buffer, and the bounce buffer is sent to the device. This same technique will work in 32bit PCI as well where the physical address limitation is a little less tight: 0 - 4GB, but copies are expensive.

The other approach is to take advantage of the hardware address translation services of an IOMMU. When a buffer exists in physical memory outside the range addressable by the device, some adddress space is allocated from the region managed by the IOMMU. This IOMMU DVA, DMA virtual addresses, is then given to the device, and the DVA is entered into the page table of the IOMMU to point to the "real" physical address.

When the device goes to transfer the buffer, the IOMMU intercepts the transfer, finds the real physical address and puts that on the bus instead. The device is completely oblivious to the translation taking place. When the device is done, the translation entry is removed from the table.

IOMMU is present in all "real" AMD64 machines, but not the Intel clones. Unfortunately, OpenBSD support for IOMMU on the AMD machines is not quite ready for primetime (code exists, but "real life" has consorted against me finishing it). IOMMU is also present on sparc and sparc64, and has been working there since the ports were brought up. Similiar translation mechanisms exist on HPPA64 and the 64bit PowerPC's.

(Comments are closed)

Comments

  1. By Anonymous Coward (66.11.66.41) on

    I realize alpha support in openbsd isn't so great, but its also 64 bit.

    Comments

    1. By Anonymous Coward (151.136.100.2) on

      there is no iommu on alpha

      Comments

      1. By Martin Reindl (62.178.75.222) martin@ on

        > there is no iommu on alpha

        And physical memory is currently limited to 1GB.

        Comments

        1. By Anonymous Coward (66.11.66.41) on

          > > there is no iommu on alpha
          >
          > And physical memory is currently limited to 1GB.

          Why 1GB? What is cause of this limit?

      2. By Anonymous Coward (70.27.15.123) on

        > there is no iommu on alpha

        Well, that's a good start. Anyone know any more specifics? How does alpha handle 32 bit devices accessing memory > 4GB? Or does it just not offer anything to help with that?

        Comments

        1. By Jason Wright (65.202.219.66) jason@openbsd.org on http://www.thought.net/jason

          > > there is no iommu on alpha
          >
          > Well, that's a good start. Anyone know any more specifics? How does alpha handle 32 bit devices accessing memory > 4GB? Or does it just not offer anything to help with that?


          Alpha has something similiar to amd64: the sgmap page table (scatter gather map). I didn't comment on alpha because my understanding of sgmaps is rather limited, and I don't own any alphas to play with (this is on purpose).

  2. By Anonymous Coward (209.147.112.69) on

    Does this mean OpenBSD supports/will support over 4GB of RAM?

    Comments

    1. By Berk D. Demir (85.108.62.31) on

      > Does this mean OpenBSD supports/will support over 4GB of RAM?

      mickey@ talked about PAE support in NYCBUG May 2006 meeting.
      MGP slides are located at: http://www.openbsd.org/papers/nyc2006/

    2. By Jason Wright (65.202.219.66) jason@openbsd.org on http://www.thought.net/jason

      > Does this mean OpenBSD supports/will support over 4GB of RAM?

      amd64 doesn't require PAE mode as such

  3. By David Mazieres (66.92.14.60) on

    > IOMMU is present in all "real" AMD64 machines but not the Intel
    > clones.

    Unfortunately, this is not yet true. While it's now possible to buy AMD64 machines that support vmrun (in the Athlon-64 FX-62 CPU), as far as I can tell the motherboards supporting IOMMU won't be available for another few months. If I'm wrong, please tell me a vendor who will sell me an IOMMU-enabled machine, as I'll buy one immediately.

    Note that while unfortunately intel and AMD may be incompatible with this, intel also is coming out with a technology called VT-d that will provide somewhat similar functionality. I'd also love to get a VT-d-enabled machine, if anyone knows a vendor carrying them already.

    For more information on intel's plans, see the following document.

    ftp://download.intel.com/technology/computing/vptech/Intel(r)_VT_for_Direct_IO.pdf

    For more information on AMD's IOMMU architecture, see this document:

    http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/34434.pdf

    Note that one potential huge benefit of these architectures is the prospect of untrusted or less trusted device drivers. Currently drivers are a potential source of kernel bugs. If you can control where a device may access memory through DMA, this can be used to mitigate the thread.

    David

    Comments

    1. By Jason Wright (65.202.219.66) jason@openbsd.org on http://www.thought.net/jason

      > > IOMMU is present in all "real" AMD64 machines but not the Intel
      > > clones.
      >
      > Unfortunately, this is not yet true. While it's now possible to buy AMD64 machines that support vmrun (in the Athlon-64 FX-62 CPU), as far as I can tell the motherboards supporting IOMMU won't be available for another few months. If I'm wrong, please tell me a vendor who will sell me an IOMMU-enabled machine, as I'll buy one immediately.
      >

      The amd64 machine I bought over a year ago has it. The extended GART is an iommu. Intel's EM64T stuff doesn't have the extended GART.

      Comments

      1. By Jason Wright (65.202.219.66) jason@openbsd.org on http://www.thought.net/jason

        > > > IOMMU is present in all "real" AMD64 machines but not the Intel
        > > > clones.
        > >
        > > Unfortunately, this is not yet true. While it's now possible to buy AMD64 machines that support vmrun (in the Athlon-64 FX-62 CPU), as far as I can tell the motherboards supporting IOMMU won't be available for another few months. If I'm wrong, please tell me a vendor who will sell me an IOMMU-enabled machine, as I'll buy one immediately.
        > >
        >
        > The amd64 machine I bought over a year ago has it. The extended GART is an iommu. Intel's EM64T stuff doesn't have the extended GART.

        Ah, right, ok... the iommu architecture you're talking about is more like sparc64 where accesses are FORCED through iommu. What's available today is a voluntary system (bad devices and drivers can still scribble around in main memory).

        Comments

        1. By Kostik Belousov (212.82.216.227) kostikbel@gmail.com on

          > > > > IOMMU is present in all "real" AMD64 machines but not the Intel
          > > > > clones.
          > > >
          > > > Unfortunately, this is not yet true. While it's now possible to buy AMD64 machines that support vmrun (in the Athlon-64 FX-62 CPU), as far as I can tell the motherboards supporting IOMMU won't be available for another few months. If I'm wrong, please tell me a vendor who will sell me an IOMMU-enabled machine, as I'll buy one immediately.
          > > >
          > >
          > > The amd64 machine I bought over a year ago has it. The extended GART is an iommu. Intel's EM64T stuff doesn't have the extended GART.
          >
          > Ah, right, ok... the iommu architecture you're talking about is more like sparc64 where accesses are FORCED through iommu. What's available today is a voluntary system (bad devices and drivers can still scribble around in main memory).

          Could you, please, point to the documentation of this facility
          (GART) in current amd processors ?

          Comments

          1. By Anonymous Coward (65.202.219.66) on

            > > > > > IOMMU is present in all "real" AMD64 machines but not the Intel > > > > > clones. > > > > > > > > Unfortunately, this is not yet true. While it's now possible to buy AMD64 machines that support vmrun (in the Athlon-64 FX-62 CPU), as far as I can tell the motherboards supporting IOMMU won't be available for another few months. If I'm wrong, please tell me a vendor who will sell me an IOMMU-enabled machine, as I'll buy one immediately. > > > > > > > > > > The amd64 machine I bought over a year ago has it. The extended GART is an iommu. Intel's EM64T stuff doesn't have the extended GART. > > > > Ah, right, ok... the iommu architecture you're talking about is more like sparc64 where accesses are FORCED through iommu. What's available today is a voluntary system (bad devices and drivers can still scribble around in main memory). > > Could you, please, point to the documentation of this facility > (GART) in current amd processors ? > Try this. Round abouts page 154 or so.

          2. Comments

            1. By Kostik Belousov (212.82.216.227) on

              > This should be more readable.
              > here, starting on page 154.
              >
              >
              Oh, thanks. I have rev. 3.24, that seems to be silent on the issue.
              Rev. 3.30 is OK.

    2. By Freddie Cash (142.22.16.57) fcash@ocis.net on

      > > IOMMU is present in all "real" AMD64 machines but not the Intel
      > > clones.

      > Unfortunately, this is not yet true. While it's now possible to buy
      > AMD64 machines that support vmrun (in the Athlon-64 FX-62 CPU), as
      > far as I can tell the motherboards supporting IOMMU won't be
      > available for another few months. If I'm wrong, please tell me a
      > vendor who will sell me an IOMMU-enabled machine, as I'll buy one
      > immediately.

      All the server-class Opteron boards we've purchased in the last year include an IOMMU option in the BIOS. Not knowing what it was, or did, I've never enabled it. These are all Tyan Thunder K8-series (K8S, K8SR, K8SD, K8SD Pro) boards using AMD 8000-series and Via chipsets.

      Freddie

  4. By Bob Beck (129.128.11.43) beck@openbsd.org on http://seizurerobots.com


    It should be good *bytes* come in groups of 8

    -Bob

Latest Articles

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]