OpenBSD Journal

[c2k8]: The amd64 port can now use memory above 4GB

Contributed by johan on from the all-your-mem-are-belong-to-us dept.

Tobias Weingartner (weingart@) has committed largemem support for the amd64 OpenBSD port.

We asked Toby to elaborate a little bit on the topic, here's what he told us:

The amd64 memory handling and detection was still using the old BIOS routines to retrieve the amount of base and extended memory. So the re-writing of the memory detection routine required the use of the "new" E820 bios routine. This has a number of positive effects.

  • it will tend to discover memory above 4G.
  • It will tend to not use memory that is assigned to ACPI (which is a good thing).
  • Many new BIOS will tend to do the right thing with E820, but screw up the others.

Unfortunately, rewriting the memory load routines ended up snowballing in a number of places:

ptoa() needed a cast to make sure that integers passed (constants) to it would sign extend the right way.

Various constants needed "UL" suffixes added to them to make things not complain.

The memory handling routine needed to be made aware of more holes in the memory map (to fix allowing mmap() of various X11/device things)

Along with the new memory loading things, the maximum data segment limit was bumped, so right now a process can use up to 8G of data segment. This of course means that malloc(3) may return pointers that are larger than 32 bits, which I'm sure some ports programs will not enjoy. In particular, the amd64 is an I32LP64 architecture, which means that an "int" will not be large enough to hold a pointer.

(Comments are closed)

Comments

  1. By guilherme m. schroeder (lero) on

    Working fine here:

    OpenBSD 4.3-current (GENERIC.MP) #0: Tue Jun 10 02:30:27 BRT 2008
    root@plasma.loop.int:/usr/src/sys/arch/amd64/compile/GENERIC.MP
    real mem = 4245254144 (4048MB)
    avail mem = 4122628096 (3931MB)

    ;)

    Comments

    1. By Anonymous Coward (2001:6f8:94d:4:2c0:9fff:fe1a:6a01) on

      > real mem = 4245254144 (4048MB)

      That's less than 4 GiB.

      Comments

      1. By guilherme m. schroeder (lero) on

        > > real mem = 4245254144 (4048MB)
        >
        > That's less than 4 GiB.

        Yeah, but now it recognizes 4GiB. Some days ago just 3GiB.

      2. By Chris Kuethe (142.244.224.68) ckuethe@ on

        > > real mem = 4245254144 (4048MB)
        >
        > That's less than 4 GiB.

        Yes, that's less than 4G, but it's important to test for regressions, and near various boundaries.

        real mem = 8509796352 (8115MB)
        avail mem = 8264597504 (7881MB)

        Works nicely with processes using 4100MB of physmem.

        Comments

        1. By Oliver (62.178.151.180) on

          > Works nicely with processes using 4100MB of physmem.

          EMACS? ;-)

          Comments

          1. By Anonymous Coward (70.61.44.242) on

            > > Works nicely with processes using 4100MB of physmem.
            >
            > EMACS? ;-)
            >
            >

            this joke is no longer relevant

  2. By Anonymous Coward (213.221.123.170) on

    AMD supports the execution of 32Bit code in 64Bit mode.
    of course this wont help if you compile except you may specify that you'll build for 32Bit. If OpenBSD adds such a layer there should be no problems for ports.

    But except this: What did Theo write? All my memory belongs to you ;-p
    THANKS!

    Comments

    1. By Anonymous Coward (212.20.215.132) on

      > But except this: What did Theo write? All my memory belongs to you ;-p
      > THANKS!

      I guess Tobias meant "All your memory _are_ belong to us." ;-)
      (http://en.wikipedia.org/wiki/All_your_base_are_belong_to_us)

    2. By Anonymous Coward (80.152.237.153) on

      > AMD supports the execution of 32Bit code in 64Bit mode.
      > of course this wont help if you compile except you may specify that you'll build for 32Bit. If OpenBSD adds such a layer there should be no problems for ports.

      I don't think this will happen. Just see the comment for revision 1.75 of sys/arch/amd64/amd64/machdep.c.

      Comments

      1. By Anonymous Coward (128.171.90.200) on

        > I don't think this will happen. Just see the comment for revision 1.75 of sys/arch/amd64/amd64/machdep.c.

        http://www.openbsd.org/cgi-bin/cvsweb/src/sys/arch/amd64/amd64/machdep.c

        - remove USER_LDT, it was never in a state where it would copile, nor will we support i386-compat mode on amd64.

        agreed by beck@, dlg@, kettenis@
        ok deraadt@, tom@

  3. By Paul 'WEiRD' de Weerd (weerd) weerd@weirdnet.nl on http://www.weirdnet.nl/

    This is cool ! 

    OpenBSD 4.3-current (GENERIC.MP) #1701: Mon Jun  9 03:09:09 MDT 2008
    deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 8576643072 (8179MB)
avail mem = 8329785344 (7943MB)

    Fresh dmesg sent in to dmesg@, thanks Tobias and others !

  4. By Brynet (Brynet) on

    I'm a fairly cheap individual, so I'm wondering, What's the maximum amount of memory anyone has seen on a OpenBSD supported machine?

    Just curious.. and if your wondering, yes, that was a request for dmesg pr0n. :-)

    Comments

    1. By Anonymous Coward (142.244.224.31) on

      No dmesg porn... 56GB is the most I've seen.

      -Toby.

    2. By Matthieu Herrb (142.244.224.96) on

      > I'm a fairly cheap individual, so I'm wondering, What's the maximum amount of memory anyone has seen on a OpenBSD supported machine?
      >
      > Just curious.. and if your wondering, yes, that was a request for dmesg pr0n. :-)

      OpenBSD 4.2-current (GENERIC.MP) #0: Tue Nov 13 20:50:59 CET 2007
      root@brutus.xxxx.xx:/usr/src/sys/arch/amd64/compile/GENERIC.MP
      real mem = 222788734976 (212467MB)
      avail mem = 216185495552 (206170MB)
      mainbus0 at root
      bios0 at mainbus0: SMBIOS rev. 2.3 @ 0xfbf20 (83 entries)
      bios0: vendor American Megatrends Inc. version "080012" date 04/19/2007
      bios0: Sun Microsystems Sun Fire X4600 M2

      Comments

      1. By Terrell Prude' Jr. (151.188.18.43) tprude@cmosnetworks.com (this is a spamtrap address) on http://www.cmosnetworks.com/

        > OpenBSD 4.2-current (GENERIC.MP) #0: Tue Nov 13 20:50:59 CET 2007
        > root@brutus.xxxx.xx:/usr/src/sys/arch/amd64/compile/GENERIC.MP
        > real mem = 222788734976 (212467MB)
        > avail mem = 216185495552 (206170MB)
        > mainbus0 at root
        > bios0 at mainbus0: SMBIOS rev. 2.3 @ 0xfbf20 (83 entries)
        > bios0: vendor American Megatrends Inc. version "080012" date 04/19/2007
        > bios0: Sun Microsystems Sun Fire X4600 M2
        >

        Ay, chihuahua! That is a beautiful thing, indeed. The most I've got is 3.5GB DRAM on a dual Athlon MP box from way back. But it works. :-)

        --TP

  5. By Yusuf (58.152.232.171) yusufg@gmail.com on

    Would this help in putting large number of IP addresses/CIDR in pf(4) tables via pfctl(8). This would be for (ab)using spamd :)

    I recollect that pf(4) had to use kernel memory which limited the number of entries in tables and that was limited to 1GB (or maybe that limit went away a long time ago)

    Comments

    1. By Ryan McBride (2001:240:581:69::218) mcbride@openbsd.org on

      Would this help in putting large number of IP addresses/CIDR in pf(4) tables via pfctl(8). This would be for (ab)using spamd :)

      Not yet, the amount of memory accessible to the kernel is still limited. But work is being done to improve this wherever possible.

      However, I just committed a change to PF which avoids storing 64bytes of byte/packet counters in each table entry, saving around 40% memory. So you should be able to (ab)use spamd that much further in the existing memory limits.

      I recollect that pf(4) had to use kernel memory which limited the number of entries in tables and that was limited to 1GB (or maybe that limit went away a long time ago)

      That's basically correct; the actual limit varies a bit per architecture and is a little more complicated than just x GB or y MB. I'm not sure exactly what amd64's limit is currently, but it definately has a lot of room for improvement.

  6. By sthen@ (142.244.224.106) on

    Some third-party software written without due regard to portability will break at runtime after this change. Now is the time to test ports/packages that you depend on and work with the maintainer and/or ports@ to help get problems fixed.

    Comments

    1. By Anonymous Coward (67.42.90.210) on

      > Some third-party software written without due regard to portability will break at runtime after this change.

      I don't know if it has to be "written without regard to portability". It's easy for assumptions about type sizes to creep in, even if you are normally very mindful of these things. That's why it's so important to test.

Latest Articles

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]