OpenBSD Journal

Bigmem enabled on amd64

Contributed by jj on from the half-meg-half-gig-half-tera dept.

Since this commit:
CVSROOT:	/cvs
Module name:	src
Changes by:	beck@cvs.openbsd.org	2011/04/02 11:36:45

Modified files:
	sys/arch/amd64/amd64: machdep.c 

Log message:
Enable bigmem by default on amd64.
ok deraadt@
amd64 machines may now use more than 4G of ram.

Given the fact that amd64 machines commonly have more than 4G (or rather the 3.5G you end up getting usable after everything else has been deducted) nowadays, this is really good news.

It has required some changes to the buffer cache in order for it to only pick memory in the low 4G range for I/O buffers, and similar changes to various other device drivers, but apart from that, it is ready for testing by you!

I took a few of my local amd64 machines for a spin and they all ran as expected so I decided to take a box at work for a quick run. It got stuck here for a long while before continuing on to this dmesg.

The whole installation went fine, with a record comp49.tgz unpack in 7 seconds (probably attributed to the fast disks more than the memory) but unfortunately, the final installation when booted with all 64 cores and 512G memory enabled left me with these crashes.

I don't have access to the box anymore, so I couldn't test with a bunch of cores (or HT) disabled, but at least the whole installation (which does run in single-processor mode) went through, so there's definitely hope even for big boxes. I have no idea how OpenBSD scales, but if anyone buys me another one of these lovely Dell 910 boxes, I'll surely find out.

(Comments are closed)


Comments
  1. By Amit Kulkarni (amitkulz) amitkulz@gmail.com on http://forestlaw.blogspot.com

    You probably hit some kind of bigmem bug. Try again if you can get access to that machine.

    Theo's message to enable panics to happen, have it in, if you can tolerate problems in your setup
    http://marc.info/?l=openbsd-tech&m=130204512403769&w=2

    The team is slowly going over and fixing the DMA issues. Look at dlg@ and beck@ commits.

    http://marc.info/?l=openbsd-cvs&m=130267980603747&w=2
    http://marc.info/?l=openbsd-cvs&m=130265404616833&w=2
    http://marc.info/?l=openbsd-cvs&m=130265369316363&w=2
    http://marc.info/?l=openbsd-cvs&m=130263762831630&w=2

    Test, test, test as this will make more likely that bigmem with km_malloc sticks and is not backed out.

    Thanks

    Comments
    1. By Janne Johansson (jj) on http://www.inet6.se

      > You probably hit some kind of bigmem bug. Try again if you can get access to that machine.
      >

      I'm positively thinking it has to do with the CPU limits rather than the memory, but I promise to try to get longer access to the box next time, so I can run evil tests on it and grab the whole dmesg out of it. But for now, it was just more fun to get 15 minutes remotely on it than not at all so I took the chance.

  2. By Amit Kulkarni (amitkulz) amitkulz@gmail.com on http://forestlaw.blogspot.com

    Hmmm ISA bus? Why is Dell even packaging those anymore? I thought ISA was ancient history, maybe maybe, is it for a serial console? or IPMI or ILOM or something along those lines?

    The Apr 3-5 snapshots were having too many changes, now those have been backed out, and slowly going back in.

    The CPU's should be able to handle, Xeon 7560 is a year old, but you never know. The install always uses a single CPU, AFAIK.

    Thanks

  3. By Bryan (brakeb) on

    I picked up 8GB for my laptop, and I've been building -current nearly everyday since BIGMEM was default. I've even been building by putting /usr/obj, /usr/xobj, and /usr/dest in RAM.

    Dmesg for e6500 laptop (8GB of RAM, dual core)

    http://pastebin.com/raw.php?i=h79qxn6M

    Server HP m9400t (4GB, quad core) It will take up to 8 GB, but I've not upgraded yet.

    http://pastebin.com/raw.php?i=ti56JGiC

    So far, the only issue I've had is some gcc compiler issues, but if you start the compile of userland again, it goes through without a problem. Don't know if it's a bug, or something else. It's intermittent, and I never know when it will happen.

  4. By chris cappuccio (chriscappuccio) chris@nmedia.net on www.nmedia.net/~chris/

    You can reproduce these crashes...Or vaguely similar ones...On an amd64 box with a slightly different kernel config.

    Basically, get rid of config <blah> and then add a ramdisk to GENERIC or GENERIC.MP:

    option RAMDISK_HOOKS
    option MINIROOTSIZE=7480
    config bsd root on rd0a swap on rd0b and wd0b and sd0b
    pseudo-device rd 1

    NKPTP needs to be raised if the resulting kernel takes more than 16MB, too.

    Depending on whether you do this in 4.8, 4.9, i386, amd64, GENERIC, or GENERIC.MP, any various combination will give you one or the other crash. If you strip out isadma/isadma-dependent devices then of course you can skip the isadma problem, which in the RAMDISK case is predictable as the kernel overlaps the isadma space. So this may not be the same issue, even if the crashes end up looking similar.

    With 4.9, amd64 GENERIC and GENERIC.MP both give a similar uvm crash, with 4.8 only amd64 GENERIC.MP did, and with 4.9 i386 only GENERIC.MP does. Under 4.8, i386 works for GENERIC or GENERIC.MP. Not sure if any of this is interesting, but the isadma crash that you can reproduce is pretty obviously the same one, but the reasons for it are more obvious.

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]