OpenBSD Journal
Home : : Add Story : : Archives : : About : : Create Account : : Login :
Developer blog: cnst@: fixing make
Contributed by deanna on Tue Jun 19 10:42:41 2007 (GMT)
from the that's-a-lot-of-j dept.

Constantine A. Murenin (cnst@) writes:

A 10-year-old pointer-arithmetic bug in make(1) is now gone, thanks to malloc.conf and some debugging. The bug prevented people from running make -j on OpenBSD for fear of it looping on one the CPUs forever, without actually building anything.

Personally, I've encountered this bug a few times after todd@ told me about the -j option at c2k7. I immediately noticed performance improvements in building kernel with `make -j4` — which was obvious from the fact that now both CPUs were at close to 0% idle time for the duration of the build. However, I've also noticed that shortly after being started with a big -j number, like 16 or 24, the make process would become mysteriously quiet and would start to consume 100% of one of the CPUs (without actually running any jobs) every time I ran it.

Independently (or maybe due to the influence) of the above make(1) bug, one day I've decided to give the malloc options a try, with running `ln -s 'AFGHJPRX' /etc/malloc.conf` to set it up. After setting these malloc options, I've noticed that make was crashing quite often when used with the -j option.

To make the long story short, I've discovered that the problem had to do with pointer arithmetic. A number of bytes was added as an offset for a pointer of type fd_set, which has a size of 128. I.e. the offset in the number of bytes was erroneously multiplied by a factor of 128, thus doing a memset on the unallocated piece of memory, leaving the allocated part uninitialised.

Afterward, millert@ also identified that the original code from 1997 seems to have a potential memory leak due to incorrect realloc usage, so we've fixed that problem too, and a complete patch was committed to 4.1-current

Happy hacking!

[topicblog]

<< OpenBSD: Free As In Air | Reply | Flattened | Expanded | OpenBSD runs great on Soekris 5501 >>

Threshold: Help

Related Links
more by deanna


  Re: Developer blog: cnst@: fixing make (mod 10/56)
by Anonymous Coward (193.63.217.208) on Tue Jun 19 11:48:18 2007 (GMT)
  Excellent, thanks. Will this be back-ported to -stable and 4.0 as a reliability fix?
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: Developer blog: cnst@: fixing make (mod 5/45)
by Anonymous Coward (24.37.242.64) on Tue Jun 19 12:13:03 2007 (GMT)
  Awesome! Thank you!
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: Developer blog: cnst@: fixing make (mod 10/48)
by Anonymous Coward (88.64.133.128) on Tue Jun 19 16:15:22 2007 (GMT)
  Damn it!

I thought that the bug of "looping on one the CPUs forever, without actually building anything" I encountered on many ports I compiled with my distcc port were related to distcc!

Now I have to reevaluate and rebenchmark distcc for ports. I hope this works out and faster port building with many "crap" machines on my home network becomes reality.

Thank you very much, great work!
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: Developer blog: cnst@: fixing make (mod 7/53)
by Anonymous Coward (149.169.206.225) on Tue Jun 19 19:28:48 2007 (GMT)
  How much of a speed increase is this? Are we talking like a double in compile speed because you can multithread?
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: Developer blog: cnst@: fixing make (mod 1/45)
by Anonymous Coward (88.64.159.34) on Wed Jun 20 07:19:52 2007 (GMT)
  Now the only thing left to do is making the system build make -j safe, but I guess this would be a huge amount of work.
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

  Re: Developer blog: cnst@: fixing make (mod 11/51)
by Steve Shockley (68.80.137.106) on Wed Jun 20 23:50:48 2007 (GMT)
  malloc.conf kicks ass!
  [ Show thread ] [ Reply to this comment ] [ Mod Up ] [ Mod Down ]

[ Home | Add Story | Archives | Polls | About ]

Copyright © 2004-2008 Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to April 2nd 2004 as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. Some icons from slashdot.org used with permission from Kathleen. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. Search engine is ht://Dig. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]