OpenBSD Journal

The Beaver Challenge

Contributed by jose on from the how-fast-is-it? dept.

Dmitry Dorofeev was one of many who sent this in: "People at http://osuosl.org/ are going to bencmark most OSes ! Any OpenBSD gurus to setup our server ?

See http://osuosl.org/benchmarks/bc/

How can we best achieve this benchmarking across many different distributions? Simple, we gather the finest minds and administrators from each camp and bring them together to build the machines themselves.

We have selected the following distributions. This list is not final and if people want to ante in to try this with their favorite distro, let us know at bc2004 at osuosl dot org or in #beaverchallenge on the Freenode.net IRC network.

  • Debian GNU/Linux
  • Fedora Linux
  • FreeBSD
  • Gentoo Linux
  • NetBSD
  • OpenBSD
  • Red Hat Linux
  • Slackware Linux
  • SuSE GNU/Linux

(Comments are closed)


Comments
  1. By Anonymous Coward () on

    OpenBSD shouldn't even be a contender....the system to be tested on is a dual processor.

    Comments
    1. By Noryungi () n o r y u n g i @ y a h o o . c o m on mailto:n o r y u n g i @ y a h o o . c o m


      Yep. Dell SMP system 2xXeon @ 2.0 GHz. Stupid, Stupid, Stupid.

      Here is a link to the machine that will be used.

      FreeBSD may have a chance. NetBSD may be able to install pre-2.0 (1.6.2?) with SMP support, but high performances is not OpenBSD main concern. If they tested reliability and security, OpenBSD may stand a chance...

      Comments
      1. By Job () on

        You hardly have the right to complain that someone decides to measure OS performance on hardware that most people are going to use in their work.

        If they really do a good job and publish decent reports, it may be helpful the developers to see which operations take longest; they can then examine the code to see if they can speed it up.

        There is no reason that something cannot be secure and perform well. A local root vulnerabilty doesn't happen because someone is trying to push data through too fast, it happens because of a design flaw or coding error.

        I've patched a lot of security holes on my own apps over the years, and the patches never make the system slower. Designing an app for security can often put extra steps into the process, but it is almost always possible to find a way to make those steps faster so that the security has the minimum effect on performance.

        Comments
        1. By Kim () on

          I think you miss the point. OpenBSD doesn't have SMP support, so every other contendant has another full CPU to take advantage of. Even if OpenBSD was an equal with the other contenders in terms of speed, it's only going to perform at best, half as well. I think that using a multi-proc system to do benchmarking is going to primarily focus on the SMP code. As far as saying that most people use multi-processor machines in their work - I personally think that is an overstatement. Anyways, code can be secure and effecient at the same time, but I think OpenBSD is plenty effecient. It's just not what they focus on.

          Comments
          1. By Anonymous Coward () on

            Actually, I'm sure that for many benchmarks, dual processing offers at best no advantage (at worst extra overhead). Even so, no SMP code will of course be a show stopper for benchmarks where this is an advantage.

            Still, I hope they decide to run lone processor tests in addition to dual ... most of us don't have dual processor machines.

        2. By Shane () on

          There is no reason that something cannot be secure and perform well. A local root vulnerabilty doesn't happen because someone is trying to push data through too fast

          Hang on a minute. This is not just secure code that been audited. This is code that has additional mechanisms which go to incredible lengths to provide consistency checking and randomisations where they are appropriate.

          No matter how fast you can make that code, it cannot be faster than the same code without the consistency checking and randomisations.

          So as such, OpenBSD cannot be the fastest as long as it retains these mechanisms. It can be fast and secure, but not fastest. And that is okay because the goal of OpenBSD is security, first and foremost.

          I've patched a lot of security holes on my own apps over the years, and the patches never make the system slower.

          This is not about changing a "255" to "254" or a "greater than" to a "greater than or equal". It is about much larger additions to code with the intent of greater security. Extra code hurts "performance".

          But what is "performance" anyway? Web pages served per second? For me security performance is most important. Buy a computer which can saturate your internet connection while running OpenBSD and be happy.

    2. By LB () l0rd@xs4all.nl on http://www.xs4all.nl

      OpenBSD definately isn't a contender as it's built for stability & security. You can't have everything unforunately :(

    3. By Anonymous Coward () on

      I don't care that it isn't the fastest. I use OpenBSD for security. But I would like to know what the relative performance difference is so that I can understand what I am giving up.

      It would be more comparable if it was on a single processor machine. I have no plans to get a dual processor machine for the forseeable future. I just stack up cheap single CPU boxes.

  2. By Anonymous Coward () on

    I think this is stupid... What are they trying to prove? That Linux can be faster than say... OpenBSD? So what if so, if someone needs that, then use Linux I say. That's not OpenBSD's goal(s).

    OpenBSD's goals aren't to be the fastest OS or even speed at that. It's mainly security.

    FreeBSD on the other hand I think would be a good contender, but then again it would be better to test FreeBSD 5.3-stable (once it's out) with better SMP support than 4.x.

    NetBSD, personally, I have next to no experience with so I couldn't comment.

    Just my $0.02.

  3. By Chad Loder () on

    What's with the defeatist attitude? If this is done properly, it's a good opportunity to identify areas where performance can be improved. Remember the last benchmarking fiasco, even though the author was criticized, we did see some commits by (I think) tedu@ and others to fix performance in certain cases.

    We shouldn't be afraid of the truth. Is OpenBSD going to come out on top? I doubt it...but if OpenBSD is shown to be less slow than most people assume, that's a good thing. If performance bottlenecks are identified, it will help prioritize development. I say we should do our best to configure the server, with no excuses, and see what the results are. What are you afraid of?

    Comments
    1. By Matt Fleming () splitscreen@splits.mine.nu on mailto:splitscreen@splits.mine.nu

      I completely agree. Let the benchmark event occur, review the statistics, if there are noticable bottlenecks anywhere, it can only help to improve development.

    2. By Anonymous Coward () on

      Which some doesnt account is to make people see it as a viable option.
      I always think in openbsd to solve my problems and not to be fastest os in the hill.

    3. By tedu () on

      actually it's markus who deserves a lot of the credit for faster socket code.

      Comments
      1. By Anonymous Coward () on

        Faster socket code is cool ... but I really wish someone* would work on parallel disk accesses. I don't know if it is just my machines(?), but with softdeps turned on, my system performance goes absolutely to hell on disk writes (say, from detarring a large archive) -- and there seem to be large (10s) delays for any other process that even wants to read a few bytes from the disk. It seems to me OpenBSD suffers far more from these niggling interactivity issues than any real lack of raw performance.

        *actually I wish I had been gifted with brain cells sufficient to do more than criticize ... but alas it wasn't to be ...

        Comments
        1. By Paul () spawn@maltliquor.ca on mailto:spawn@maltliquor.ca

          Your not alone, a few people including myself have seen this issue before (See mailing list archives).

          I haven't done alot of testing on this, but its been suggested it might something specific in our setups; as a few people aren't able to re-produce these results. Though I've seen this on Intel and Sun boxes with various configurations.

          -Paul

          Comments
          1. By Anonymous Coward () on

            There's definately more to it than just softdeps. All of my servers use softdeps, even a mail server doing thousands of messages per hour. None of them have this problem. If I just take a random machine, do a default openbsd install, and then enable softdeps, it doesn't have this problem. Does a default install + softdeps have this problem for you?

            Comments
            1. By Anonymous Coward () on

              Yes. A stone default install on my older i386 machines of the ~400 mhz era (2 of 2 machines so far) have this problem, and not just under 3.4 but previous releases as well.

              Softdeps has also never worked right on my SparcStation 20.

              At this point I'd just stopped using softdeps entirely -- but since everyone doesn't have this problem I guess I'll keep trying on each individual machine I use.

  4. By Anonymous Coward () on

    It would still be good to see what kind of performance optimizations the OpenBSD gurus would use on their systems.

    Comments
    1. By Anonymous Coward () on

      What was done, and why, would be valuable to the OpenBSD community.

      Comments
      1. By Oliver Neubauer () on

        Agreed. I'd love to be a fly on the wall as crack admins and system gurus tweak their respective systems to the most optimal config possible for the task at hand. If I actually felt I'd have something to offer besides curiosity, I'd sign up just to soak it all up.

        Hopefully they will publish methods along with results (as I think the former will prove more interesting than the latter)....

  5. By Anonymous Coward () on

    i vote for henning

    Comments
    1. By Anonymous Coward () on

      I'd vote for henning, theo and daniel, except that theo would probably yell at the organizers about how they don't know anything

      Comments
      1. By Shane () on

        I'd vote for henning, theo and daniel, except that theo would probably yell at the organizers about how they don't know anything

        And the more we use and enjoy OpenBSD, the more we appreciate Theo's determiniation.

        And BTW, regarding this defeatism crap, OpenBSD actively improves security with a willingness to do this at the expense of performance if that is what is required. It is not a defeatist attitude to admit to the reality that OpenBSD can be very quick, but not quickest by default as long as all these consistency checks and what not are in place.

        I'd be happy to see improvements made to OpenBSD due to areas that might be highlighted in these benchmarks, but I know a lot of slashdot kids will be running around claiming OpenBSD sucks and some arseholes will come in here to gloat over something that matters little to the project in the grand scheme of things.

        Comments
        1. By Anonymous Coward () on

          "but I know a lot of slashdot kids will be running around claiming OpenBSD sucks"...

          And the people who think it's true, deserve to continue in ignorance...or is that bliss?

          Comments
          1. By Anonymous Coward () on

            "but I know a lot of slashdot kids will be running around claiming OpenBSD sucks"...

            And the people who think it's true, deserve to continue in ignorance...or is that bliss?


            I agree. As long as they stay away from the lists and deadly, I'm happy.

  6. By Dave () on

    Just out of curiosity... shouldn't all various Linux distro's be comparable to each other? I mean, aren't they all just the same OS with different packaging? Why would RedHat be measurably faster of slower than Slackware, for example? Of course the different distributions may be at slightly different software releases which might account for minor variations, but overall they should be equal. Or am I completely ignorant about the Linux Universe and the value each distribution adds to the base OS?

    Comments
    1. By asdfg () on

      There are some differences in the way the binaries are compiled and optimized. For example, Linux distributions like Debian tend to be more conservative and are compiled for i386 CPUs. Distros like Gentoo are customized and optimized according to the spec of the machine that they're installed on. Yet others like Slackware take a semi-conservative approach, where it's targeted for i486 CPUs and compiled with the -O2 optimization flag.

      Of course, sites like OSNews claim that Slackware is the fastest and snappiest for desktop use (http://www.osnews.com/story.php?news_id=5307), whatever that means. :-)

      In the end, however, it's up to the experts from each camp to tweak each distribution to the max. I expect that they would probably end up with the "same" optimized binaries for each system with the "same" kernel, so there shouldn't be any difference.

    2. By Anonymous Coward () on

      Well, RedHat for instance has their own substantially patched version of the Linux kernel. For that matter, so does Gentoo. I dunno about the others. If the gurus decide to use the Redhat kernel versus the Gentoo kernel versus, say Slackware running a vanilla kernel, the Linux competition could prove to be very interesting.

      If, on the other hand, they all go to kernel.org and grab 2.6.2 and compile it for i686, then testing multiple Linux distros is pretty pointless.

  7. By Sergey Sokolov () siddkharta@yandex.ru on mailto:siddkharta@yandex.ru

    That is an amasing idea! If U wonna waste your time to persuade somebody in smth, it's may be OK. But what for? That is eternal talk. It will never come to it's end.

  8. By IDunno () on

    Henning & Daniel & the other dude come to mind, they have optimized systems out the wazoo!

  9. By dawg () on

    With the smp disadvantage and some additional overhead for security, I expect openbsd will be slower than most.

    In fact I hope it is slower and I hope all the slashdot sheep make a big deal out of it.

    If they do, hopefully the idiots that can't figure out why won't degrade the openbsd genepool :-) with their presence.

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]