OpenBSD Journal

Computerworld: 'Nightmare' drove desperate user to open source

Contributed by dhartmei on from the success-stories dept.

Rodney Gedda writes in an article on Computerworld about an OpenBSD success story:
IT managers who want to deploy an open source solution but are worried about company politics should go ahead and do it without asking, according to PricewaterhouseCoopers (PWC) Japan IT manager Mark Uemura.

Faced with an unreliable network, Uemura went ahead and migrated systems from Windows to OpenBSD on the premise that management would trust his judgement.

(Comments are closed)

  1. By Fábio Olivé Leite ( on

    "So we had to put an OpenBSD firewall in front of Checkpoint."
    I guess this point says it all. Nice read for this monday morning!

    1. By Fábio Olivé Leite ( on

      Heh, half of my comment subject went away. Perhaps the CGI does not like double quotes in the subject? It was supposed to read 'OpenBSD protects the "Corporate" firewall'.

  2. By Confused ( on

    <quote>"Microsoft just happens to be one of our clients and Checkpoint is our standard firewall," Uemura said. "Checkpoint on Windows was unmanageable but after a few months of using OpenBSD we were told to put Checkpoint back."</quote>

    This paragraph is inpenetrable. I am left with questions like "Huh?" followed by "If it was unmanageable why was OpenBSD better?" and "If it was unmanageable why put it back?" I've seen more depth in a puddle. Anyone care to explain for poor stupid people like me?

    1. By Anonymous Coward ( on

      Sounds like a fairly typical upper management decision to me =) Often stupid decisions are forced onto good tech people by managers with no real understanding of technology (but with absolute authority over IT decisions anyway). Undeniably a really crappy model for managing network/server infrastructure, but none-the-less it is probably the most common model =) I suppose it is also *possible* they had a reason he didn't elaborate on in the article. I think Checkpoint-one has some application-level network proxies these days...

    2. By SleighBoy ( on

      If I had not read the article, I might have been dumbfounded by your posted quote. However, skip to the next sentence..

      Then PWC was hit with a virus affecting network traffic and the Checkpoint firewall was running at 100 percent CPU capacity which was effectively a denial of service.

      "So we had to put an OpenBSD firewall in front of Checkpoint," he said.

    3. By Anonymous Coward ( on

      The way I read that is that even though OpenBSD worked better, they were forced to put the Windows/Checkpoint device back in place because Microsoft is a client of PriceWaterhouseCoopers. Basically some PHB at PWC felt that if Microsoft got wind that they were actually using something OTHER than an MS product, the sh*t would hit the fan. Therefore, to prevent that from happening, just put the crappy Windows box back into production.

    4. By Lars Hansson ( on

      What he is saying is that even thoughOpenBSD worked better they "had" to put Checkpoint back. No, it can't be rationally explained. Upper management decisions are not based on technical realities.

  3. By Anonymous Coward ( on

    Running Checkpoint FW on anything other than a Nokia IPSO box is just plain silly. Quite why one would choose a Wintel box for this is highly questionable. If you put Checkpoint on Windows, RedHat or Solaris, then that does not remove the need to harden the OS and keep it up-to-date with patches. IMHO, the IPSO boxes are designed *for* a specific purpose and are generally quite good. The patching frequency is FAR less. The current batch of Nokia VPN devices are an unmitigated disaster with device failure rates through the roof, but that's a story for another day.

    1. By Anonymous Coward ( on

      Actually, even IPSO is a bad idea, IMHO. I wouldn't run it on anything but Secure Platform (SPLAT). It's CheckPoint's own hardened Linux distribution.

      1. By Anonymous Coward ( on

        For sure! IPSO costs way too much, and honestly nokia's "platform" is nothing more than a pc. After finding OpenBSD, I'll never use checkpoint again.<p>What an overpriced peice of junk.

        1. By Anonymous Coward ( on

          It is simply a PC if you open it up... It's OS is a modified version of FreeBSD; unless they've changed that recently?

          1. By Anonymous Coward ( on

            It is simply a PC if you open it up... It's OS is a modified version of FreeBSD; unless they've changed that recently?

            The last one I touched was running FBSD 2.1.7. It was interesting... No ssh on there, and no https on the web interface. So I took sshd from a FBSD box and installed it there and tunneled web requests over ssh. Amusing thing was that you could pull the FW-1 kernel module off the box and load it up on a stock FBSD box. Last I heard, Nokia was moving to Linux. Odd choice really, it seems like *BSD would be much easier to track/maintain.

            1. By bert ( blambert at thepresidency dot org on

              "Last I heard, Nokia was moving to Linux. Odd choice really, it seems like *BSD would be much easier to track/maintain"

              Yes, but 'BSD' isn't a buzzword, and therefore aren't compliant.

              1. By Anonymous Coward ( on

                That sounds right. Linux as a firewall is a disaster waiting to happen. Not to mention linux on the i386... ugh.

                1. By Anonymous Coward ( on

                  Yes, iptables syntax is really a nightmare (sucks) compared to OpenBSD's PF...

                  1. By scarynetworkguy ( on

                    Keep up at the rear there. This isn't about iptables but rather Firewall-1 running on Linux. Which still sucks. But for a new and different set of reasons.

    2. By djm@ ( on

      Nokia IPSO boxes are overpriced PCs with underpowered processors, running an ancient and lobotomised version of FreeBSD. I.e. there is nothing "magic" about them at all. If I was forced to run Chequepoint again, it would be on just about anything *but* a Nokia.

      1. By Anonymous Coward ( on

        Dont forget the fact that unless you know the sales/support team in person, you are basically fscked for support. That "firewall" solution is the worst.

        Truely the "windows" of the firewall world.

    3. By Mike Murphy ( on

      I really like the IPSO boxes. I run FreeBSD on mine. (1 IP330, 2 IP440, total of 26 Ethernet interfaces). On the IP330, autodetect doesn't work, so, in /etc/rc.early ifconfig fxp0 ether 00:a0:8e:0b:bf:b8 ifconfig fxp1 ether 00:a0:8e:0b:bf:ba ifconfig fxp2 ether 00:a0:8e:0b:bf:ba Look at the motherboard to determine the MAC addresses or choose your own LAN unique ones if you don't want to use a screwdriver. If you use a screwdriver, might as well put in a big disk drive while you're at it. I put a CD on the box for the duration of the install. I'm lazy. For the IP440, do a sysgen with the "device dc" commented out and the 4-port NIC's will correctly identify. You could use OpenBSD if you felt like it, but I kind of like FreeBSD. Sorry for being terminally lazy and not fixing the Ethernet drivers, but the procedure above works OK.

  4. By Nathen Hinson ( on

    Not to detract from the goodwill of the story, but It could very well be that this situation is the difference between a really bad sys admin and a really good one. Although following this logic the really good ones choose OpenBSD ...

    1. By Anonymous Coward ( on

      Yeah, because they actually know something! :)

    2. By Jim ( on

      Because the best admins are lazy. They choose the best tool that will cause them the least pain, automate the rest, and then get a good nights sleep!


  5. By Anonymous Coward ( on

    Another cliched, uninformative "X is better than Windows, we won't actually explain why but trust me" blurb masquerading as an article/news. I really wish tech news sites would stop posting these.

    1. By Luiz Gustavo ( on

      Do you have any idea of how much addon security software for each windows server? Believe me when I say that whole families live by the cost of such stubborn OS.

      And before you claim something, fw-1 is far from the fastest kid outside.

      Btw I serious doubt that SPLAT is hardened or whatever they call it anyway.

      1. By Chris ( on

        That doesn't change the fact that this is a terrible news piece. No depth, no explanation or background into any of the statements made. Just a piece of fluff to pad out the real "story", which is "hey, an IT manager is using OpenBSD, yaay!"

      2. By Anonymous Coward ( on

        SPLAT used to be a seriously cut-down Red Hat kit, I'm not sure if that's still the case. Having said that, Check Point do publish a document which tells you how to build your own "appliance" with RHEL 3: Minimum OS Install guidelines They haven't updated this for NGX and RHEL 4 yet, but this tells you how to build a very basic, locked-down Linux setup that you can tailor to your own needs. Nokia/IPSO isn't the only game in town for high-end deployments - they also offer it on Nortel Alteon, and CrossBeam X-Series hardware.

    2. By Lars Hansson ( on

      Hey, just like the even more common "Windows is better than X because random MS drone Z says so" articles.

      1. By Anonymous Coward ( on

        Maybe I'm reading the wrong sites but I wouldn't say they're equally as common. But yeah, they're just as bad.

  6. By Lars Hansson ( on

    I have done exactly the same thing, ie replaced Checkpoint on Windows with OpenBSD. Granted, this was a few years back and the installation (done by IBM personell) was completely botched to begin with but the situation wasn't helped by Checkpoints general suckiness either.
    Long story short, changed insanely overpowered NT+Checkpoint box into web hosting machine and replaced it with a p2 300 running OpenBSD that is still running flawless to this day.

  7. By Joel Sing ( on

    Mark presented the entirety of this at AUUG2005 last week (as mentioned in the article), along with two tutorials on the topic. Unfortunately a lot of the technical information has been lost in the translation and as per usual, the journalist has picked out key pieces that are good for news (which is to be expected). A copy of his paper would provide much more information with facts rather than just quotations. He knows his stuff.

    1. By Darren Tucker ( on

      One of the presentations is available online. Reducing several 40-minute presentations with live demos to a short article meant there was a lot left out.

      1. By Anonymous Coward ( on

        Sweet! I didn't know you could use CARP for web servers (apache, etc.)? Anyone know of any articles doing such a setup or is it relativly easy? Taken from:

        1. By Anonymous Coward ( on

          Yeah, that looks awesome. I wonder if anyone has audio or video from the presentation that would be great. Also, and I know it's trivial, but I am interested in the hardware he shows there for the carp/pfsync simulation.

        2. By Joel Sing ( on

          You can easily use CARP for any setup which needs failover based redundancy. All you need is to have an additional IP address and create /etc/hostname.carpX files (where X equals 0..9) on each host, with the appropriate configuration. You then use the CARP'd IP address for inbound connections. The only complexity is that of keeping the data and configuration synchronised (eg. same web content when website updates occur).

          The OpenBSD FAQ contains all of the information you should need:

          There are also a couple of sysctl's which you can tweak, as documented in teh FAQ. Also, "man carp" is your friend.

  8. By Mark T. Uemura ( on

    It's unfortunate that reporters such as this guy would sensationalize
    a talk by carefully crafting his story from bits and pieces mostly
    taken out of context. So, in all fairness to my firm and to those who
    were not present, I feel compelled to set the story straight.

    First off, the story is not an interview even though it may come across
    as such. The title is rather sensational but I certainly wasn't
    desperate. There were problems and they were fixed and our team was
    just very resourceful in doing so.

    Gedda writes:
    > IT managers who want to deploy an open source solution but are worried
    > about company politics should go ahead and do it without asking,
    > according to PricewaterhouseCoopers (PWC) Japan IT manager Mark Uemura.

    No, this is taken out of context. What I said was that we had very big
    and important changes that we needed to make in order to restore network
    and application stability. My reference to just going ahead and doing it
    referred to making the necessary changes behind the scenes. It wasn't
    about company politics and it wasn't about migrating services from Windows
    to OpenBSD. My experience was that we did ourselves a disfavour by trying
    to inform and explain to users and management the technical reasons for
    the changes that needed to be made. In fact, all of the pushback had
    nothing to do with OpenBSD. We needed to migrate from an old Domain
    Controller with a corrupt Active Directory to a new one. We also
    introduced the concept of working on Application Servers in Terminal
    Services to take advantage of server power for resource intensive
    applications that ran very slowly on users' PCs. So, the push back was
    related to things like "you'll have to login to this new Domain rather
    than the old one from tomorrow onwards." or getting users to change the
    way they work and use applications running on a Terminal Servers for speed.
    In the end, when all was sad and done, users and management realized the
    difference that we had made; no more downtime or data loss. Furthermore,
    they've never had everything running so smoothly and as efficiently for
    as long as they could remember. Their IT problems went away as a result
    of our efforts and the decisions that we made.

    In fact, all of the migrations to OpenBSD were either behind the scenes
    where the users were oblivious to the changes. Well, almost oblivious.
    Often times we would get "Hey, the Internet is really fast today, cool!"
    or "Man, can you guys like spill some coffee in the server room or
    something? We're not used to this much uptime. It means we can't go
    home early anymore!"

    In those cases where users did have to interact with OpenBSD, it was
    always well received and positive such as moving off of a very slow VPN
    for remote access on to a quicker and more user friendly alternative
    such as port forwarding applications through OpenSSH.

    > Faced with an unreliable network, Uemura went ahead and migrated systems
    > from Windows to OpenBSD on the premise that management would trust his
    > judgement.

    Once again, migrating services to OpenBSD was not an issue. So long as
    we did not compromise security in doing so. Generally, we did so to
    improve security and that's what OpenBSD is famous for and yet there's
    so much more.

    > "PricewaterhouseCoopers is a Windows shop but we were forced to use open
    > source," he said. "I inherited a real nightmare with servers going up
    > and down. There were e-mail outages and on top of that there was a bad
    > relationship between our users and IT."

    Well it's either replace Windows with Window for Internet facing servers
    or find a more secure alternative that didn't have to be patched and
    rebooted so often. Bringing back network and application stability
    was important to the business as much as increasing security wherever it
    was possible to do so. I feel that stability is a result of good security.

    We concentrated on network perimeter security. Hence anything that was
    public facing was considered so long as it satisfied four main criteria:

    1) If security was a concern, then we used a more secure alternative to

    2) If cost was a factor either for software licenses, service/support
    contracts or hardware, then we considered the Open Source alternatives.

    3) If stability and uptime was important, then this was taken into account.

    4) If all three points above qualified, then the last question to be
    answered before replacing any Windows based application or service
    was the following. Will there be any interoperability issues? That
    is, will there be any downside to replacing Windows and implementing
    a more secure, stable, cheaper Open Source alternative? If the
    answer to the last question was "NO", then we used Open Source when

    Once again, we were really concerned with any Internet facing servers.

    > "My predecessor spent too much [so] I was told not to spend any money."

    We could have begged for new hardware but it wasn't necessary. I knew
    that we needed to make big changes that required applications to be
    migrated from hardware to hardware. It is true that management told us
    not to spend any more money than was absolutely needed. This is just
    good business sense and a good rule of thumb to follow for any company
    big or small. If we were given the opportunity to spend on hardware,
    we would have had twice the server power that was really needed for our
    office in the end.

    > When asked what argument he used to convince management to use an open
    > source solution, Uemura said: "They didn't have an argument because they
    > said don't spend any money." "They trusted me," he said. "The whole
    > office was relying on one domain controller which was dying."

    Again, we are not talking about the migration from Windows to OpenBSD.
    The journalist is really good at combining different parts of my talk
    and the answers to questions following it in order make his story that
    much more sensational. This news story is a great example proving that
    you shouldn't believe everything that you read. At face value, it's
    very misleading.

    > Uemura said a lot of work was done "behind the scenes".

    My team did a lot of work behind the scenes for which I am grateful.
    I didn't do it alone.

    > "My experience is that if something has to be done, just do it - don't
    > ask! They will thank you later," he said.

    If you give users a choice, change or no change, they'll tend to favour
    the status quo.

    > commercial lock-in.

    There are many companies that have clued into this however many large
    financial institutions still have big support contracts with Open Source
    Vendors liken to a kid's security blanket where they just don't want to
    let go. I'm not against it, I just don't think it adds much value.
    Rather than hire smart and experienced Admins, they now feel that they
    can skimp on the higher salary candidate for someone not so qualified
    because they have a million dollar support contract in hand. My
    experience has been that hiring inexperienced IT Admins will cost
    you more in third party vendor support as just about anything that
    is remotely difficult gets outsourced.
    Regardless, even with contracts, the savings are still substantial after
    the decision is made to integrate Open Source.

    > "We had a lot of downtime and data loss before we migrated over. After
    > five months that was eliminated," he said. "There is a lot about open
    > source that people don't know. Many corporations tend to lump open
    > source into one basket, which is a shame."

    Sadly, this is my biggest gripe when discussing the merits of OpenBSD.
    It's almost as if having the two words "Open Source" associated with
    OpenBSD just seems to have a negative effect. I've come to realize that
    this is mainly due to a mis

    1. By Mark T. Uemura ( on

      I've come to realize that
      this is mainly due to a misunderstanding or misconception about OpenBSD
      and Open Source.

      > After the five-month migration, PWC's servers are now equally split
      > between Windows and OpenBSD.


      > "Microsoft just happens to be one of our clients and Checkpoint is our
      > standard firewall," Uemura said. "Checkpoint on Windows was unmanageable
      > but after a few months of using OpenBSD we were told to put Checkpoint
      > back."

      Another glaring example of professional journalism at its best :(
      After one month, I was informed that OpenBSD was not the firm standard
      Firewall. No, problem. I just rebuilt the Checkpoint Firewall and
      put that back into production...

      > Then PWC was hit with a virus affecting network traffic and the
      > Checkpoint firewall was running at 100 percent CPU capacity which was
      > effectively a denial of service.

      This was the only case in all of the Windows to OpenBSD migrations that
      I had to struggle with. As much as this Checkpoint was a new
      installation, it didn't sit well with me...

      > "So we had to put an OpenBSD firewall in front of Checkpoint," he said.

      This then satisfied our firm policy and also let me sleep at night
      knowing OpenBSD was the "Titan" out in front taking the worst of it
      without breaking a sweat.

      > "We saved seven salaries worth over one year. It was so dramatic they
      > gave me a big raise and I was promoted from system administrator to IT
      > manager. And because of the savings we get more productivity out of old
      > hardware."

      The savings are all relative but whether you're a small business where
      every penny counts or a large multi-national with huge IT budgets, the
      saving are substantial in either case. Any company that doesn't have
      some sort of Open Source adoption strategy is just throwing money away.

      > Despite this Uemura is adamant the move wasn't made because he wanted
      > to. "As much as I love OpenBSD, we had no choice," he said.

      Just because it can be done, doesn't mean it should be done. We could
      have done a lot more in terms of our migration of services to OpenBSD.
      It's not about pushing the envelope, but getting a balance that makes
      sense while ensuring the utmost in security and interoperability
      when doing so.

      It took about five months with very long days and sacrificing most of
      our weekends in order to get it all done. However, it was worth it in
      the end. Not only did we save the firm money, we increased security,
      stability and restored user confidence in our IT systems and IT team.
      Our great sense of accomplishment was knowing that we did it, all the
      while maintaining business as usual.

      1. By Anonymous Coward ( on

        Hey ... thanks for taking the time post a clarification in this forum! It was much more informative than the Computerworld article.

      2. By Anonymous Coward ( on

        nice post.. i work for the same company, different theater and the "vibe" for open source software here is completely different. Mostly driven by FUD from Microsoft and a bit of ignorance thrown in for good measure. The first question asked of us when outlining a solution to a problem is "is it open source?".

        I saw a nice little draft paper at one stage outlining that we go through and remove ALL open source software anywhere in the company. Not sure how were goint to connect to any of our unix boxes without our consoles, and i am sure we will be fine without firewalls and other such appliances.. but by damn we'll be free of that cancerous open source! Least thats the impression given.

        The short sightedness and extreme political nature of some of the technical decisions made astounds me at times, sometimes its as if we'd rather doom ourselves to an expensive unworkable solution that is completely unlike anything anyone else is running than learn from our peers and run something contempory that fits the problem, even if it is open source.

        my hat is off to you for getting something like that accomplished in this medieval firm we work in.


Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]