OpenBSD Journal

What about other bug-squashing tools like SPLINT and Valgrind

Contributed by jose on from the automated-bug-discovery dept.

(anonymous coward) writes :
"I was wondering if there was a reason that OpenBSD didn't make more use of tools like Splint http://www.splint.org?

Or possibly someone is working on a project like (or port of) valgrind? http://developer.kde.org/~sewardj/

Thanks !"

Having used and written some automated code checking tools, I do wonder if some of the project members use them or why they don't. While you can't catch several bug types with automated tools, as a general first pass it seems to do a pretty good job. Any insight about the OpenBSD project's use of such tools?

(Comments are closed)


Comments
  1. By Christian Gruber () cgruber@nospam-israfil.net on mailto:cgruber@nospam-israfil.net

    ... and implementing proper bounds checking, etc.

    Not that I'm against auditing tools and correcting tools, but I have to fight against the tendancy in my employees to rely on such tools to the detriment of their understanding of their own code, and to the detriment of good coding habits.

  2. By Anonymous Coward () on

    man 1 lint

    Comments
    1. By Anonymous Coward () on

      lint is much more limited in the tests that it applies to your code. Which is not to say that it is not worth running, just that it has limitations.

      "gcc -Wall" is another good tool - if your code generates a single warning then you should sort it out.

      Several papers describing automated tests and static analysis also suggest that gcc be improved to (optionally) apply these tests and generate warnings accordingly. Which would be nice. However bugs will still get through - demonstrating the need for good coders.

    2. By tedu () on

      you'll never find a real bug. do you know how many printfs there are that aren't cast to void?

      Comments
      1. By Jeffrey () on

        Being that my C skills are limited... Could you (or anyone) perhaps enlighten me as to the real purpose of casting printfs to void. Be gentle, I'm an idiot ;)

        Or maybe recommend something I might read..?

        Is the reasoning something like...
        If you do not intend to use the return value of a function in further computations, then you should cast it to void.

        That's my best guess anyway =)

        Comments
        1. By Jeffrey () on

          ... But is it a "real" bug if you don't cast it to void..?

          Sorry for the stupid question.

          Comments
          1. By Anonymous Coward () on

            Yes and no. As far as lint is concerned, printf() could be malloc(), and discarding its return value could be a serious bug. We happen to know that it's not, and we could hack lint to make it know the difference, but then what happens if someone writes my_malloc() and then somewhere accidentally discards the return value? Again, lint won't know if that's important or not, and if the programmer is expecting lint to catch that sort of bug, he'll be surprised. The only safe thing is for lint to flag every discarded return values, even ones (like printf) that we know are safe.

        2. By tedu () on

          you cast it to void to shut up lint. it gets annoying REAL fast. not just printf, but memcpy and lots of other functions. your reasoning is correct, and that's what lint checks for, but in the real world, it looks ugly and wastes time. hth

          Comments
          1. By Jeffrey () on

            Ah yes, now I see.
            Your statement, along with some docs I read yesterday,
            clears my previous confusion about this.

            Thanks! =)

  3. By Anonymous Coward () on

    ( cd /usr/ports/devel/splint && make install ) && splint

  4. By Anonymous Coward () on

    We know that the tools (lint, splint) are there on BSD, the question is why aren't things like OpenSSH, OpenSSL, PF userland utils, etc. etc. annotated to USE them.

    If they are integrated into the build, it makes it less likely that patch submitters dork something up they will have to annotate and get it working within the confines of (sp)lint, it also can prevent a maintainer from dorking themselves at 2:00 AM(if that is a usual hour for sleeping :).

    Valgrind can be another useful tool to help detect bugs and would make a GREAT addition to OBSD.

    Comments
    1. By djm () on

      Submit a patch, then we can talk about it.

    2. By Daniel Hartmeier () daniel@benzedrine.cx on mailto:daniel@benzedrine.cx

      I just ran src/sbin/pfctl/*.c through splint.

      With -checks (the default), you get no warnings at all.

      With -strict, you get an overwhelming number of warnings.
      After spending 30 minutes manually disabling harmless warnings, more than 75% remained, and I haven't found a single bug due to a helpful warning.

      The next step would be to annotate the code to disable warnings in more detail. But there's a limit of far that would go, I doubt anyone would be willing to go through /usr/include and annotate all headers, it's just too much work.

      The question is, how much time do you have to invest to find how many bugs. That's the point, finding bugs. And by just reading the code I find more bugs in the same time.

      And try running splint on kernel sources, it doesn't seem to like that at all. :)

      So, yes, I'd like to get the 'interesting' warnings that help find bugs. But if it costs a week to set up the tool to get only the interesting warnings, it doesn't get used. Try it, you'll see what I mean.

  5. By Chad Loder () on http://www.loder.us/

    The biggest obstacle to doing stuff like this is Theo himself. I speak from experience. There's only a handful of OpenBSD developers who are capable of, e.g., porting valgrind to OpenBSD (valgrind is VERY Linux specific and mucking with the instrumentation is non-trivial), and Theo's point of view is that he'd rather have them doing other more important things.

    Theo has a good point -- when people start focusing on tools like this, people start believing that the GOAL of the whole exercise is to get rid of the warnings. This is of course a complete waste of time -- the goal is not getting splint (or whatever) to shut up, the goal is to find and fix bugs. It's a complete waste of time to go through some code and sprinkle (unsigned) casts here and there to shut up the diagnostics tools. It's also a very good way to introduce new bugs (or permanently hide old ones).

    That having been said, here's what I think. First: lint sucks. NetBSD's xlint is a *teensy* bit better but...it still *sucks*. 99% of the stuff it complains about is not important, and it doesn't find 99% of the kinds of bugs that usually occur.

    Splint is loads better than lint. If you're going to use this class of tools, then splint is the way to go. You can easily customize it, and the output it yields can be pretty high quality if you are willing to put the work into customizing the splint language libraries.

    The biggest issue with splint right now is that it doesn't grok a lot of the BSD structures and functions. It also barfs on some of the newer GCCisms. However it's not hard to add that stuff, it just takes time.

    I'm going to paste a note I sent to another OpenBSD developer a few months back when I started playing with splint. This is not a complete TODO list, there are lots and LOTS more things that need to be done, but you could make splint useful if you wanted to, and here's a start:

    - splint doesn't know about:
    strl*(3)
    err(3), warn(3), errx(3), warnx(3), etc.
    daemon(3)
    getaddrinfo(3), which is supposedly part f IEEE POSIX 1003.1g
    gai_strerror(3), etc.

    - splint seems to have a different prototype for syslog() than OpenBSD does.
    splint says arg 3 of vsyslog must be a char*, not a va_list

    - splint says bzero(3) expects arg 2 to be an int, we say size_t

    - splint doesn't know about certain non-standard BSD types:
    register_t (defined in machine/types.h)
    sig_t (defined in sys/signal.h)

    u_quad_t
    v_addr_t (defined in machine/types.h)
    u_int16_t splint/lib/standard.h knows about uint16_t (c99) but not u_int16_t
    u_int32_t same
    u_int64_t same
    u_int8_t same
    vsize_t defined in machine/types.h
    quad_t

    - some attributes cause problems:
    __volatile

    - h_errno's are not recognized (TRY_AGAIN, etc. from )

    - LOTS of identifiers aren't known about, e.g. IN_MULTICAST from netinet/in.h

    - I had to fix the BYTE_ORDER #ifdef'ing to use #elif in the following files:
    /usr/include/netinet/ip.h
    /usr/include/arpa/nameser.h

    - struct stat has extra fields that splint doesn't know about:
    st_mtimespec, etc.

    --------

    You can actually USE splint already in many cases, if you take the results with a grain of salt. Just use the existing LINT target in the BSD build system, like so:

    make LINT=splint LINTFLAGS="+unixlib -Dsig_t='void(*)(int)' -Dregister_t=int32_t -Du_quad_t='unsigned long long' -Dva
    ddr_t='unsigned long' -Du_int16_t='uint16_t' -Du_int32_t='uint32_t' -Du_int64_t='uint64_t' -Du_int8_t='uint8_t' -Dquad_t='
    int64_t' -Dvsize_t='unsigned long' -D__volatile= -Dlint +weak" lint

    It won't work in all cases but it should give you decent results in some cases.

    Just remember: the point is NOT to shut splint up, the point is to use the tool to find and fix bugs. If you don't find and fix bugs, you are wasting your time -- it would be better spent reading through source code manually and looking for bugs.

    I also think valgrind is cool, but it's going to be a lot of work to port that to OpenBSD. There's already a semi-functional NetBSD port here:

    http://www.onnanifujiyuu.org/~lonewolf/valgrind/


    I would stress -- if you can make the tools work well and make them worth using, you can overcome the initial skepticism of some of the developers. However, if you are just asking in the hopes that someone else will do it, don't hold your breath. There's already a ton of more important things we need to work on and there's only so many hours in a day.

    Comments
    1. By Anonymous Coward () on

      Wow, Great answer!

    2. By Anonymous Coward () on

      Concur, Well done answer.

      This was the answer the poster was looking for.

      Hope the discussion was helpful.

  6. By Anonymous Coward () on

    i have used most of these security auditing tools (not valgrind, it may be different) and generally they are crap.

    they are basically grep. they generate so much crap you end up reading the source and thinking about possible problems yourself thats its easier & quicker to not use them.

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]