OpenBSD Journal

afl-fuzz - American Fuzzy Lop

Contributed by tbert on from the Fuzz Aldrin dept.

I wanted to test the afl fuzzer that sort of recently entered the ports collection, ever since this webpage talked about how they give a jpeg decoder the string "Hello" in a file which it twists and mutates until the jpeg decoder no longer croaks on it, and it ends up actually being a valid jpeg image (though not very pretty).

Still, the general idea of instrumenting the code and then fuzz the inputs in such a way that it detects when a program starts taking new (perhaps less tested?) paths through the binary was new to me.

So, I dug in to how to set this up in an OpenBSD environment. First of all, whatever porting effort needed to make it run was already fixed, so "sudo pkg_add afl" takes care of that. Then you need to have a space to run the tests in, and since the fuzzer is going to create a huge amount of junk files to throw at your program, you really want this to be inside a tmpfs or mfs. This affects the speed a lot. It doesn't need to be very big, just fast in creating and deleting files.

Next step is to make a "in" and an "out" dir (for each program you intend to fuzz, in case you want to exercize a bunch of cores in parallel), and lastly to recompile your program with the appropriate afl-gcc wrapper.

In theory, this part could have been hard and cumbersome if the OpenBSD system makefiles were looking for stuff in odd places or if they generally didn't honor CC or whatever since the docs basically assumes everyone will run ./configure and have the flags trickle down into the generated makefiles, but I found it to be very easy on OpenBSD. Just copy the /usr/src/usr.bin/name_of_program directory next to the in and out dirs in your tmpfs and go:

prompt$ cp -r /usr/src/usr.bin/name_of_program .
prompt$ cd name_of_program
prompt$ CC=afl-gcc make

For many of the programs this will just work fine, since they are very self-contained in their directories. Some have several directories so you may need to copy a bunch of extra files (mopchk for instance, which is the mopd/ directory). In this example I chose cap_mkdb.

Prep a infile, not too big (so it can change each bit and each string and so on), in this case, I took a small part out of the main termcap (which normally is 21k lines long):

dumb|80-column dumb tty:\
        :am:\
        :co#80:\
        :bl=^G:cr=^M:do=^J:sf=^J:
unknown|unknown terminal type:\
        :gn:tc=dumb:
lpr|printer|line printer:\
        :bs:hc:os:\
        :co#132:li#66:\
        :bl=^G:cr=^M:do=^J:ff=^L:le=^H:sf=^J:
glasstty|classic glass tty interpreting ASCII control characters:\
        :am:bs:\
        :co#80:\
        :bl=^G:cl=^L:cr=^M:do=^J:kd=^J:kl=^H:le=^H:nw=^M^J:ta=^I:
vanilla|dumb tty:\
        :bs:\
        :bl=^G:cr=^M:do=^J:sf=^J:

then we start the fuzzer:

prompt$ nice afl-fuzz -i in -o out -f termcap -- cap_mkdb/cap_mkdb -f outfile @@

The two @ signs tells afl-fuzz to add the temporary name of the current fuzzed infile there, if you omit them afl-fuzz will feed input via stdin instead.

As it runs, it gives you a simple ascii info pane to look at, which tells you how it is doing. As per the documentation, it tells people to have some patience, which will be needed, since it can take a day or two to find all possible variants for that particular input, all depending on your CPU and I/O of course.

For me, it ended up like this:

                      american fuzzy lop 0.73b (cap_mkdb)

┌─ process timing ─────────────────────────────────────┬─ overall results ─────┐
│        run time : 2 days, 20 hrs, 28 min, 11 sec     │  cycles done : 4      │
│   last new path : 1 days, 15 hrs, 54 min, 26 sec     │  total paths : 82     │
│ last uniq crash : 2 days, 14 hrs, 21 min, 13 sec     │ uniq crashes : 4      │
│  last uniq hang : 2 days, 19 hrs, 52 min, 53 sec     │   uniq hangs : 8      │
├─ cycle progress ────────────────────┬─ map coverage ─┴───────────────────────┤
│  now processing : 77 (93.90%)       │    map density : 84 (0.26%)            │
│ paths timed out : 0 (0.00%)         │ count coverage : 5.58 bits/tuple       │
├─ stage progress ────────────────────┼─ findings in depth ────────────────────┤
│  now trying : havoc                 │ favored paths : 39 (47.56%)            │
│ stage execs : 4983/7500 (66.44%)    │  new edges on : 11 (13.41%)            │
│ total execs : 13.0M                 │ total crashes : 166 (4 unique)         │
│  exec speed : 52.63/sec (slow!)     │   total hangs : 22.7k (8 unique)       │
├─ fuzzing strategy yields ───────────┴───────────────┬─ path geometry ────────┤
│   bit flips : 12/405k, 0/405k, 0/405k               │    levels : 4          │
│  byte flips : 0/50.6k, 0/50.6k, 0/50.5k             │   pending : 34         │
│ arithmetics : 2/3.55M, 0/330k, 0/16.6k              │  pend fav : 0          │
│  known ints : 0/452k, 0/1.87M, 0/2.52M              │ own finds : 81         │
│       havoc : 70/1.47M, 0/1.36M                     │  imported : 0          │
│        trim : 39.0 kB/27.8k (44.08% gain)           │  variable : 0          │
└─────────────────────────────────────────────────────┴────────────────────────┘
^C
+++ Testing aborted by user +++
[+] We're done here. Have a nice day!
prompt$

At this point, you can see it has 4 unique crashes, which means that in my out/ dir, there now exist 4 fuzzed versions of the above termcap file (quite damaged versions, I might add), that will crash cap_mkdb if you ask it to make a DB file out of them:

prompt$ ls out/crashes/
README.txt
id:000000,sig:11,src:000048,op:havoc,rep:16
id:000001,sig:06,src:000059,op:havoc,rep:8
id:000002,sig:11,src:000059,op:havoc,rep:128
id:000003,sig:06,src:000059,op:havoc,rep:32

Now, the not-completely-automated-part enters, where you need to use your own smarts and rebuild this binary with debug info and run it from gdb to figure out where in the code it bombs out on you:

$ gdb cap_mkdb/cap_mkdb
GNU gdb 6.3
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-unknown-openbsd5.6"...
(gdb) run out/crashes/id:000000,sig:11,src:000048,op:havoc,rep:16
Starting program: /home/jj/afl2/cap_mkdb/cap_mkdb out/crashes/id:000000,sig:11,s
rc:000048,op:havoc,rep:16
cap_mkdb: no name field: lpr?
cap_mkdb: ignored duplicate: lpr
cap_mkdb: ignored duplicate: lin?print?
cap_mkdb: no name field: gl
cap_mkdb: no name field: g
cap_mkdb: ignored duplicate:
cap_mkdb: ignored duplicate: glasst?y
cap_mkdb: Record not tc expanded:
...
<a lot of output from cap_mkdb goes here, since it complains about the mangled
entries a lot>

Program received signal SIGABRT, Aborted.
0x00000d10f694a34a in kill () at <stdin>:2
2       <stdin>: No such file or directory.
        in <stdin>
Current language:  auto; currently asm
(gdb) bt full
#0  0x00000d10f694a34a in kill () at <stdin>:2
No locals.
#1  0x00000d10f69839e4 in __stack_smash_handler (
    func=0xd10f6ab0aaa "cgetnext", damaged=Variable "damaged" is not available.
)
    at /usr/src/lib/libc/sys/stack_protector.c:61
        sdata = {log_stat = 0, log_tag = 0xd0e597086a0 "cap_mkdb",
  log_fac = 8, log_mask = 255}
        message = "stack overflow in function %s"
        sa = {__sigaction_u = {__sa_handler = 0, __sa_sigaction = 0},
  sa_mask = 0, sa_flags = 0}
        mask = 4294967263
#2  0x00000d10f6952cc3 in cgetnext (cap=0x7f7ffffd5530, db_array=0x6)
    at /usr/src/lib/libc/gen/getcap.c:801
        len = Variable "len" is not available.

the binary "mopchk" also has issues with strange input:

(gdb) run out4/crashes id:000000,sig:11,src:000009,op:flip1,pos:14
Starting program: /home/jj/afl2/mopd/mopchk/obj/mopchk out4/crashes id:000000,sig:11,src:000009,op:flip1,pos:14
Checking: out4/crashes

Checking: id:000000,sig:11,src:000009,op:flip1,pos:14
Unknown file.

Program exited normally.
(gdb) run out4/crashes/id:000000,sig:11,src:000009,op:flip1,pos:14
Starting program: /home/jj/afl2/mopd/mopchk/obj/mopchk out4/crashes/id:000000,sig:11,src:000009,op:flip1,pos:14
Checking: out4/crashes/id:000000,sig:11,src:000009,op:flip1,pos:14

Program received signal SIGSEGV, Segmentation fault.
0x0000174079c0a588 in GetMopFileInfo (dl=0x7f7ffffd9400, info=1)
    at /home/jj/afl2/mopd/mopchk/../common/file.c:280
280                             isize = (header[isd+EISD_L_SECSIZE+3]*0x1000000 +
(gdb) bt full
#0  0x0000174079c0a588 in GetMopFileInfo (dl=0x7f7ffffd9400, info=1)
    at /home/jj/afl2/mopd/mopchk/../common/file.c:280
        header = "?\0000", '\0' <repeats 11 times>, "\200\000\001", '\0' <repeats 197 times>, "t", '\0' <repeats 295 times>, "\005"
        image_type = Variable "image_type" is not available.

Looking at the mangled infiles and of course all the normal debug procedures will be of help to fix these issues. stdhosts(8) seems to have issues with a lot of single-char lines (i.e lots of blank lines), but after trying out this patch for it:

diff -u stdhosts*/stdhosts.c
--- stdhosts/stdhosts.c Tue Dec  2 13:19:02 2014
+++ stdhosts2/stdhosts.c        Fri Jan  2 21:24:35 2015
@@ -46,6 +46,9 @@
                        int len = strlen(buf);

                        done += len;
+
+                       if (len == 1 && buf[0] == '\n') continue;
+
                        if (len > 1 && buf[len-2] == '\\' &&
                            buf[len-1] == '\n') {
                                int ch;
it seems that the fuzzer still beats it at times (not as fast though) and produces even stranger inputs for stdhosts to croak on:
Program received signal SIGSEGV, Segmentation fault.
main (argc=2, argv=0x7f7ffffbd718) at stdhosts.c:118
118                     while (!isspace(*p))    /* find first "space" */
(gdb) bt full
#0  main (argc=2, argv=0x7f7ffffbd718) at stdhosts.c:118
        data_line = "0.0.5\000\000\000st\000\000\000\000\000l\000$O\000$\000\000\225\225\225\225\225\225\225\225\000\200\000\000", '\225' <repeats 35 times>, "#t\000\000\2\
25\225\225\225\225\225\225\225?", '\225' <repeats 43 times>, "?\225\225\225\225\225\225\225.5\000\000", '\225' <repeats 129 times>, "\221\225\225\225\225\225\225\225\225j"\
, '\225' <repeats 36 times>, "\206\225\225\225\225\225?", '\225' <repeats 29 times>, "?", '\225' <repeats 65 times>, "?", '\225' <repeats 67 times>, '?' <repeats 22 times>\
, '\225' <repeats 162 times>, "\235\225\225\225", '?' <repeats 17 times>, '\225' <repeats 23 times>, ":16d", '#' <repeats 11 times>...
        p = 0x7f7fffffc000 <Address 0x7f7fffffc000 out of bounds>
        line_no = 79
        len = Variable "len" is not available.

So, this afl-fuzzer is a neat way for me and you to become that elephant in the porcelain store that smashes something everytime you turn around. The hard part is actually fixing the found issues. The cap_mkdb seems to relate to libc line parsing code, so it could potentially affect other similar programs too.

If anyone wants to take a stab at fixing the things I found so far, I've posted the crashy inputs here: http://c66.it.su.se:8080/afl-fuzz/

(Comments are closed)


Comments
  1. By mirabilos (2a01:238:4200:4342:321e:80ff:fe12:4223) tg@mirbsd.org on https://www.mirbsd.org/cvs.cgi/src/usr.bin/cap_mkdb/

    All four cap_mkdb cases do not crash on MirBSD. I think I've plugged
    a number of parsing issues back in... 2006, it appears.

    Maybe this time, the OpenBSD people will get over their apparently
    still present hate and actually take some fixes...

    Comments
    1. By Anonymous Coward (2003:56:c600:2f00:b84a:6b8:e100:243e) on

      > All four cap_mkdb cases do not crash on MirBSD. I think I've plugged
      > a number of parsing issues back in... 2006, it appears.
      >
      > Maybe this time, the OpenBSD people will get over their apparently
      > still present hate and actually take some fixes...

      I didn't even know that MirBSD still exists.

  2. By brynet (Brynet) on http://brynet.biz.tm/

    This is really great, thanks tbert!

    I was looking for a write-up like this. I couldn't quite work out the instructions.

    Comments
    1. By brynet (Brynet) on http://brynet.biz.tm/

      > This is really great, thanks tbert!

      Apparently jj wrote this, thanks jj!

  3. By Rob (10.32.151.184) on

    Thanks for this, incredibly useful tutorial for a tool I've struggled to understand properly up until now.

  4. By Anonymous Coward (209.181.89.124) on

    Thanks for posting the article which points to the pursuit of real proactive stability and security work to find both problems and solutions. The more tools that are used, the better, due to the time savings for developers that otherwise get tied up staring at code for inspiration.

    Comments
    1. By Anonymous Coward (79.227.16.105) on

      > time savings for developers that otherwise get tied up staring at code for inspiration.

      All these fuzzers are very good software products, but I don't fully agree that they should be time-saving for code reviewers, yet alone developers. A code review can reveal much more than "just" core dumps, like race conditions, memory leaks, information disclosures, etc. Haven't seen good fuzzers/analysers for that, although Valgrind will help with memory.

      Instead, these tools are a great components for users who are inexperienced with code reviews (yet). They get pointed to interesting code pathes which they can start fixing. Or, if they can't, write bug reports.

      Then, developers can take care of it. There won't be just a bug report but also a nice input file attached that reproduces the problem.

      Comments
      1. By Anonymous Coward (209.181.89.124) on

        > > time savings for developers that otherwise get tied up staring at code for inspiration.
        >
        > All these fuzzers are very good software products, but I don't fully agree that they should be time-saving for code reviewers, yet alone developers. A code review can reveal much more than "just" core dumps, like race conditions, memory leaks, information disclosures, etc. Haven't seen good fuzzers/analysers for that, although Valgrind will help with memory.
        >
        > Instead, these tools are a great components for users who are inexperienced with code reviews (yet). They get pointed to interesting code pathes which they can start fixing. Or, if they can't, write bug reports.
        >
        > Then, developers can take care of it. There won't be just a bug report but also a nice input file attached that reproduces the problem.

        http://comments.gmane.org/gmane.os.openbsd.misc/192303 lists (from 2012):
        coverity
        clang's static analyser
        cppcheck
        parfait

        One tool that doesn't seem to get mentioned is:
        http://compcert.inria.fr/

  5. By Damien Couderc (91.135.188.215) on

    Honestly your comments become really boring.

    If you're not happy of how things have been managed, then just go away.

    If you seek accountability, then you're using the wrong methods.

    You want to be recognized ? Then propose something and listen to the feedback. From what i've seen you've heavily failed on the second part.

    Have fun,
    Damien

    Comments
    1. By Jorden Verwer (84.105.195.188) on

      Can Bitrig build base without using packages yet? Furthermore, I wish you good luck getting Clang to compile on all supported architectures. It's just not gonna happen. Even today there have to be two versions of GCC in the tree, because VAX and M88k don't get correct results out of GCC 4.2.1. That particular version of GCC, by the way, isn't as horrible as you claim it to be. So adding Clang would just mean going back to having to support three different compilers.

      Personally, I think there's a certain timelessness to having a compiler that can even be compiled with a pre-ANSI C compiler. GCC 3.3.6 was the last version to achieve that. Nowadays you even need a C++ compiler, for crying out loud!

      Comments
      1. By Anonymous Coward (209.181.89.124) on

        Ignoring the architecture issue and focusing only on security, there is a serious reason to have a different compiler in OpenBSD (that is hopefully simpler): How do we know the compiler isn't inserting exploits into the OpenBSD source code? We know it is possible, just have no proof of it happening. A better choice for handling this problem seriously is
        http://compcert.inria.fr/ or something similar. That type of change would help OpenBSD pursue security and reliability on a stable foundation.

      2. By Chris Cappuccio (204.80.187.232) on


        > > Personally, I think there's a certain timelessness to having a compiler that can even be compiled with a pre-ANSI C compiler. GCC 3.3.6 was the last version to achieve that. Nowadays you even need a C++ compiler, for crying out loud!
        >
        > Ok listen.. you love to get things wrong, or?
        > Are you all realy so blinded or narrow minded?
        >

        I think you need to start taking your medications again. Perhaps then the way things actually work would become obvious to you. How it can not be obvious that OpenBSD has graciously accepted (and occasionally duplicated) clang fixes from Bitrig, I don't know how. With a few small changes, you can compile working kernels with clang today. Many other things were lovingly adopted from Bitrig too.

        Bitrig has legitimacy. MirOS, not as much.

  6. By Anonymous Coward (2003:5b:4e50:8c02:dce4:7682:ee93:3a9d) on

    > Offtopic:
    >
    > Maybe interesting improvement of grep -n
    > http://blog.fefe.de/?ts=aa3c0cd3
    >
    > From 15k CPU cycles down to 2500...

    Given your choice of words and pathetic attention whoring combined with insults, I don't think it's a good idea to take anything you say seriously. Probably, any potential customer of your company, who happens to read your posts here, will think the same.

    Just sayin'. Reputation is a volatile good these days.

    Comments
    1. By Anonymous Coward (141.113.85.95) on

      > > Offtopic:
      > >
      > > Maybe interesting improvement of grep -n
      > > http://blog.fefe.de/?ts=aa3c0cd3
      > >
      > > From 15k CPU cycles down to 2500...
      >
      > Given your choice of words and pathetic attention whoring combined with insults, I don't think it's a good idea to take anything you say seriously. Probably, any potential customer of your company, who happens to read your posts here, will think the same.
      >
      > Just sayin'. Reputation is a volatile good these days.

      Completely agree here.

      Fefe's work to speed up grep is totally worth the look, but sometimes the messengers rep just kills the message.

      This belongs to tech@ *including* a diff.
      (I would love to implement it. Maybe during the summer...)

  7. By Gilles Chehade (gilles) gilles@poolp.org on https://www.poolp.org/~gilles/

    Very nice article.

    Is there a way for logged in users to filter morons in comments ?

    Comments
    1. By Theo de Raadt (199.185.136.55) on

      > Very nice article.

      I thought so too. It shows there is a lot of room for new people to participate in the process of improving code.

      > Is there a way for logged in users to filter morons in comments ?

      Unfortunately there seems no way to avoid the people who don't improve the code. But undeadly sometimes identifies them clearly to us. Small win, I know.

      Comments
      1. By phessler (phessler) on http://www.openbsdfoundation.org/donations.html

        > It would be kind if you stop stealing my hints/work and also name me..
        > I gave up to report specific details since Henning Brauer (and you..) stole my work and did not named me.

        Please provide references to where a) you reported this, and b) this was fixed without naming you.

        Comments
        1. By tbert (tbert) on

          Great, you've had your say. Now be content with the things in life you can't change.

          I certainly hope that you're not this unhinged in person, but my guess would be you're about the same in meatspace.

          If you keep staying in a private home where you continually insult the friends of the owner, and generally making a nuisance of yourself and refuse to leave, they call the cops.

          You've disinvited yourself from this community, by the behavior you've displayed here and elsewhere.

        2. By Anonymous Coward (80.153.96.240) on

          > > > It would be kind if you stop stealing my hints/work and also name me..
          > > > I gave up to report specific details since Henning Brauer (and you..) stole my work and did not named me.
          > >
          > > Please provide references to where a) you reported this, and b) this was fixed without naming you.
          > >
          >
          > Ok Peter as you wish... I provide you an example, I start with the oldest one:
          >
          > Generaly:
          > http://www.cvedetails.com/cve/CVE-2009-0687/
          >
          > http://archive.cert.uni-stuttgart.de/bugtraq/2009/05/msg00010.html
          > (This includes that the OpenBSD Project, back then I contacted Henning and others, fucked it up and proofs that you give a fuck about your project principles on the cost of others..)
          >
          > NetBSD named me with my Nickname back then:
          > https://mail-index.netbsd.org/security-announce/2009/06/23/msg000021.html
          >
          > Wich is:
          > http://marc.info/?l=openbsd-security-announce&m=123949585205081&w=2
          >
          > Aka:
          > http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/net/pf.c?f=u&only_with_tag=OPENBSD_4_5&logsort=date

          What exactly is your problem?

          "Avoid dereferencing a null pointer when pf attempts to translate a
          specifically crafted IP datagram.

          Problem noted by Sebastian Rother.

          fix from jsing. ok henning@ mcbride@"

        3. By phessler (phessler) on http://www.openbsdfoundation.org/donations.html

          > > > It would be kind if you stop stealing my hints/work and also name me..
          > > > I gave up to report specific details since Henning Brauer (and you..) stole my work and did not named me.
          > >
          > > Please provide references to where a) you reported this, and b) this was fixed without naming you.
          > >
          >
          > Ok Peter as you wish... I provide you an example, I start with the oldest one:
          >
          > Generaly:
          > http://www.cvedetails.com/cve/CVE-2009-0687/
          >
          > http://archive.cert.uni-stuttgart.de/bugtraq/2009/05/msg00010.html
          > (This includes that the OpenBSD Project, back then I contacted Henning and others, fucked it up and proofs that you give a fuck about your project principles on the cost of others..)
          >
          > NetBSD named me with my Nickname back then:
          > https://mail-index.netbsd.org/security-announce/2009/06/23/msg000021.html
          >
          > Wich is:
          > http://marc.info/?l=openbsd-security-announce&m=123949585205081&w=2
          >
          > Aka:
          > http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/net/pf.c?f=u&only_with_tag=OPENBSD_4_5&logsort=date
          >


          You were SPECIFICALLY NAMED in that commit. From your own evidence, your claim is proven false.

    2. By Paul 'WEiRD' de Weerd (weerd) on

      > Is there a way for logged in users to filter morons in comments ?

      The moron-detection code in undeadly isn't up to the modern levels of stupidity. Fortunately, pf brings solace.

  8. By Anonymous Coward (12.39.252.110) on

    > So censorship is now your tool of tollerance?
    > Some of you are just retarded fashistic fucktards...
    >
    >
    > You removed all the "unconfortable" posts by your Devs.. how pittyfull..
    > What you feared? That I gonna sue those german fucktards because they broke german laws?

    For the record: An apology was posted yesterday, but got deleted.

    By insulting others (cite "german fucktards"), you break German law as well. Calm the fuck down.

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]