Contributed by tbert on from the Fuzz Aldrin dept.
I wanted to test the afl fuzzer that sort of recently entered the ports collection, ever since this webpage talked about how they give a jpeg decoder the string "Hello" in a file which it twists and mutates until the jpeg decoder no longer croaks on it, and it ends up actually being a valid jpeg image (though not very pretty).
Still, the general idea of instrumenting the code and then fuzz the inputs in such a way that it detects when a program starts taking new (perhaps less tested?) paths through the binary was new to me.
So, I dug in to how to set this up in an OpenBSD environment. First of all, whatever porting effort needed to make it run was already fixed, so "sudo pkg_add afl" takes care of that. Then you need to have a space to run the tests in, and since the fuzzer is going to create a huge amount of junk files to throw at your program, you really want this to be inside a tmpfs or mfs. This affects the speed a lot. It doesn't need to be very big, just fast in creating and deleting files.
Next step is to make a "in" and an "out" dir (for each program you intend to fuzz, in case you want to exercize a bunch of cores in parallel), and lastly to recompile your program with the appropriate afl-gcc wrapper.
In theory, this part could have been hard and cumbersome if the OpenBSD system makefiles were looking for stuff in odd places or if they generally didn't honor CC or whatever since the docs basically assumes everyone will run ./configure and have the flags trickle down into the generated makefiles, but I found it to be very easy on OpenBSD. Just copy the /usr/src/usr.bin/name_of_program directory next to the in and out dirs in your tmpfs and go:
prompt$ cp -r /usr/src/usr.bin/name_of_program . prompt$ cd name_of_program prompt$ CC=afl-gcc make
For many of the programs this will just work fine, since they are very self-contained in their directories. Some have several directories so you may need to copy a bunch of extra files (mopchk for instance, which is the mopd/ directory). In this example I chose cap_mkdb.
Prep a infile, not too big (so it can change each bit and each string and so on), in this case, I took a small part out of the main termcap (which normally is 21k lines long):
dumb|80-column dumb tty:\ :am:\ :co#80:\ :bl=^G:cr=^M:do=^J:sf=^J: unknown|unknown terminal type:\ :gn:tc=dumb: lpr|printer|line printer:\ :bs:hc:os:\ :co#132:li#66:\ :bl=^G:cr=^M:do=^J:ff=^L:le=^H:sf=^J: glasstty|classic glass tty interpreting ASCII control characters:\ :am:bs:\ :co#80:\ :bl=^G:cl=^L:cr=^M:do=^J:kd=^J:kl=^H:le=^H:nw=^M^J:ta=^I: vanilla|dumb tty:\ :bs:\ :bl=^G:cr=^M:do=^J:sf=^J:
then we start the fuzzer:
prompt$ nice afl-fuzz -i in -o out -f termcap -- cap_mkdb/cap_mkdb -f outfile @@
The two @ signs tells afl-fuzz to add the temporary name of the current fuzzed infile there, if you omit them afl-fuzz will feed input via stdin instead.
As it runs, it gives you a simple ascii info pane to look at, which tells you how it is doing. As per the documentation, it tells people to have some patience, which will be needed, since it can take a day or two to find all possible variants for that particular input, all depending on your CPU and I/O of course.
For me, it ended up like this:
american fuzzy lop 0.73b (cap_mkdb) ┌─ process timing ─────────────────────────────────────┬─ overall results ─────┐ │ run time : 2 days, 20 hrs, 28 min, 11 sec │ cycles done : 4 │ │ last new path : 1 days, 15 hrs, 54 min, 26 sec │ total paths : 82 │ │ last uniq crash : 2 days, 14 hrs, 21 min, 13 sec │ uniq crashes : 4 │ │ last uniq hang : 2 days, 19 hrs, 52 min, 53 sec │ uniq hangs : 8 │ ├─ cycle progress ────────────────────┬─ map coverage ─┴───────────────────────┤ │ now processing : 77 (93.90%) │ map density : 84 (0.26%) │ │ paths timed out : 0 (0.00%) │ count coverage : 5.58 bits/tuple │ ├─ stage progress ────────────────────┼─ findings in depth ────────────────────┤ │ now trying : havoc │ favored paths : 39 (47.56%) │ │ stage execs : 4983/7500 (66.44%) │ new edges on : 11 (13.41%) │ │ total execs : 13.0M │ total crashes : 166 (4 unique) │ │ exec speed : 52.63/sec (slow!) │ total hangs : 22.7k (8 unique) │ ├─ fuzzing strategy yields ───────────┴───────────────┬─ path geometry ────────┤ │ bit flips : 12/405k, 0/405k, 0/405k │ levels : 4 │ │ byte flips : 0/50.6k, 0/50.6k, 0/50.5k │ pending : 34 │ │ arithmetics : 2/3.55M, 0/330k, 0/16.6k │ pend fav : 0 │ │ known ints : 0/452k, 0/1.87M, 0/2.52M │ own finds : 81 │ │ havoc : 70/1.47M, 0/1.36M │ imported : 0 │ │ trim : 39.0 kB/27.8k (44.08% gain) │ variable : 0 │ └─────────────────────────────────────────────────────┴────────────────────────┘ ^C +++ Testing aborted by user +++ [+] We're done here. Have a nice day! prompt$
At this point, you can see it has 4 unique crashes, which means that in my out/ dir, there now exist 4 fuzzed versions of the above termcap file (quite damaged versions, I might add), that will crash cap_mkdb if you ask it to make a DB file out of them:
prompt$ ls out/crashes/ README.txt id:000000,sig:11,src:000048,op:havoc,rep:16 id:000001,sig:06,src:000059,op:havoc,rep:8 id:000002,sig:11,src:000059,op:havoc,rep:128 id:000003,sig:06,src:000059,op:havoc,rep:32
Now, the not-completely-automated-part enters, where you need to use your own
smarts and rebuild this binary with debug info and run it from gdb to figure out
where in the code it bombs out on you:
$ gdb cap_mkdb/cap_mkdb GNU gdb 6.3 Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-unknown-openbsd5.6"... (gdb) run out/crashes/id:000000,sig:11,src:000048,op:havoc,rep:16 Starting program: /home/jj/afl2/cap_mkdb/cap_mkdb out/crashes/id:000000,sig:11,s rc:000048,op:havoc,rep:16 cap_mkdb: no name field: lpr? cap_mkdb: ignored duplicate: lpr cap_mkdb: ignored duplicate: lin?print? cap_mkdb: no name field: gl cap_mkdb: no name field: g cap_mkdb: ignored duplicate: cap_mkdb: ignored duplicate: glasst?y cap_mkdb: Record not tc expanded: ... <a lot of output from cap_mkdb goes here, since it complains about the mangled entries a lot> Program received signal SIGABRT, Aborted. 0x00000d10f694a34a in kill () at <stdin>:2 2 <stdin>: No such file or directory. in <stdin> Current language: auto; currently asm (gdb) bt full #0 0x00000d10f694a34a in kill () at <stdin>:2 No locals. #1 0x00000d10f69839e4 in __stack_smash_handler ( func=0xd10f6ab0aaa "cgetnext", damaged=Variable "damaged" is not available. ) at /usr/src/lib/libc/sys/stack_protector.c:61 sdata = {log_stat = 0, log_tag = 0xd0e597086a0 "cap_mkdb", log_fac = 8, log_mask = 255} message = "stack overflow in function %s" sa = {__sigaction_u = {__sa_handler = 0, __sa_sigaction = 0}, sa_mask = 0, sa_flags = 0} mask = 4294967263 #2 0x00000d10f6952cc3 in cgetnext (cap=0x7f7ffffd5530, db_array=0x6) at /usr/src/lib/libc/gen/getcap.c:801 len = Variable "len" is not available.
the binary "mopchk" also has issues with strange input:
(gdb) run out4/crashes id:000000,sig:11,src:000009,op:flip1,pos:14 Starting program: /home/jj/afl2/mopd/mopchk/obj/mopchk out4/crashes id:000000,sig:11,src:000009,op:flip1,pos:14 Checking: out4/crashes Checking: id:000000,sig:11,src:000009,op:flip1,pos:14 Unknown file. Program exited normally. (gdb) run out4/crashes/id:000000,sig:11,src:000009,op:flip1,pos:14 Starting program: /home/jj/afl2/mopd/mopchk/obj/mopchk out4/crashes/id:000000,sig:11,src:000009,op:flip1,pos:14 Checking: out4/crashes/id:000000,sig:11,src:000009,op:flip1,pos:14 Program received signal SIGSEGV, Segmentation fault. 0x0000174079c0a588 in GetMopFileInfo (dl=0x7f7ffffd9400, info=1) at /home/jj/afl2/mopd/mopchk/../common/file.c:280 280 isize = (header[isd+EISD_L_SECSIZE+3]*0x1000000 + (gdb) bt full #0 0x0000174079c0a588 in GetMopFileInfo (dl=0x7f7ffffd9400, info=1) at /home/jj/afl2/mopd/mopchk/../common/file.c:280 header = "?\0000", '\0' <repeats 11 times>, "\200\000\001", '\0' <repeats 197 times>, "t", '\0' <repeats 295 times>, "\005" image_type = Variable "image_type" is not available.
Looking at the mangled infiles and of course all the normal debug procedures will be of help to fix these issues. stdhosts(8) seems to have issues with a lot of single-char lines (i.e lots of blank lines), but after trying out this patch for it:
it seems that the fuzzer still beats it at times (not as fast though) and produces even stranger inputs for stdhosts to croak on:diff -u stdhosts*/stdhosts.c --- stdhosts/stdhosts.c Tue Dec 2 13:19:02 2014 +++ stdhosts2/stdhosts.c Fri Jan 2 21:24:35 2015 @@ -46,6 +46,9 @@ int len = strlen(buf); done += len; + + if (len == 1 && buf[0] == '\n') continue; + if (len > 1 && buf[len-2] == '\\' && buf[len-1] == '\n') { int ch;
Program received signal SIGSEGV, Segmentation fault. main (argc=2, argv=0x7f7ffffbd718) at stdhosts.c:118 118 while (!isspace(*p)) /* find first "space" */ (gdb) bt full #0 main (argc=2, argv=0x7f7ffffbd718) at stdhosts.c:118 data_line = "0.0.5\000\000\000st\000\000\000\000\000l\000$O\000$\000\000\225\225\225\225\225\225\225\225\000\200\000\000", '\225' <repeats 35 times>, "#t\000\000\2\ 25\225\225\225\225\225\225\225?", '\225' <repeats 43 times>, "?\225\225\225\225\225\225\225.5\000\000", '\225' <repeats 129 times>, "\221\225\225\225\225\225\225\225\225j"\ , '\225' <repeats 36 times>, "\206\225\225\225\225\225?", '\225' <repeats 29 times>, "?", '\225' <repeats 65 times>, "?", '\225' <repeats 67 times>, '?' <repeats 22 times>\ , '\225' <repeats 162 times>, "\235\225\225\225", '?' <repeats 17 times>, '\225' <repeats 23 times>, ":16d", '#' <repeats 11 times>... p = 0x7f7fffffc000 <Address 0x7f7fffffc000 out of bounds> line_no = 79 len = Variable "len" is not available.
So, this afl-fuzzer is a neat way for me and you to become that elephant in the porcelain store that smashes something everytime you turn around. The hard part is actually fixing the found issues. The cap_mkdb seems to relate to libc line parsing code, so it could potentially affect other similar programs too.
If anyone wants to take a stab at fixing the things I found so far, I've posted the crashy inputs here: http://c66.it.su.se:8080/afl-fuzz/
(Comments are closed)
By mirabilos (2a01:238:4200:4342:321e:80ff:fe12:4223) tg@mirbsd.org on https://www.mirbsd.org/cvs.cgi/src/usr.bin/cap_mkdb/
a number of parsing issues back in... 2006, it appears.
Maybe this time, the OpenBSD people will get over their apparently
still present hate and actually take some fixes...
Comments
By Anonymous Coward (2003:56:c600:2f00:b84a:6b8:e100:243e) on
> a number of parsing issues back in... 2006, it appears.
>
> Maybe this time, the OpenBSD people will get over their apparently
> still present hate and actually take some fixes...
I didn't even know that MirBSD still exists.
By brynet (Brynet) on http://brynet.biz.tm/
I was looking for a write-up like this. I couldn't quite work out the instructions.
Comments
By brynet (Brynet) on http://brynet.biz.tm/
Apparently jj wrote this, thanks jj!
By journeysquid (Tor) on http://www.openbsd.org/donations.html
By Rob (10.32.151.184) on
By Anonymous Coward (209.181.89.124) on
Comments
By Anonymous Coward (79.227.16.105) on
All these fuzzers are very good software products, but I don't fully agree that they should be time-saving for code reviewers, yet alone developers. A code review can reveal much more than "just" core dumps, like race conditions, memory leaks, information disclosures, etc. Haven't seen good fuzzers/analysers for that, although Valgrind will help with memory.
Instead, these tools are a great components for users who are inexperienced with code reviews (yet). They get pointed to interesting code pathes which they can start fixing. Or, if they can't, write bug reports.
Then, developers can take care of it. There won't be just a bug report but also a nice input file attached that reproduces the problem.
Comments
By Anonymous Coward (209.181.89.124) on
>
> All these fuzzers are very good software products, but I don't fully agree that they should be time-saving for code reviewers, yet alone developers. A code review can reveal much more than "just" core dumps, like race conditions, memory leaks, information disclosures, etc. Haven't seen good fuzzers/analysers for that, although Valgrind will help with memory.
>
> Instead, these tools are a great components for users who are inexperienced with code reviews (yet). They get pointed to interesting code pathes which they can start fixing. Or, if they can't, write bug reports.
>
> Then, developers can take care of it. There won't be just a bug report but also a nice input file attached that reproduces the problem.
http://comments.gmane.org/gmane.os.openbsd.misc/192303 lists (from 2012):
coverity
clang's static analyser
cppcheck
parfait
One tool that doesn't seem to get mentioned is:
http://compcert.inria.fr/
By Damien Couderc (91.135.188.215) on
If you're not happy of how things have been managed, then just go away.
If you seek accountability, then you're using the wrong methods.
You want to be recognized ? Then propose something and listen to the feedback. From what i've seen you've heavily failed on the second part.
Have fun,
Damien
Comments
By Jorden Verwer (84.105.195.188) on
Personally, I think there's a certain timelessness to having a compiler that can even be compiled with a pre-ANSI C compiler. GCC 3.3.6 was the last version to achieve that. Nowadays you even need a C++ compiler, for crying out loud!
Comments
By Anonymous Coward (209.181.89.124) on
http://compcert.inria.fr/ or something similar. That type of change would help OpenBSD pursue security and reliability on a stable foundation.
By Chris Cappuccio (204.80.187.232) on
> > Personally, I think there's a certain timelessness to having a compiler that can even be compiled with a pre-ANSI C compiler. GCC 3.3.6 was the last version to achieve that. Nowadays you even need a C++ compiler, for crying out loud!
>
> Ok listen.. you love to get things wrong, or?
> Are you all realy so blinded or narrow minded?
>
I think you need to start taking your medications again. Perhaps then the way things actually work would become obvious to you. How it can not be obvious that OpenBSD has graciously accepted (and occasionally duplicated) clang fixes from Bitrig, I don't know how. With a few small changes, you can compile working kernels with clang today. Many other things were lovingly adopted from Bitrig too.
Bitrig has legitimacy. MirOS, not as much.
By Anonymous Coward (2003:5b:4e50:8c02:dce4:7682:ee93:3a9d) on
>
> Maybe interesting improvement of grep -n
> http://blog.fefe.de/?ts=aa3c0cd3
>
> From 15k CPU cycles down to 2500...
Given your choice of words and pathetic attention whoring combined with insults, I don't think it's a good idea to take anything you say seriously. Probably, any potential customer of your company, who happens to read your posts here, will think the same.
Just sayin'. Reputation is a volatile good these days.
Comments
By Anonymous Coward (141.113.85.95) on
> >
> > Maybe interesting improvement of grep -n
> > http://blog.fefe.de/?ts=aa3c0cd3
> >
> > From 15k CPU cycles down to 2500...
>
> Given your choice of words and pathetic attention whoring combined with insults, I don't think it's a good idea to take anything you say seriously. Probably, any potential customer of your company, who happens to read your posts here, will think the same.
>
> Just sayin'. Reputation is a volatile good these days.
Completely agree here.
Fefe's work to speed up grep is totally worth the look, but sometimes the messengers rep just kills the message.
This belongs to tech@ *including* a diff.
(I would love to implement it. Maybe during the summer...)
By Gilles Chehade (gilles) gilles@poolp.org on https://www.poolp.org/~gilles/
Is there a way for logged in users to filter morons in comments ?
Comments
By Theo de Raadt (199.185.136.55) on
I thought so too. It shows there is a lot of room for new people to participate in the process of improving code.
> Is there a way for logged in users to filter morons in comments ?
Unfortunately there seems no way to avoid the people who don't improve the code. But undeadly sometimes identifies them clearly to us. Small win, I know.
Comments
By phessler (phessler) on http://www.openbsdfoundation.org/donations.html
> I gave up to report specific details since Henning Brauer (and you..) stole my work and did not named me.
Please provide references to where a) you reported this, and b) this was fixed without naming you.
Comments
By tbert (tbert) on
I certainly hope that you're not this unhinged in person, but my guess would be you're about the same in meatspace.
If you keep staying in a private home where you continually insult the friends of the owner, and generally making a nuisance of yourself and refuse to leave, they call the cops.
You've disinvited yourself from this community, by the behavior you've displayed here and elsewhere.
By Anonymous Coward (80.153.96.240) on
> > > I gave up to report specific details since Henning Brauer (and you..) stole my work and did not named me.
> >
> > Please provide references to where a) you reported this, and b) this was fixed without naming you.
> >
>
> Ok Peter as you wish... I provide you an example, I start with the oldest one:
>
> Generaly:
> http://www.cvedetails.com/cve/CVE-2009-0687/
>
> http://archive.cert.uni-stuttgart.de/bugtraq/2009/05/msg00010.html
> (This includes that the OpenBSD Project, back then I contacted Henning and others, fucked it up and proofs that you give a fuck about your project principles on the cost of others..)
>
> NetBSD named me with my Nickname back then:
> https://mail-index.netbsd.org/security-announce/2009/06/23/msg000021.html
>
> Wich is:
> http://marc.info/?l=openbsd-security-announce&m=123949585205081&w=2
>
> Aka:
> http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/net/pf.c?f=u&only_with_tag=OPENBSD_4_5&logsort=date
What exactly is your problem?
"Avoid dereferencing a null pointer when pf attempts to translate a
specifically crafted IP datagram.
Problem noted by Sebastian Rother.
fix from jsing. ok henning@ mcbride@"
By phessler (phessler) on http://www.openbsdfoundation.org/donations.html
> > > I gave up to report specific details since Henning Brauer (and you..) stole my work and did not named me.
> >
> > Please provide references to where a) you reported this, and b) this was fixed without naming you.
> >
>
> Ok Peter as you wish... I provide you an example, I start with the oldest one:
>
> Generaly:
> http://www.cvedetails.com/cve/CVE-2009-0687/
>
> http://archive.cert.uni-stuttgart.de/bugtraq/2009/05/msg00010.html
> (This includes that the OpenBSD Project, back then I contacted Henning and others, fucked it up and proofs that you give a fuck about your project principles on the cost of others..)
>
> NetBSD named me with my Nickname back then:
> https://mail-index.netbsd.org/security-announce/2009/06/23/msg000021.html
>
> Wich is:
> http://marc.info/?l=openbsd-security-announce&m=123949585205081&w=2
>
> Aka:
> http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/net/pf.c?f=u&only_with_tag=OPENBSD_4_5&logsort=date
>
You were SPECIFICALLY NAMED in that commit. From your own evidence, your claim is proven false.
By Paul 'WEiRD' de Weerd (weerd) on
The moron-detection code in undeadly isn't up to the modern levels of stupidity. Fortunately, pf brings solace.
By Anonymous Coward (12.39.252.110) on
> Some of you are just retarded fashistic fucktards...
>
>
> You removed all the "unconfortable" posts by your Devs.. how pittyfull..
> What you feared? That I gonna sue those german fucktards because they broke german laws?
For the record: An apology was posted yesterday, but got deleted.
By insulting others (cite "german fucktards"), you break German law as well. Calm the fuck down.