b2k13 hackathon report: Florian Obser (florian@) on nginx.conf(5), slowcgi

Our sixth b2k13 hackathon report comes from Florian Obser (florian@), who writes:

Some time ago Reyk Floeter (reyk@) mentioned that it would be really nice to have a nginx.conf(5) man page. I started looking into generating mdoc(7) from upstream's XML files (nginx XML docs) with XSLT and some heuristics in perl. Ingo Schwarze (schwarze@) was kind enough to look over an early draft of the first section ("Core Functionality") and giving me a lot of valuable input on all the things I was doing wrong.

Since we are not living in the future yet it was clear that I would not have reliably working Internet connection during the 5 hour train ride to Berlin. So I pulled down everything I would need from the net to be prepared to work through all the information from Ingo Schwarze (schwarze@) while offline.

At the end of the trip I was at the point where I thought that's all I can do with XSLT and it's time to start to do more clean up by hand. Since I was too tired to do any focused work I went for this tedious task for the rest of the day.

There is this long standing bug in slowcgi(8). I first noticed it when running ab (apache benchmark) against it. With 10 parallel requests and doing 1000 requests there were always exactly 100 requests not answered. Really puzzling. Right before the hackathon I was again hunting for this bug with no luck though. However I found some others. With Bret Lambert (blambert@) in Berlin too, we could quickly go over the diffs and get them in.

This however had the downside of making it harder to trigger the bug for some reason. Is slowcgi(8) using libevent wrong in some way? Is there a bug in nginx(8)? Does nginx(8) not like the rather creative interpretation of the FastCGI spec slowcgi(8) implements? It was clear that the bug was somehow timing dependent.

So I went democratic with my problem and told random developers about it. Claudio Jeker (claudio@) suggested: Well, maybe you are looking for a bug in kqueue(2). Switching libevent to poll(2): Nope, same problem. Switching to select(2): Aha! Progress! slowcgi(8) would exit with "[warn] bad file descriptor" or something like that.

Breaking out ktrace(1) and it became immediately clear that slowcgi(8) would close(2) file descriptors two times, the second time by the unconditional close(2) calls in cleanup_request().

Since fds get recycled there is a chance that the second close(2) in cleanup_request() would close a fd in another request. Oh you poor bastard, what were you thinking when writing that code?!

So this was one of those bugs which are either not yet found or trivial. Talking to other developers and explaining the problem helped me greatly to refocus on the code I was staring at for hours.

At dinner Bret Lambert (blambert@) asked me: "What are you working on today, sir?"

Being incredibly happy about finding the bug I replied tongue-in-cheek: "Well, with that bug in slowcgi found, it's now bug free. My job here is done."

Apparently Theo de Raadt (deraadt@) accepted the challenge and found some more bugs for me.

Other than that I put some more work into slowcgi(8) regress tests which were started by Bret Lambert (blambert@), tricked Sebastian Benoit (benno@) into bringing over accept_reserve() from relayd(8), read and tested some bgpd(8) and pf(4) diffs from Claudio Jeker (claudio@) and Henning Brauer (henning@), climbed the store front of the hackroom and was having a great time in Berlin.

On the train ride home more work on nginx.conf(5) was waiting for me.

Thanks to everyone who made this hackathon possible.

Thank you Florian for sending in your hackathon report, and it's great to know that slowcgi is finally bug fre^w^w working better (Never tempt fate, or Theo).