Rthreads Hackathon Part the Epilogue

OpenBSD Journal: You initially began working on rthreads due to performance issues in the existing uthreads implementation. Why did the userland-only threading library last so long?

tedu@: It was almost good enough? I think also the amount of work involved in switching to kernel threads, which even though we've tried to keep it simple is still a lot, slowed things down. For many programs, rthreads will be an incremental improvement at best, and it's a lot of work. I'll get to this more in a bit, but lots of programs just use threads as a state machine abstraction, and uthreads were well suited to that task.

OpenBSD Journal: There's been discussion by the OpenBSD developers that, especially in the realm of networked daemons, threading is an inferior solution to asynchronous solutions. Where do you come down in that debate?

tedu@: If you're pushing traffic in and out, without much processing, I would agree. If you're spending time on the CPU, however, threads will let you use more CPUs. And if you're doing disk IO, you can't do eventing, because the file system layer just doesn't (in OpenBSD) support async ops.

There's also some debate about the preference for threads or multiprocess. I think they amount to about the same thing. And if you've designed your program nicely, you won't even know if your message passing interface is a pipe between processes or a shared memory queue between threads. You could even swap one backend for another. Threads are a little easier to work with for heterogenous tasks and small scale (think desktop), while multiprocess readily becomes multimachine if you want to go all web scale.

I would also like to say that if you go the async/evented route, you really have to know what you're doing. There are blocking APIs *everywhere* that you must call with caution, if ever. I think the people writing web services with things like node.js are rediscovering this, over and over. Just as with threading, it's easy to write code that mostly seems to work, until it doesn't.

OpenBSD Journal: There has seemingly been a rise in the use of threads over the past decade or so; is that an illusion, or is it real? And, if real, is it something to be embraced, or merely endured?

tedu@: Rise of multicore? As a user, if the program works, I don't care about the internals. To the extent shitty programs use threads, I don't think the shit is a direct consequence of threading.

OpenBSD Journal: Is there a logical follow-on to the work being done on finishing rthreads? Is there anything that is now possible which wasn't before?

tedu@: I think there is going to be some performance work necessary. Other than things working better, I'm not sure anything is more "possible". Maybe java becomes more viable for some people.

In terms of some lessons learned: When I gave the original rthreads presentation way back in 2005, part of the talk focused on the fact that we were going with a 1:1 threading model because it was simpler. Developers from both NetBSD and FreeBSD came up to me and explained how we would regret that choice, and the M:N models they were both building were so much better. Well, today they have either entirely scrapped (NetBSD) or mostly sidelined (FreeBSD) the M:N threading systems in favor of 1:1 models.

I can't laugh too hard, because they still shipped first and I had to be carried across the finish line on the backs of others, but it's worth reflecting on the OpenBSD development philosophy. Try to solve the real problem you have, not problems you imagine someone else having. Also, I think the simple model is precisely what's allowed other people to step in and finish rthreads. M:N with scheduler activations is just so hairy, it doesn't attract developers to fix bugs, it attracts developers to rewrite everything as 1:1.

OpenBSD Journal: What did you think of The Avengers?

tedu@: ok, so the hulk in the air was out of control, but then down on the ground he's like, it's all cool? How did that happen? They made it seem like he faked the whole out of control thing, but in that case why'd he go tearing the shit out of the place?

Truly, a question for the ages.