Publicly Accessible Server Inside qemu

Undeadly reader Ted Walther writes in with a story about how he set up qemu for live testing:

I have a server at a colocation site that hosts various company software. A new hire came in and I wanted him to get some practice at configuring Apache, DNS, and various things like that. Instead of buying a new computer, I decided to use qemu to emulate a computer on the colocated server. I wanted the emulated computer to be accessible to the internet at large, just as a regular server with static IP is. Here is how I did it.

We assume you already have OpenBSD installed and running on the raw disk image that you will be booting with qemu.

The main network interface (NIC) on the colocated box is re0. The colocated server is a vanilla OpenBSD 4.7 install; if the following files already exist, pick new names as appropriate.

Fill in these two files:

$ cat /etc/hostname.bridge0
add re0
add tun0
up

$ cat /etc/hostname.tun0
link0 up

Substitute your real interface for re0 in the bridge file.

Add the following to /etc/rc.local

$ cat /etc/rc.local
if [ -x /usr/local/bin/qemu ]; then
    echo -n 'Qemu: vmi386'
    sh -c "sudo -C 4 -u qemuuser \
        /usr/local/bin/qemu \
        -daemonize \
        -nographic \
        -net nic,vlan=0,model=rtl8139,macaddr=aa:aa:aa:aa:aa:aa \
        -net tap,vlan=0,fd=3,script=no \
        -m 512 \
        -hda /path/to/raw/disk.img \
        -serial null \
        -monitor null \
        -no-fd-bootchk 3<>/dev/tun0"
    echo "."
fi

The above is almost identical to the README. It starts qemu and then drops root privileges. There was a recent bug fix in sudo on March 11, 2010 and the -C 4 option to sudo works around the bug. Without it, the tun0 device is never opened, and qemu has no network access.

Change qemuuser to the user you want the qemu instance to run as. Make sure that user has read/write access to the raw disk image.

Change the aa:aa:aa:aa:aa:aa to some random MAC address; this is to prevent network sniffers from guessing you are a qemu instance based on your MAC address.

With the -serial and -monitor options set to null, you can only ssh in. You will need completely different options when doing your fresh OpenBSD install on the raw disk image. I eventually broke down and installed a vnc client on another local box; then I was able to connect to the qemu instance on OpenBSD and install OpenBSD on the raw disk image.

There is a kernel module and package called kqemu which accelerates qemu performance. It is apparently very unstable on multi-cpu and on non-i386 architectures. Without kqemu, qemu runs at 1/4 of the speed of the host processor. A KVM specific to OpenBSD for accelerating qemu would be nice. I wonder what it would cost in developer time.

Why did I feel the need to write this document? After several days of struggling and not knowing about the sudo bug, I was questioning everything, and wondering what exactly the network model of qemu is. But it is simple; qemu connects to the tun0 device. tun0 is bridged to the bridge0 device. From there, tun0 sees all packets that come into your real interface (re0 in my case). re0 also forwards the packets coming out from tun0. Magic. There is some stuff in the README about a trunk device; it isn't necessary in this simple configuration.

No modifications to /etc/pf.conf need to be made. You do not need have to have net.inet.ip.forwarding enabled in /etc/sysctl.conf.

Thank you Todd Fries (todd@) for your effective and reasonably priced consulting services. After I'd gotten as far as I could, Todd just Made It Work.

The /usr/local/share/doc/qemu/README.OpenBSD file and the qemu manpage are still essential reading; this little writeup is just to add clarification.