OpenBSD Journal

Coming Soon to OpenBSD/amd64: A Native Hypervisor

Contributed by pitrh on from the its-puffies-all-the-way-down dept.

Earlier today, Mike Larkin (mlarkin@) published a teaser for something he's been working on for a while. Then a little later in the day, an announcement appeared on tech@:

TL;DR - a native hypervisor is coming. stay tuned.

For the last few months, I've been working on a hypervisor for OpenBSD. The idea for this started a few years ago, and after playing around with it from time to time, things really started to take shape around the time of the Brisbane hackathon earlier this year. As development accelerated, the OpenBSD Foundation generously offered to fund the project so that I could focus on it in more earnest.

At this point, I think I've made sufficient progress that a public announcement is in order. I've also reached the point where I think other developers can step in and help out as much of the gooey bits in the core of the vmm are functioning the way I want.

Presently, the vmm code I've built is capable of launching a kernel and asking for the root filesystem; it doesn't do much more than that for now. However, getting to this point invovled building most of the scaffolding needed to finish the rest, and like I stated before, more people can now more easily help with what's left (basically the virtio(4) emulation for disks and network interfaces).

One might ask - why not port one of the other hypervisors out there instead of rolling your own from scratch? Fair question. However, for various technical reasons, choosing to port an existing vmm just didn't make a whole lot of sense. For example, I've been baking in support for things that the other implementations don't care about (namely i386 support, shadow paging, nested virtualization, support for legacy peripherals, etc) and trying to backfit support for those things into another hypervisor would probably have been just as hard as building it from the ground up.

The inevitable questions:

Q. What OSes can I run?
A. To start, OSes that support virtio-based devices (most of the CPU goo is done and is basically the same across most OSes once you get that part finished). Later, we'll see if we can expose a place for qemu to attach (maybe mimic KVM) to run legacy OSes or OSes that require BIOS/UEFI boot support. There is no legacy-free mandate in this vmm.

Q. When will it be ready?
A. Hard to say. I hope by the end of October, but no promises. A lot also depends on how much help I get writing some of the virtio backends.

Q. What will be the CPU requirements?
A. Any AMD or Intel CPU that supports hardware virtualization (SVM for AMD, and VMX for Intel). For those CPUs that don't support RVI or EPT, we'll use shadow paging.

Q. What will be the OS requirements?
A. The core tools (vmd(8) and vmmctl(8)) and the main vmm (vmm(4)) will be built in to base, on i386 and amd64. This should be sufficient to run virtio based guests without any additional software requirements.

Q. Yuck, i386?
A. Yes. It helps find bugs. And I urge you to review my last dozen or so commits to i386 for another reason.

And now, the most important question:

Q. How can I help?
A. I would have likely had no time to work on this project for the past few months had it not been for the generous sponsorship of the OpenBSD Foundation. Your donations made that possible, and other projects like this. So, you can help by donating. If you think a hypervisor in OpenBSD is something you'd like to take advantage of, I urge you to go make a donation right now.


And finally, a sample boot:

Enable vmm mode (ala apm/apmd):

# vmmctl -e
cpu0: entered VMM mode

Start a VM:

# vmmctl -S -c openbsd_amd64.vmrc

Attach to VM console:

# cu -l /dev/ttyp0
exit save va/pa  0xfffff000bcff000 0xbcff000
exit load va/pa  0xfffff000bd1e000 0xbd1e000
entry load va/pa 0xffffff000bd2c000 0xbd2c000
vmd: new vm with id 1 created
loading 64 bit kernel
warning: bcopy during ELF kernel load not supported
warning: bcopy during ELF kernel load not supported
mark[start] = 0x1000000
mark[entry] = 0x1000160
mark[nsym] = 0x1
mark[sym] = 0x1939000
mark[end] = 0x1a1de00
mark[random] = 0x18a4000
mark[erandom] = 0x18a6408
vmd: all vcpu run threads for vm 3 launched
vcpu_run_vmx: yielding the cpu
vmd: waiting on thread 0
vcpu_run_vmx: yielding the cpu
vcpu_run_vmx: yielding the cpu
loading 0x1a44000-0x4000000 (0x1a44-0x4000)
loading 0x100000-0x1000000
avail_start = 0x26000
avail_end = 0x4000000
First_avail = 0x1a44000
[ bsd ELF symbol table not valid: bad magic ]
[ no symbol table formats found ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
        The Regents of the University of California.  All rights reserved.
Copyright (c) 1995-2015 OpenBSD. All rights reserved.

OpenBSD 5.8-current (GENERIC) #204: Mon Aug 31 01:13:40 PDT 2015
vmd: i8253 PIT: 16 bit counter I/O not supported
RTC BIOS diagnostic error ff
real mem = 50331648 (48MB)
avail mem = 45101056 (43MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0
acpi at bios0 not configured
cpu0 at mainbus0: (uniprocessor)
cpu0: Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz
pvbus0 at mainbus0: OpenBSD
vmx_handle_cpuid: unsupported rax=0x40000100
pci0 at mainbus0 bus 0
isa0 at mainbus0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns8250, no fifo
com0: console
vmm at mainbus0 not configured
vmx_handle_cr: mov to cr8 @ ffffffff8131854a
nvram: invalid checksum
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root device:

This means that OpenBSD/amd64 is on its way to becoming a virtualization host operating system with its very own native hypervisor, joining the exclusive club that so far consisted of only OpenBSD/sparc64 with its support for logical domains (LDOMs).

Note that this is still not committed to the regular tree, this is just a teaser for upcoming goodness. To help the work along, head over to right now and do the right thing!

(Comments are closed)

  1. By brynet (Brynet) on

    Nice work mlarkin@ and everyone who donated so he could work on this!

  2. By Anonymous Coward (2003:5b:4e56:d159:c851:d127:b76:2a28) on

    Every cent and euro donated, every CD bought - I always knew that the money was spent for a good case (and will be, as long as I work and live ;))

    I encourage everyone to support this project, even when it's 2 bucks per month.

  3. By Just Another OpenBSD User ( on

    Jumped the seat reading this on the tech mailing list, held breath at the subject setting expectations lower thinking about preparation work, proposals even jails sprang to mind for a split second but then some, a full blown native hypervisor has been cooking and is coming to our favourite OpenBSD very soon!

    Most impressive and welcome news, just like in the song lyrics "so much better every time". Thanks to the foundation and developers making these and many more wonderful improvements possible, donations really do efficiently serve the project goals, exciting and much anticipated!

  4. By Renaud Allard (renaud) on

    Awesome. This is the thing that was missing. At last we will be able to build VMs with a correct OS at the core.

  5. By Anonymous Coward ( on

    but... but... I thought Theo was against virtualization.

    1. By Anonymous Coward ( on

      > but... but... I thought Theo was against virtualization.

      He can speak for himself, but I think that that's a massive oversimplification of the points he's made in the past regarding what virtualization is and is actually able to deliver. I think that this email from him[1] shows a more comprehensive view than just the "FIRE BAD" that trolls around the net are attributing to him:

      > If people were saying:
      > "Yes, it increased hardware utilization, and the nasty
      > security impact might be low"
      > it would be fine.
      > But instead we have many uneducated people saying:
      > "Yes, it increased hardware utilization, and it improved security too".
      > And that's complete and utter bullshit.


  6. By X-rayS (X-rayS) on

    It would be interesting if libvirt ( would come to support this as well.

  7. By grey (2001:5a8:4:72d0:559a:f753:a09b:ef73) on

    I am curious how this will be secured; for example:

    will it be able to be invoked and run as a non-root user?

    My experiences with bhyve by comparison, require that it be run as root. ;-/

    VMWare Fusion, on OS X can be run as a non-root user, but the installer I think puts some hooks in (when invoked as an administrative user).

    VMs have a very . . . *complex* security posture associated with them, regardless, I am excited to see OpenBSD get a native hypervisor!

  8. By Blake (2a01:e34:ec06:8f90:5a55:caff:fef5:2a75) on

    Congrats that this is seeing the light of day. Off to the donation till again I go.

    On a completely tangental note: personally I had anticipated that we would see OS-level virtualization before machine-level virtualization on OpenBSD.

    Given that we already have a Linux "emulation" ABI, has there been interest in developing an operating system level virtualization implementation (e.g. libcontainer, which has already made significant inroads towards working under FreeBSD, or perhaps simply multiple userland namespaces)?

  9. By Laurence Rochfort ( on

    Will this support windows?

    1. By Anonymous Coward ( on

      > Will this support windows?

      See the "What OSes can I run" question.

  10. By Anonymous Coward ( on

    I would love to see this eventually run Windows.

    In addition, I wonder if this means the eventual end of the Linux emulation layer in the i386 version of OpenBSD.


Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]