KVM proposal lets VMMs host VMware guests without a host-wide switch

A proposed KVM change would let userspace virtual machine monitors run unmodified VMware guests without turning on a host-wide compatibility switch. Zack Rusin of Broadcom, whose VMware Workstation product already runs on KVM, posted the work on the kernel mailing list.

VMware guests talk to their hypervisor through a backdoor of roughly a hundred commands that VMware implements in userspace. Older guests reach it with port I/O and a few fake performance counters, both of which KVM already emulates, and which VMware allows from user mode even when the guest's I/O permissions would deny them. Newer guests use the same commands through VMCALL or VMMCALL. Today the port and counter paths sit behind a module parameter that applies to every virtual machine on the host, whichever monitor created it.

The proposal replaces that blunt switch with a per-VM capability. Existing setups keep working: the module parameter still seeds the default. Monitors that want VMware compatibility can turn the legacy backdoor on only for the guests that need it, and can ask KVM to exit to userspace when a guest issues a VMware hypercall. KVM reports the privilege level and the registers; userspace finishes the call or asks KVM to inject a fault. Calls that do not carry VMware's magic value are left alone, so the same VM can still use KVM, Xen, or Hyper-V hypercalls. Doug Covelli reviewed the command list and found none that KVM itself has to handle.

Nested guests are the sharper case. VMware Tools issue backdoor port I/O from user mode. When Windows virtualization-based security puts Hyper-V under the guest, those tools run as a nested guest, and the resulting fault is delivered to Hyper-V, which does not implement the backdoor. An opt-in flag lets KVM claim only those accesses and hand them to userspace, while a nested hypervisor that does implement the backdoor, such as ESXi, can keep receiving them. A Hyper-V nested TLB flush that KVM handles itself stays a Hyper-V hypercall even if VMware's magic value happens to sit in a register.

Kernels that do not need VMware guests could compile the emulation out, the same way they already can for Hyper-V and Xen, shrinking the interface exposed to guests. The option would default on so current configurations keep the backdoor. Encrypted guests are a limit, not a target: KVM still cannot bypass I/O permissions for SEV, and VMs whose register state is protected cannot enable the hypercall exit.

Rusin and Covelli are proposed as maintainers of the new code. The series is not yet merged.