Virtual Machines in Kubernetes? How and what makes sense?
Happy new year.
I stopped by saying that Kubernetes can run containers on a cluster. This implies that it can perform some cluster operations (i.e. scheduling). And the question is if the cluster logic plus some virtualization logic can actually provide us virtualization functionality as we know it from oVirt.
Can it?
Maybe. At least there are a few approaches which already tried to run VMs within or on-top of Kubernetes.
Note. I'm happy to get input on clarifications for the following implementations.
Hyper created a fork to launch the container runtimes inside a VM:
docker-proxy | v [VM | docker-runtime] | + container + container :
runV is also from hyper. It is a OCI compatible container runtime. But instead of launching a container, this runtime will really launch a VM (libvirtd, qemu, …) with a given kernel, initrd and a given docker (or OCI) image.
This is pretty straight forward, thanks to the OCI standard.
frakti is actually a component implementing Kubernetes CRI (container runtime interface), and it can be used to run VM-isolated-containers in Kubernetes by using Hyper above.
rkt is actually a container runtime, but it supports to be run inside of KVM. To me this looks similar to runv, as a a VM is used for isolation purpose around a pod (not a single container).
host OS └─ rkt └─ hypervisor └─ kernel └─ systemd └─ chroot └─ user-app1
ClearContainers seem to be also much like runv and the alternative stage1 for rkt.
RancherVM is using a different approach - The VM is run inside the contianer, instead of wrapping it (like the approaches above). This means the container contains the VM runtime (qemu, libirtd, …). The VM can actually be directly adressed, because it's an explicit component.
host OS └─ docker └─ container └─ VM
This brings me to the wrap-up. Most of the solutions above use VMs as an isolation mechanism to containers. This happens transparently - as far as I can tell the VM is not directly exposed to higher levels, an dcan thus not be directly adressed in the sense of configured (i.e. adding a second display).
Except for the RancherVM solution where the VM is running inside a container. Her ethe VM is layered on-top, and is basically not hidden in the stack. By default the VM is inheriting stuff form the pod (i.e. networking, which is pretty incely solved), but it would also allow to do more with the VM.
So what is the take away? So, so, I would say. Looks like there is at least interest to somehow get VMs working for the one or the other use-case in the Kubernetes context. In most cases the Vm was hidden in the stack - this currently prevents to directly access and modify the VM, and it actually could imply that the VM is handled like a pod. Which actually means that the assumptions you have on a container will also apply to the VM. I.e. it's stateless, it can be killed, and reinstantiated. (This statement is pretty rough and hides a lot of details).
VM The issue is that we do care about VMs in oVirt, and that we love modifying them - like adding a second display, migrating them, tuning boot order and other fancy stuff. RancherVM looks to be going into a direction where we could tnue, but the others don't seem to help here.
Cluster Another question is: All the implementations above cared about running a VM, but oVirt is also caring about more, it's caring about cluster tasks - i.e. live migration, host fencing. And if the cluster tasks are on Kubernetes shoulders, then the question is: Does Kubernetes care about them as much as oVirt does? Maybe.
Conceptually Where do VMs belong? Above implementations hide the VM details (except RancherVM) - one reaosn is that Kubernetes does not care about this. Kubernetes does not have a concept for VMs- not for isolation and not as an explicit entity. And the questoin is: Should Kubernetes care? Kubernetes is great on Containers - and VMs (in the oVirt sense) are so much more. Is it worth to push all the needed knowledge into Kubernetes? And would this actually see acceptance from Kubernetes itself?
I tend to say No. The strength of Kubernetes is that it does one thing, and it does it well. Why should it get so bloated to expose all VM details?
But maybe it can learn to run VMs, and knows enough about them, to provifde a mechanism to pass through additional configuration to fine tune a VM.
Many open questions. But also a little more knowledge - and a post that got a little long.