
Virtlet: run VMs as Kubernetes pods - ivan4th
https://www.mirantis.com/blog/virtlet-run-vms-as-kubernetes-pods/
======
darren0
The real reason for these projects (Virlet, KubeVirt, RancherVM) is that
OpenStack is too damn hard. Even with all this focus on containers these days
people still need an onprem VM solution. But the requirements are really
pretty simple for most use cases and the cost and complexity of OpenStack is
not justified.

Edit: disclaimer: my company does RancherVM.

~~~
jell
The reason for this project was that people want to migrate to containers as
industry is pushing them forward to it, but - there will always be cases in
which "normal" containers will be out of scope - when you will be dependent on
piece which is: a) based on other than linux based operating system b) you
need a specific linux module which for some reason should not be loaded on
host kernel c) you need a hardware separation using virtualization because of
security reasons All that can be achieved using "somewhere around" openstack,
but what about integration/common interface with other parts of stack? That's
the reason why we did Virtlet - to have the same interface for "normal"
(docker image based) pods and VMs (hard disk image based), with the same
kubectl command, with the same api (so it can be used for deployments,
daemonsets, statefulsets and so on), with the same thing configuring
networking in whole cluster (CNI, hopefully any of your choice - if it's not
working, please fill the bug on GH).

~~~
darren0
I don't fully buy that explanation. If OpenStack was better they would have
already had success on that platform and then would be asking for a bridge
between k8s and OpenStack (openstack could easily be a virtual kubelet).
Instead people are looking to abandon OpenStack in favor of something
hopefully better.

~~~
jell
I'm not saying that OpenStack is better/worse - it was only out of our goals.
For sure OpenStack has better tenancy support, much more mature approach to
networking or storage (in containers world we are still working on
[https://github.com/container-storage-
interface/spec](https://github.com/container-storage-interface/spec) ) but in
the same time - we only needed to have a qcow2 based vm running between
already deployed pods, with the same user interface. That's all, without other
story around that.

------
rwmj
Interesting to see how this _really_ compares to KubeVirt, which seems to be
doing the same thing. I don't think KubeVirt is "just" pets as far as I
understand. [disclaimer: I've been very peripherally involved with KubeVirt
because they asked me about integrating virt-v2v support].

~~~
ivan4th
You can't have a StatefulSet of VMs on KubeVirt, although you may want one if
you're using unikernels or want to run a nested Kubernetes cluster for
testing, like here:
[https://github.com/Mirantis/virtlet/blob/master/examples/k8s...](https://github.com/Mirantis/virtlet/blob/master/examples/k8s.yaml)
On the other hand, you can't have VM migrations with Virtlet.

~~~
rwmj
Be nice if there was just one "virt for Kube" technology. I guess virt-v2v
will end up having to support both.

~~~
wmf
Given that the container ecosystem is a huge land rush with many obviously
missing pieces, there are going to be multiple projects for every piece.

------
s0l1dsnak3123
I'm using kubevirt for this. My usecase is to allow preconfigured windows VMs
to boot up inside of kubernetes - so that both Windows VMs and Linux
containers are manipulated using the same API. It works very well!

~~~
jell
the key is that with Virtlet you can use you windows VM as other pods in
replicasets, daemonsets, statefulsets, what can not be done using CRD offered
by kubevirt. In the same time, assuming that your windows image supports cloud
init (e.g. using [http://cloudbase-init.readthedocs.io](http://cloudbase-
init.readthedocs.io) ) - you can setup it during the boot, configuring users,
passwords, running arbitrary scripts - you have full control about what will
be done during cloud init phase using only pod annotations (you don't need
prepared specific image with predefined content as in KubeVirt case). In case
of Virtlet you have a possibility to merge data from different sources (config
maps, secrets, annotations) and use the result in output cloud init image.

At the end - KubeVirt provides you a way to run a VM as a custom resource in
an app on k8s (virt-launcher), while Virtlet provides a way to run VM as first
class cluster citizen, as a pod, with the same interfaces as for other pods
(so you can use kubectl logs, kubectl attach -it, as with other "normal"
(docker image based) pods).

ok, with windows vm you will see in kubectl logs only cloud init logs as thats
the only part which is using serial console to output something :P

with other types of vm images (based on linux, bsd or unikernels) you would
probably appreciate more possibility to use kubectl attach -it, or kubectl
logs ;)

~~~
jell
btw. both solutions have they strong points - virtlet is a bit closer to k8s
interfaces, while kubevirt provides closer to libvirt options (e.g. TAL on
[https://twitter.com/dummdida/status/992037913352392705](https://twitter.com/dummdida/status/992037913352392705)
)

------
edaemon
What's the key difference between Virtlet/KubeVirt and just running VMs
alongside your Kubernetes pods? Is the main goal to centralize management?
That seems reasonable, I'm just wondering.

Also, how does a Kubernetes-managed VM compare to a plain VM from, say, AWS
EC2? I imagine it's a little less efficient, since the situation I'm imagining
involves various VMs running a Kubernetes pod running the VM, but I may have
this all wrong.

~~~
ivan4th
Besides using kubectl to manage VMs, you also get VMs that join the cluster
network as first-class citizens. In this blog post you can see a k8s service
pointing to a VM and accessing k8s services from within the VMs. Also,
compared to EC2 / GCE / etc., you can use Virtlet for bare metal on-prem
clusters, too.

~~~
edaemon
That makes sense, thanks.

------
agumonkey
reminds me of vmlets term from the virtual virtual machines
[https://pages.lip6.fr/vvm/publications/0008SBAC.pdf](https://pages.lip6.fr/vvm/publications/0008SBAC.pdf)

