
Minimega – A distributed VM management tool - voxadam
http://minimega.org/
======
floren
I'm one of the minimega authors. A friend pointed out that we made it on the
HN front page, so I thought I'd come in and offer to answer any questions
people may have.

~~~
DrPhish
Hello! This looks like a very slick project, but before going down this
particular rabbit hole, I'm wondering if you can tell me if I'm set up for
success or frustration: I'm running a Debian based hand-rolled 3 node kvm
cluster with sheepdog/corosync for my vm image store/cluster management and a
fully-connected p2p infiniband network for replication. Would it be possible
to use minimega for management of this environment? What features rely on a
regular openstack environment?

~~~
floren
To address the last question first, there's no dependency on openstack at all.
If you want to run VMs across the three nodes, it should be fine, and in fact
I _think_ the "sheepdog:windows7" disk format might just work in minimega. You
should be able to use the Infiniband network for inter-VM communication
provided you've got IP-over-IB set up.

Now, to address the unspoken question: _should_ you use minimega for this
environment? It depends what you use it for. If you're hosting your mail
server on the VM, I'd probably go with something else--we haven't focused on
long-running unique/essential VMs, although stability is quite good (I've run
experiments for weeks at a time). If you're playing around with network
topologies, or developing distributed software, or any other situation where
you're running lots of VMs that you may want to tear down and re-launch
frequently, it's a great option.

------
catern
Does this use libvirt? It doesn't look like it. Why not? libvirt already
provides almost all the functionality you need, you just need to layer a
distribution mechanism for configuration across multiple hosts on top.

~~~
floren
Nope, no libvirt. One big reason for that is, unless I misremember, libvirt
needs a daemon of its own. That adds yet another step to deploying minimega on
a cluster; as it is, you just copy the minimega binary to every node and run
it. If you're netbooting a cluster, adding libvirt is going to make that
initrd fatter and make bootup slower. If you're running on a "borrowed"
cluster, you may have to convince the owner to install libvirt for you.
Basically, we try to avoid external dependencies when possible, to the point
where we've even eliminated the two C libs we used to link in.

Additional reasons:

* We already had a lot of experience driving QEMU/KVM manually (before minimega, we'd write shell and python scripts to launch our big virtual environments)

* If we want to do something QEMU supports but libvirt does not, we have to work around that. Since libvirt just wraps QEMU anyway, we avoid the middle-man.

* There's XML in libvirt, and we don't like XML :)

~~~
catern
But according to a sibling comment, you support containers too (like
libvirt)... you must already have an abstraction layer between you and QEMU.
libvirt exposes pretty much every QEMU feature in existence, anyway, and
allows passing through raw command line arguments to QEMU if you need it.

How does minimega run, if not with a daemon? You can certainly copy a libvirt
binary + deps to a bunch of hosts and run it, nothing stopping that...

My guess is that minimega just grew out of your bespoke internal tools, and
you never seriously tried to get rid of those tools and replace them with a
more standard tool, like libvirt. It's disappointing. We're never going to get
a viable alternative to Openstack if people keep writing things from scratch
instead of reusing existing tools.

~~~
mdekkers
_My guess is that minimega just grew out of your bespoke internal tools, and
you never seriously tried to get rid of those tools and replace them with a
more standard tool, like libvirt. It 's disappointing_

I must have missed the toolchain-police memo that you can only work on one
toolchain. This stuff works for them, has a few cool features, and sounds
interesting - I will be trying it out. Nothing wrong with growing out of your
internal tools, and open sourcing them. By your reckoning, we'd all still be
messing around with 386BSD or MINIX

~~~
catern
There's also nothing wrong with wanting tools to reuse existing platforms
rather than reinvent the wheel. minimega looks neat, but I don't want to
sacrifice all the other libvirt-based tooling that's out there.

If minimega's virtualization abstraction for a single host was somehow
superior to libvirt's, then I would be happy to see them forging ahead. But
it's not.

~~~
mdekkers
_minimega looks neat, but I don 't want to sacrifice all the other libvirt-
based tooling that's out there_

Great! So libvirt is a good fit for you, awesome. Maybe other people have
other needs or wants, and now they have an alternative that might suit them
better. Isn't Open Source neat?

TBH, your argument is starting to sound very much like "libvirt is my tool of
choice and I'm pissed that not every available person is working on the
betterment of my preferred tool, instead choosing to work on something that is
of no interest to me"

------
tyingq
Seems like a rough equivalent to LXD, but using VM instances instead of
containers, and more actual cluster functionality versus just remote commands.

~~~
floren
It does containers too; we're especially proud of how it uses common
configuration when possible. For instance:

    
    
      # configure the disk image used to boot KVM VMs
      mm$ vm config disk /home/floren/debian.qcow2
      # give it 4 gigs of RAM instead of the default 2
      mm$ vm config mem 4096
      # configure the virtual network, in this case we call it "foo"
      mm$ vm config net foo
      # launch three KVM VMs
      mm$ vm launch kvm my-vm[1-3]
    
      # now configure the container filesystem
      mm$ vm config filesystem /home/floren/containerfs/
      # launch 10 containers; they will also get 4GB of RAM and be connected to the "foo" network
      mm$ vm launch container my-container[1-10]
    
      # start all the VMs that we've launched so far
      mm$ vm start all

~~~
tyingq
Ahh, that's great, and helps explain what makes it different.

With containers in addition to VM, it now looks more like a stripped down
proxmox that's much more scriptable.

Looks like it would be a good underpinning to implement your own VPS service
with "resource pools", where end users get some amount of cores, memory, disk,
but can slice them however they want.

~~~
floren
Yeah, there's a lot of fun things to do with it. Namespaces are especially
cool. First you set up your minimega cluster (the nodes can auto-discover each
other), then you run e.g. `namespace foo`. This creates a namespace containing
all other nodes in the cluster (your machine is considered the defacto 'head'
node but you could also add it to the namespace if desired). Now if you launch
VMs, they'll actually get distributed among the nodes of the cluster based on
available memory & current load. This makes it super easy to go from
prototyping on your desktop to running the whole thing across a cluster--add a
'namespace' call at the top of your script and it'll Just Work.

------
nik736
Sounds cool.

I am currently managing hundreds of KVM VMs with libvirt, what would convince
me to change to Minimega? Is this production ready?

