
Vagrant core no longer tied to VirtualBox - BummerCloud
https://github.com/mitchellh/vagrant/commit/391dc392675c73518ebf04252d824fe916e8860b
======
snprbob86
> Showing 122 changed files with 4,156 additions and 2,915 deletions.

Wow, that's a lot of code for a tool which is doing little more than creating,
starting, stopping, deleting, and zipping-up VMs. Looking at this diff, all I
can say is: Holy Java-ish Ruby, Batman!

Most of these classes contain two non-helper methods: init and call. That's
not a class; that's a function! I'm reminded of this talk, titled "Stop
Writing Classes": <http://news.ycombinator.com/item?id=3717715>

I've used Vagrant before, but it always felt like overkill. When I had an
issue with it, I cracked open the code and was quickly deterred from digging
deeper. I hacked my VM back to working by hand and decided to avoid mucking
with Vagrant.

The next time I need automated development VM provisioning, I plan to write a
very small shell script. Here's an example of getting a box up and running
with 7ish commands: <http://www.linux-mag.com/id/7673/> For a complete
development environment, you only need to additionally mount a shared
filesystem and write an entry into your ssh config.

All that said, I do think that there is value in having a command line tool
which abstracts various machine provisioning APIs. I know that such a thing
exists for cloud providers, but I suspect local VMs have some unique concerns.
Regardless, it seems like a task that a shell script is best suited for.

~~~
mitchellh
Vagrant author here. Allow me to address both of your points.

First, the "Java-ish" Ruby: I am using the middleware pattern. The classes are
used to build up complex sequences of actions[1]. They are reusable and very
easily testable. Why classes? Why not modules with functions? It actually
started out as modules with functions, but middleware was so _painfully_ the
RIGHT way to do things, I can't imagine doing it any other way. Instead of
explaining it all, I guide you to this talk I gave on the pattern:
[http://confreaks.com/videos/961-mwrc2012-rack-middleware-
as-...](http://confreaks.com/videos/961-mwrc2012-rack-middleware-as-a-general-
purpose-abstraction)

Next, on Vagrant being "overkill." Write a shell script that does the
following: Import a virtual machine, setup host only networking, setup shared
folders, boots the machine, runs a provisioner within the machine. Wow that is
simple. Oh, wait, I need that script to work on Mac, Linux, and Windows. Oh,
VirtualBox has some bugs so can you work around that?[2] Also, host only
networking requires some guest OS specific setup (Ubuntu different from Fedora
different from FreeBSD etc.), so you have to handle that. Right, and it'd be
nice if the script didn't setup networking that collided with other machines
on the network, so please write in checks that ports and IPs don't collide.
Maybe the script should also verify that Guest Additions are properly
installed. AND shared folder performance in VirtualBox sucks, so can you
modify that script to use NFS? But only on non-Windows, because Windows
doesn't support NFS. Did I mention that NFS requires specialized configuration
depending on the host OS AND guest OS? So if you're running Mac + Ubuntu its
totally different from Linux + FreeBSD. Have fun!

Now, make all of that REUSABLE and EXTENSIBLE, while adding in some other
features (other types of networking, host name setting, and so on).

Yeah, I'd say Vagrant's 3000 line of code count is pretty spot on, and pretty
good.

[1]:
[https://github.com/mitchellh/vagrant/blob/391dc392675c73518e...](https://github.com/mitchellh/vagrant/blob/391dc392675c73518ebf04252d824fe916e8860b/plugins/providers/virtualbox/action.rb#L124)

[2]:
[https://github.com/mitchellh/vagrant/blob/391dc392675c73518e...](https://github.com/mitchellh/vagrant/blob/391dc392675c73518ebf04252d824fe916e8860b/plugins/providers/virtualbox/action/clean_machine_folder.rb)

~~~
snprbob86
Mitchell: First allow me to apologize for the tone of my comment. Re-reading
it now, I realize that I glossed over a key point: I used Vagrant every day
for a team of 5 people. In general, it worked very well, rarely broke in any
way, saved us a whole bunch of time getting up and running. So thank you for
your small, but real, contribution to my startup.

> middleware was so _painfully_ the RIGHT way to do things

I did see how you were composing the operations and I'm familiar with the
middleware pattern; thanks for sharing your talk. However, I think this is one
of those cases where OOP has a name for something that FP just takes for
granted: function composition. Regardless, Clojure's Rack equivilant does
provide middleware as functions of request -> response maps. It's much nicer
to work with, but that might be a result of Clojure's design vs Ruby's and
unrelated to Vagrant: <https://github.com/ring-clojure/ring/wiki/Concepts>

> Write a shell script that does the following

I don't have any of those requirements. I only need to support OSX as the host
OS. The team's MacBook Airs don't have enough resources to run more than one
VM, so I don't ever worry about port collisions. I need to configure my
network adapters anyway for my production deployment, so I just need an if
statement or two in that config file template. Hell, I don't even need the
ability to turn off a VM gracefully, since kill -9 seemed like a decent way to
test crash recovery.

Even thought I don't have those requirements: I used Vagrant. It saved me some
time up front.

But that complexity has a cost. Running `vagrant ssh`, for example, took 3ish
seconds on my machine. Again, this is probably due to RubyGems, which is
painfully slow. I had to explain to the team to run `vagrant ssh-config >>
~/.ssh/config`, then to go tweak that a bit, so that they could write `ssh
dev` and have it be instantaneous.

Worth noting however, is that I _already needed_ a way to generate local ssh
config for each production box to be able to talk to specific other production
boxes, and for client machines to talk to our cluster. Vagrant's ssh-config
command was just one more extra moving part to concern myself with, so I
manually put the correct config into a cat script along with the production
needs, and added it to the build process.

There was one or two more of these little things that escape my memory now.

> make all of that REUSABLE and EXTENSIBLE

I think my main complaint about Vagrant is one which you seem to be fixing,
starting with the very commit that this HN thread is attached to: You're
pulling out the REUSABLE bit of abstracting away various virtualization
platforms across a variety of host and guest operating systems.

However, I'm a big believer that EXTENSIBLE does not belong in library code.
Extensibility belongs at user layers. Furthermore, extensibility is _really
fucking hard_. 99% of the time, I'd rather the core be more reusable and allow
me to bypass extensibility.

I think that Git is a perfect example of this design. They call it plumbing
and porcelain. The plumbing commands offer zero extensibility. They don't have
any creature comforts, as their output is tuned for parsing. The goal of these
commands is to manipulate the abstractions of the underlying system. The
porcelain, by contrast, does offer extensibility via git-foo.sh scripts on
your path, aliases and configuration, commit hooks, etc. I'd see a lot of
value in something like this for virtual machine management.

Another example would be Chef & Puppet's facts systems. I forgot what they are
called. Ohai? Something like that. They provide OS-independent interfaces for
gathering facts about machines. Those components should be hoisted to first
class project status!

Unfortunately, Vagrant does not, yet, provide a plumbing vs porcelain split.
In many ways, I view Vagrant like SVN: It's a damn solid piece of software.
Lot's of people get a ton of milage out of it. However, it's underlying
concepts are slightly muddled together and there are _huge_ gains to be reaped
from deep thought and careful modularization. When those big gains do come,
they will only work on a single platform, maybe one or two guest operating
systems, people will bitch about it's low ease of use, despite it's inherent
simplicity, but the people who deeply understand machine provisioning and
management will adopt it in droves, until eventually Windows support doesn't
suck too bad, all the various use cases are more or less covered, and people
are wondering "How can you possibly still use that old thing?"

I think you've got a good start towards that goal in Vagrant, but my
complexity sense is tingling.

> I'd say Vagrant's 3000 line of code count is pretty spot on

I just ran cloc on the master branch locally; I see 12355 lines of Ruby ;-)

~~~
mitchellh
Thanks for your response. Lots of points in here so I'll try my best to
respond to each in turn. I apologize if I miss anything (just point it out,
I'm not trying to avoid anything).

1\. Function composition middleware

Actually, I simply use classes as a way to do function composition. The reason
I chose classes over functions I think doesn't really matter in the grand
scheme of things, but in my talk I mention that what I am doing is function
composition, and I spent a solid 5 to 10 minutes explaining how function
composition solves the problem. :) We're on the same page here.

2\. The fact you don't have the requirements for the situation I posted.

I understand this, and that is why Vagrant works for you. But it also works
for people who do have requirements other than yours. There is a _huge_ amount
of Windows users out there that love Vagrant because it now makes Ruby on
Rails and so on development work well! If I catered Vagrant to only _my_
requirements (Mac OS X host, Ubuntu guest), then Vagrant would indeed be a lot
simpler, but Vagrant as a tool gets a lot of its power and usefulness out of
its ability to work in heterogenous environments.

3\. `vagrant ssh` is slow. This must be caused by complexity.

This has nothing to do with complexity. This is caused by poorly optimized
code paths and how things work up to that point. I just want to note that
nowadays `vagrant` commands across the board are a LOT faster. But there is
still a lot of work to be done. Specifically the `vagrant ssh` case will get a
massive speedup due to cached SSH info.

I specifically have issue with "complexity causes slowness." Sure, it happens,
but it doesn't HAVE to, and I'll show that to be true.

4\. Plumping and porcelain Git vs Vagrant

Perhaps there is an option here to use unix-type piping to provide an API via
plumbing commands. I've decided to take a different approach. Will this work
well? Time will tell, but I don't think its fair to criticize Vagrant at this
point before this is proven/disproven yet.

5\. 12000 lines of Ruby.

~3500 is test code.

~5500 is "plugins" but there is a lot of boilerplate here. The boilerplate is
on purpose, I didn't want any "magic," hence it being somewhat Java-like, I
suppose, but this is a lot more friendly to non-Ruby developers, and a vast
majority of Vagrant users are not Ruby developers. Note that ~3000 of this is
ONLY the VirtualBox plugin, which is a pretty complex piece, so that makes
sense. The other ~2500 is _every other_ feature of Vagrant.

~3300 is core code. This is mostly a glorified plugin runner that handles
plumbing for you, such as hooking up the right host/guest combination,
choosing the right communication (SSH, WinRM, etc.) for the machine, error
handling, etc.

~~~
bitshift
Hi Mitchell, it's really great to have you explain all of these things here.
Thank you both for this and for writing Vagrant, there is no doubt that it is
a valuable piece of software. I have a few more questions if you don't mind:

1\. When you first started with Vagrant, did you consider using Python at all?
Did you consider using libvirt? Because VirtualBox offers a Python API out of
the box (or it does now) and libvirt comes with direct support for Python
(<http://libvirt.org/bindings.html>). I like Ruby and generally I prefer it to
Python but I am interested in your view of whether it was the right tool for
the job in this case or whether the language choice didn't matter much.

2\. In your view what is the overlap in functionality between libvirt and
Vagrant and what are the differences? As far as I can see libvirt does let you
spin up boxes, talk to them and tear them down. The notable difference is that
there is no Puppet of Chef provisioning, is there anything else beyond that?

3\. veewee in version 0.3.0 (currently in beta) seems to offer the same
functionality as Vagrant plus the ability to create boxes using templates,
minus the ability to run provisioners. If they add that say in 0.4.0 then it
will be 1:1 in terms of features with Vagrant. Any thoughts on this?

Thank you again for your time and for answering in such detail :-)

~~~
mitchellh
No problem, I'm glad you're getting value out of the responses. In response to
your questions:

1\. When I first started Vagrant, I was a full time Ruby developer, so Ruby
was really the obvious choice for me. I don't really see language as a barrier
for using a tool, my opinion is generally use the best tool for the job. And
as a Ruby developer that at the time had 4 years of Ruby experience, that was
the best tool for the job since I'd get the most productivity out of it. To
date, I don't regret this decision. There are some things Ruby is bad at, some
things it is good at, but I think I can overcome the bad with time.

2\. I think the difference is that Vagrant is a tool focused on user
experience and workflow. i.e. it'd be possibly to build Vagrant on top of
libvirt. I think libvirt is technically stronger than Vagrant, but Vagrant
provides a better overall experience. The overlap is small, we both have a lot
to gain from working with each other.

3\. A handful of people keep saying that Vagrant is just a "VM setup +
provisioning" tool. Vagrant does quite a lot more. The main example I always
use is networking because it is usually the most complex. Vagrant makes
networks work across Linux, Mac OS X, and Windows, and sets up the hosts and
guests properly. VeeWee doesn't do this and won't do this. VeeWee at its core
was built to be a VM image creation tool. VM lifecycle control was bolted on
later. VeeWee filled a major gap in the Vagrant ecosystem for a long time.

I'm good friends with the creator of VeeWee and we talked all the way back in
Oct, 2011 of merging the functionality. He was all for it. Its almost been a
year but this work is finally going to get started.

I don't want to talk too much about it until I really start coding, but I can
say that no one in his thread has really seen the true scope of what I'm
trying to build here. They will. :)

------
jacques_chester
This is exciting news.

I know one fellow who built a large private cloud for his organisation (about
two racks of topline gear) so that they could simulate an architecture of
several dozen servers and thousands of complex, concurrent transations.

What's it built on? VMWare. He was not totally amused when I sent him a link
to Vagrant.

(I understand VMWare have similar fleet-orchestration tools, but not
development-oriented as vagrant is -- awesome for production though).

Putting aside my friend, I can also see performance gains from in-kernel
virtualisation schemes such as KVM^w lxc or Solaris Zones.

Bravo to the Vagrant team. This tool has become one of the centrepieces of how
I personally do development.

~~~
justincormack
KVM is a full virtualization solution, Zones is a container (same kernel).
Maybe you mean lxc, the Linux container system?

~~~
jacques_chester
Thanks for the correction.

------
bitshift
I'm not sure what are the benefits of supporting many VM providers for a tool
that is meant for development (as far as I understand at least). Isn't
VirtualBox sufficient? It is open source and runs on Linux, OS X and MS
Windows. Perhaps I am missing some core idea.

Then there is veewee[1] which started life as a Vagrant plugin but as of
version 0.3.0 it can be used standalone and one can ssh to the boxes using it,
just like with Vagrant. If they were to add provisioning it would probably be
an alternative to Vagrant feature for feature so maybe we're seeing a bit of
competition between these two tools there.

On Ubuntu there is VMBuilder[2] written in Python but it doesn't offer
provisioning as far as I know.

Last, as others have mentioned there is libvirt[3] which offers tools for
communicating with different VMs but I don't think it offers a tool that
matches what Vagrant does 1:1 (most notably the provisioning part).

1\.
[https://github.com/jedi4ever/veewee/blob/master/doc/README-c...](https://github.com/jedi4ever/veewee/blob/master/doc/README-
changes.txt)

2\. <https://launchpad.net/vmbuilder>

3\. <http://libvirt.org>

~~~
davnola
veewee is being merged into vagrant.

~~~
bitshift
I wasn't aware of this. Have you seen anything that supports it?

------
Axsuul
This is really great news. Vagrant has really made my life easier as a
developer, thanks Mitchell! I prefer VMware so this is awesome!

I personally think vagrant needs a UI manager of some sorts (I 'm thinking
about doing this when I have more free time). I have about 10-15 vagrant boxes
now and it can be a pain to manage. Having to go into each project directory
and manually up'ing each vagrant box can be tedious.

------
shell0x
Good to hear this. I had several issues with Virtual Box in the past on OS
X(Lion, Mountain Lion) and it even produced 3-4 times a kernel crash on my
MacBook. I switched to VMWare Fusion then and all the complains were gone.

------
rwmj
Reimplementing libvirt, badly ...

~~~
mitchellh
Vagrant author here.

This is incorrect. I am not trying to reimplement libvirt at all. If anything,
I'm providing the infrastructure to use libvirt WITH Vagrant. Actually,
surprise! I'm working with RedHat to do JUST THIS.

I applaud the libvirt project for everything they've done. We've had our
difference, but I have no intention of competing with that project. The power
in Vagrant is the workflow of Vagrant, not Vagrant's ability to abstract away
the hypervisor (and I certainly don't want to play that game).

The power in Vagrant is the workflow, and that is the game that I'm playing.

~~~
rwmj
Well, I take it back. This is great news.

