
Virtualization: good idea for startups? - martin
In a presentation he gave (slides at <a href="http://brainspl.at/xen_and_the_art_of_rails_deployment.pdf" rel="nofollow">http://brainspl.at/xen_and_the_art_of_rails_deployment.pdf</a>), Ezra Zygmuntowicz, the founder of Engine Yard, suggested that folks hosting web apps should base their infrastructure on Xen virtualization from the ground up, with a separate VM for each service. In other words, you start with one physical server, on which you have a web server VM, a database server VM, an email server VM, etc. Backing up your server configuration is now as easy as copying over one (albeit large) file, you can more easily isolate performance issues within your application's dependencies, and when you need to start worrying about scaling, it's a cinch to move a resource-hungry VM over to a new machine, boot up a new web server instance, etc.<p>I'm just curious whether anybody's tried hosting their app in a virtualized environment like this, as well as what people's thoughts are about this in general. Is it a good idea? Is Xen stable enough and is the performance good enough for this to work? Is it premature optimization for a startup not having scalability issues yet to be doing something like this, or do the other benefits help make it more justifiable? Would this really be all that much more work to set up compared to running all the services on a single box anyway?
======
myelin
Phil Pearson here from Broadband Mechanics. We use Xen for all our hosting...
the killer feature for us is the performance isolation. If one site gets too
busy for its VM, it might go down, but it doesn't hurt the other sites on the
same physical server nearly as much as without virtualization.

The ability to (relatively) quickly deploy a blank VM for development is
great, too. It means you can give people more control over things. If you have
lots of dev sites used for demos, there's the danger that someone might make a
mistake editing httpd.conf and bring it all down, but if they're all in
separate VMs, you don't need to worry.

~~~
rams
How does this work with Xen ? I haven't tried performance isolation yet. Is
there a way to allocate CPU weights, guarantee minimum CPU cycle to VM's etc.
Resource hogging by one VM could be really problematic in a hosting
environment.

~~~
myelin
I think there is, although all I've been doing is limiting how many CPUs a VM
is allowed to use. Every VM starts out with vcpus=1, and we add more when the
load factor gets up too high.

This is quite effective on a server with 8 cores (two quad Xeons) and 4G of
RAM, as by the time you've given each VM 512M of RAM, there's plenty of cores
to go around. It would be more problematic on a single or dual core machine,
though.

There are still a couple of ways a single VM can kill the box, though: it can
max out the disks (so you really don't want to be running out of memory,
unless you have a dedicated swap drive, so swap doesn't compete with normal
disk access) and it can max out the network bandwidth.

In a hosting environment you'd probably want to set up some sort of internal
traffic shaping so the each VM has a fair go at the available bandwidth, and
you'd also want to keep swap partitions on a separate disk to data. This way
if a couple of VMs run out of memory, they compete for the swap disk but other
VMs that are within their limits and don't touch the swap disk don't see a
problem.

I don't know, but I imagine you might be able to do some magic with LVM to
balance disk transactions/throughput between logical volumes...

------
cstejerean
Xen is great for many things, especially for performance. With help from newer
processors it can essentially run a virtualized OS at close to native speeds.
You could do the same with commercial offerings from VMWare but as far as free
solutions go it doesn't get much better than this. Having separate VMs for
each service does indeed make it easy to scale but you don't need to go
through this overhead from the start. Instead try to focus on building share
nothing architectures that can scale horizontally and then you can worry about
VMs at a later point.

------
DanielBMarkham
What you build should be scalable, whether you are on one server or a
thousand. I'm a huge VM fan, mainly because of the complexity of some systems
we use. If you solve your app for a few VMs somewhere, it becomes trivial
(mostly) to scale. Then you can do backups easily, share snapshots of complex
systems, allocate hardware resources virtually, etc.

Not sure about Xen, though. I've been using VMWare for a bit and I'm a happy
customer.I would be interested in hearing how the various VM tools compare.

------
eserorg
We're running ESer.org (<http://www.eser.org/usa>) on over 120 different
virtualized linux servers.

Moving a virtual machine from one server to another involves a simple rsync
operation and takes just a few minutes over gigabit ethernet. Same goes for
cloning virtual machines.

~~~
ratsbane
I just looked at eser.org. Neat-o. Have you thought about adding a
comments/ratings system or features matrix?

~~~
eserorg
There is a comments system on every page. Just scroll down to the bottom, type
in your comment, and hit submit!

I should probably make that feature more obvious.

~~~
ratsbane
Yes, thanks. I see that now. Perhaps if you made the fonts smaller and
separated the questions from the answers?

~~~
eserorg
Definitely. Good suggestion. I didn't think of that. Thanks.

Unfortunately, I'm more of a low-level kernel hacker and math geek (machine
learning algorithms, cluster file systems, robotics, etc..) than I am a UI
person.

Which is a _major_ problem since the first thing people see is the UI.

And since I wrote my own web server and database, it wasn't really possible to
use an off-the-shelf pretty-looking CMS.

So, my current project is a web-based satistical datamining system. And the UI
is ... a text box. Which means it will be more difficult for me to f-up the
UI. Although, I still might manage to do that.

~~~
eserorg
Actually, there's a good reason why the UI is so odd:

* My web-browser of choice is Lynx :-)

------
chmike
VM is generally considered for servers, but I'm interested for desktop
clients.

I would like to have one VM for off network developments or document editing
where I can ensure viruses, spyware or trojans are unable to contaminate the
system.

I would like a VM for web surfing, where I could easily throw away the VM and
restart with a fresh one. Same for mail or other internet applications.

I would also like to have one VM dedicated to do security sensitive tasks like
monitoring my bank accounts and so.

Now browsing reddit can result in installing a trojan or whatever that puts
all my applications in danger.

Pushing this logic a bit further, we can see that the logic is to go for a
more lightweight desktop with eventually more smartness on the remote service
than today. But also to combine multiple OS on a same machine. Such systems
would work as well on lightweight hand held devices as on big PC with 128
CPUs.

~~~
majimojo
I haven't actually gotten around to trying this out and getting it working,
but Enomalism.com claims to have open source software that lets you to manage
your VMs and ease deployment between VMWare and AWS EC2. Seems too good to be
true.

------
iamelgringo
I've been using VMWare on the desktop for over a year. It's just good
software, and a joy to develop on.

I can develop on whatever OS I want, and not have to worry about multiple
monitor support, or other sticky install issues. I don't have to worry about
not having a decent Office suite.

I can set up a server on a VM, poke around on it, and throw it away and start
over. I can clone a server and try different setups that vary from the base. I
can move a server from one box to another without problems.

When I get to the point of having my apps out there, I'm certainly going to
have my servers virtualized. It's a no brainer for me.

If you want to try it out, VMware server is a free dowload.
<http://www.vmware.com/products/server/>

------
patrickg-zill
I have started using Xen and like it so far. There are a few glitches, like
Xen will hide memory from the dom0 and not give it back without a reboot, etc.
Overall however I would recommend Xen, especially if you want Linux.

For really robust, high-performance stuff I heartily recommend Solaris and
Zones. CPU impact is 1%, filesystem performance difference is practically
zero. If you use packages from Blastwave you can have almost all of the
software you care about installed.

------
brl
We've built our development infrastructure with virtualization technologies.
Our main development server is a Qemu image which runs as a process on my
desktop under linux kernel virtualization (KVM). Inside that server, various
resources we use to organize development such as subversion, trac, irc, build
system, etc. are isolated into separate vserver partitions. Later when we need
to move everything onto dedicated server(s) it's clear and obvious how to do
it.

------
aston
If your services are already parallelizable to the degree that they could sit
on separate VM's, couldn't you just run them side by side on the same machine
without all of the extra overhead a VM gives you? Maybe this is a Rails-
related problem, but in general the hard part is not the move from
parallelized tasks on one machine to many. The big problem is taking some non-
parallel code and trying to distribute it much later on.

~~~
wmf
In this case we're really talking about configuration isolation. If you're
running a database and a Web server on one machine and then want to move them
onto separate machines, you have to untangle and move the config files. But if
they're separate VMs from the start, then there's nothing to untangle.

~~~
aston
Again, the process of getting your services to the level that they can run
independently (read: non-tangled config files) means that you don't need
separate VM's.

------
nailer
Anyone know a good Xen configuration tool? There doesn't seem to be a good
equivalent of say, VMware Server for Linux.

Don't care whether it's Open source or proprietary, just want something that
makes the image creation, starting / stopping, networking and reconfig simple.

~~~
mrevelle
XenSource released a product called XenCenter that looked similar to the
VMWare Server Console. Can watch a demo after signing up to receive their
spam: <http://www.xensource.com/Pages/XenEnterprise_Demo.aspx>

Note that it's written on .NET and is Windows only.

~~~
rams
crap, I have been using it at work for the last couple of months. The console
doesn't refresh correctly and they prob. went the dot-net way to impress
microsoft. Then they got bought by Citrix. Overall, I feel that the value-add
on top of the open source xen is rather weak, to technically savvy users at
least.

------
jdavid
Anyone that is using EC2 is using a customized version of zen. We have started
the process to move our services over to ec2.

first, vmware, and then we plan to make AMIs for ec2.

