
Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack - vegasbrianc
http://www.forbes.com/sites/reuvencohen/2013/03/26/paypal-to-drop-vmware-from-80000-servers-and-replace-it-with-openstack/
======
3am
I hate cloud headlines. What's the hypervisor? VMWare is a hypervisor (ESX)
and management applications (vCenter/vSphere, vCloud Director, etc). Openstack
is only the management applications. And VMWare contributes the most
interesting part of OpenStack, anyway, which is the virtual networking based
on Nicira's OpenFlow.

My personal experience with OpenStack (admittedly late Diablo release
timeframe) was that it was borderline unusable outside of the developer setup
of a standalone node on Ubuntu 11.4 backed with KVM (edit: and that was only
Nova/Compute, not even including Swift/Storage).

Anyway, very interested to see if this is KVM, Xen, or something else backed.
Also very interested to see when they reverse their decision (probably after
VMWare comes down on pricing by 10-20 percent).

~~~
sc68cal
>What's the hypervisor?

OpenStack supports multiple hypervisors. They are implemented as drivers - and
are in this part of the OpenStack codebase:

<https://github.com/openstack/nova/tree/master/nova/virt/>

See my slides from a talk I did yesterday.

[https://github.com/sc68cal/openstack-theory-and-practice-
pre...](https://github.com/sc68cal/openstack-theory-and-practice-presentation)

~~~
easytiger
Then they should say that. Reading their website tells me their are doing that
bullshit thing were they divorce the tech from the marketing completely.

> OpenStack is a cloud operating system that controls large pools of compute,
> storage, and networking resources throughout a datacenter, all managed
> through a dashboard that gives administrators control while empowering their
> users to provision resources through a web interface.

Should it (??) really say :

> Openstack is a ton of scripts and tools and things which pull lots of
> various bits of software together and allow you to have redeployable virtual
> machines all over the place

------
ffk
A few things going on here,

1\. According to Business Insider (BI), PayPal is using Fuel from Mirantis to
manage its OpenStack deployment.

2\. BI also announced that Fuel was released on March 25 under the Apache 2.0
License. However, the fuel website points to a form that appears to give free
access to fuel under a creative commons attribution, noncommercial, share
alike license, with additional licensing options available from Mirantis.

3\. To top it all off, the CEO of Mirantis contacted BI and said the
information about PayPal migrating over was incorrect and based off of second-
hand knowledge (as seen on the update in the bottom).

4\. BI also spoke to paypal who stated they are diversifying their VM
infrastructure to "enable choice and agility", not replacing VMware.

So, based on all this, it sounds like what really happened is PayPal decided
to implement OpenStack to allow for a more diverse internal set of tools to
develop infrastructure. A miscommunication occurred somewhere, which resulted
in the BI and Forbes story.

[http://www.businessinsider.com/a-dangerous-sign-for-
vmware-p...](http://www.businessinsider.com/a-dangerous-sign-for-vmware-
paypal-chooses-rival-openstack-2013-3#ixzz2OeJgF5JI)

<https://fuel.mirantis.com/>

------
babarock
Slightly relevant:

If you're interested in discovering OpenStack, you should check out
devstack[1]. It's a bash script that (git) pulls different projects of
openstack into a machine and install them.

The script itself is very readable, thoroughly commented, on purpose. You're
encouraged to read it while the whole thing installs (which can take several
minutes).

Feel free to launch it inside of a VM (preferably running Ubuntu or Fedora),
in order to avoid polluting your current system.

[1]: <http://devstack.org/>

~~~
batgaijin
how does this compare to puppet?

~~~
xt
It's an alternative to puppet. Biggest difference is built-in remote
execution.

~~~
babarock
Not exactly, it doesn't have the same goal at all.

From the first line of the script: "stack.sh is an opinionated OpenStack
developer installation". It doesn't provide nearly the flexibility of Puppet
(or Chef).

Devstack does one thing, and one thing (almost) only: get a simple
installation of OpenStack up and running as fast as possible (predominantly
for developers to test their modifications against).

Puppet can handle much more complex scenarios, and is really meant to deploy
production.

------
niggler
Can we at least link to the original article:
[http://www.businessinsider.com/a-dangerous-sign-for-
vmware-p...](http://www.businessinsider.com/a-dangerous-sign-for-vmware-
paypal-chooses-rival-openstack-2013-3)

I'm stunned by the extent to which the forbes.com rehashing of articles crest
the front page compared to the original articles.

------
mkoz
_"UPDATE: We have heard from Adrian Ionel, the CEO of Renski's company
Mirantis, who says that Renski is "exaggerating the use case" of OpenStack at
PayPal and that Renski's "knowledge of the project is second hand and
therefore limited." Renski's title at Mirantis is executive vice president."_

It looks like they are adding OpenStack to the suite of tools they use at
PayPal and not replacing anything(at least, not yet). Sounds like Renski is
just a little over-excited when commenting about a platform he believes in.

------
blantonl
Paypal runs on _80 THOUSAND_ servers? And it takes hours for me to even get a
quarterly transaction log prepared for download?

That just boggles the mind....

~~~
ditoa
Is that 80,000 physical or virtual servers though? That ratio could easily be
10:1 if it is the VM count which is more what I would be expecting.

~~~
bluedino
The article says 10,000 servers, so maybe there's an 8:1 ratio

~~~
ditoa
Unless I read it wrong the article says 10,000 servers going live this summer
with the plan being to replace all 80,000 servers in total.

edit: just noticed they sneak "and eBay" into the 80,000 total server count.
Makes a little more sense now :)

~~~
enigmo
eBay doesn't have anything close to 80,000 servers.

------
hp50g
I think this only solves the cost problem. VMware is damn expensive for what
it is.

I'm not really a fan of virtualization myself. We canned it at our
organisation a couple of years ago. We had 30 ESX/vSphere managed hosts across
3 data centers which hosted app servers, web front end servers, virtual load
balancers and some other minor infrastructure. The final straw was the upgrade
costs. Not only that we had problems with volume size limits on our SAN which
would require more expense and hackery to work around. Also, we couldn't host
our database servers on top of it due to performance problems so we had to
have dedicated machines there anyway.

The whole thing at the end of the day just added complexity, expense and
didn't improve reliability, security or load distribution as our architecture
was sound on that front already.

We've gone back to the original virtualization system: processes and sensible
distribution of them across multiple machines. Performance, cost and sanity
have improved.

I can understand that virtualization is useful when your resource requirement
is less than one machine but above that, I doubt there are any real benefits.
It's snake oil.

~~~
hhw
Exactly. Virtualization/Cloud are only good for scaling down, not scaling up.
The 20-30% hit to IO, additional latency of SAN vs local storage, management
overhead, and added complexity to infrastructure more than negate any
potential benefits of improved load distribution when you're dealing with
applications that use anywhere near the capacity of physical servers.

~~~
druiid
20-30% hit in IO? It hasn't been at that level for a long time. With new KVM
versions and good Intel processors there is a performance hit of as low as 5%
these days. That's not to necessarily say your particular workload will see
that, but 10% is generally an 'at-worst' level at this point.

~~~
hp50g
Its definitely 20-30% for realistic workloads using VM based tech on top I.e.
CLR/JVM or a database engine which is realistic. This is on top of VMware. I
can't speak for Xen.

The outcome is pretty grim.

~~~
hhw
Exactly. I often see claims of 5-10%, but I've yet to see any reliable set of
benchmarks done with those results. Too often, people are using dd and testing
throughput instead of actual IOPS. Even the benchmarks that show 20%+ tend to
be skewed in favour of virtualization, as they tend to be run with a single VM
instead of multiple VM's.

Even if there was 0% performance penalty from virtualization, you'd still see
suboptimal allocation of hardware resources just from trying to take an
abstracted view of the hardware. Different applications have different
performance profiles. You either end up with overbuilt hardware to support the
virtualization environment and the different performance profiles of the
different applications, or with multiple VM's for the same application on the
same hardware which is totally unnecessary overhead. Virtualization is just
not meant for large scale.

~~~
amscanne
Here [1] is a great paper about _nested_ virutalization for KVM. This combines
hardware capabilities with software tricks to allow running multiple levels of
VMMs. It may not have intense IOPs testing but it's got a couple benchmarks
that would be representative of real-world workloads. Keep in mind that this
paper was published in 2010 and virtualization performance has been on a
dramatic rise the last several years.

Jump to the results section. The more relevant bullets here are 'single guest'
(either virtio or using direct mapping).

Highlights (or lowlights, depending on your perspective): kernbench - 9.5%
overhead SPECjbb - 7.6% overhead

I don't agree with your point about suboptimal allocation of hardware
resources. Virtualization does not require you to divide a machine in a
different way than processes do (you could easily have one VM consume nearly
all CPU cycles, one consuming nearly all I/O capacity, etc.) IMO, the key
difference is that virtualization _lets_ you easier establish hard,
enforceable limits and concrete policies around resource usage (not to mention
the ability to _account_ for usage across all kinds of different applications
and users). And, it lets you do that for arbitrary applications on arbitrary
operating systems. So users don't have to write to one particular
framework/language/runtime/OS whatever. That's all pretty important for large
scale.

[1] [http://static.usenix.org/event/osdi10/tech/slides/ben-
yehuda...](http://static.usenix.org/event/osdi10/tech/slides/ben-yehuda.pdf)

~~~
tmzt
What distinction does KVM or the kernel make for a single guest?

Is there a system that would allow for mapping part of an IO device (such as a
block range or a LUN) to a guest when multiple guests are running with the
same level of overhead?

~~~
amscanne
I'm not sure I follow your question 100%, but I'm gonna take a stab...

The distinction being made here isn't for a single guest or multiple guests,
it's for a single guest OS or _nested_ guests (i.e. a VM running another VM).
To expose the hardware virtualization extensions to the guest VMM, then they
must be emulated by the privileged domain (host). There are software tricks
that allow this emulation to happen pretty efficiently (and map an arbitrary
level of guests onto the single level provided by the _actual_ hardware). It's
not a common use-case, but for a few very specific things it's very useful.

There are a few different ways to map I/O devices directly into domains. Some
definitely allow for _part_ of an I/O device. For example, many new network
devices support SR-IOV -- which effectively allows you to poke it and create
new virtual devices (which may be constrainted in some way) which can be
mapped directly into guests.

------
reidrac
So they're going to move to OpenStack Compute but... which supervisor are they
going to use? I guess is not VMware (KVM? Xen?), but I still find the article
a little bit confusing.

Besides the quotes suck ("OpenStack is not really free"; well, if you don't
know how things work you'll have to pay who knows), so I'm not sure if I can
trust that information.

------
mullr
"VMware’s RabbitMQ, an open source middleware-type application deployment
project, ..."

Perhaps it's mean to poke fun at reporters, but this is pretty brazen.

~~~
lflux
Not quite sure what you're getting at here.

~~~
lifeisstillgood
RabbitMQ is a messaging queue, Erlang based, not sure if it is owned by
VMWare. I get the idea that either it is not owned by them, or the description
is wrong enough

Oddly quite a few build systems are now turning to MQ's for build systems -
beats make and jenkins it seems

~~~
mullr
> quite a few build systems are now turning to MQ's for build systems

References? I'm curious as to who's doing this and why. I'm trying to think of
a reason and not coming up with much.

~~~
mindjiver
We have started using a MQ to send messages between different Jenkins
instances. For example to have long running test executions on a dedicated
jenkins-instance so the main build-jenkins can be restarted without affecting
the tests.

~~~
ternaryoperator
Curious how you've set this up. What role does the MQ play? Does it ship
results to the mother ship or a console? Or are you sending notices of changes
to the test machine? etc. TIA

~~~
mindjiver
We have custom plugin to send message to the MQ. Other Jenkins-instances
running the same plugin will receive the message and can act on them according
to the configuration.

Currently we mostly use this to send internal release notifications around,
e.g. "foobar-1.0.0 released, find it here <path to RPM>". Kind of neat with a
n * 1000 developer organization spread over the entire world.

Currently the plugin is not open source but hopefully it will be later on.

~~~
ternaryoperator
Very cool. Thanks for the explanation!

------
outside1234
Is anyone in the startup world using OpenStack? If so, how is it working for
you?

I'm always skeptical of abstractions but it'd be great to hear about any real
world experience.

~~~
lifeisstillgood
Yes.

It runs.

Its a VM. I can add and delete and image. Remotely.

Its admin system is all REST(ish) and I did have a fully working auto-build
system till they changed it and I have to rebuild (known API breaker, well
telegraphed, unlikely to be more, I was just lazy)

------
sgloutnikov
I think VMWare is going to be in some trouble in the next few years. A lot of
opensource projects are picking up speed. One of my favorites, oVirt
(<http://www.ovirt.org>) especially. Red Hat mainly runs the project, and is
pretty much the open source version of RHEV.

~~~
yardie
I don't think they are in that much trouble. They are just making a boatload
of money while they can get away with it. But they are working on multiple
tiers: free, starter, and "OMG, Money"! That last one might not be around for
much longer.

If you look at their other offerings some of it is reasonable (VMWare Fusion)
and others are set at whatever the market will bear (VMWare Workstation,
vsphere Enterprise).

------
wilhil
Well, it makes sense... OpenStack is getting better and better with each
release and whilst VMWare do add some cool new features every now and then,
the price simply sky rockets and you don't really need the new features.

For the customers who do not need much more than basics (e.g. just
virtualisation, storage server etc.), it is just so hard to justify the price
now.

Just the other day I had a client who wanted basic fault tolerance and by
changing versions, the quote goes from 6k to 47k! It is crazy pricing... :(

------
BroNamath
I can confidently say forbes.com is the worst website of all time

------
neya
If I'm correct, Openstack is different from Openshift and Openshift actually
runs on top of Openstack, right? It's more like an appserver that can run Ruby
on Rails, Play framework, etc., right?

Because very often I tend to get confused between the two..because of the
vague similarity in their names..

~~~
cacois
OpenShift (<https://www.openshift.com/>) is a PaaS, made by RedHat. OpenStack
(<http://www.openstack.org/>) is an IaaS cloud platform, maintained as an open
source offering by Rackspace, and originally created by a partnership between
NASA and Rackspace.

OpenShift could theoretically run in an OpenStack cloud, as just about
anything could, but the two products are in no way related to my knowledge.

~~~
neya
Thank you!

------
SqMafia
In other news, VMware has announced plans to accept only Square payments for
all $4 billion of its revenue.

------
ericcholis
I'm very interested to see how this all plays out. Despite their customer
service issues, PayPal has a huge ecosystem. I'm curious to see if this speeds
up their infrastructure at all, considering that their API and website have
been notoriously slow for years.

------
daigoba66
What's wrong with VMware? Why would someone not choose VWware over something
else?

~~~
samspenc
In my experience, VMWare is great and has a nice GUI, which lead to early
adoption. BUT: it has licensing fees for each copy. This can be a big bummer
for companies like PayPal that have a fleet of servers - much easier to
replace to replace all the VMWare installs and get rid of licensing costs to
switch to an open-source VM solution.

At the same time, I could be that OpenStack's API is superior to VMWare's.
With a fleet of servers, you want to automate as much as possible - meaning
that at the end of the day, API trumps GUI.

~~~
waps
Switch to virtualbox ... Nice gui, better features, open source, no licencing
per copy.

~~~
tacticus
Not really the same thing.

they would be talking about vmware ESX\vsphere not workstation.

------
polskibus
previously @ <http://news.ycombinator.com/item?id=5439770>

~~~
GFischer
Yep, that was the original source, it didn't gain any traction or comments.

It's pretty interesting, I hadn't heard of OpenStack before, and the company I
work for is migrating TO VmWare.

------
dcc1
Internet to drop Paypal and replace with Bitcoin :D

