
Moz Dumps Amazon Web Services, Citing Expense and 'Lacking' Service - rbudd
http://www.xconomy.com/seattle/2014/01/30/moz-dumps-amazon-web-services-citing-expense-and-lacking-service/
======
mgkimsal
For what _many_ orgs need to do, a myriad of lesser-known VPS providers would
be just fine, and be a fraction of the cost. But most people I've known don't
investigate or educate themselves much - it's been a herd mentality towards
AWS, irrespective of needs or costs (because often, it's not them paying -
it's "the company").

Perhaps as a freelance consultant right now I take a sharper look at the costs
involved - if a client has, say, $50k to budget on a project, I really don't
want $10k of that to go to hosting if it doesn't need to - I'd rather it be
available for my services directly (obviously) or to give us more play for
marketing, testing, whatever.

I've known a couple of small to medium sized companies where they were using
AWS rather strategically as part of their services - dynamically spinning up
test environments for running parallel testing, off-peak data crunching, etc -
but even in those cases, there weren't gigantic savings vs other options.

What bugs me most is the false dichotomy I get when I talk about this with
friends; their choice is presented as either use AWS or "hire a full-time
sysadmin, buy multiple server-grade boxes and manage all this in a rented data
center". I've been told Amazon is saving people "easily" $100k/year by not
having to hire staff and buy expensive hardware. Hrmm... not all hardware is
expensive, and instead of hiring a sysadmin, you've now got multiple
developers spending time learning the AWS API, and building critical business
processes around AWS, tying you to Amazon for the long haul, all while they
_could_ be focusing more on line-of-business stuff that you hired them for in
the first place.

I don't doubt that in some cases there's robust cost savings, but it's never
struck me as a terribly cost-efficient way of doing any web stuff. In some
cases it may be the _only_ way to deal with certain problems, but in those
cases cost probably isn't a consideration anyway.

~~~
d23
> if a client has, say, $50k to budget on a project, I really don't want $10k
> of that to go to hosting if it doesn't need to

On a $50k project, how are you even dropping $10k on AWS? I've found a medium
instance (~$100/mo) handles with ease 100k+ visitors/mo without breaking a
sweat.

~~~
mgkimsal
$50k/$10k is possibly a bit extreme - say $80k/$8k perhaps.

Typically when my clients do a project, they're budgeting out for the year, so
that's $8k/year, not per month.

AWS would be cheaper with reserved instances, but then (IIRC) you're prepaying
in advance for something you may not need. Even if it's cheaper, it's sort of
defeating the purpose of "i can scale on demand". (which, yes you can still
do, but you're back to 'on demand' pricing).

As someone else pointed out, some of where AWS gets you is the storage and
traffic. I've got a setup right now where I need to have at peak 2-3 servers
to handle traffic, as there's a moderate amount of computation going on - it's
not all the time, nor every day, but when I need it, I need it. And it's
processing documents (photos, etc) and we need to store them. Then show them
back to the user. With on demand pricing, and the amount of traffic, this can
be ~$600+ per month (*12 = ~$7000/year).

Could I optimize the code more to use less CPU, and perhaps require 1 less
server? Possibly. Could I optimize some other stuff? Possibly, but that will
cost more time, in order to save money with AWS.

Reserved instances would probably be the best, but if we ended up needing more
later, it's harder to go back 6 months later and say "hey, we need more $ for
X", as the budget was already done and processed for project Y.

I've got some projects like this at digitalocean, and one on month-to-month
dedicated hardware, and the pricing is about 50%, mostly because of the built-
in storage that AWS line items separately.

Can AWS do more functionally? Sure. Do we need it? Usually not, and paying for
that infrastructure when it doesn't match the core needs isn't usually an
option. Elastic IPs would be nice, but we could get much of that utility at
linode if need be, and still pay less.

~~~
ballard
Reserves instances long-term don't make much p&l sense, it makes more sense to
rent dedicated hw where most of the other costs are cheaper.

~~~
mgkimsal
Agreed, which is why I don't do it or recommend it. I had a client look in to
it once, and it was a toss up between renting dedicated servers vs reserved
instances, and they went dedicated.

------
askinner
To be clear, we haven't dumped AWS for short running stateless processing. We
dynamically spin up spot instances as needed for all our stateless processing
using AWS and other cloud services. At our current size our AWS bill will not
be 7 million but closer to 500k-1 million a year. Not exactly pocket change.
Some of the cost savings we are realizing is due to working with AWS as well
on best practices.

For our longer stateful processing or apps that need to be available 24/7 with
no variability in load we have purchased our own hardware (a process that has
been going on for over 18 months). Owning the equipment plus the data center
will run us approx 1.2 million including growth to build a hot back up.

It should be noted, staffing cost was not a factor. We must have staff to
manage 1000s of servers at AWS or at our own data centers. The biggest factor
was paying for compute on boxes that crashed and yielded nothing we could use
to move our business forward. Well, I take that back, we got really good at
check points and rollbacks. Other than that, not much.

No matter how you slice it, AWS and other cloud services are a great service
for the right types of processing, and applications.

Skinner Moz CTO

~~~
jack57
Thanks for the additional information. It was difficult to decipher your
problems from the original article. I'm guessing the "no variability in load"
processes are where AWS was the most unnecessary.

~~~
askinner
Yes! 95% of our services were running at AWS. Paying AWS for running our
website is not a good use of money. Many cheaper ways to host a website.

Paying for 70-100 hours of compute and having the server crash in the middle
of calculating your predictive analytics not exactly all that bright either.
So, we bought our own gear.

AWS works and worked with us pretty damn closely as we pulled apps out of the
cloud.

~~~
bigiain
I'm curious to know if you considered other hosting providers as an
alternative to building out your own hardware/datacenters? I think it's
reasonably well known that AWS is around 3 times more expensive than
alternatives where you pay by-the-month instead of by-the-minute. Was a
Rackspace/Linode/whoever implementation costed against a buy-your-own-boxes
solution, and if so, is there anything you could share about why you chose the
way you did?

(Oh, and thanks for the information you've already shared - even if you can't
answer my curiosity here…)

~~~
askinner
We use other cloud services as well. Rackspace and Nimbix are a couple of
them. We even looked at Azure figuring no contention for boxes but we don't
have any MSFT in our stack (sorry to my whole neighborhood of MSFT employees).
We never depend on just one and have a detailed cost breakdown before we
decided to buy or move a service. The ROI needs to be there.

As an aside, AWS has everyone beat when it comes to regions however. We can be
close to our customers in Europe, US and so on.

To date, no one is spinning up cloud fronts and services in more areas than
AWS. It will take the MSFT , IBMs and the like to move the global cloud along.
MSFT just needs to realize not everyone wants Sharepoint, SQLServer and .net.

Which service you use is really situational. Plenty of good ones out there.
AWS is just one!

~~~
chrissnell
I'm very curious about your ROI analysis for colo vs. a service like
Rackspace's "Managed Colocation" [1] where a hosting provider runs the DC and
provides the hardware and your folks manage the OS layer and up.

I run technical operations for a company with a fraction of your footprint
(but growing quickly). We're at the point where we are growing out of the RAX
public cloud but by my calculations, the decision to run our own private cloud
in colocation vs. lease one from a service provider is (financially) a wash.

From a practicality standpoint, the scales tip towards leasing bare metal from
a provider. I'm curious to hear your experiences with colo. How many folks do
you have working in your colocation facilities doing hardware maintenance?
What about network engineering? I presume you also keep a sizable stock of
spares?

1:
[http://www.rackspace.com/managed_hosting/managed_colocation/](http://www.rackspace.com/managed_hosting/managed_colocation/)

~~~
askinner
All good questions. The answer is pretty long winded. In short, we love our
bare metal and you are not going to be able to rip it out of our cold dead
hands. It is stable, fast and cost efficient.

Now for the long winded:

1\. We do have a decent amount of spares but not a ton. Our contracts require
replacement parts within hours to a day. Some items like F5 gear we have two
and no spare. They just replace their gear in hours.

2\. Each data center has 24/7 support that can do some minor tasks.

3\. Yes, Networking is a pain and you need the right people to do it. It is
not cheap either! Luckily our VP of Tech Ops is a networking guru.You mess up
networking and you are hosed. Our first networking guy wasn't exactly Tops!
So, we know first hand.

Having said all of that, for us there are economies of scale. It is the case
if we want to test different machines, databases or any other combination at
scale it could cost us several hundred thousand dollars just to run the tests.
Yep, we have dropped over 100k for testing at scale. Its simply not
sustainable and an irresponsible way to spend investors cash. Also, when you
add in multiple environments for dev, test, staging and integration you can
quickly see we consume a lot of boxes. So, many in fact, a lot of
colocation/cloud services will not work with us unless we plop down large
amounts of cash. Let's also factor in AWS wants large up front spend for
reserved instances. Thus, if you need the capital to get the amount of compute
you need to run your business it is not very hard just to call Dell, Cisco,
Nimbix, Equinix or any other vendor and negotiate our own deals. If any of
these companies can get half our spend a year they are willing to at least
talk.

------
rgrieselhuber
Not surprised to see this, although I am surprised more companies aren't
following suit. We abandoned AWS a year and a half ago and cut our server /
infrastructure costs by 90% in the process.

~~~
ballard
Yup. Folks like Netflix and Zynga are showing off how much money they are
willing to spend for more potential agility. They could save a ton for
predictable capacity by deploying dedicated boxes + openstack. That's where
the savings are obvious, because AWS is like living at a hotel.

~~~
dagw
You're assuming Netflix and Zynga are paying the prices that are published on
the webpage. I'm fairly confidant that that isn't the case.

------
jhartmann
Public cloud is not for everyone. If you have a large number of machines that
have to run all the time and are being taxed, then public cloud services do
not make sense. If you are doing complex machine learning, large scale web
crawling/scrapping, or generating large quantities of data (images/video/etc.)
then services like AWS might not be a fit. They are incredibly awesome though
for certain types of workload, but for others it is a very expensive option
that isn't economical.

Personally I think the best is some mix of both public and private, so you
have the scalability when you really spike but have cheaper more capable
machines handling baseline and things like analysis or storage workloads.

~~~
rhizome
It's for people not making enough money (or not having the clue) to hire
systems talent.

If you read the Moz report, they appear to have gotten a new CEO and are now
building out a datacenter presence. They spent 7MM+/yr on AWS in 2013, there
is no way that won't be viciously slashed by renting racks.

~~~
vidarh
> It's for people not making enough money (or not having the clue) to hire
> systems talent.

That may be true compared to doing your own racks, but not compared to renting
managed servers at a monthly basis.

Every time I price out AWS vs alternatives, I end up with the same: If you
need a server for more than about 8 hours a day, renting a managed server at a
monthly basis tends to come out substantially cheaper. And that doesn't
increase sysadmin workload; in fact many hosting providers now offers APIs for
provisioning monthly billed services too, just generally with lead times in
hours instead of minutes/seconds, so it takes really bursty traffic before
taking the hassle of auto-scaling with cloud instances becomes worthwhile.

------
programminggeek
What I find most perplexing is that people don't really understand that the
point of Amazon is elasticity to make capacity planning easier for high growth
situations.

That elasticity only makes financial sense when you use it properly. Like, if
you keep your entire infrastructure turned on all the time, and cranked up to
11, but you aren't using reserved instances, then you are missing the point
and you might as well just build/rent your own datacenter space and run your
own servers (or pay someone like Rackspace to do it).

If you are using AWS, then you need to make use of reserved instances and the
elasticity to spin down things that are being underutilized. If your
infrastructure is too fragile to safely handle spinning up and down instances,
then you probably are missing the cloud or aren't taking full advantage of
what it has to offer.

I'm not saying Moz wasn't doing all of these things, I'm just saying that if
you are spending $7 million a year, but aren't doing the appropriate capacity
planning, then you are making a mistake.

At the very least, the exercise of doing real capacity planning by moving into
physical data centers probably had a lot to do with the cost savings as well.

~~~
chrissnell
What you're saying may apply to simple SaaS companies who might build a two-
tier application that can be easily scaled up and down by adding front-end
nodes and r/o database nodes, but it doesn't always apply to data analysis
companies like Moz, who may use large ElasticSearch clusters that don't
autoscale. I run technical operations for such a company. The vast majority of
our infrastructure remains static, simply because the underlying technologies
don't allow for autoscale. The best place for a company like ours is in a
private cloud, be it at Rackspace, SoftLayer, or in colocation.

~~~
programminggeek
Right, and that makes total sense. I haven't ran the numbers, but I know that
if you have some amount of fixed capacity required, you would absolutely want
to do reserved instances on all of those and consolidate as much as you can
where it makes sense.

However, if the numbers don't work out on reserved instances and you have a
pretty steady demand, going with your own infrastructure makes total sense.

------
vosper
My employer spends a small fortune with AWS, and we make use of many of the
services. There's no way our small team could have built our product on any
other platform.

Still, as my colleague observed, it must be the only business in the world you
can spend $1m a year on and get only basic support, with web-only tickets that
take hours or days to get a resolved.

~~~
wahnfrieden
Why aren't you spending even the $50 support tier if not enough for one of the
higher tiers if you're dropping that much in usage costs? I don't understand
why anyone expects free support when every little thing in AWS is charged for
as a utility.

~~~
fbags
The paid support tier is also atrocious and close to useless. AWS routinely
denies the existence of major flaws for months or years, until they eventually
admit it, often by just documenting it instead of fixing it.

They're horrifically expensive, and low quality.

It's really, really sad that so many startups have been convinced that it's
"cheap" when it's almost comically expensive and lousy.

~~~
wahnfrieden
I don't mean to claim otherwise, but I'd be interested to see what instances
led to your observation.

~~~
fbags
As a simple example, if you have a moderate number of hosts in a single
security group (e.g. 200+ hosts), you can expect to have intermittent
communication problems between the nodes. They used to deny this was a
problem.

here's a source, since you're attacking and disbelieving everybody who doesn't
love AWS:
[http://searchcloudcomputing.techtarget.com/news/2240203992/N...](http://searchcloudcomputing.techtarget.com/news/2240203992/Networking-
security-issues-still-irk-large-AWS-shops)

~~~
dsl
It may or may not be the same issue, but from what I have been able to gather
a modification of a security group is basically a "delete, recreate,
repopulate" operation. All of our intermittent network issues could be
reasonably tied back to SG modifications.

~~~
fbags
That's a different issue. This one is instances that are launched into an sg
that is then left unmodified. If there are a moderate to large number of
instances in that sg, intermittent network connectivity problems will ensue.

The issue you mention (where you have to treat sg's as being immutable if you
want them to work reliably) is another problem with the sg's.

------
abrowne
Did anyone else think "Moz" was short for Mozilla?

~~~
dangrossman
Not I, but I'm part of their demographic --
[http://d2v4zi8pl64nxt.cloudfront.net/2013-in-
review/52e9b643...](http://d2v4zi8pl64nxt.cloudfront.net/2013-in-
review/52e9b6437a2607.51764974.jpg)

~~~
bradleyjg
Weird. I recognize only half the companies on that chart, and I'm not sure
what Adobe or salesforce have to with marketing.

~~~
icedchai
so you're telling us you don't know what content creation or CRM has to do
with marketing?

------
ecaron
I bet one of the leading factors is the number of sites blocking the AWS
servers. As a scraping service, the fact that Amazon makes all their public
IPs known
([https://forums.aws.amazon.com/ann.jspa?annID=1701](https://forums.aws.amazon.com/ann.jspa?annID=1701))
is really inconvenient for anyone "crawling" the web. Rackspace also makes
their list available, albeit incomplete
([http://www.rackspace.com/knowledge_center/article/cloud-
site...](http://www.rackspace.com/knowledge_center/article/cloud-sites-
outbound-ip-addresses)).

As more websites setup default blocks for all accesses from AWS services, the
Moz exit was inevitable.

~~~
randfish
We've actually never crawled out of Amazon (mostly because it was expensive to
do so), so the crawl blocking stuff is unrelated, at least for us. As Sarah
noted, it's really been costs, service, and support issues.

~~~
ecaron
I'd redact my suggestion then if I could. Thanks for answering so directly and
promptly! (and great seeing you on HN)

~~~
randfish
Thanks dude! Totally fair question & assumption BTW :-)

------
beachstartup
there are dozens, if not hundreds of smaller TRUE managed hosting providers
popping up to service these kinds of accounts. that's what my startup does. we
routinely steal large $5k-50k/month accounts from AWS and rackspace and
softlayer (IBM) and the like. they just do a shitty job for an exorbitant
price, plain and simple.

folks... _NOBODY_ gives a fuck about you at those companies, and you better
believe it. not the account manager, not the engineer working on your 5 day
old ticket, not the sales guy who probably isn't even paid a decent
commission, and certainly not the executive management who is pulling down
300k+/year annual salaries plus huge stock grants and bonuses (check out the
SEC filings!)

a well managed cluster of computing equipment running baremetal operating
systems, hypervisors of your choice, and maybe something like openstack or
docker is both profitable for the vendor and a MASSIVE price reduction for the
customer.

if you know what you're doing, you're spending $10k/month for $100k/month
worth of equivalent amazon services. yes, you heard me right. it is an order
of magnitude of cost savings. once you scale past $10k/month, using amazon for
all your needs is an INCREDIBLY STUPID business decision.

the genius of AWS, of course, is that they disguise all their marketing as
technical spec sheets and deliberately design their collateral and sales
process to appeal to the "engineer" types with acronyms, numbers, and highly
specific jargon.

it's quite clever. but not clever enough to fool everyone. and those that are
fooled, learn eventually that maybe paying 300% more than you should isn't
that smart.

~~~
quicksilver03
I'm interested in knowing more since we do development and system integration
for client in this kind of budget, and we didn't find a decent hosting
provider in years of trying. Could you name some of those managed providers,
including the one you work for?

~~~
beachstartup
i put my email in my profile. i don't promote my company by name on HN, but
send over your requirements and we can take a look.

thanks.

------
apapli
I'm curious about what is behind Moz's statement "The business impact is
profound. We’re spending less and have improved reliability and efficiency."

I don't have the experience with large AWS setups to comment about the cost
implications, but improved reliability and efficiency strikes me as
questionable - AWS's reliability has improved markedly over the last 18 months
to the point I cannot remember when their last major outage was.

I'm keen to hear other peoples' thoughts on this..

~~~
justinsb
There may not have been a recent total outage of the kind that hits the news,
but I would bet Moz are referring to degraded service (slow CPU/disk/network)
that causes their application to fail (or their error rate to rise).

I'm not very familiar with Moz, but it looks like they have an analytics
product, which is probably particularly sensitive to slowdowns.

~~~
res0nat0r
Sounds like they are more at a size that makes sense to have an on site setup
vs. a cloud based one.

The cloud market is maturing now and a hybrid private/public model I think is
going to become more of the norm, for companies that are large enough that
this makes sense.

It is cheaper to run your own hardware when you are big enough, and then use
AWS for failover / load spikes / cool tech, etc.

~~~
corresation
The scale where doing your own thing makes financial sense can be much lower
than many think.

About two thousand dollars a month gets half a rack with a high-speed, highly-
reliable connection in co-location. The lease payments on several extremely
high performance, flash storage rack servers would be a couple thousand
dollars a month. Throw on some shared nothing redundant virtualization, and
you have the equivalent of an army of 4xmegaxlargemongos.

AWS has a place (such as being the redundant offsite backup for that scenario,
or for extremely variable service requirements), but endlessly on here we see
people disbelieving alternatives make sense.

~~~
AznHisoka
I own 15 OVH servers (32 GB RAM, 2 TB HD, quad-core), and the cost monthly is
less than $1000. What can that buy more in Amazon? Maybe a huge instance? But
I surely can't run thousands of distributed crawlers and store a distributed
search index.

~~~
gtaylor
You can get some decent beef with the new C3 instances, but you have to pay a
good chunk of the year upfront (called a reservation) to get the sane rates.
If you do that, the prices are OK.

But you don't come to AWS just for EC2 instances. You're there to combine
their portfolio of services in various ways. Because of this, it'd probably be
wrong to just compare $1000 on AWS vs what you are paying now.

------
gesman
Cloud premise is an ability to fire expensive sysadmins and let cloud babysit
your needs. Amazon was the first in the world to create the cloud but it's
still a bare-metal cloud today. "Lacking service" ? Of course - they didn't
promise any service whatsoever to begin with.

In other words you pay more to use Amazon cloud + you still have to hire and
pay sysadmins to babysit the whole pile of Amazon cloud mess.

That's what MSFT is perfectly positioned to solve with Azure + their excellent
set of mature tools and services.

MSFT came late to the cloud game (as Ballmer was mostly doing in all other
games for the last 10 yrs) - but they are getting this enterprise play right
step by step.

------
ballard
(I was part of a preferred AWS solution provider shop that demonstrated AWS'
enterpriseness.)

Pair (US), ByteMark (UK) and Linode (global) are top choices for budget stuff.
ByteMark has setup VMware boxes with freenas storage for us in the past. (For
us being sysadmins, budget dedicated is way cheaper than AWS.). Pair and
Bytemark support are top notch.

For enterprisey stuff self-service/remote hands/support... OpSource, SoftLayer
and RackSpace.

Also the killer cost in a datacenter is power followed closely by labor, so
reducing needs by using the simplest / COTS solution can be much cheaper.
Servers tend to sprawl, not decrease... So cull viciously.

------
bane
At some point, I'm going to guess that even moderately large AWS
configurations would be better off getting a high capacity business FiOS line
(or whatever the next tier is), get a couple racks full of home rolled high-
end systems and staff 2 or 3 guys full time to run everything. The systems
will be faster, probably about as reliable, everything will be cheaper even
with extra people on payroll.

I'd love to see some numbers on this though.

~~~
matwood
Where do you just throw a couple racks of computers? Closet? Sounds great, now
let's deal with cooling them and the maintenance around that. Then there is
power. You think your standard office had all the necessary circuits to drive
a couple racks full of servers?

There is so much more to running a server farm than just buying a couple racks
of computers and upgrading the FiOS line.

~~~
__david__
Yes, to all of that. Apparently you weren't around 15 years ago when startups
pretty much _had_ to do that if they wanted servers and an internet presence.
I lived through a startup during that era and that exactly what we did. We had
our fancy ISDN line and could server our web site to the world at a blistering
128K baud.

We bought a $300 air conditioner from Home Depot, cut a whole in the server
room door with a sawzall and installed the air conditioner. Ok technically we
only used the air conditioner in that configuration on the weekends when there
was a human in the room and the main building air was off (we couldn't
override it) but I like that story.

You're not going to get 5 nines this way, but you would be surprised at how
much you can do, even when there exactly zero dedicated sysadmins.

------
antirez
While AWS can be a good solution depending on the needs and the costs/earnings
ratio (for example there are companies netting tens of millions every year
that have relatively limited computational requirements: if they spend
200k/year on EC2 is not the end of the world), I bet that part of its
popularity is due to the following interesting feedback: at the end of the
day, the sysops/devs picking EC2 are not paying for the bill. At the same time
they are not going to be fired if the service is down, as "eveybody is down
right now". This creates an environment where EC2 is not just a popular pick,
but also praised by most, even when actually for the specific company needs
other solutions could be way more cost effective or even provide a way better
environment.

~~~
ballard
AWS is best for PoC, bursting capacity, batch jobs and getting around
corporate policies.

------
aritraghosh007
It makes total sense for a small team (whether a startup or in a large
organization) to be on AWS rather than going the DIY route by renting VPS out
of some datacenter and managing everything from patching to migration to
troubleshooting. What most fail to realize is the fact that a complete AWS
package comes at a much cheaper cost than all of the above tasks combined. It
just requires one to use some smart infrastructure planning in order to keep
the bills low by tuning most of the configurable services which AWS provides.

~~~
ilaksh
AWS doesn't automatically upgrade your OS image or anything like that. If you
want to do that you have to do it yourself. And AWS isn't going to
troubleshoot shit unless you pay a huge amount for support which is hard to
even find the option for that.

However, on Ubuntu server at least there are literally about two commands to
run for upgrading the OS.

------
jl6
Their m1.small is still a pretty good deal if you're in need of root access to
an OS image, and your capacity needs are very small. Anyone aware of anything
comparable?

~~~
mattspitz
DigitalOcean is fabulous. It's very nice hardware and not very expensive. No
AWS services, but if all you need is root access to an OS image, I can't
recommend it enough.

------
booruguru
If Moz is a privately held company, why do they publicly release their
financial data?

~~~
mooreds
This link would probably help answer that:
[http://www.feld.com/wp/archives/2012/05/seomoz-tagfee-and-
me...](http://www.feld.com/wp/archives/2012/05/seomoz-tagfee-and-me.html)

