

Why we've doubled down on AWS and the cloud - lfittl
http://blog.codeship.io/2013/09/24/why-we-double-down-on-aws-and-the-cloud.html

======
necro
I wanted to throw my experience into the ring because there seems to be such a
fear of colocation. We knew nothing about colocation and decided to build some
supermicro servers ourselves and install them and a switch in a colo 4 years
ago. I read the all stories "i had to get up in the middle of the night to
drive to the colo. it was the worst move ever to colo", and they are total
bull. Even the biggest noob can setup things so it's totally remote. Servers
have a dedicated ipmi port ( remote console over ethernet ) that will make it
as if you're sitting at the server remotely. You can even mount a cd/image on
your laptop, remotely so the hardware thinks that cd is in that machine. Hell,
I can reinstall the bios on the server remotely, OS, everything. Why on EARTH
would you have to drive to your colo? You can get servers that have 4 ether
ports that you can bond in pairs to different switches. You can have hardware
raid so loosing 2 drives in a server is no big deal, and you can take care of
it at a later time. We have drives fail sometimes, but things keep on ticking.
With the costs you save you can have triple redundancy if you like, and the
benefit of consistent latency and better performance always. We have 250TB of
storage and its double redundancy and also a remote backup. It cost ONCE what
we would have to pay for a few months on the cheapest storage service.

We run straight kvm virtualization on our own hosts for flexibility. We run
dbs on bare metal. I hear all these stories of people vms "crashing" all the
time but i can tell you we have only had 1 instance of a vm, or in this case
host dying in 4 years. Happened to be one of the video conversion hosts that
is pinned 24/7 and it turned out it just hit some un recoverable memory
hardware error. No big deal there were others.

Flexibility? We can clone and spin up VMs at will. We can live migrate and
upgrade hosts. We can automate things with virtlib to our hearts desire.

Costs? $1500/month for direct equinix colo ( includes power, full rack, and
gigabit connection from tier1 provider ) Never had a power issue, never had a
network issue. We also use a CDN for static stuff and thats extra. We started
with 3 servers, now are at a dozen, and adding a new one does not add a new
monthly expense.

You can have a E3-1240 V2 @ 3.40GHz server built for $1500 and that as a host
can run most of our front end stack. Sure we have 6 of those for backend crap,
redundancy, but we actually run most of our stack on 1 of them. Mostly we do
that for shits and giggles, but also because the interaction between the www,
redis, mcd, zeromq is a few ms faster when it does not go over physical net.
So if you over optimize like us, and want 30ms page gen times, you can nerd
out like that.

s6 CPU: 8 MEM: 32080MB total running CPU: 16 MEM: 16384MB r-fp1 running CPU: 2
MEM: 1024MB r-mcd2 running CPU: 2 MEM: 1024MB r-www2 running CPU: 2 MEM:
4096MB r-www3 running CPU: 2 MEM: 4096MB r-red1 running CPU: 2 MEM: 2048MB
r-red2 running CPU: 2 MEM: 2048MB r-zmq running CPU: 2 MEM: 1024MB r-zmq2
running CPU: 2 MEM: 1024MB

front end proxy, www front ends, redis, zeromq, memcached, etc. Excluding
mysql db which is on bare metal. This serves our site that handles about 200
page views per second peak day, and that is at 25% host utilization. Our pages
generate ( no caching ), including redis, zmq, and maybe 25 db mysql calls per
page in about 30ms. You can optimize things too like...you know that the
default config on a server will kick down the cpu to 1.6Ghz if its not really
loaded, and that means page gen times in our case would be 15ms slower. Hell,
we dont have to try to save power, so we can kick that sucker to 3.4Ghz all
the time and make sure users get the benefit of that. Nice to be in control of
the host.

We never needed remote hands or anything like that, but that is available a
phone call away. I visit the colo in San Jose once a year and I schedule it
with my motocycle trip down there. Sometimes I just dust the servers off, pet
them a little and look at the pretty lights.

Of course ec2 has it's use. If your html traffic spikes higher than 1 gbps,
then it's nice to have the flexibility of a fatter distributed pipe. If you
want to optimize for rtt then it's nice to be able to spin up in a different
geographical areal.

I think what bugs me the most is that a lot of companies use the argument of,
if you get high traffic, like slashdotted or hackernews you can spin up a 100
front ends easily and handle it. We've been on the top of hackernews and the
change in traffic was in the noise floor as compared to 200 r/s we normally
handle. The point I'm trying to make is that if you engineer your app better,
and understand and fix issues with generating your pages faster, you wont need
the fancy scale to 100 front ends bullshit. ( tip. it's probably your database
queries anyways so optimize that. it's not the print/echo statement that is
outputting html on the front end ) Of course some do require webscale and it's
a good way to go with ec2 and all the extra costs and engineering, but it
seems that every joe blow and his blog or app seems to think they need so spin
up to 100 front ends.

Sorry for the rant. I actually think that ec2 and the likes are the future and
as tech gets better and prices get better I can see it making sense for more
and more. I just wanted to give a contrast with our current setup.

~~~
druiid
$1500/month for Equinix + Bandwidth? We got quotes from them before and rack
was low, but I wasn't finding super cheap bandwidth like that. Did you go with
Cogent or similar?

~~~
necro
We actually went via Bandcon, which was then bought by highwinds. BW is around
the going price $2.5/Gbps and it seemed to be level3 at the beginning and now
it's seems more of a mix. ( I should specify that we have a gbps port but we
only use about 100 mbps as it's only the html we serve from there ) We use
another 2 Gbps of traffic via CDN for all the static/video content but that is
of course a different cost ) But it's nice when the CDN ingest point is in the
same physical DC as we are.

I just looked what 250TB would cost us on s3, $20k/month, or $240k/year. ( im
not even counting the put/get usage )

You can build it, for ease of math, 100x 3TB seagate constellation. 100x $250
= $25k, another $5k easily covers a 45 jbod and raid card and server with ssd
zil and arc for zfs and you're done. so $30k. Get 2 more for redundancy and
backup as you see fit.

So over 3 years, 720k vs apples to apples 90k ( if you got 3 of those servers)
so you save say $600k. You can get a decent remote dev for $200k/year for that
time.

------
kogir
Hosting your own servers doesn't require foregoing virtualization and
containers. All the tools listed in the article that aid deployment can be
used on your own hardware, and should be! Also, if you suddenly need 10 or 100
new machines, EC2 is still there - you've not sworn it off forever.

Using the cloud, your own servers, or both should be a deliberate decision
based on cost and your real business needs. Not thinking about this seriously
is doing yourself a disservice.

~~~
druiid
This indeed. A smart company will do a cost-benefit analysis, not blindly go
to the cloud, or 'double-down'. There are great benefits to doing things
(maybe even your entire environment) on 'The Cloud', but sometimes physical
hardware (rented or purchased) will be a better investment in the long-run.
Also, Hybrid Cloud is big at the moment (physical servers for day-to-day
traffic and either dev tasks on the cloud, or expansion/failover).

~~~
fragsworth
I think we've reached a point where most small companies _can and should_
blindly use the cloud for everything that can be done remotely. There are only
two situations where the cloud shouldn't be used:

1\. If you're using tons of servers, you might benefit from hosting your own
hardware on a large scale. But even then you can probably negotiate with the
cloud service for a better price (like Netflix did). If you're a big customer
and you think you should move off the cloud, the provider _can_ beat your
expected costs and still make a profit. Your team doesn't have thousands of
years of combined hardware experience, and you're not buying hardware at a
better price than Amazon got.

2) You don't want The NSA snooping on your company's data. This is a moot
point because they will obtain the data anyway, with a gag order, if they
really need to.

~~~
oijaf888
Why should a small company use the cloud for everything instead of a dedicated
server or VPS?

Most small companies aren't going to benefit from the elasticity that the
cloud provides. The administrative overhead will be similar on each except in
the case of the cloud you need to worry about the persistence of data either
via EBS, replication or having an acceptable loss level.

~~~
fragsworth
I would think remote dedicated servers and VPS services fall under the
definition of "cloud"?

The question is whether or not you're paying up-front for hardware, and
managing it locally somewhere. If you lease a $10/month VPS, you're using
cloud services.

------
timr
_" AWS is expensive when you compare only the raw server costs and do not use
reserved instances. But that premium lets you focus on building your product
far more than going with a self managed machine."_

Sometimes it's faster and more cost-effective to use AWS. Other times -- such
as when you spend a lot of time and/or money making up for the performance
limits of the platform -- you lose. It all depends on what you need, and there
are clear engineering trade-offs at play.

The problem comes when you don't make _rational_ choices. If you're writing
blog posts about how you're "all in" on one platform or another, you're either
linkbaiting (this seems most likely to me), or you're not being rational about
your choices.

Let's be honest: it isn't that hard to manage a rack of servers. Any competent
engineer should be able to do the basic sysadmin tasks necessary, and aside
from the setup costs, the amortized maintenance costs should be on the order
of a few hours a month. Certainly, the wildly overblown claims of _entire
teams_ spending _months_ to set up a few colocated servers...well, those teams
are either completely incompetent, or there's some exaggeration going on.

Don't be afraid of colocating. Just be rational about what you need.

~~~
btgeekboy
We don't use AWS/etc, we have 3 colo spaces in various locations around the
country. We've never visited two of them; a small fee gets your equipment
racked and connected, and remote hands does the rest as needed. The local one
we visit maybe 3 times a year, usually to install new equipment.

The overhead of dealing with hardware, especially for applications that don't
have massive exponential growth, is really not as big of a deal as it sounds.

~~~
spongle
Indeed.

We have 6x 42U racks across three datacentres in the UK packed with network
hardware, servers, san and other kit.

Stuff doesn't really go wrong that often. Our ops guys probably go there once
a quarter. Most of the time, you can manage it remotely. The only main failure
point appears to be disks but you can just mail them to the DC and get them to
stick it in for a small fee.

------
clemc
Great post! Just discovered another nice piece about CI on their blog >
[http://blog.codeship.io/2013/09/12/the-codeship-workflow-
par...](http://blog.codeship.io/2013/09/12/the-codeship-workflow-
part-5-continuous-deployment-of-immutable-build-servers.html)

Seems damn legit!

~~~
matlock
Thanks, going immutable with all of our infrastructure made our system way
more stable and productive

------
Lowgain
That Cloud -> Butt Chrome extension is doing a great job

------
druiid
This ending of the article kind of had me going???:

> AWS is expensive when you compare only the raw server costs and do not use
> reserved instances. But that premium lets you focus on building your product
> far more than going with a self managed machine. Especially for early stage
> startups or projects this makes all the difference in the world.

Well, which one is it? The article spoke from the beginning about that they
use the idea of tearing down VM's on a daily basis. This is kind of a not
having your cake or eating it scenario. If you want to have reserved instances
you, well, have to reserve them. Once you've done that (unless I've missed
something looking into reserved instances on EC2 before) you can't really
remove and re-create a new instance just like that without paying the up-front
cost for the instance again.

Really that then takes away a lot of the power, even though you're saving
money in the long run. At that point, why not just go with a managed dedicated
server provider? You get a whole lot of hardware these days for not a whole
lot of money (People have talked endlessly about the different solution
providers out there, so I won't repeat any).

~~~
matlock
Reserved instances are not bound to any specific VM. When you start a virtual
machine and there are reserved instance slots free it will charge you the
reserved instance amount. If you provision more machines it will charge the
standard amount.

~~~
druiid
Well what do you mean by it will charge the reserved instance amount? Do you
mean by that just the hourly reserved costs, or the up-front payment that
varies based on the instance period or instance type? If the former, then I
stand corrected!

~~~
matlock
It will only charge the hourly reserved costs.

From: [http://aws.amazon.com/ec2/reserved-
instances/](http://aws.amazon.com/ec2/reserved-instances/)

Easy to Use: Reserved Instances are easy to use and require no change to how
you use EC2. When computing your bill, our system will automatically apply
Reserved Instance rates first to minimize your costs. An instance hour will
only be charged at the On-Demand rate when your total quantity of instances
running that hour exceeds the number of applicable Reserved Instances you own.

~~~
druiid
Then I stand corrected!

