
Ask HN: Anyone used Amazon EC2 with a database or other high IO operation? - MichaelApproved
I'm thinking of using Amazon's EC2 to run my database and web farm but I'm concerned about the intense IO that's required to run the database. I've shared resources on a vps in the past and it's been a nightmare. I know they offer high IO servers that have limited sharing but it's still sharing.<p>Has anyone had good or bad experience running a high IO operation using EC2?
======
jedberg
We run all of reddit on ec2, which includes a bunch of postgres servers. Each
one is running with a single EBS. However, I've heard horror stories of people
trying to run much less busy databases and having lots of problems, usually
with MySQL.

Those databases are all on XLarge instances, so there is minimal sharing, and
we've also gone to great lengths to make sure all of our normal queries are in
indexes, so the disk gets hit less.

We also have a read slave for every database to alleviate read loads.

One thing you might want to do is run 'iostat -xtc' on your current box and
put that in a log file. Then go back and analyze it and see what your average
and peak reads and writes are. Amazon's max for a single EBS appears to be
about 1000 ops / second (at least, that is what we were doing when they told
us we maxed out the performance of the disk).

Good luck!

Edit: I forgot to mention that on all the database disks, we use ext2 and
noatime. Both decrease the total number of writes necessary, and have very
little downside (the biggest being that you have to fsck on a crash).

~~~
MichaelApproved
I was just talking to someone wondering how you guys at reddit were able to do
it. The article about taking your last server offline really pushed me to move
forward on this.

I should mention that I'm running Windows with MSSQL. I imagine the
performance should be the same but there must be some differences of course.

I like the option of rackspace allowing for a dedicated server next to the
cloud but I should probably give EC2 a few test runs of my own too. I think if
EC2 is going to work it wont be an overnight solution. I'll have to move over
slowly, learning to tune the box as I use it more.

~~~
jedberg
If you're using Windows, I would stay away from EC2 right now. They aren't
quite mature enough for Windows yet. If you look at the EC2 forums, the
majority of complaints are about Windows bugs.

Actually, you might want to try and get in on Microsoft's Azure beta.

~~~
MichaelApproved
Azure is tempting, especially since MS is trying hard to get into this market,
but I worry about putting all my eggs in the MS basket this early is Azure's
life.

~~~
JasonM80
Are you able to build something quickly to do some test runs against Windows
Azure to see if it has the performance you need? You may also want to check
out SQL Azure (part of the Windows Azure Platform), also in beta, to see if it
can do what you need for DB performance.

As far as Windows Azure being early in its life, you are correct. However, it
has been in beta for about a year and it is running Windows Server 2008, which
has been in the market for almost two years. You may also want to look at the
Windows Azure forums at <http://bit.ly/MSDNWinAzureForum> to get an idea of
other people's experiences with it.

You can also get more info about Azure and its beta availability at
<http://bit.ly/WinAzurePlatformDevCenter>

(Jason - working for M80, representing Microsoft)

~~~
MichaelApproved
I'm going to PDC this week and hopefully there will be people there with some
good real world experience with Azure that I poll.

~~~
JasonM80
Have a great time at PDC. I'm sure finding Windows Azure experts there will be
like finding coffee in Seattle :-), and you'll definitely find plenty of
experienced Azure users at the Azure presentations.

------
JoeH
Here at RescueTime we use EC2 for all of our servers, with the exception of
one server outside EC2 for monitoring.

All of our MySQL instances are currently each running on 4x EBS volumes using
LVM stripped across all 4 volumes.

Our primary transactional table is nearing a billion rows, and just this
weekend we performed a migration that segmented this table into 40 partitions,
containing data back to 2007.

We are currently handling roughly 3,000,000 inserts and 2,500,000 updates on
this one table daily. IOWait on our DB holds steady at about 5% throughout our
peak (5am - 4pm PST).

It is certainly not Fibre Channel backboned speeds, but the other benefits of
using EBS CURRENTLY outweighs any IO performance hits we might have. The
ability to backup production and refresh any of our other DB environments in
less than 15 minutes, using EBS snapshots is worth it. Especially for the
price we pay monthly for this infrastructure.

If you need any more details on our setup or tips in troubleshooting/setting
up your own, feel free to ping me.

------
qhoxie
We have a highly trafficked and large dataset that we aimed to move from our
datacenter to EC2. This is all in MySQL. Upon doing so, we had to spend
significant effort trying to get our system stable and we ended up narrowing
it down to EBS simply not being able to handle the IOPS we needed.

We explored all the options available and worked with both Percona and Amazon
on the issues, but it was clear that EBS and EC2 just are not meant to handle
such load. Our database is currently back running in our datacenter.

There have been a number of reports with similar results using EBS. It all
depends on your particular IO profile, but since you said it was high, I
encourage you to tread lightly.

If you want more information you are welcome to email me.

~~~
jedberg
I'm beginning to thing that MySQL does something very different with disks
than Postgres, because everyone I've talked to that said they had a problem
had been running MySQL, and everyone who has been successful has been running
Postgres (myself included).

~~~
qhoxie
This really would not be all that surprising, but it also would not surprise
me if the average Postgres setup was better tuned/architected than the
majority case of MySQL uses.

Our setup was surely not ideal, but even with significant tuning, it was not
sufficient.

It is really good to hear successes like Reddit's, though. How long have you
folks had your databases in EC2?

~~~
jedberg
We had some of databases at EC2 since November of 2008, and moved the rest of
them over in May 2009.

------
jbellis
I work on the Cassandra distributed database for Rackspace. A _lot_ of people
start out trying Cassandra on EC2 but the universal conclusion has been that
I/O performance is miserable.

Anecdotally, Rackspace Cloud Servers I/O works much better, although the only
serious head-to-head comparison I have seen is this one:
<http://pl.atyp.us/wordpress/?p=2240> (tl;dr: < 20MBps on writes with local
storage or EBS on EC2 vs 100MBps+ on RCS)

~~~
JoelPM
At OpenX we're using Cassandra on EC2 and it has served our needs well so far.
Our use case is a little different in that we set hard timeouts on the client
side reads and can tolerate a small percentage of requests that don't complete
in time.

According to Amazon the I/O depends on the instance type
(<http://aws.amazon.com/ec2/instance-types/>) so the OP may want to take that
into consideration when picking an instance type.

~~~
JoelPM
Unfortunately, last night I completely failed to remember that Rackspace and
EC2 are direct competitors in this space so what was intended to be a simple
data-point may be construed as more than that - my apologies.

What I intended to say was "you can make Cassandra work on EC2." However, if
the I/O throughput of EC2 is a fraction that of some other provider, you'll
need more machines in EC2 to achieve the same throughput.

My personal site is hosted on the Rackspace Cloud and I've been very pleased
with it, but my workload is different so I can't give a comparison.

I also know that we've benefited directly from jbellis' support on IRC and the
mailing list, so this should be considered an endorsement of Cassandra, if
anything.

~~~
jbellis
Of course no offense taken. Glad EC2 is working out for your workload. :)

------
mark_l_watson
I can't answer your question from personal experience because I always just
use plain old EBS volumes with my EC2s.

That said, here is a good writeup on RAID use with EBS by someone at Heroku:
[http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs...](http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs/)

~~~
MichaelApproved
That's got some really good information. Most important to me is that it does
confirm my fears of flaky IO performance. He does point out some ways to tune
it but it still seems like a rough experience.

These cloud services seem great for low IO apps but once you have to hit the
database hard it seems like anyone's guess if your db will perform.

~~~
bretthoerner
That's why I'm a big fan of Rackspace. You can get bare metal servers for
heavy I/O and simultaneously have "Cloud" servers on the same local network
(example: Github).

~~~
z8000
Anyone have any experience with NewServers? They offer dedicated server
provisioning in about 5 minutes. On paper they look great but I have not found
much commentary about their product anywhere.

~~~
vbuser
I've been hosting with them for five years (last two @ newservers). Their
support is great and they even helped installing my scripts. I ordered 7
differents servers since the beginning to create farms and their load
balancing mechanism is also very cool. Cheap and quality service. I'm a non-US
customer, my site is a top 100 site in my country.

------
dryicerx
Seconding the EBS Raid0 option. Some comparison here
[http://blog.mudy.info/2009/04/disk-io-ec2-vs-mosso-vs-
linode...](http://blog.mudy.info/2009/04/disk-io-ec2-vs-mosso-vs-linode/)

Also, if you have a part of a DB that just requires unholy amounts IO, there
is of course the new 68gig instances, move that part and run it purely on
memory with a combination of frequent dumps to disk.

~~~
MichaelApproved
Maybe a combination of DB file in RAM and transaction logs to disk?

------
spudlyo
It's worth noting that you can speed up write operations on EBS volumes by
first initializing them by writing data to every block on the volume.
Subsequent writes will be faster due to the way Amazon virtualizes disks.

[http://docs.amazonwebservices.com/AWSEC2/2007-08-29/Develope...](http://docs.amazonwebservices.com/AWSEC2/2007-08-29/DeveloperGuide/instance-
storage.html)

------
nearestneighbor
I'm curious about the economics of this. If you reserve an extra large high-
CPU instance for 3 years, you end up paying $3,000 per year ($2,800 initial
fee + hourly rate). Isn't it cheaper to have your own servers?

~~~
jedberg
To buy physical hardware equivalent to that instance, yes, it would be
cheaper. However, you need to also factor in the costs of keeping that machine
in a rack for 3 years with power and cooling. When I worked out the costs, it
ended up being cheaper to use EC2 by about 30%. My datacenter was a bit
expensive, but that is because I was looking at datacenters in San Francisco,
since I don't want to have to drive to the middle of nowhere when I have to do
maintenance on the physical machine.

Also, I was comparing against 1 year reserved instances, which is what I use
now.

~~~
nearestneighbor
If it's not a commercial secret, I wonder, are sites like Reddit and Digg low
margin businesses, or are hardware/bandwidth costs a tiny speck compared to
the ad revenue generated?

In your "IAMA" thread on reddit, someone estimated that Reddit must be pulling
on the order of $1M per month, making the costs neglibile. Digg must be
earning even more, because of its greater number of users and their higher
susceptibility to ads.

~~~
jedberg
I'm not at liberty to discuss reddit's revenues, but what I can tell you is
that the entire operational cost of the site (about $15K last month) is small
compared to the human cost (ie. salary and benefits)

------
garyrichardson
I am using MySQL on EC2 c1.medium instances. I haven't noticed an IO limit
yet, but that could be how I'm using it plus the data set is still under 2GB.

I use the database as a raw data source and have a cron that generates read
only data sets for the web servers to use. This works for my application, but
probably not most.

------
charlesju
With any high IO web application, it all comes down to horizontal sharding to
make it work. But, we recently switched from a high powered DB to Amazon and
we're doing fine.

------
hendler
Good luck so far using EBS for MySQL, logs etc. I/O costs not significant yet.

You can Raid EBS devices btw.

~~~
MichaelApproved
Doesn't Amazon already RAID the storage they offer you? Is drive failure
something I need to worry about? I thought they had a beefy RAID system
already in place.

~~~
dryicerx
They do, but I think he's talking about doing a raid0 just to leech more io,
not safety.

~~~
MichaelApproved
I see now with all the comments that everyone is referring to getting more
platters spinning the data.

------
bretthoerner
You wouldn't want to use the virtual hard drive they provide you, but rather
EBS (Elastic Block Storage). With that you can setup a mirrored RAID
(redundancy is a given already) of as many devices as you please.

------
scumola
Just write to /mnt (150GB ramdrive on the small instance) then occasionally
sync to an EBS volume. That'll probably give you the best I/O.

~~~
jotto
/mnt is not a ram drive, it's ephemeral hard disk, and EBS has better IO than
the /mnt. You only get 1.7GB RAM on the small instance.

