
What Powers Instagram: Hundreds of Instances, Dozens of Technologies - zio99
http://instagram-engineering.tumblr.com/post/13649370142/what-powers-instagram-hundreds-of-instances-dozens-of
======
japhyr
I love that companies like Instagram are happy to share this kind of
information. I don't have the problem of scaling yet, but it's reassuring to
know that when I do, there are good models like this one to follow.

~~~
brlewis
The Instagram founders are exceptionally cool. After the $1B Facebook
acquisition Mike Krieger kept right on answering questions on the instagram-
api mailing list without missing a beat, as if nothing had happened.

~~~
citricsquid
I might be wrong, but as far as I know they haven't sold _yet_ (it's still got
to clear regulators?) so although he technically _could_ have walked away I
don't think it would been sensible as he isn't going to be able to cash out
yet?

~~~
zio99
Some sources (the NY Post for example) say the deal may take up to 6 months
with a 50% chance of going through. There's some concerns as to whether
Instagram may prevent sharing with social networks other than facebook (flickr
for e.g.). I'm already booted off instagram without upgrading to the new app,
but fear that I'll be forced to share all my instagram pictures on my facebook
timeline.

~~~
brlewis
To prevent sharing out except to Facebook, Instagram would have to disable its
API and stop saving photos to Camera Roll. I don't expect either of those to
happen.

Even if the deal might fall through, I could easily imagine Mike taking a day
or two to celebrate instead of jumping back in and answering questions,
particularly on the instagram-api list, since nurturing a slowly-growing
ecosystem is not Instagram's best path to a successful exit. Logically, I
think he did it because he's a nice guy.

------
alpb
This is a fairly old blog post and it is posted to HN by Kevin Systrom – he's
co-founder of Instagram. Here's his post:

<http://news.ycombinator.com/item?id=3306027> (223 points, 220 days ago) so
I'd like to say this is a repost. Though, I have no idea how the same link can
be reposted to HN.

~~~
achompas
I don't understand how this topic comes up so frequently, but HN is configured
to allow reposts after a certain amount of time. That isn't really a problem
for old posts with excellent information, so I think a repost is warranted for
this.

------
yumraj
I don't see Amazon SimpleDB in this list, rather Postgres and Redis.

Can someone who knows more about various DBs opine on this. Is it better to
run your own DB as Instagram seems to be doing or is relying on SimpleDB good
enough if you don't need such high performance.

Also, as happens with many startups, how easy/difficult is data migration when
startups try to scale and need to scale fast.

~~~
seldo
AWS has more than one DB option.

<http://aws.amazon.com/simpledb/>

SimpleDB is a non-relational store that automatically indexes everything, but
can only store up to 10GB. It's very flexible but limited, good for
prototyping.

<http://aws.amazon.com/dynamodb/>

DynamoDB is like SimpleDB's grown-up version. At greater cost and with more
up-front configuration, it scales automatically to huge workloads. It's still
a non-relational store, with the drawbacks that implies.

<http://aws.amazon.com/rds/>

Relational Database Service is literally managed MySQL instances. Amazon spins
them up for you, manages configuration, backups and restoration. One of RDS's
primary value-adds is that it can automatically partition your data across
multiple EBS volumes (like hard drives). This helps get around the relatively
low I/O performance of EBS volumes.

So pick your poison -- if you don't need high performance, simpleDB or a small
RDS instance will work; it depends whether you want relational data or not. I
can't speak for the difficulty of migrating; we stuck with running our own
MySQL instances from the start.

For a lot of reasons (that I can enumerate if you'd like), we think running
your own DBs is the best option.

~~~
cluda01
I would very much appreciate it if you elaborate on why managing your own
MySQL database is the best option. I'm currently moving from back end
proprietary systems development to web development and would like to hear your
considerations.

~~~
seldo
The short answer is that we believe running your own instances gives better
performance and reliability than RDS. However, the cost is complexity: I'm a
relatively experienced DBA, and we've since hired a second person with deep
MySQL experience. If MySQL admin is something you don't want to spend much
time doing, you may be willing to make the performance sacrifice of RDS.

The longer answer is that we don't use RDS because it relies on EBS, and we do
not trust EBS for any critical applications. Instead, we put our data on
instance storage (aka "ephemeral" storage).

This has two big disadvantages:

a) portability: you can't detach the drive and move it to a new instance like
you can with EBS -- to clone or backup, you have to copy over the network,
which is much slower (and obviously, if you kill the instance, you lose the
data).

b) storage: you are limited in how big your DB can be. An AWS large instance
these days gives you nearly 1TB of instance storage, but if you have a single
DB larger than that, you need to use EBS if you're on Amazon. (Of course, if
you care about performance and your database is > 1TB, you should probably be
looking at sharding across multiple machines anyway)

However, using instance storage has two big advantages that we think outweigh
those:

a) performance. EBS is basically a network drive. Total I/O operations per
second (iops) is punishingly low. If you have a high transaction rate on your
database you're going to really hate it. As I mentioned, RDS tries to mitigate
this by using multiple EBS drives, but we consider that a band-aid on a pretty
fundamental problem with EBS. Instance storage on the other hand is physically
local to the VM's host machine, and is therefore much faster.

b) reliability. After 3 years on AWS, our trust in EBS is zero. It fails too
often, and its failure pattern is awful: you tend to lose big batches of EBS
drives at the same time, and whenever there been a major EBS failure, the API
used to launch replacement volumes has failed at the same time, making
replacement impossible. Again, we think this is a fundamental problem with the
nature of EBS and unlikely to change.

~~~
ralph
Thanks seldo, interesting. Just been looking at a client's Amazon dashboard
where they have a small set-up running, not something I normally deal with but
I see their RDS is billing over 2e9 I/Os/month and ends up being a significant
part of the non-fixed bit of their bill. I suspect their MySQL queries are
doing table scans and building temporary tables for some of the queries; these
would both up the I/O count as all RDS storage is EBS, even temporaries?

So if your MySQL storage is ephemeral how do you cope with outage? Replicate
it off AWS?

~~~
seldo
I believe MySQL's working directory is on EBS, so yes, even temporary tables
would be on EBS -- don't quote me on that, though.

Re: outages, we use multiple replicated servers in different availability
zones -- an outage is usually (though not always!) limited to a single zone.
For a region-wide outage, we have emergency backups being sent to a different
AWS region (east -> west), and if shit completely hits the fan we have off-AWS
backups.

------
mistercow
I'm pretty sure that "dozens of technologies" is a meaningless thing to say.
Instagram is powered by _thousands_ of technologies; it's just that a few
dozen of those happen to have trendy buzzwords attached.

~~~
shoo
Instagram! Powered by the polio vaccine and domestication of wheat.

~~~
skeletonjelly
Instagram! Powered by a lack of earth destroying quantum vacuum metastability!

~~~
mistercow
I'm pretty sure that one's not a technology, but just a happy and not very
unlikely set of circumstances.

------
vegas
To sum up: it's gonna fall over at any minute without a continuous injection
of money.

~~~
josephcooney
My take-away was "this is why when EC2 when down it wasn't just a case of
rebooting the server". My other thought was "and how did all of this help you
when EC2 went down?"

~~~
jbigelow76
That was my thought too, the multiple mentions of "stuff" in different zones
had me wondering why the recent EC2 outage was able to get them.

------
peterwwillis
I don't feel very good about the idea of DR just being duplicate servers in a
different zone. You don't know what kind of problem could ripple out and
affect more zones, or all of them. A completely different host/cloud/colo
provider seems like a safer bet.

~~~
pandemicsyn
When you come down to it earth is a single point of failure, its about how big
of a disaster you expect to occur and how big of one you expect to recover
from.

~~~
sbov
Yes but it's a lot easier and cheaper to protect against e.g. a major bug
surfacing in amazon's cloud management api vs protecting against an asteroid
hitting earth.

------
gagabity
Could someone plug these into the AWS monthly calculator and try to figure out
the monthly cost, when I try I get around $10,000 per Months, but I think I am
wildly underestimating the bandwidth usage. AWS Calculator
<http://calculator.s3.amazonaws.com/calc5.html>

------
insaneirish
The idea of mdadm on top of EBS volumes makes me cringe.

Mark Mayo's thoughts on abstracted block storage are spot on:
[http://joyeur.com/2011/04/24/magical-block-store-when-
abstra...](http://joyeur.com/2011/04/24/magical-block-store-when-abstractions-
fail-us/)

------
gsingers
A couple of comments as one of the authors of Solr's spatial code:

\- I don't think any of us have compared it to PostgreSQL, but I can tell you
we have clients doing 500 queries per second+ with it including text search.
I've yet to see a DB do good text-based fuzzy matching and combining two
systems (DB + search) via a join is usually slow. YMMV.

\- The main goal of the implementation is to add point-based search to text
search. It is not a general purpose replacement for an r-tree, etc.

\- You are not required to use haversine. The distance function is pluggable
and we have other options implemented. Also, in many cases, you have other
clauses in your query that restrict down the set of documents that need to be
scored by distance.

------
resrc
"The photos themselves go straight to Amazon S3"... Is there no image resizing
involved? It's pretty hard to find stuff about scaling image resizing, It's
hugely CPU intensive, that's a sure thing.

~~~
lallysingh
I'd imagine it's easy enough to do before transmission.

~~~
riobard
Actually they have to do the compression on the mobile clients before
uploading, otherwise it will take forever and cost a real fortunate on the
mobile data usage.

------
Naushad
Well explained, and an awesome way to say "We are hiring a DevOps"

------
rurounijones
Do they have any application monitoring stuff a la NewRelic? I saw stuff like
munin but that isn't really the same.

------
timaelliott
> For our geo-search API, we used PostgreSQL for many months, but once our
> Media entries were sharded, moved over to using Apache Solr. It has a simple
> JSON interface, so as far as our application is concerned, it’s just another
> API to consume.

Does anyone have particular insight to share on this? Last I checked, Solr's
geospatial searching methods are rather inefficient -- haversine across all
documents, bounding boxes that rely on haversine and Solr4 was looking into
geohashes (better but have some serious edge-case problems where they fall
apart).

Meanwhile PostgreSQL offers r-tree indexing for spatial queries and is blazing
fast.

Am I missing some hidden power about Solr's geospatial lookups that make it
faster/better than an r-tree implementation?

~~~
awj
It probably was the database sharding. If the Solr setup could handle the geo-
search-related data without the need for sharding it probably can beat out
Postgres with sharding.

Having this exposed through an api that is standardized and maintained by
someone else is also nothing to sneeze at. I'd trade a bit of performance for
that kind of standardization and turnkey use in the right scenario.

~~~
rbranson
The reason we use Solr for this specific task is because PostgreSQL cannot
efficiently and quickly merge two index queries (time & space). It can do this
to a limited degree, but both of these dimensions potentially match 10s of
millions of documents, and PG falls over at this.

~~~
timaelliott
So you make the r-tree 3 dimensions (lat,lng,time). PostgreSQL supports this.

I dunno I can't envision Solr being more efficient than a properly designed
RDBMS for these situations. If you were integrating a full-text search I'd
absolutely believe that to be the case but...

~~~
rbranson
We need independent time & geo searches as well. The indexes are vastly
smaller in Solr. We use PostgreSQL extensively and prefer it, so it's not a
matter of simply wanting to use something different.

~~~
fdr
That's very interesting. Could you share your story with the mailing list
pgsql-hackers a little bit? The guys who work on indexing are quite active on
those lists.

Also, there's some new thing I don't understand super well, sp-gist, do you
have any thoughts on that?

------
dclowd9901
Color me surprised: No NoSQL? Instagram seems to be a prime candidate for such
a storage method.

~~~
gregburd
You don't consider Amazon/S3 or Redis NoSQL databases?

~~~
jammmuel
Since when is S3 a 'database'?

~~~
philjohn
It's got an API, you can store data in it and retreive it later by key, since
when was S3 not a database? It's a key/value store that lets you store very
large values.

