
Amazon bares its computers - DLay
http://bits.blogs.nytimes.com/2013/11/15/amazon-bares-its-computers/?_r=0
======
jsnk
>“All of our engineers are focused” on reducing the costs of computing

If this is so, how is it that a startup like Digital Ocean that's only 2 years
old find a way to provide $5/month plan that's running on SSD while m1.small
on EC2 which is slower than DO costing me around $40/month? At this rate, I
should probably buy a server myself and run it at home.

I like AWS and its ecosystem, but I am tired of Amazon continuing to provide
shitty service for a ridiculous cost.

~~~
chaz
I use DO and AWS for different projects because they meet different needs. Of
course there are going to be places where one is better suited than the other
in both performance and price.

For example, one of my projects requires 1TB of drive space. At DO, I have no
choice but to go to the $960/mo 1 TB plan. But I don't need the SSD speed, 96
GB of RAM, 24 cores, or the 10 TB of monthly transfer. But with EC2, I can
just use EBS @ $100/mo/TB and pair it with the right-size EC2 instance for the
jobs at hand. As an aside, the real question for me is to decide whether I
want to invest in a proper dedicated server where I can get good performance
and online storage at a lower cost.

DO has done a fantastic job at finding a spot in the VPS market, which is why
I'm one of their customers. It doesn't mean Amazon is doing it wrong.

~~~
vidarh
> But with EC2, I can just use EBS @ $100/mo/TB

If all you're after is the storage, that is very expensive unless you keep
moving your storage tiers up and down very rapidly. E.g. Hetzner has servers
at 49 euro (ca. 66 USD) per month for 2TB.

------
colmmacc
Full-disclosure: I'm an engineer at Amazon Web Services.

"Mr. Hamilton later said privately that Amazon had developed original
statistical methods to limit damage from catastrophic failures."

I don't think it's what James is referring to (as it's just the tip of the
iceberg), but at Re:Invent we also released Infima - a library for service-
level fault isolation and fault tolerance using Route 53 for endpoint
discovery;

[https://github.com/awslabs/route53-infima](https://github.com/awslabs/route53-infima)

The techniques in Infima give some insight into how we think about multi-
tenancy and availability, as well as provide a framework for building other
services on top of EC2, Route 53 and so on.

~~~
simonebrunozzi
This one looks great! Any talk/video available that describe it in more
details?

------
b0b0b0b
James Hamilton's blog is not to be missed:

[http://perspectives.mvdirona.com/](http://perspectives.mvdirona.com/)

------
eitally
Interesting that they focus on all the infrastructure they design and
build/commission themselves. They are also a big customer of my company
([http://www.newisys.com/Products/products.shtml](http://www.newisys.com/Products/products.shtml)).
Coincidentally, they call out Facebook at the bottom of the article, in
context of the Open Compute Project. Facebook had airflow and overheating
problems with their intended OCP design and our Newisys engineers helped with
some design adjustments to resolve those, too. Now Facebook is buying cheap
Taiwanese-made (Quanta & Compal) OCP-design servers and shipping them to us
([http://sanmina.com/](http://sanmina.com/)) for racking, systems test, and
shipping to their DC in Lulea, Sweden
([https://www.facebook.com/LuleaDataCenter](https://www.facebook.com/LuleaDataCenter)).

So yes, these guys all do lots of tough engineering work, but it's not in
isolation and not without assistance from industry veterans whose careers it
has been to ensure properly functioning data centers and networks.

------
kevinchen
> Google publishes some technical papers about its big computers, but does not
> talk about things like the semiconductors it has developed internally to
> handle computing at this scale.

Google designs its own ICs now?

~~~
jws
I think I recall them making custom 10gbit Ethernet switch fabrics.

~~~
kijiki
They used merchant silicon in their custom switches. Modern switch ASICs are
2+ billion transistor devices with lots of fancy custom logic for SRAMs and
TCAMs and analog magic for PHYs. Even Google can't afford to do that by
itself.

~~~
twic
Pfft. They can just map-reduce over each bit in the frame.

------
venus
> 1.5 million requests a second, and holds trillions of “objects”

My god. I used to think I was a bit of a rock star, with (rails) web sites
exceeding 100 requests/second on my resume. 1.5 million requests per second
... I cannot even really imagine that.

~~~
codex
These get requests are likely sharded among a large number of machines. Viewed
that way, it's much more manageable. Want to service more requests? Just add
more machines.

1.5M relational queries on the other hand--that would be impressive.

------
locusm
A Dell account exec told me last week that Dell had built and implemented the
Sydney AWS data center - any truth to that statement?

~~~
nl
They are in Equinix SY3 (Sydney)[1]. Pretty sure Dell doesn't do DC design for
Equinix. I heard some stories about the truck loads of servers going in when
they were rolling it out, but I don't remember the vendor.

[1]
[http://www.arnnet.com.au/article/442144/equinix_expands_its_...](http://www.arnnet.com.au/article/442144/equinix_expands_its_sydney_sy3_datacentre_28_photos_/)

------
11thEarlOfMar
What is rather mind boggling is that Google, Amazon, Facebook, Microsoft,
Yahoo all have multiple, huge data center installations and, they are all
profitable companies. It does not seem to be a zero-sum game yet for market
share, and the growth in the world's appetite for storage and access is still
not fully served. Can they all wind up 'winning' with profitable operations
founded on web infrastructure?

How much storage does one Internet citizen occupy, anyway?

~~~
adventured
"How much storage does one Internet citizen occupy, anyway?"

That number chases toward infinite. Its growth may slow down, as we reach the
limit of how many 'actions' a person can reasonable take that generates
content, but it'll never stop growing just on the basis of the unlimited
desire for higher quality experiences (mixed with a general moore's law ride
that makes it all financially reasonable).

And just when you think storage demand is going to slow down, we'll jump to
some kind of virtual reality life recording, that makes it possible for others
to walk around in your daily life as though they're actually there, and
storage demand will explode another magnitude.

You can remove Facebook and Yahoo from that list though. Their operations are
funded by their core businesses and they don't compete in the AWS-style cloud
segment. That is, it doesn't make sense to discuss Facebook in regard to
whether it can win or make money, compared to AWS or Azure.

So is there room for Google, AWS, Azure, Rackspace, IBM etc? Yep. It's a
future hundred billion dollar plus business (AWS will probably be at $10b+ in
a few years). No reason there can't be three or four big players in such a
large market.

------
rottyguy
Not really meant to be a slam, but startling to me that even after all these
years, Amazon still has a PE of 1330+.

~~~
3825
I assume this means the market expects Amazon to make more money (a lot of
money) in the future.

~~~
rottyguy
Yep, I get that. But they IPO'd in 1997. Again, not to detract, and in fact
I'm amazed by how they've been able to compete both on the Brick & Mortar
stage and Digital Services stage. I've just never seen a company given such
love/leniency for so long by Wall St.

~~~
adventured
On Wall Street, the fake love often turns to a company being out of favor
quickly (with both directions being equally irrational, in the overvaluing and
then undervaluing). Indeed, the less fundamentals you have propping you up,
the more likely this is to eventually happen, and happen hard. That exact
scenario has played out numerous times with Netflix, for example.

Amazon's valuation is almost purely an emotional basis, as all future returns
are being pulled forward to the extent that they're overvalued and the
calculations about their future earnings expectations are correct. Is Amazon
worth $50 billion? $100b? $168b? That's an emotional judgment today, as they
have no profits and little history of profitability, and if you want to
speculate on tomorrow... well if their valuation keeps bubbling up, they'll
soon catch Walmart, which has $17 billion in annual profit (meaning it may
only take... 20 years or so for Amazon to fully realize that market value via
profitability). Amazon is going to be an excellent short at the end of this
stock market bubble, they'll lose half their value on the down swing (my
opinion).

However keep in mind, this is not a company that has always been loved by Wall
Street. For most of its life, it has been hated by Wall Street. If you run an
average price from early 2003 through the end of 2008, the stock was
essentially flat lined for five years. And obviously it was viewed with near
universal disgust after the dotcom bubble imploded, and the stock cratered to
the $5's after 9/11\. Amazon.Toast and all that.

~~~
bennyg
It's a better business idea for Amazon to build and invest in the company than
it is to go profiteering right now. How valuable would a company be if it
could get same day deliveries anywhere in America? In Europe? In China? All of
the world? One of these days Amazon is going to stop building as much and
their profits are going to shoot through the roof.

~~~
phinnaeus
Yes, that is bet that every Amazon investor is making.

------
kraemate
Anyone have the link to the original presentation? Particularly intrigued by
the statistical fault tolerance.

~~~
dbarlett
Not the whole presentation, but some slides and better notes:
[http://gigaom.com/2013/11/15/how-amazon-is-building-
substati...](http://gigaom.com/2013/11/15/how-amazon-is-building-substations-
laying-fiber-and-generally-doing-everything-to-keep-cloud-costs-down/)

------
badmadrad
The price will bottom out eventually and Amazon knows this. What we are really
seeing is an infrastructure arms race and that is why AWS is costly. They are
not building AWS for the individual user long term. They are building it for
the enterprise. Eventually as bandwidth and computing power increases and cost
decreases it may be feasible to run your personal blog out of your house or
use a VPS on Digital Ocean. However, as we become a global community you will
have to think about your application on that scale to be successful and that's
where AWS can find longevity.

------
bluedino
>> Mr. Hamilton also said Amazon was not interested in participating in
industry standards discussions, which could publicize its many technical
advances.

What are the side effects of this in a few years, when all of the high-end
stuff has been designed 4 different ways by 4 different companies?

What happens when Dell and HP's server businesses are shells of their former
selves? When Cisco is but a memory as their biggest customers have began
designing solutions in-house?

You'll have to go with the cloud. You just won't be able to get the best of
the best equipment without doing so.

------
mwilcox
"You've reached your limit of 10 free articles a month." ugh.

~~~
medell
Open in a private browsing window.

~~~
natrius
Or, you know, pay for the product. I don't understand the logic behind
complaining about paying $3.75 a week for a product that you use so
frequently. It should be embarrassing.

~~~
jlgreco
10 articles a month isn't exactly frequently. At the low end, you are
suggesting that people throw more than a dollar per article at them. You
shouldn't find it surprising that many people will baulk at that.

------
anan0s
I'm a bit skeptical on the fact that such companies do not disclose their
custom infrastructure design. So their value is both in hardware optimizations
as well as software ones ?

being a grad student myself, I'm wondering if we'll ever see any specific
technical aspects on infrastructures that could drive systems research _and_
provide real problems to be attacked...

~~~
XorNot
I'm skeptical that there's anything _that_ innovative going on behind the
veil.

I suspect if you had access you would look at a whole bunch of evolutionary
improvements and probably trace them back to which papers you've seen it sort
of mentioned in.

Probably not as much custom silicon as the article implies either, simply
because on the scale of things Amazon is not a semiconductor company and Intel
pretty unambiguously own the cutting edge there.

------
dschiptsov
VmWare is a much greater waste.

