
Amazon is building substations, laying fiber and keeping cloud costs down - antr
http://gigaom.com/2013/11/15/how-amazon-is-building-substations-laying-fiber-and-generally-doing-everything-to-keep-cloud-costs-down/
======
happycube
Everything at Amazon is about creating scale.

Prime exists to get shipping volumes up far enough to pull things like
trucking items daily from Vegas to Southern California, directly into the USPS
system at the sorting facilities - and eventually direct deliveries.

The other thing - which lead to AWS, FBA, etc, was that each Amazon department
had to make a globally usable API, even for (then!) inside facing stuff.

Combining those and you get a giant. Hopefully Bezos will stick around for a
while yet - I can't see Amazon continuing to not maximize profits in exchange
for volume very long after he's gone.

~~~
pyre
What happens to the kingdom once the "good king" passes away...

Edit: I was angling more at the fact that Amazon has been very dominant, and
that's not going away soon. So if it starts getting bad, it's not like there
will be something just as good overnight. Look at how long it took to erode
MS's dominance from the 90's.

~~~
WesternStar
Honestly, if I'm Amazon or Google I'm in that business not really to make
money. Hell, I'll run the same business at a loss frankly because I have to
build that business anyway to run the business I'm already in. I need the
infrastructure to run a scalable website if my competitors and others want to
pay me to do that ,great.

------
malandrew
Their comments about storage cluster density are interesting. I'm still
shocked that hard drive design basically hasn't changed at all.

What I'd like to see is hard-drives that don't need to be screwed in and don't
need sleds. Instead they are machined with grooves on the tops, bottoms and
sides to be "slide in, slide out" and have the grooves be multipurpose in the
sense that they are designed for airflow. i.e. the top of one hard drive mates
with the bottom of an identical hard drive to create channels through which
air can be forced through and cool both drives maximally, simultaneously.

Basically take the level of machining Apple puts into designing the the
Macbook Pros and apply it to hard drives. If designed correctly, you should be
able to pack them together, side by side in a way that they lock (like a
jigsaw puzzle or the Heineken World Bottle[0]) in a stable "lattice" of hard
drives that only permit sliding out to one side and permit efficient cooling.

Screws and sleds are so yesteryear.

[0] [http://www.fastcodesign.com/1672684/heineken-s-lost-plan-
to-...](http://www.fastcodesign.com/1672684/heineken-s-lost-plan-to-build-
houses-out-of-beer-bottles)

[1] I hope that HN comments like these constitute prior art in case someone
decides to go out and do this and patent the idea.

~~~
sigstoat
since the tops are thin sheet metal, and the bottoms are circuit boards, i
don't think we'll see mating surfaces there.

but they could probably make some side grooves, and little cutouts for spring
loaded latches to catch on.

add some cutouts on the front that something could grab onto, and you might be
able to avoid the need for the sled and screws, without building up an entire
latticework out of drives.

there probably needs to be structure to hold the back plane(s) and line them
up with the drives, anyways.

~~~
malandrew
I get the impression that the circuit board would be best placed on one of the
sides or entirely on one end. Placing it on the bottom pretty much nullifies
efficient heat exchange along one of the broadest surfaces to use for cooling.

------
aquadrop
I wonder how come Amazon didn't mop up the floor with his competitors (e.g.
DigitalOcean, Rackspace) price-wise if AWS infrastructure is so optimized? Of
course, they have more features, but that's all kinda software-related, and
they say software doesn't cost them that much.

~~~
eloff
In high-end servers they blow the competition away on price. I did a
comparison recently and they came out over 4x cheaper than Rackspace for two
256GB of RAM servers + heavy bandwidth.

~~~
adventured
This isn't even remotely accurate.

I could give you a dozen examples, but one will suffice.

From a hosting provider I've used for many years, WebNX. Here are two example
configurations from them:

E5-1650v2, 64gb ram, 4x 450gb SAS 15,000 rpm enterprise drives and hardware
raid 10 with battery backup, 20tb of bandwidth - $289 / month, no setup fee

Dual 2690v2, 256gb ram, 6x 600GB 15k sas raid 10 w/ 2x SSD Cache - $950 /
month, no setup fee

The bandwidth alone would cost you $2,300 at Amazon. And 20tb is nothing these
days, countless quality providers offer 100mbps standard with $150 machines.

The AWS cr1.8xlarge with just 20tb of bandwidth is $5,134 per month. A one
year locked in contract gets it down to $2,815 ($7,200 up front) - three times
more expensive. And there are far cheaper providers than WebNX.

AWS is almost interesting, if you never need any bandwidth.

~~~
eloff
That's also an over-simplification.

For example at AWS you can get a fully bidirectional 10gbps link, regardless
of what other tenants are doing on the network. Nobody else that I'm aware of
can match that. And you'll quickly find that 20TB of bandwidth at those
network speeds are nothing, you can cross into petabytes without breaking a
sweat (1 petabyte at 50% of bandwidth 24x7 for a month for just one server.)
At that kind of network volume bandwidth costs dominate everything (about $65K
for an petabyte of bandwidth at Amazon.) I challenge you to beat that
anywhere.

Also if you have bursty traffic patterns, AWS comes out a lot cheaper because
it's designed around quickly scaling up and down capacity.

In the end it depends on matching a provider to your particular needs because
they vary wildy in strengths and weaknesses and costs.

------
alexkus
> Hamilton didn’t talk a lot about AWS’s custom-built storage, but he did
> share one tidbit. The densest storage servers you can buy commercially today
> come from Quanta, and a rack full of them would weigh in at about three-
> quarters of a ton. “We have a far denser design — it is more than a ton,”
> Hamilton said.

Interesting. For reference; a Backblaze pod is 4U and weighs 145 pounds, so 10
of them in a standard 42U rack would be 1450 pounds = 0.715 (short) tons (or
~648kg).

~~~
sparkman55
For many colo centers, there are both weight and power limitations on a per-
rack (or per-cabinet / per-row) basis. One wouldn't want a ton of disks
crashing through the (often raised) floor!

For compute clusters, power is often the limiting factor; for storage
clusters, weight is often the limiting factor. If you put these clusters in a
general-purpose colocation facility, you'll end up with half-full racks.
Density on a per-rack-unit basis is not necessarily important for most of us.

Of course, if you're Amazon's size, you build your own facilities, and you can
make sure to support the weight and power requirements of whatever custom
hardware you can dream up!

------
taway2012
Pretty interesting nugget there. I've always thought Amazon cloud services
were overpriced. But I thought maybe the software engineering labor is
expensive. The comment that "software engineering costs rounds down to zero"
was extra information I didn't have that strengthens my gut feeling.

Just my two cents. I don't expect to change anybody's mind. :)

~~~
Patrick_Devine
Everyone gets a raise!

Seriously. I'm not sure how well compensated AWS engineers are, however if the
price engineering is just noise, why not pay the best in the industry?

~~~
thinkling
Have you looked at what Amazon stock has done over the past few years?

~~~
Patrick_Devine
Sure, but realistically what's the upside? Maybe your stock doubles if Amazon
gets to be as big as Microsoft, but that's unlikely given how razor thin the
margins are. Sure, it could happen, but if Amazon is spending several billion
on data centers and warehouses every year, that's likely to make a dent in the
stock price.

------
polysing
This is all very interesting and impressive, but why doesn't anyone talk about
how much pollution and environmental damage this is causing? It's terrifying
to think about how much fossil fuel will be burned to keep these resources
running. How many servers and cables they will be throwing away every year.
How much packaging trash is being generated by Amazon every day. Do these
inconvenient by-products simply disappear? No, they show up in your landfills,
and in your oceans, and they get "exported" (dumped) in China and Africa.

The only metrics people are looking at is revenue, profits, growth, bandwidth
and storage. But how about we start measuring pollution, waste, and
environmental destruction for every company, right alongside the traditional
metrics?

It's great when the stock price doubles. But does anyone think about the fact
that this probably also means the amount of plastic trash in our oceans, and
poisonous gas in our atmosphere doubles?

~~~
wmf
The environmental impact of cloud computing is hotly debated. To the extent
that it's replacing "OldOps" kind of infrastructure it's probably more
efficient. OTOH the Jevons Paradox predicts that as the cloud gets cheaper it
will be used for increasingly higher volumes of lower-value tasks that
previously would not have been done at all. Ultimately I favor internalizing
the environmental costs and letting Amazon deal with them how they will.

------
protomyth
"Back in the day, Hamilton used to lobby for just having one or two SKUs from
a server vendor in order to minimize complexity, but times have changed. Once
you master the process, going straight to server manufacturers with custom
designs can lop 30 percent off the price right away, not to mention the
improved performance and faster turnaround time."

So, they get the server manufacturer to do the work instead of building it
themselves. I guess the scale goes:

    
    
      low volume: buy -> build own -> custom outsource :high volume
    

I wonder how long each design lasts before they move on.

------
antr
I might be naïve with the following comment:

I would love to see AWS moving a bit further down the customer chain,
providing one-click installations to non-savvy customers (I'm thinking on-
click wordpress, magento, docker, etc.). Given the recent acquisition of
MediaTemple by SlowDaddy I'm confident many GridServer customers would of
moved from MT to AWS.

~~~
jgreen10
[https://aws.amazon.com/marketplace](https://aws.amazon.com/marketplace)

tip: search for wordpress, magento

The biggest drawback of AWS is that you miss features not because they don't
have them, but because they have so many of them. :)

