
Why I, Jeff Bezos, Keep Spending Billions On Amazon R&D - rrrgggrrr
http://www.sfgate.com/cgi-bin/article.cgi?f=/g/a/2011/04/27/businessinsider-why-i-jeff-bezos-keep-spending-billions-on-amazon-rd-2011-4.DTL
======
vaidhy
Disclaimer:I work at Amazon as a dev manager.

Working at Amazon does mean that you need to work hard, long and smart. There
is a lot of technical debt to deal with too. But the technical debt did not
come from bad design, but from the way business grows and changes. There is a
20% growth YoY. Inventories run into millions of products across multiple
warehouses. We ship through multiple means and we ensure the product shipped
reaches the customer on the promised date at over 95% of the time.

In real life, this means that you are going to have trade-offs and some of the
trade-offs do turn into technical debt. What we do with the debt is more
important and amongst my groups, we are decreasing the debt by leaps and
bounds.

If you want to see how it is, give me a shout. We are hiring, as usual :) I
can take resumes in any format too :)

~~~
senthil_rajasek
How long have you worked at Amazon and where are you based?

~~~
vaidhy
I have been there for about a year and based out of Hyderabad.

------
jordanb
Judging from things Steve Yegge has written about Amazon, and from friends of
mine who have worked there, I don't get the impression that Amazon is a
company dedicated to technical excellence.

In particular I seem to recall something about 40 million lines of C++ code in
their core product, and about mediocre engineering talent and high levels of
burnout due to the technical debt they're required to deal with...

In fact, ever-increasing engineering costs would be just as indicative of
mounting technical debt as it would of an increasing commitment to R&D.

~~~
haberman
I worked at Amazon from 2004-2007. On one hand, Amazon does some really
excellent and novel work like Dynamo and AWS (S3, EC2, EBS, etc). They
organize developer conferences and bring in really good people to lecture.
They hire really solid distributed systems people like Pat Helland and give
them reign to work with engineering teams to improve their designs.

On the other hand, when you're in the trenches of a product team at Amazon,
good engineering takes a back seat to features, deadlines, and keeping
everything running. There was an incredible amount of pressure from the top to
maintain high levels of service, but not a lot of support for taking the time
to refactor or rework those systems to address the underlying causes of
service disruption. Everything was always an emergency, but we always had to
make the shallow fixes.

Overall, I'm much happier at Google where there is a lot of support for good
engineering at all levels of the organization, not just as a means to an end,
but as an end in itself.

~~~
puredemo
Too bad that Google doesn't touch the customer service standards at Amazon.

~~~
VB6_Foreverr
I bought an xbox and a game for it at the same time from amazon. The xbox was
fine the game disk was defective. I contacted amazon via my account, got a
call from a human in less than a half hour (mainly I'd say to confirm the xbox
itself was OK), received the new game a couple of days later and a refund of
my return costs for the game itself. The only (very) slight quibble I'd have
was that the postage on the return exceeded what they refunded. They made the
return process itself simple by emailing me barcoded return labels and
frequent reminders. I feel very confident buying off amazon a lot more so than
ebay for example

------
prpon
That letter is so geeky, I doubt if 90% of amazon investors would get the
things he was explaining. As a geek though, I love it.

~~~
michaelochurch
My feeling about it was that it was written to appeal to top programmers, not
investors except so far as "You can read this letter, in which I prove that I
know the language one needs to speak to hire smart people".

~~~
athst
Good point - maybe this is just the greatest recruiting tactic ever.

------
px
Investors want higher margins but obviously Amazon is investing substantially
in itself. So far, though, Bezos seems to be winning the tug of war. Many
analysts are glad to see higher revenue and feel that Amazon is cementing its
position at the top of the ecommerce world. AMZN shares are up over 5% despite
the earnings miss.

------
TWADDLE
Its funny how people seem to be whining about Amazon.

In fact Amazon is one of the best places in terms of innovation. In addition
to having a strong core business, they also have AWS, kindle, android app
store.

Who cares if their legacy codebase is 40 milllions lines of C++ and not your
hip Ruby or LIPS nonsense. The truth is that they get work done. AWS is one of
the best cloud computing system out there. Nothing else comes even closer.
Dropbox, Heroku and Reddit all rely on them. Who cares if they take resume in
word format, they pay well.

Its sad to see HN turning into a language fanboi and apple fanboi club.

The AWS EC2 outage did not even affect 0.01% of their revenue.

~~~
jacques_chester
My understanding is that the original code for Amazon was written in hip Lisp
nonsense with some ironic-retro C thrown in.

------
javanix
_The end result of all this behind-the-scenes software? Fast, accurate search
results that help you find what you want._

Has Bezos ever actually used Amazon's search functionality?

I know it isn't the easiest problem in the world to solve, but their search is
by far the worst of any of the major shopping sites that I use on a regular
basis.

~~~
pkteison
Specific examples? It seems pretty good to me. Which shopping site has a
better search, and what makes it better?

Exact keyword match is obviously fine. "harry potter and the sorcerer's stone"
gives me various options for buying that book and a bunch of other harry
potter related books, and i can sort by some useful options like reviews or
bestselling.

Harder problems, Google shopping vs amazon: Misspellings: Hairy Potter -
Amazon suggests 'did you mean Harry Potter?', but does not initially put that
book in front of me. When I check again later they are now providing a second
list, "Results for "harry potter" (corrected from "hairy potter")"

Google shopping shows me the Did you Mean question, search results are the
lego sets and one movie.

Concepts/descriptions instead of exact thing: "Boy Wizard Series" - both fail
to suggest harry potter to me, but they do start with "Harry, a History" which
might be enough to jog my memory. Still, no real points here, but this is a
hard problem, and I don't know any shopping site that does it well. Even
Google's regular web search is not fantastic at this, hitting mostly news
articles and not reading my mind and popping out with "I bet you want to see
information about Harry Potter..."

So where's the trouble?

~~~
Splines
In particular, searches for computer equipment is generally frustrating. I've
also seen sorts result in a large number of irrelevant outdated used items
getting filtered to the top.[1]

I'm willing to cut them slack since it's good enough, but look at Newegg to
see how it should be done. It's not a completely fair comparison (since Newegg
specializes in this), but they do it so much better.[2]

[1]
[http://www.amazon.com/gp/search/ref=sr_nr_i_0?rh=k%3A16gb+sd...](http://www.amazon.com/gp/search/ref=sr_nr_i_0?rh=k%3A16gb+sdhc%2Ci%3Aelectronics&keywords=16gb+sdhc&ie=UTF8&qid=1303934061#/ref=sr_st?keywords=16gb+sdhc&qid=1303934069&rh=k%3A16gb+sdhc%2Cn%3A172282&sort=price)

[2]
[http://www.newegg.com/Product/ProductList.aspx?Submit=ENE...](http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&IsNodeId=1&Description=16gb%20sdhc&bop=And&Order=PRICE&PageSize=20)

~~~
pkteison
Thanks, great example. It looks like they don't account for the importance of
certain terms in tech searches well enough. Like they're far too willing to
throw away the "16gb" part of "16gb sdhc" when sorting by price, and sorting
by relevance shows you expensive stuff, so there is no good way to say "show
me cheap stuff but I'm not kidding about the 16gb part".

~~~
nitfol
Also note that sorting by price doesn't actually sort correctly. In this
search, a $6.99 card comes before a $5.99 card (both with free super-saver
shipping). This is so broken (and has been for so long) that I almost think it
must be intentional somehow.

------
cmelbye
This is only slightly on-topic, but regarding the service-oriented
architecture that he mentions at the beginning of the article, how would one
implement this internal? Specifically, how would you communicate with your
internal services? Just an HTTP API? Is that fast enough? Another thing I'm
wondering is what's the best way to handle internal authentication between
your frontend and some backend service for example? Sorry if this is too
specific for this discussion, but I've just always wondered this and I figure
that HN knows.

~~~
shykes
DotCloud [1] is built from the ground up around distributed services. We use a
custom rpc mechanism called ZeroRPC. It uses ZeroMQ [2] for transport, msgpack
[3] for serialization, and some extra glue for RPC mechanics, discovery, event
dispatching, etc.

Among other neat tricks, it allows us to handle synchronous RPC calls and
asynchronous message passing in a unified way. It also maps transparently to
any ZeroMQ topology.

Combine this with the recent release of gevent support for zeromq [4] and you
get a fun, fun playground :)

[1] <http://www.dotcloud.com>

[2] <http://www.zeromq.org>

[3] <http://msgpack.org>

[4] <https://github.com/zllak/gevent-zeromq>

~~~
cmelbye
Oh, awesome, thank you!. I never thought to use ZeroMQ, but that would work
perfectly. Ironically, it doesn't seem like DotCloud supports these sorts of
architectures very well yet, any plans to fix that? ;)

~~~
shykes
Absolutely - our job is to make developers happy and productive. Whenever we
find a sweet tool that makes _our_ lives easier, we're practically tripping
over ourselves to share it with _you_

Our network team is working on some major goodies which should make you happy.
Personally they bring tears to my eyes :)

In the meantime, my recommendation is to start with http. If you keep the RPC
semantics simple, you can easily migrate to a different transport later. And
you'll need HTTP to authenticate and encrypt at the boundaries anyway.

------
czhiddy
Interesting how AMZN jumps 5% after missing earnings, yet GOOG drops nearly
10% with nearly the exact same results (increased hiring and expenses
resulting in lower than expected profit).

~~~
Eliezer
Stocks don't react to how results in announcements compare to last year,
stocks react to how results in announcements compare with how Wall Street
priced the stock yesterday based on their guess to what the results would be.
Efficient markets 101 - otherwise you could make money by buying, one day in
advance, stocks that obviously probably did well compared to last year. If
this is not obvious to you then you should never invest in anything besides
index funds (seriously, I've seen people do really badly by not understanding
elementary implications of the inexploitable markets _hypothesis_ ).

~~~
anatari
Um, missing earnings means earnings was less than what the wall street
analysts expected, it doesn't mean it was less than last year.

~~~
Symmetry
I thought it generally meant that it was less than the company's own
predictions, not the analysts? In any event it shouldn't matter which estimate
they're falling short of as long as it isn't the market's implicit estimate.

------
sahillavingia
$AMZN stock is flying right now:
[http://www.google.com/finance?client=ob&q=NASDAQ:AMZN](http://www.google.com/finance?client=ob&q=NASDAQ:AMZN)

~~~
JabavuAdams
Lol. Sold a bunch of AMZN last week to buy AAPL, just before AAPL announced
earnings. Looks like I should have kept AMZN.

As a Canadian, the whole currency conversion thing adds another source of
risk.

~~~
3pt14159
I was mulling over that exact trade, but figured the relatively small amount
of AMZN I had didn't justify getting off my tush. Sold a couple hours ago with
a solid 60% up in 10 months (usd -> usd) and 41% in CAD terms. I'm fairly
bullish on AAPL, but I already own 200 shares (selling medium term call
options, hence the 100s multiple) so buying some more probably isn't worth it.
Just going to keep it in cash until I see something.

Do you have any experience in hedging your US investments by shorting the USD?
I'm wondering if the additional fees are worth it.

~~~
JabavuAdams
Re Hedging FOREX: Nope. It's something I'm interested in learning about,
though.

------
revorad
PR damage control?

~~~
matthavener
I think so, but not for the reddit/EBS outage.

~~~
mfringel
Exactly. This is all about their earnings. Given a general trend towards
decreasing or outsourcing R&D, Amazon is still doing it in-house, because it's
their competitive advantage. Explaining that to a bunch of panicky
stockholders who just saw a Big Ugly show up on the cashflow statement is
mandatory, no matter how much they care about the long-term.

~~~
icey
I feel like I'm missing something. As of this moment (9:42AM PDT) Amazon's
stock is up 5.6% for the day and near a 52 week high. That doesn't seem like
panicky shareholders to me.

I'm not a stock maven, so it's possible that I'm missing some important
detail.

~~~
groovylick
AZMN stock climbed over 8% during the AWS outage, the market is disconnected
from reality.

~~~
jmatthews
Just reading his run up regarding their distributed server technology it read
to me like a brief ad for AppEngine.

I'm thinking about all of the hardcore neuron work that went in to their
infrastructure and how I'm getting a competing product for a song.

~~~
neuroelectronic
for liberal definitions of "competing" The only similar cloud services are
from Microsoft and Google and they are about the same price, definitely not "a
song" territory.

~~~
jmatthews
I meant that given your app works within the restrictions of GAE you get the
benefit of a lot of brainpower from men with long beards.

Maybe the association was temporal as I'd just finished watching an IO
presentation about how to scheme data for appengine where the detail was
primarily the same as Bezos' first few paragraphs.

~~~
neuroelectronic
Like many conversations, this conflict was only a result of miscommunication.

------
aphexairlines
I wish the company would reveal some information on the internal systems he
outlined in his letter. Amazon's architecture is impressive and its
infrastructure teams build great stuff to help developers get their work done
& deployed.

Exposing these things would be a powerful recruiting tool.

------
spinchange
This is instructive. Google's stock might not have tanked as hard the other
week if LP had come out and placated the street with some jargon like this,
re: Google's also ballooning expenses.

~~~
athst
Bezos explicitly said it in this letter - he expects all of this stuff to make
the company gobs of money in the future. For a lot of the stuff Google spends
money on, I don't think investors are necessarily very confident that it will
have the same impact on future profits.

