
Amazon makes 14 cents from every Lyft ride - skilled
https://twitter.com/mohapatrahemant/status/1102401615263223809
======
chomp
This is a bunch of FUD.

* No, using a colo saves you way more than just 20%. In one of our facilities, our org has maybe ~15k servers under management and dc costs are ~ 1 million/mo. Build out of the cages and racks wasn't incredibly expensive, I think ~2 million. Power isn't insane either.

* I have no idea what he's talking about re: the fiber statement. We have a blend of different backbone providers that give us ~100 gigabit for ~$0.60 per gig. Compared to bandwidth costs for AWS, it's dirt cheap.

* You don't need ultra fast hot swappable robotic arms if you're not FAANG. With a handful of dc techs and a simple monitoring system, you can swap out disks just as easily, at the expense of a little more opex.

* PCI requires no consultants, its terms are pretty black and white - you will need an ISA though. HIPAA and can be more of a bear, but your org will need this consultation regardless of whether you own a dc or put it in the cloud. AWS does not give you automatic compliance.

* Yes I can totally believe that working at Google, you'd give TCOs that make Google look good.

Literally none of us except Lyft's operations knows what's good for Lyft. 100
million a year is a ton of cash, but gives them presence around the world. My
company has datacenters around the world and it is very operationally complex
to maintain them. It's definitely not for everyone, but this guy is making it
sound worse than it actually is.

~~~
wnevets
On an unrelated note when did FAANG before a thing? I don't recall seeing this
used anywhere a mere year ago.

~~~
marton78
Also, why is there no M for Microsoft in that abbreviation?

~~~
wnevets
probably lack of growth in their stock prices?

~~~
kerng
I think it's because of base compensation of employees, which still holds true
still, because Microsoft pays way less then FANG.

Microsoft is the most valuable company in the world. When it comes to stock
Facebook probably would be not in FANG anymore because of their drastic stock
drop over last year.

~~~
taobility
A for Amazon is well-known for its terrible work env for the employees.

------
SilasX
I don't think the title is a fair representation of what's going on. Hear me
out.

The way it's phrased, it's like each ride directly translates into 14 more
cents for Amazon. But really, the figure comes from averaging Lyft's Amazon
costs over Lyft's total rides.

Now, in _some_ cases, that would be valid accounting, if _all_ (or nearly all)
of your costs (with respect to that service) are variable. For example, most
of the cost of sending a letter is the truck/plane/employee, which are
variable, so it makes sense to divide total letters over total truck etc
costs.

But would Lyft pay twice as much to Amazon if they served twice as many rides?
Would they have to double _every_ AWS service they're buying? I don't think
so. They might have to add more of one or two kinds of server/services, but
most of that 14 cents figure comes from amortizing fixed costs, which will go
down if they scale up further.

Am I off base here?

~~~
hinkley
This theory has unfortunately never panned out for me. To the extent that I
get a little anxious about my employer getting too aggressive with PR. Organic
growth always plays out better for the engineers in the short to medium term.

IT's version of the Laws of Thermodynamics is that nearly every interesting
action you can do programatically is going to take _at least_ O(log n) time,
where n is the amount of data you have (at rest, in flight, or both).

If I have 16x as much traffic as I used to deal with, my load isn't 16x
higher, it's 64x higher - if I'm fortunate and we've engineered for scale. But
it could easily be three orders of magnitude higher (16 _16_ 4) if I'm not.

For the 64x scenario I tell my boss to buy more hardware and I put medium to
high priority tickets in the backlog to improve the worst cases. In the 3
orders situation, those tickets bump all of our other priorities, and there
are more of them. Performance is now a feature. It sucks up a significant and
very visible fraction of our engineering budget, which comes with its own sort
of opportunity costs.

If you try to scale vertically, we all know that except at the low end, buying
a server that's twice as beefy costs far more than twice as much. If you go
horizontal, there are plateaus where your network topology has to get more
complex (hardware cost, maintenance cost, speed, pick two). Personally, I
think the biggest lie in cloud computing is that 10G ethernet fixes all of
your network topology problems (ie, it's treated as magic that you don't have
to worry about). As disks and PCIe get faster over the next couple years I
think that'll be back on people's radars.

~~~
hexane360
If your complexity is O(log n), an increase of 16x only increases your time by
log(16)=1.2 (constant, not a multiple).

~~~
naniwaduni
I'll say, I'm so used to seeing "log" denote either lg or ln (depending on
whether the context is closer to maths or computing) that seeing the claim
that log(16)=1.2 threw me for a loop.

This despite it being fundamentally meaningless because of course it needs to
be multiplied by some time-dimensioned value anyway.

------
g9yuayon
People seem have forgotten the productivity loss of running their own data
center. When I was in my previous company, most of things are just
excruciating compared with what I enjoyed in Netflix. Dockers did not come
with persistent volumes, provisioning machines involved lengthy approvals and
sometimes long waiting time; building and deploying services that involved
multiple clusters that talked to each other required serious devop-fu;
production clusters got broken because infra upgraded system dependencies
using a background job and mitigating such simple mistakes may take hours if
not days. Deploying any non-trivial service required multiple meetings with
some infra teams, for resources, for concerns with stability, or whatever. PS,
did I mention that we built tens of thousands of Puppet files that very few
people understood and any updates to my system took half an hour to refresh
yet the early members of the infra team thought Puppet was the best f@#$ tool
in the world? The list can go on and on. PPS, did I mention that we had a
500-person infra team and later probably more than a thousand for a fraction
of traffic/complexity compared with what Netflix had to deal with, while
Netflix didn't even have an infra team (to be fair, Netflix had a build team
and a platform team and a monitoring team, which had fewer than 30 people
combined)?

In contrast, EC2 alone would have to save us lots of pains.

You may say that my previous company was not technically great. Maybe. If
that's the case, how many companies can be really better? I think the
fundamental problem is that companies tend to underestimate the effort to
build a world-class infrastructure from scratch as well as the opportunity
cost due to loss of productivity. Netflix's leaders were truly wise by
claiming that Netflix didn't want to work on undifferentiated heavy lifting
early on.

------
sisk
The thread is much more interesting than the initial tweet.

[https://threadreaderapp.com/thread/1102401615263223809.html](https://threadreaderapp.com/thread/1102401615263223809.html)

~~~
altmind
This thread only compares ROI of hosting in cloud with building own datacenter
and laying own oceanic fiber(!!). The option of using a colo and magistral ISP
somehow fell through the cracks.

Of course building one datacenter and own oceanic fiber to host your own
services is a bad option for almost every business - not every business can
fill a full DC, and 1 DC is not enough to guarantee global operations.

~~~
altmind
The orignal author mentioned how good AWS worked for netflix, but did not
mention that netflix rolled their own content delivery system in part because
of high traffic cost with all the major cloud providers.

~~~
drewg123
My understanding is that most commercial CDNs (Limelight, Akamai, etc) are
also more cost-effective than serving directly from AWS. However, commercial
CDNs don't know a-priori what files are going to be popular, and how to
distribute them in different regions and on different types of storage to
maximize efficiency. But that's one of the things that Netflix's CDN team can
predict. See [https://medium.com/netflix-techblog/distributing-content-
to-...](https://medium.com/netflix-techblog/distributing-content-to-open-
connect-3e3e391d4dc9)

------
msoad
Lyft is horribly inefficient in terms of using their AWS resources. I know
this first hand. They don't have to build their own DS. They just need to
build a better system. Possibly take advantage of Lambdas because a lot of
compute heavy operations like geo en/decoding are done in services that run at
like 25% CPU in production.

~~~
ghaff
Not to pick on you specifically as others are saying more or less the same
thing... But if you're such a genius that you know how to slash Lyft's AWS
spend without knowing basically a thing about their infrastructure, you should
be knocking loudly on their door because I'm sure they have an _extremely_
well-paid job waiting for you.

Otherwise, maybe consider voicing opinions like these about things you
actually know something about.

ADDED: Meant to respond to a different comment that didn't suggest any inside
knowledge. (Though I'd still point out that even insiders often can't
understand how their own company has so many employees in X department, is so
inefficient, etc. And this is more or less the case everywhere.

~~~
ceejayoz
"I know this first hand" implies they _do_ know something about their
infrastructure.

Given that they work for "one of few >10b unicorns" per their comment history,
I have a suspicion you may be telling a Lyft employee they don't know anything
about Lyft.

~~~
ghaff
I actually meant to respond to a different comment which merely said this was
a few hours of an AWS instance.

~~~
ceejayoz
Well, that's factual - you _can_ get a couple hours time on an entire AWS
instance for $0.14. That certainly _seems_ to point to an cost-inefficient
infrastructure.

~~~
ghaff
I don't really have an opinion on how efficient or inefficient their
infrastructure is. AWS has lots of services and I'm sure Lyft is using a heck
of a lot more of them than just a bunch of EC2 instances.

------
mv4
Some quick math: last I heard, Lyft was doing 1M rides a day. That's 1M
transactions per day, or "only" 11 transactions per second. Of course there's
lot more (assuming 10x) supporting transactions to enable one ride - search,
matching, status updates. Let's say that adds another 100 transactions per
second. Rounding up to 150/sec. What am I missing?

In terms of data usage and storage, matching is by geography, meaning their
data is easy to shard.

So, how's this $8M per month?

edit: fixed a typo

~~~
thinkingkong
Your math isnt wrong but the load isnt simply averaged out over the entire
day. There will be peak capacity times way way higher than the lowest times. I
would guess orders of magnitude larger. You have to build for peak capacity.
If their target response time is 100ms or something then that adds in lots of
complexity.

~~~
mv4
I agree the load will fluctuate - but will mostly follow predictable patterns
daily and weekly. However, that's what the cloud's elasticity is for - and AWS
makes it easy (and inexpensive) to expand and contract when implemented
correctly. Man, I would love an opportunity to optimize something like that!

~~~
MisterPea
Right?! The cost is way too high for their use case, even considering a large
padding to OPs quick maths.

Part of my team one month stopped all development and solely focused on
reducing AWS costs and we cut our monthly bill by 50% (probably around 4 dev's
yearly salary)

It's not easy, but optimizing AWS resources is an absolute must.

------
sek
Are we talking just about machines? A huge chunk of AWS is actually enterprise
SAAS.

Analytics, authentication, support etc. you don't know what Lyft uses exactly
here from AWS. For example if they use Auth0 instead of Cognito they pay that
part to Auth0. AWS prices are very competitive most of the time.

AWS is the Walmart of enterprise IT, they sell you everything else too.

~~~
oblio
One could say: AWS is the... Amazon.com of enterprise IT, they sell you
everything else too :D

------
ddorian43
These posts always make it like "cloud is better vs building datacenters on
mars".

There are middlegrounds, like colocation & dedicated servers. If you get
dedicated servers 4x cheaper than cloud-shared-vps with remote-hdd, then you
can overprovision.

Especially now that hardware is getting bigger you need even less space
(assuming your software scales vertically).

And they ALWAYS make it like the next hour you will have 10x requests and your
database will autoscale that quickly.

~~~
hpkuarg
Agreed. It's disingenuous of the article OP to:

\- only parenthetically mention colocation, as if it isn't absolutely normal
to rent a rack/cage/suite as needed from an existing DC operator (which in
addition would make datacenter rent an OpEx and actual server equipment be
amortized over 2-3 years, not the 10 for real property)

\- somehow present "intercontinental traffic" as something both necessary and
tremendously expensive, as if all the major public clouds aren't charging 10x
what the market rates for bandwidth are

\- imply that "the cloud" is immune to outages, as if GCP didn't have multiple
major global- or region-wide outages over the past few years

I mean, I understand why most companies don't go on-prem despite all that, but
this series of tweets is borderline FUD.

~~~
kemitche
I'm not sure the article OP is being disingenuous. It's a thread
_specifically_ about Lyft's costs - and Lyft is clearly at a scale where they
would NOT be colocating.

~~~
Slartie
Of course they would. They are a taxi service, not a Dropbox or YouTube. They
do neither have obscene demand for storage, nor for bandwidth, they need some
servers tracking their users coordinates, running a "who is nearest" search in
case someone wants a ride, some GPS navigation/pathfinding during the ride and
some billing code after it.

If you don't blow this up to stupid proportions, that should be able to run on
modern hardware in a few racks at a colo for millions of users per day,
especially since you can neatly shard the load geographically, thus
distributing load (and rented racks) over multiple DCs, ideally with failover
in place for emergencies. The only thing they really need to merge is the
billing data at the end, but handling billing data of a global userbase of
tens or hundreds of millions of users in a single system is a solved problem
nowadays and does not even constitute a case worthy of the overused "big data"
moniker.

------
prepend
Amazon got 2 cents from me typing this note and reading ycombinator for an
hour.

I love when indirect costs are naively calculated for marginal cost estimates.

~~~
toomuchtodo
HN runs on a dedicated box colo'd at M5 Hosting :-P

~~~
buboard
A single box ?

~~~
toomuchtodo
Yes.

[https://news.ycombinator.com/item?id=11916168](https://news.ycombinator.com/item?id=11916168)

~~~
buboard
that is awesome

~~~
ddorian43
A single core inside 1 server!

------
quantumhobbit
14 cents per ride is nothing! Maybe 1% of the cost of a ride.

Maybe they could save a bunch by colo or something else. But would 14 cents
per ride really matter at all for their competitiveness. I’m not going to
notice a 14 cent difference even if I do bother to price compare Uber with
Lyft.

This is a VC fueled market. It isn’t really about small margins of this size.

~~~
ceejayoz
It may be 1% of the _cost_ of the ride, but it's likely a substantially larger
part of the _profits_ of the ride.

Lyft lost a billion dollars on $2B of revenue in 2018.

~~~
isostatic
So saving $8m a month, or $100m a year, isn't going to turn that round.

~~~
ceejayoz
Cutting 10% of losses is hardly something to sneer at.

~~~
isostatic
If you're still losing $100m a month you have bigger problems then saving a
couple million in infrastructure costs.

Moving from AWS, or doubling AWS spending, will make no difference to the
company's viability, it's not worth the time in meetings to discuss it.

~~~
slededit
Man this industry is crazy. Most traditional companies fight hard to gain even
1% in net margin.

~~~
isostatic
Most traditional companies are based on having revenues higher than costs.

dotcom v3.0 companies are all about the potential and cornering the market.
Amazon was exactly the same - it was founded in 1994 but didn't make a profit
until 2001.

~~~
slededit
For Lyft having a taxi anywhere I want one with low wait times at low cost is
going to secure their success. Not new features in their app, not even latency
to the data center.

There's no reason they need to be spending such crazy amounts on servers -
ostensibly to allow faster iteration. A new version of their app just isn't
going to move the needle. Signing up new drivers will though.

I fail to see how the benefits AWS provides are so important they need to
spend such crazy amounts.

~~~
isostatic
They need to be seen to be doing something other than bleeding money.

Every extra driver costs them money, every lift costs them money. If they can
hold out the illusion of "we know we can save money here when we've got time
and have won the market", maybe it keeps the money rolling in.

------
devit
Maybe they could have a reasonably efficient setup instead?

Not sure how you can possibly pay 0.14$ in computing for a single ride (if
that's accurate). That's more than 3 hours of a t3.medium instance for
example...

~~~
kemitche
The tweet took Lyft's total AWS costs and divided it by total rides. It's not
a literal accounting of the compute cost for a single ride.

AWS costs that wouldn't really fit into a per-ride accounting:

* Redundancy of instances (regions/AZs) * Data-storage/duplication/backups * Non-ride related AWS costs (hosting/processing of analytics, test & automation infrastructure, etc.)

I'm sure there are others. I'm not saying that there's no way they could get
their AWS costs down, btw.

------
so_tired
Well, each driver needs a mobile+plan, at ~$100/month fully amortized.
Definitely does <1000 rides/month.

APPL, VRZN each get 5c/ride - just for showing up.

I started this comment as a joke. But now i am not so sure.

~~~
so_tired
And while i am at it...

Suggesting lyft needs 5-9s is ridiculous for 1000s of servers is ridiculous.
Its entire event stream comes from a mobile network which is probably less
reliable.

~~~
charleslmunger
It's not that simple - if Lyft won't connect and everything else works, riders
and drivers will turn to competitors. Once they've installed the app and
signed up for an account with Uber, they're a lot less likely to stick
exclusively or primarily with Lyft.

~~~
marcosdumay
Well, 5 9's are 5 minutes of disruption an year. Drivers won't be able to get
into Uber on that timeframe. 4 9's would be 50 minutes an year. Unless it is
all concentrated on a couple of events it may not even make the clients time-
out.

------
jgalt212
OK, but that's just the AWS expense. How much per ride does Lyft spend on it's
own devs (and admin staff) and boxes (and electric and RE costs)?

------
syntaxing
I always see the motivation for the cloud mainly because of financial
reasoning but is there a benefit to stay on-site in terms of knowledge
retention (for the lack of a better word)? For instance, there is a big push
to outsource manufactured parts and good mainly due to "cost savings".
However, the are long term cost savings when we keep the parts made in-house
which management always overlook. Sure, it cost money to maintain the
machines, inventory, and tools but this allows greater flexibility and also
retain the knowledge within the company. In addition, it allows the engineers
to drive technology within the company faster. Is this the case for data
center infrastructure?

~~~
perlgeek
You are a bit more flexible in the data center. Like, if you really need very
low latency and high bandwidth between two machines, you could buy an
infiniband thingy. Expensive AF, but may less expensive than setting a horde
of software engineers on it for two years to deal with the higher latency.

But, all in all, there aren't all that many practical use cases that really
benefits from stuff like that. Most projects are quite happy with some
storage, compute, network, and a few managed services. Most companies don't do
rocket science with their software and infrastructure.

And there are things that you simply can only afford at scale, like security
management for your supply chain, dedicated automation for updating firmware
of every tiny controller that's in your hardware, and so on.

------
amaccuish
"...you'll never have hot-swappable everything managed by ultra fast robotic
arms that replace hard drives in seconds."

Does anyone know anything more about these robotic hard drive replacers?

~~~
ceejayoz
I've seen it done for tape libraries.

My understanding is that the biggies - Google, Amazon, Facebook, etc. - don't
bother with individual drive replacements (and certainly not with an expensive
robotic arm system). They wait for an entire rack to fault over a certain
percentage and then just swap out the whole thing.

~~~
ADefenestrator
Depends on the case. Facebook at least generally replaces drives as they go.
In a pipelined fashion rather than as they fail, though. So, a given drive
failure doesn't lead to immediate emergency replacement, but sometime in the
next few days as someone makes a drive-replacement pass through that area.

------
rhacker
Our (small) company has a $500-$1000 AWS bill. About 2 months ago it was
double that. This is a 3-4 developer company. When we hire another person
we'll probably increase our EKS instances by one more box. We already paid
down our debt - and it definitely cost us about 2 months with half our dev
staff. But our goal wasn't to reduce the AWS cost, it was to reduce the time
it takes to replicate an environment in K8s with one script. I'm guessing Lyft
has not taken a path like that.

------
somethoughts
I think what's interesting to know is of that 14 cents, how much of that 14
cents does Amazon keep as profits and how much goes directly to costs. Some
analysis suggests AWS has 25% profit margin which means of that 14 cents,
Amazon really only pockets ~2-3 cents per Lyft ride.

Another interesting question is whether that profit margin includes the
original capex of when AWS was not reporting the revenue/profit of AWS in its
early years. There's also all sorts of creative accounting methods to hide
capex such that I wouldn't be surprised if Amazon is selling at cost to buy
marketshare in a way somewhat analogous to Uber subsidizes rides to gain early
marketshare.

------
thisisit
I am surprised no one is talking about Lyft's risk assessment of their
business. Specifically, this line:

 _Our results of operations vary and are unpredictable from period-to-period,
which could cause the trading price of our Class A common stock to decline._

 _Our results of operations have historically varied from period-to-period and
we expect that our results of operations will continue to do so for a variety
of reasons, many of which are outside of our control and difficult to
predict._

I find the term "period-to-period" rather vague. It could mean quarter-to-
quarter or even year-to-year.

Still what they are effectively saying is that Lyft cannot be trusted to
provide any growth assessment.

------
_bxg1
A really elaborate (and admittedly interesting) way of repeating the constant
refrain: Things that work for the extremely large-scale usually aren't ideal
for the small- or medium- scale. This applies to everything from codebase size
to infrastructure size, and the opposing force tends to be engineers' inherent
enthusiasm for engineering things. Be honest about your actual needs and know
when not to chase the N=∞ case.

~~~
buboard
Is lyft considered extremely large scale?

~~~
_bxg1
Well, that's the question. The guy basically makes the case that it isn't, for
this purpose, and that Netflix and Dropbox are. Good insight into the hairy
details, but the overall thesis is pretty simple.

~~~
solidasparagus
I would say he's making the case that Lyft doesn't have a single component
with ridiculous scale like Netflix CDN or Dropbox storage. That could just
mean Lyft's costs are more evenly distributed.

------
ProAm
This is why Twitter is terrible for having real conversations. This was
painful to read because of formatting and structure.

~~~
tim333
There's the threadreader version
[https://threadreaderapp.com/thread/1102401615263223809.html](https://threadreaderapp.com/thread/1102401615263223809.html)

------
buboard
I dunno, hetzner and contabo sell boxes for $20/mo

I still find it weird thaf lyft needs so much processing per ride, it just
doesn’t sound efficient

And a lot of ppl seem to find this normal. Its not, this could be money that
pays the driver more

------
yalogin
Certification has nothing to do with DC right? It depends on the product. You
have to get pci done even if you use AWS

------
gesman
Something to consider in all estimates - if cloud provider could run into
conflict of interest with you.

------
encoderer
Their payment processor also takes a big cut. This is just a cost of doing
business.

------
OrgNet
how much per ride goes in taxes?

------
craigkilgo
How are they paying this much?

------
ousta
Why would you need to build underwater fiber????????? I don't understand.

~~~
mattkrause
Especially for Lyft, where you can be pretty sure most stuff won’t cross
oceanic boundaries: “Raoul is arriving in a white Cunard QM2-G32”

~~~
aserafini
Look out for ‘Uber for Ferries’ in this summer’s YC batch!

------
tomtompl
The title is clickbaity.

They get paid that ~14 cents but they don't 'make' that much.

What they make is what is left after costs.

~~~
tudelo
If you were to say how much you get paid, you would in general say you make x
an hour. You wouldn't say "Well, after taxes, rent, food, gas, insurance, etc,
I make x-y an hour."

