>In January 2019, we entered into an addendum to our commercial agreement with AWS, pursuant to which we committed to spend an aggregate of at least $300 million between January 2019 and December 2021 on AWS services. If we fail to meet the minimum purchase commitment during any year, we may be required to pay the difference, which could adversely affect our financial condition and results of operations.
Not as bad as snap but what could they possibly be spending $100 million a year on?
100M/year is ~8M/month. Some perspective on that, it could by you one of:
~400PB of data in S3.
~2600 bare metal "x1 type" ec2 instances running 24/7, 3 year upfront reservation.
~60M Write IOPS in dynamodb
~300M Read IOPS in dynamodb
~3500 16xl RDS aurora instances
Again, each of those is spending the entire budget on a single service, but that seems like a nonsense level of spending.
Maybe they really have that much data. Maybe they have 100PB of data in S3. Assuming 1B rides since day 1, that's 100MB per ride, which seems high. If the average ride is 20 minutes, that's 80KB per second. That would be 25% of the budget.
But assuming they generate 80KB/s/ride, that's ~1MB/second (assuming 1M rides/day). So maybe all of that hits DynamoDB, and between duplicate data, secondary indexes, and size of dataset we have 6 million write iops. And then we do big data processing jobs and have 5x the read load. That's 20% of the budget.
And to process all these events there is a massive EMR cluster of bare metal instances. About 1750 of them. That's 50% of the budget.
Leaving 5% (a measly 400k) for load balancers, and the like.
Those numbers are all a little outrageous to me, but I can see how they might be using that much.
~3500 16xl RDS aurora instances? I worked at one of the 100 biggest websites on the internet (a search engine), and we only had 3. Why/how would Lyft need 1000x that!?
Basically, we had three types of data: 1) everything associated with the SERPs, 2) most things associated with payments, and 3) everything else.
Category three was relational, IIRC like ~2TB. We originally stored it all on a custom MySQL cluster, and we started running into pretty bad replication lag. Amazon came around with Arora and promised we'd never have any replication lag. We switched over. Still had replication lag.
Lyft's load varies wildly, with significantly higher traffic on Friday and Saturday nights than e.g. 4 AM on a Tuesday, plus spikes on certain evenings like Halloween and New Year's Eve. Having cloud hosting where we can dynamically grow and shrink based on load saves us a lot compared to having fixed infrastructure that is always provisioned for the NYE peak.
Why not employ an hybrid architecture of bare metal for base load augmented by cloud-based infrastructure for peaks, constructed via a polyglot union of taped-together tools and lubricated by the daily tears of a hundred college hires only to regret it after the engineers who designed it have successfully used it as a springboard for promotion and departed with their accumulated arcane knowledge (and vested shares) for greener pastures?
What sort of promotion metrics did your peers experience? Are we talking lateral moves, with more interesting technologies, or are we talking about movement into leadership with significant pay bumps? Also: based on this experience, what's the best technology to invest in and then abandon completely because nobody else wants to deal with it?
It still stands... $8mm monthly is over 350 full time engineers making $250k annually (ya that doesn't cover hardware, but it just illustrates the point). That would be a 22% increase in Lyft's total number of employees.
If you have engineers making less than $250k annually, then you have a lot more staff. $8mm monthly is a LOT...
Amazon's clearly making a profit after $8mm monthly.
But it's not like Lyft would suddenly have 350 full time engineers developing new features. A large chunk of those engineers would be working on building and maintaining infrastructure that AWS provides.
The point stands... 350 engineers is an army of engineers... for $8mm monthly, it wouldn't be unreasonable to achieve 500+ engineers depending on salaries.
Lyft could definitely build and maintain their own infrastructure for this kind of money... probably do it better (customized to their needs) and cheaper.
All of that is ignoring payroll taxes (for your new, very large staff), shifting all of your tax-deductible operational expenses into tax limited capital expenses.
Businesses don't flagrantly throw around money just to upset people. There are huge advantages to offloading non-primary business costs to other businesses.
Netflix is doing this too. I think we can assume not all of them are just idiots that haven't figured out they could build this themselves.
> Businesses don't flagrantly throw around money just to upset people.
But businesses do throw around money for the wrong reasons, and keep on doing so if that's the status quo. No one gets fired for buying IBM.
> Netflix is doing this too.
IBM stuff was bought by a lot of people.
> I think we can assume not all of them are just idiots that haven't figured out they could build this themselves.
That statement is very misguided and misses the problem. For example if you built your infrastructure around a specific solution then you also end up building a team of professionals whose livelihood is tied to a specific supplier of said infrastructure.
> But businesses do throw around money for the wrong reasons, and keep on doing so if that's the status quo. No one gets fired for buying IBM.
Businesses are wasteful because that's the natural status of a bureaucracy. They aren't throwing away money on infrastructure because they are unaware, they are spending more than they potentially have to because infrastructure isn't their core business.
> IBM stuff was bought by a lot of people.
That's such a tired argument. Just because they could save money doesn't mean it's a good idea, and with Enterprise pricing from Amazon combined with tax advantages, you honestly have no idea how much "cheaper" it really is.
> That statement is very misguided and misses the problem. For example if you built your infrastructure around a specific solution then you also end up building a team of professionals whose livelihood is tied to a specific supplier of said infrastructure.
No, the fact that you think this is a "problem" is the problem. Do you honestly think dev ops guys couldn't figure out how to use a different tool? By your own logic, you also shouldn't build data centers because you end up building a team of professionals whose livelihood is tied to managing your own infrastructure.
> Businesses are wasteful because that's the natural status of a bureaucracy. They aren't throwing away money on infrastructure because they are unaware, they are spending more than they potentially have to because infrastructure isn't their core business.
That's not true at all. The "isn't their core business" argument is meaningless and absurd. Any company, big or small, does not want to waste 300M dollars on something they don't need, whether it's their core business or not, particularly when said company is still far from turning a profit.
> That's such a tired argument. Just because they could save money doesn't mean it's a good idea
You are aware you're stating that baseless assertion on a discussion on how a company which is burning through cash and looking for investors is needlessly wasting 300M on infrastructure costs.
> No, the fact that you think this is a "problem" is the problem.
Needlessly spending 300M dollars is a problem in every single business in any corner of the world. I have a hard time understanding how someone can throw around the baseless assertion that this sort of inefficient while operating at this particular scale is not a problem, and pointing out this problem... is the problem? That's crazy.
Probably they will do a lot of optimization once they go public. That will improve financials over a year which will help the stock unless it comes at the expense of growth. The name of the game is growth that’s what gets you the high multiples.
> constructed via a polyglot union of taped-together tools and lubricated by the daily tears of a hundred college hires only to regret it after the engineers who designed it have successfully used it as a springboard for promotion and departed with their accumulated arcane knowledge (and vested shares) for greener pastures?
I think the parent was pretty clearly sarcastic and suggesting that this was a bad idea.
so, all workers at Lyft have to do continue to grow and protec to stand point performance, at some point company will face to choose between continue to grow and except paying extra cost whatever AWS bills to them or stop growing until contract end. i guess we all gonna watch.
Maybe. Running your own infrastructure at large scale is actually really hard and therefore surprisingly expensive in risk and talent cost. If you look at how big the internal infra teams are at companies that host their own infrastructure, they're often individually the size of growth stage companies.
The problem is that provisioning, reliability, and security are by themselves really tough problems. If those issues aren't in your company's core competencies, it's not necessarily efficient to invest in building out all of that.
I look at it as the question: can you get the same set of agility/reliability/security guarantees for your narrower set of use cases by paying for your own hardware and engineering? I won't even begin to pretend I have any answers there, but I think that's the calculus.
> If those issues aren't in your company's core competencies, it's not necessarily efficient to invest in building out all of that.
Maybe that's just the story cloud providers tell you.
Until you try, do you really know if it's all that complicated? People have been running datacenters for a long time, and not all of them work for Amazon.
But there may be also a beneficial side effect of having gearheads around, and maybe that's the real cost to going cloud.
Having done a bunch of bare metal, I can tell you the calculus isn't really that hard. Bare metal will save you money.
Operating bare metal at scale requires talent that doesn't exist, not necessarily at an engineering level, but at all levels.
As an example, I worked at a place that had a large bare metal deployment, i.e. >1MW worth of compute. It was woefully inefficient and costly to operate. The product that they offered required network QOS and compute with real time capabilities, neither of which was available from any cloud provider at the time.
One of our executives (formerly a leader in the DC ops org at AWS) left the company to be replaced by another executive by another well-known silicon valley org who then insisted we should migrate everything to the cloud.
I showed him the relatively easy math that efficiently utilized bare metal was way less costly and that the aforementioned QOS and RT requirements would be a deal breaker anyway. He failed to fully grok this and remained insistent. When I quite, he seemed surprised. After the fact, I discovered that they'd made a deal with IBM to move everything into their cloud. A year later it was an utter failure and they abandoned the project.
There are lots of folks in the valley with lots of experience on their resumes that suggests that they should be capable of understanding these kinds of things that simply don't. Lacking that understanding leads to poor decision-making, which leads to failure, which leads to risk-aversion, which leads to everyone believing that it must be cheaper in the cloud.
Or so goes the old adage, "nobody ever got fired for buying IBM."
Feel free to ignore this, but would you shoot me an email? I'm working on a project that uses bare metal and would like to pick your brain.
EDIT: To whoever downvoted this, the commenter hasn't listed an email address, or I would have reached out directly. This is an honest attempt at communication that doesn't require someone to break anonymity.
From a Lyft engineering perspective would rather focus on things like how do I make sense, process, extract the ton of data. How do I focus on customer experience rather than
how do I save money in data-center, how do I keep my data-center stack updated and many more
You can't just eliminate that 10%. Even if going to fully bare-metal lowers costs it takes a lot of time and manpower to make that transition. When that investment can be made in other areas that have much more impact it really doesn't make sense.
Bare metal works when your workload is well-defined and understood. Then you can actually put reasonable estimates for what you need and hire/purchase infra accordingly.
The balance here is tricky. Based on public data, it seems that Netflix has ~$16B in revenue against $300m/yr cloud spend. 2% seems much more reasonable to me.
I feel like a drive toward efficiency is a worthwhile endeavor for a startup in terms of establishing a competitive advantage.
The description here seems to be more of the compute and storage. How about the boat load of services that are offered with AWS. Plugging and playing with services maintained by AWS makes it easier for companies to focus on their product logic. The major expense is actually engineering.
My forecasting on RDS is that it’s a dead-end product – all the future hotness is going to be in Aurora Serverless. Multi-region active/active Postgres with totally usage-based pricing and totally elastic performance is going to be a game-changer.
But if you need bleeding-edge Postgres performance, you hire a DBA, and they probably build something on EC2 or bare metal.
———
As I understand it, RabbitMQ is probably a better point of comparison for SNS/SQS, and Kinesis is the Kafka peer.
Regardless, the reason you don’t “just” run Kafka is: you don’t have a team that knows how to tune, deploy, and operate a production Kafka cluster. I learned enough about SNS and SQS to get it running in an afternoon, and I really haven’t needed to think about it since. Kafka (or RabbitMQ, or ActiveMQ, or etc) need instrumentation and monitoring and patching and quorums and capacity planning and etc, and at some scale those are worthwhile, but that scale is MUCH larger than what most Kafka clusters are actually serving.
———
The theme here is: if you have a business requirement for 90th percentile specialized performance, great! Hire domain specialists who can make your systems run at that tier! But for everyone else in the world, when you can get usage-based pricing, elastic resources, and automatic durability and patching... why would you go to the trouble of learning how to deploy and manage a service?
I've been on both sides and it's not as simple as "bare metal saves you money." It really depends on the company and the type of applications being hosted and where the business is growing (or not growing). An established company with an established workload, especially if it's simple, will probably do better on bare metal, but cloud is popular in the Valley because ideas are still being developed and iterated on heavily where you don't want to be stuck on multi-year hardware leases that might not sync with what that future business looks like. Lyft is probably in that in-between stage, but I still think it's an enormous undertaking to become an infrastructure company over simply being able to hire full-stack developers, which is a lot easier and cheaper. Right now their time is better spent elsewhere.
I remember how hard it was to hire senior operations people. There are not many of them, and there are not many of them at the level of being able to deliver something amazing. The ubiquity of the cloud has only made these kind of experts less common.
Every place I've worked that did bare metal was always drowning in maintenance instead of working on the next big thing. And no big surprise, our internal infrastructure was nowhere near as high quality or capable as AWS. And most of our developers had experience working directly with cloud providers, without ops people, so we were delivering them a worse experience and slowing them down, and we required more ops people to help them and maintain it and keep everything online.
Also, a move to IBM's cloud isn't the greatest example. I had hundreds of bare metal servers in an IBM-owned datacenter and their cloud offering was consistently behind AWS/GCP; if anyone recommended IBM cloud to me I would have laughed at them. It seemed to me that IBM was trying to up-sell on the "cloud" buzz word without actually delivering anything except higher prices, just like how they're now trying to ride the buzz of the blockchain.
Dropbox is a good example of a company that took quite a while to move to their own platform, away from AWS (and they still have 10% of their stuff in AWS to this day). Dropbox is basically a storage infrastructure company, unlike Lyft, but it still took them years to invest in the development (and migration) of that custom platform to replace AWS, an investment that not many companies are going to want to gamble on, especially if their primary business is not storage:
And I think it's telling that Dropbox started on AWS, grew the business on AWS, and moved to a custom platform once their business model was perfected and they wanted to cut costs prior to going public. If Dropbox had started on bare metal from day one, would they have been able to pull it off?
IBM cloud is a joke. That's why I put it in there. The aforementioned executive was clearly not thinking.
There's nothing you've written that I disagree with. It's easy to do the math that shows where bare metal saves money inclusive of the labor costs. For some reason most everyone seems to fail at it. I could expound one why, but this:
>I remember how hard it was to hire senior operations people. There are not many of them, and there are not many of them at the level of being able to deliver something amazing. The ubiquity of the cloud has only made these kind of experts less common.
Those folks just don't exist. Building infra is more than just buying infra. It takes actual development, which is why I think so many fail at it.
Your anecdote about Dropbox is telling. They adopted cloud, and more importantly cloud methodologies and then went back to bare metal. There are others that have done the same. I recall a talk at an Openstack conference given by Verizon in which they described their approach. Developers begin in AWS, utilize a cloud-based approach, and then when cost concerns become an issue, they aim to offer similar services in-house on bare-metal.
>Or so goes the old adage, "nobody ever got fired for buying IBM."
This is true but its never really hit me before even though I've already been operating based on the assumption that trusting the cloud is less risky than trusting my own skills.
I think the point is, Lyft doesn't want to be in the business of running enterprise could infrastructure.
They want to make money brokering rides.
Taking on their own cloud infrastructure -- in theory -- could economically make sense. But that's just an extra layer of risk and complexity they'd rather forego to focus on their core business.
After all, their core business is already losing $930M on $2B in revenue. They're cash-flow doesn't put them in a good position to make large up-front investments on data centers.
So, yeah, like a broke renter in an expensive city. In theory, it might be better to buy a house, but you don't have the down payment, and maybe you should be focused on increasing your earning power rather than saving money anyway...
Yes, I've worked with a few of those datacenters. A few examples:
- Recently had to purchase new servers, because of signed contracts the only servers we were allowed to purchase and put in the datacenter were four years old and technically EOF.
- Firewall changes, AD changes, provisioning a VM, etc. are 48 hour turnaround. Purchasing new hardware requires 4-6 weeks.
- Had an intermittent issue with their edge firewall, it'd slow certain connections to a crawl and eventually they'd timeout. Took six months to fix it, for the first three months they told us it wasn't their fault (turning off their deep packet inspection ended up fixing it).
I still remember when we opened the first ticket about it, and the reply was "no other customers are experiencing problems" and it was closed.
That's just a few examples of how painful it can be. To give you the other side of the coin, having worked with an enterprise contract in AWS, we were having an intermittent issue with DNS resolving failing for a few seconds every few days. They put an engineer on it full time till they found the problem (we misconfigured it), and it didn't cost us anything more than the enterprise support. I was actually shocked they'd invest that much on such a vague issue.
Yes AWS is expensive, but you're getting world class engineering proven at scale, and access to some very smart/motivated people to support it (and they have access to the teams who built it, when they can't solve it). I don't think I'd ever choose managed datacenter over AWS/GCP/Azure/etc. Either do it in-house where there's accountability, or use cloud providers who have proven they're competency.
To be clear, I'm talking about VPC/EC2/etc. I can't really discuss a lot of their higher level and newer managed services; they either weren't as good, or I haven't tried them. But the bedrock these clouds are built on is solid, and that's worth paying good money for.
I've done that too. It's obviously easier than running an entire datacenter, but you still need to manage all the underlying services that you deliver to your development team. For example, running things like this on your own:
- storage clusters
- database clusters
- compute clusters
They are often very easy to setup, but when things go wrong, they go very wrong. And welcome to a stressful environment because if you can't figure it out and your people can't, well, your business just sits and burns while you do.
Even when AWS has a system-wide outage, it's nice to know that I don't have to be dealing with those underlying problems anymore and I know they have the best people working on them.
I cannot put into words, after operating MySQL clusters on my own and playing back transactions after failures, how nice it is to use AWS RDS and how it's just been zero problems. Zero. I sleep through automatic updates of our database system with RDS. I would have never done that on our own system.
And in most places, even "managed" leased hardware, you still will need to purchase/lease and run your own hardware firewalls and ddos mitigation. The datacenter might offer that protection "built-in" but you'll soon find the limitations of that offering when you face a substantial attack.
Seems like your experience is a case of ”ad hoc” systems management and I know what you’re saying is all to common in the enterprise world though.
Having spent my entire working life automating infrastructure of all kinds I know you can achieve an enourmous increase in efficency rather easiliy with a few well placed automated processes.
I’ve always been baffled by the fact that at any given larger company there are 100’s of employees trying to supply the business with tools to automate business processes — the IT dept.
Yet, they are completely incapable of using these very same tools to automate their own ”business”.
And the resistance I’ve been met with at different places through the years when trying to implement the simplest of automation is massive.
I used to laugh at the ”cloud” bacause, back then, at 25 years of age, sitting at a medium size company with boatloads of cash, I assumed everyone was doing it the way we were; automating all the things.
Now, many years later I’v obviously realized that many places simply does not have the right culture and mindset as it’s not “core business”.
I believe however that this is changing, and changing quickly. In many ways thanks to the “cloud”.
Having worked in the EC2 org, the people doing the customer service tickets were usually the oncall, when nothing was on fire, or the daytime SREs. You could get hit-or-miss depending on whom is oncall and picks up your ticket, some people were non-empathic and would reply to those tickets with one word answers or you'd get lucky with people that would dig deep.
No. That is my experience too. If your company’s core value prop isn’t running and managing a datacenter, your datacenter is gonna suck when compared to folks who do it for a living.
I work for one of said teams, doing datacenter cyber-security. You could say "trying" is my day job.
Since we're internal and we manage a lot of capacity, we do often provision and roll our own equivalents of things that cloud providers will sell you, rather than just buying a cloud solution. It's often ambiguous whether it was a good use of time/money. If it weren't for the economies of scale that kick in at the sheer size of this operation, it would definitely not be worth it.
I'm actually curious how AWS is able to scale support so well with what seems reasonable quality. I had an issue (my fault in end), got excellent support - and they didn't tell us to take a hike at end when it turned out not AWS fault.
Conversely, with GCP 4 years ago now had some support issues - didn't come away impressed - I'm convinced even internally GCP isn't well doc'd or something.
I think it comes down to Amazon vs. Google's company cultures around customer support, which informs attention and resource budgeting. GCP isn't the only Google product where people hate its customer support, and AWS isn't the only Amazon product where people love its customer support.
Amazon takes the customer support very far - with one exception - bogus products on marketplace seems like a big miss.
But what I paid for and got on aws support is so far out of whack there is NO way they made money on my account for that whole year. And the person was actually competant which was a shock. So many "technical support" folks seem like idiots.
Comcast for example, I'd purchased my modem, they started charging a rental fee - I had to call these bozos every month to reverse the charge - a total waste of time. I cancelled finally - I just couldn't take it, and each one lied to me or didn't have a clue. Things like condescendingly saying - you have to pay for the modem.
Not really, at that scale it eventually will be a lot cheaper running your own thing - Lyft might still be too small, but if they go internationally and grow ×10 then AWS seems like a choice to reconsider.
With cloud vs DC you also needs to consider that re-sizing datacenter space is a very slow process, and if you're wrong about how much you need it's a massive pain. Buy too much and you're wasting money, buy too little and you're stuck throwing eng time at scrambling to keep your services from falling over, and bottlenecking your entire org with resource constraints.
That's not a judgement on whether it's worth it for Lyft or not, but especially for a growing company with spiky load the decision is not just a dollars to dollars comparison.
You don't, though. You have room for $300m of opex, coming in over time and allocated, as mentioned, around $8m a month. Less early on, more later.
If you gave me $300m to spend, largely up-front, for significant capex purchases? Sure. We could do it. The team I would build would also probably still make mistakes that AWS et al have already largely learned how to avoid, but we could do it. But capex and opex are very different beasts. By the end of that three years I'm already looking at spending way more to refresh what I bought at the start of that three year period because I'm starting to near the end of early contracts and I'm figuring out how best to wrangle, in a way that makes the rest of the business succeed most optimally, a now-heterogeneous environment, etcetera etcetera and etcetera. It's all solvable. But whether it's cheaper, at scale, and more reliable, and presents a unified tool for use by the business...that's a harder question.
Understanding how capex and opex work and how they differ is pretty critical to successfully running an engineering organization, to say nothing of a company.
If you've built a heterogenous environment at the end of three years, then you've failed.
The reason that AWS, Google, Azure, et.al do so well is that they don't just buy some servers. They do actual capacity maangement, and not a very good job of it I might add. They also manage the lifecycle of every component in the infrastructure such that the next iteration of that component is understood and interchangeable.
Network architecture, for example, should suit the needs of the application, but should also be decoupled from the underlying hardware as that hardware is going to evolve.
Compute is fairly straightforward as well. At the data center level, one makes a bunch of 400W holes. What you fill those 400W holes with is relatively irrelevant.
Fully agreed on all points. But it remains a really hard problem. And when you start to do it out at the scale of something like Lyft, you're gonna blow through your available parts of that $300m (because you can't spend it all up front, obviously) pretty quick.
The care and feeding of fleets of (physical) machines is really, really hard and not to be underestimated.
The thing is, it's not really hard. Quantum computing is hard. Managing IT infrastructure as a value-added component of the business that evolves continuously vs. a cost-center to be bought and forgot is actually not that hard. The only thing hard about that is culture shift.
It's all about leadership. The dearth of skilled leadership is the issue. I'd wager this is how some FAANG companies are managing this. They're hiring people that know what they're doing. One doesn't need to design and build their own servers and network hardware to do well at the scale of folks like Dropbox or Lyft.
Cloud adoption is all about making the issue someone else's problem, which is only kicking the can down the road.
Eventually, every company that does a thing will realize that their survival is contingent upon becoming a software company that does that thing.
Respectfully, having read your other comments: I'll answer that question if you demonstrate to me an understanding of the difference between $300M/3 years capex and $8M/month/3 years opex.
If you do that, though, my answer will be "right, so we're done here."
I'm mostly just having a laugh so I won't be able to explain the difference. If the money works in ways a lay man is familiar I'd expect I would be able to afford the necessary man power and equipment so far under 8m per month after equipment purchase that I don't really need to know the details of the finance opex/Capex difference. I really appreciate you taking the time to bring up your good points.
if I have one job in this life, it's to hang out on hacker news and repeatedly post about how it's not cost effective to run your own infrastructure. 8MM/month doesn't even come CLOSE to needing your own infra.
- person who knows how hard it is to run your own infrastructure
ha sure. I just meant I know the sort of operational burden cloud customers are able to outsource by not rolling their own, based on my time at the G inside resource planning and management.
You know anything of the burden to be reliant on someone else for your core operations? And having to invest and adapt so much just to lock yourself to one vendor? For many that alone is more than the operational burden of doing it yourself.
I promise you weren't actually doing your costing correctly to arrive at that conclusion. Engineers always badly mis-underestimate the costs of things and "rack & stack data center management" is way more costly than you are actually accounting for. Especially in terms of opportunity cost and, well, just wasted resources that aren't actually adding value to the company.
There is way, way, way, way, way more to a running a successful business than "saving money".
The savings enabled us to hire more people than it took to run. After the upfront setup (Xen and Ansible), it was pretty painless, had much of the same flexibility as cloud options, and better performance. For bulk storage, we used S3.
Did your devs have the same turnaround time for new equipment and services as AWS? When they had problems, did they have the same volume of searchable material to help them repair? Did your services evolve and adapt as quickly as AWS? Does your networking connectivity survive DDoS attacks fairly transparently? Do you have DCs all around the part of the world that’s relevant to you? Can you survive multiple DC outages?
I was a cloud skeptic and ran Tech Ops (including our DCs) for years. About 5 years ago, it dawned on me that even owning the whole budget for Tech Ops, that I wasn’t capturing the full costs of trapping my org onto our in-house solutions.
At tiny, small, and medium scale, cloud is obviously the way to go, IMO. At large and huge scale, I think letting some hybrid leak in where systems change rarely and cloud costs are WAY out of line (DropBox storage, Netflix CDN, etc) makes sense.
We did have quick turnaround - we were running everything on Xen VMs on LLVM internally, it was trivial to throw up new instances, snapshot them, etc. Our demands weren’t changing that quickly, so there wasn’t that much need to bring on new hardware all the time. A small number of dual socket machines can go a very long way these days. We didn’t have the same kind of multi-DC redundancy that AWS can give you, no, but over those years, many AWS based services were down multiple times due to being focused in Virginia, so it’s not an automatic win for AWS. The amount of time we would have spent getting that running and maintaining automatic multi-zone failover on AWS likely would have swamped the benefits for us, though, and I think are usually overkill for a small company. YMMV.
Majority of businesses aren't running insane computations at huge scales...
Most are running a few small internal-facing servers hosting some internally developed apps, and need very little resources.
Just run ESXi, XenServer, Xen or something, and spin up a few VM's on a few thousand dollars of hardware, get a couple people to maintain it, and be done.
Even at large scales, like Lyft, having your own internal team and hardware is going to save money. Amazon is profiting off your instances... which leaves room for you to do it for less. Maybe not $7mm less monthly, but even a $1mm savings is significant... but likely a lot more.
They're making $200 million / month based on their S-1 filing. I'm sure they've done the math and the effort to cost savings here simply isn't worth it to them.
Or, more likely, in the beginning AWS is what their developers knew and were familiar with, and then as time went on it was easy and convenient to just spin up "one more instance".
Fast forward to today, and now it would be a serious undertaking with serious risks to move off AWS, not to mention the costs of building up the staff and assets to reimplement their requirements in parallel of AWS until reasonably confident they can flip the switch and still have an operating company afterward.
So, they're probably stuck - beholden to Amazon's whims and pricing mood of the day. They've bought convenience from Amazon in trade for massive technical debt, one which may be even more costly to get out of... Or impossible.
AWS isn't going to get any cheaper in the future..
Those free AWS credits Amazon gives students really pay dividends.
But AWS will get cheaper in the future, if historical trends continue. To the best of my knowledge, Amazon has never increased pricing on a service. They do, however, routinely (but unpredictably) drop prices, either directly (https://aws.amazon.com/blogs/aws/category/price-reduction) or indirectly (every new generation of EC2 instance is marginally cheaper and/or marginally more performant than the previous generation).
The “whims” of Amazon’s pricing are no more unpredictable than the pricing “mood” of your colo or your hardware vendor.
What I'm saying is that <5% of revenues on technical infrastructure for a technology company at Lyft's stage doesn't seem that crazy to me.
I agree that it would be a serious undertaking to move off AWS today. But it's probably also going to provide marginal benefit. No one on the finance side of their business is probably losing sleep over it. If/once it makes sense to move off then the finance dept will tell the eng dept they need to reign in infrastructure cost...and eng will do that.
They have many teams making their own web services, so replacing AWS with in-house infra would require an AWS-like “private cloud” feature set, backed by geographically distributed datacenters. Network segmentation that is managed by configuration, user accounts with permissions to do specific things to specific services, etc. Engineers to build all this stuff are not cheap, and management to build and migrate to it cleanly it is not easy. If you can do this you ought to be selling cloud services. Even if it makes sense for Lyft to implement this, it would take a while. The RIO is probably much better on improving their core product, where there is more than $100M/year at stake.
I suspect nonprofits like the Wikimedia Foundation have much greater constraints on billables but perhaps relatively less constraints on engineering and management effort. That probably changes the ROI calculation quite a bit.
And management of such organizations might also have ideological tendencies that further skew the calculation.
No. The capex and overhead for rolling and maintaining your own infrastructure is outstanding. The only time it typically makes sense these days is when it's done in quantity and also sold to others, which is where the economies of scale kick in.
It's believed that it would be cheaper for Twitter to have used cloud services (Snapchat spends less then they do on data center operations).
There are certainly examples for big companies that benefit from having their own infrastructure (i.e. Dropbox since they have relatively specialized hardware needs compared to what cloud providers set prices around), but the number of people you need to hire to build and maintain datacenters is very high.
Twitter initially ran everything in the cloud. I should know, I was employee 13. It was much cheaper to build our own server hardware and move to our own server and network agreements. Peering and transit is cheaper at scale, as is commodity hardware.
Those numbers have changed since you left. And they could change again. It's all dynamic: these comparisons and the economically and strategically "correct" choices change over time depending on talent availability, demand, speed of scaling requirements and so on.
Strategically, you probably want to focus on what your core competencies are, even if you could in theory do something for cheaper. It's easy to ignore the foregone best alternative of iterating on your own product instead.
* If the hammer manufacturer decides not to sell you any, you'll still have hammers.
* If the hammer manufacturer gains enough power to fix prices, you won't be paying them exorbitant prices.
* If the hammer manufacturer or their country gets embargoed and you're unable to legally purchase their hammers, you'll still have hammers.
All the above grant you a strategic advantage since you'll still have the necessary tools to continue your business while your competitors won't (or will have to pay much higher prices for their supply of hammers).
Very much so. The cost benefit analysis must also question how much it would cost to educate the teams on the new hosting tools. Management has clearly decided that it’s not worth the retraining and loss of productivity.
Will add if lyft is using things like dynamodb, there is no good onprem alternatives that they can easily migrate to without significant rewrites of core business logic
This is expensive and risky and also difficult to do in piece meal
Why would they have so much data and so much computing. Would they be really dependent on that? Makes it scary to consider what could happens to their passengers if aws is down / hacked.
There is an incredible amount of data, compute and DB hits required to manage dispatching, route planning, cost estimation, dynamic pricing/LTV optimization, ETA estimation, resource positioning and many other optimizations related to every market and every request within a reasonable response time for the customer. Products like Lyft Line are particularly intensive on this front.
Who's "we" and why would they worry? I'm struggling to think of a side-effect of AWS going down that would worry me more than being unable to get a Lyft (which itself doesn't worry me very much).
Maybe a few here would cause some concern. Also note that not all companies/governments using AWS just let everybody know that they are, so they aren't listed and you can only know when the service goes offline.
> Adobe, Airbnb, Alcatel-Lucent, AOL, Acquia, AdRoll, AEG, Alert Logic, Autodesk, Bitdefender, BMW, British Gas, Canon, Capital One, Channel 4, Chef, Citrix, Coinbase, Comcast, Coursera, Docker, Dow Jones, European Space Agency, Financial Times, FINRA, General Electric, GoSquared, Guardian News & Media, Harvard Medical School, Hearst Corporation, Hitachi, HTC, IMDb, International Centre for Radio Astronomy Research, International Civil Aviation Organization, ITV, iZettle, Johnson & Johnson, JustGiving, JWT, Kaplan, Kellogg’s, Lamborghini, Lonely Planet, Lyft, Made.com, McDonalds, NASA, NASDAQ OMX, National Rail Enquiries, National Trust, Netflix, News International, News UK, Nokia, Nordstrom, Novartis, Pfizer, Philips, Pinterest, Quantas, Sage, Samsung, SAP, Schneider Electric, Scribd, Securitas Direct, Siemens, Slack, Sony, SoundCloud, Spotify, Square Enix, Tata Motors, The Weather Company, Ticketmaster, Time Inc., Trainline, Ubisoft, UCAS, Unilever, US Department of State, USDA Food and Nutrition Service, UK Ministry of Justice, Vodafone Italy, WeTransfer, WIX, Xiaomi, Yelp, Zynga [1].
That's a useless list. To take the one example I happen to work for, yes NASA uses AWS, but not for anything terribly important. Without any evidence that one of the handful of groups on that list that have safety-critical infrastructure are running that infrastructure solely on AWS, I maintain my position that Lyft going offline would be a greater inconvenience to me.
Yes, mostly useless. As I stated, and I'm sure you'd probably concur, most companies and governments don't go listing their tech stacks (especially critical tech stacks) for the world to see. That in itself can be a big security issue. For instance, AWS isn't allowed to mention, list or use the name of one of the companies I do work for.
I take it you don't use the bank listed. That's fine. Does your bank do transactions with them? Other banks? Other institutions/stores? Do you use NASDAQ? Do others? Since everything is so interconnected, it doesn't take much for one of those services to immediately or eventually affect a bunch of others. It might be relatively trivial if AWS goes down for a few hours, but what about a longer duration and the avalanche effect? Is that impossible?
You also changed the goalpost a bit from "worry me more than being unable to get a Lyft," which is the comment I responded to, to "safety-critical infrastructure." I can't give examples of that because no one in their right mind would list that anywhere.
Is Lyft itself working on self-driving cars? I thought that GM and Alphabet (which are both investors in Lyft) are doing that and Lyft itself is doing no self-driving car research.
EDIT: now I see, page 3: "Simultaneously, we are building our own world-class autonomous vehicle system at our Level 5 Engineering Center, with the goal of ensuring access to affordable and reliable autonomous technology"
Then they slowly turn into a datacenter company and lose sight of being a ride sharing company. That's the same reason billion dollar companies rent buildings instead of owning them.
Uber hosts its own infrastructure, so does Google, so does Facebook. All three of those companies have no problems remaining focused on their business models without turning into a "datacenter company."
I strongly dislike the notion that on-prem hosting is somehow a bad thing, or too cumbersome, or otherwise totally solved by cloud providers. AWS specifically is hugely convenient in a number of ways, but it doesn't come close to the cost savings from running your own infrastructure. You need a pretty large amount of capital and engineering talent, but it really is worth it even in the short term (~3-5 years).
I think people would be shocked at what the money comes out to be if they saw costs from companies doing their own physical infrastructure. AWS makes you pay through the nose, seeing the difference would change a lot of minds I'm sure.
Google have struggled to retain focus. And when they started out, they chose to compete on datacenters as a core competency: their corporate history is full of the idea that they could run a search engine cheaper/better than competitors by using commodity PCs and that kind of thing.
Facebook is a similar story: being the biggest website in the world is a core competency for them, and one of the ways they outcompeted rivals early on was by scaling their website better.
Uber has yet to turn a profit.
If datacenters are a part of your business proposition - not necessarily "we're selling datacenters to other people" but rather "we will be able to outcompete our rivals because our datacenter strategy will be better" - then self-hosting makes sense. But if the datacenter is a commodity from the point of view of your business - and I would assume that would be the case for Lyft - then it makes sense to buy off the shelf.
The trouble is when people build around Amazon, they get locked into a lot of those services. Sure you can run your own DBs instead of using RDS, but what if you start using their proprietary rubbish, like Knesis or DynamoDB?
You have to rewrite application to use something else that's open source and self-hostable.
For new startups, I honestly recommend using DigitalOcean or Vultur. You don't get all the AWS components, but you can build around flexibility. If you have to move, you can take all your Terraform and Anisble scripts, and port them to a new provider (and yes, you do have to rewrite your Terraform config. Every provider is insanely different and the magic of multi-cloud is a myth, but it's still easier than trying to move off of AWS specific services).
I remember back in the day, Stackoverflow ran everything off of a single, very expensive, dedicated server. I've worked at other shops where we've migrated stuff from AWS to self hosted solutions to reduce our $200k/month AWS bill.
> The trouble is when people build around Amazon, they get locked into a lot of those services.
The trouble when people build things that have nothing to do with their core value propositions, they get locked into those services too. It is very easy for companies to get locked into their own homebrew garbage frameworks, clustering solutions, reporting & data analysis apps, or whatever else people hacked up because "omg vendor lockin!!".
Seriously, I hear the same argument with frameworks too. At the end of the day it isn’t a choice between framework A and no framework, it’s going to be a framework A and your custom baked framework, and do you really have the time to spend reinventing the wheel?
Google is a bad comparison here IMO because Google IS a datacenter company. No different than Amazon... They run datacenters and offer IaaS to customers, and piggy back off of that.
The idea to use hosting is much older. Many companies offered services like that even in the 90s with AWS it really scaled up and the quality for hosting then improved across the globe.
SQL and Domain Names was very common even in the 90s. The other concepts hadn't really evolved. Even AWS only had EC2 and S3 in the beginning. I'm just trying to relate and point out that Google decided to self host, rather then depend on someone else - which for a business that exponentialy grows is the best way, even today.
How would they become a data center company? There are 1000s of huge companies using a mix of colocated DCs, their own DCs, and cloud providers like AWS including the majority of Fortune 500 companies. Most of them are not datacenter companies like QTS.
Most of these companies aren't managing their own data centers at least for areas that use a large amount of compute. Banks and the like that have their own data centers, generally are still using some type of contractor to manage the physical real estate, network connections, ect. even if they are the ones purchasing the servers.
Come on, that’s like saying Google has no business having datacenters because they are an Advertising and Search company. Of course large companies will have contractors to scale up and down. They will have FTEs doing System Administration and Software Engineering on the servers.
They could also just rent dedicated bare metal servers on a month-to-month basis, getting whatever hardware specs they want so long as it's not overly exotic. Then they don't have to worry about anything at the data centre or hardware level.
Given that cloud costs easily 6-7x for the equivalent amount of hardware resources as a well priced dedicated server provider, you can just buy 2-3x the resources you need for extra scalability and not have to share those resources with anyone. Or if you are in the tiny minority of companies that really does have extremely erratic load requirements, you can put your base load on bare metal and your excess load on cloud.
I don't understand why people on HN always put forth a false dichotomy between cloud and running your own data centre when there's a plethora of different mixes of infrastructure and managed services that falls in between.
This analogy somewhat fails given that buildings and equipment are a generally fixed cost/asset, whereas compute power, storeage, etc. are probably more of marginal costs for a technology company such as Lyft. It would suggest they also contract out most of their technology development as well.
I thought that was a tax law thing:
Real Estate Income Trusts don’t pay any tax on their distributions, and nor do you (immediately) if they’re in a tax-deferred or tax-free account.
In other words, holding real estate in a Corp that does other stuff isn’t efficient.
>That's the same reason billion dollar companies rent buildings instead of owning them.
If you've got billions then you can create your own limited liability company, poach a bit of top talent to fill it (overpay a bit if you must) and get a decent operation going. One that will jump when you say jump no matter what.
You can't replace AWS global scale, but for your rental example its definitely possible. Companies rent mostly due to tax & liability reasons from what I can tell.
> Then they slowly turn into a datacenter company and lose sight of being a ride sharing company.
That assertion makes no sense at all, particularly if we acknowledge the fact that they are in the business of providing a web service. IT infrastructure is critical to Lyft's core business.
Would it make any more sense to criticise Lyft for hiring developers because that would mean they would slowly turn into a software development company?
Not sure what all the downvotes are about, but spending a bunch of money on your (virtual) datacenter isn't a new or strange thing. Whereas Lyft might not spend it on processes that deal with physical products, they do have a much larger amount of connected clients and data processing.
While in theory you'd "just need a database and some REST API" it is never as simple as that. Say you have one set of systems for production, you may want one or more duplicates for engineering purposes. And then you'll want tools to managed those systems, and tools to manage those tools. Then there is AAA, versioning and storage, and you'll have some sort of forensic/auditing log.
Up to some point, what makes a system expensive isn't the one set of parts that make production, that is just the tip of the iceberg. It's that you need everything else as well.
So regardless on whether you are doing a relatively simple service (getting people from A to B), or doing buying, sales and logistics for retail, which isn't rocket science either, you get the same initial cost and overhead.
> In 2017 and 2018, certain of our named executive officers provided rides to riders using the Lyft platform in a similar manner as other drivers. We believe that these driving activities provide the named executive officers with substantial practical insight into how our platform serves drivers.
I thought this was a pretty interesting point. I was about to call it dogfooding but not quite, since it's more of an experience check than a crucial internal usage.
I had one of the VC investors in Lyft pick me up for a ride a few years ago in Menlo Park. Had a super interesting conversation - I was impressed they were scoping out their investment directly. I wasn't sure what the protocol for tipping as at the end, hah.
That's really cool. How did you know they were an investor? Did they introduce themselves, or did you recognize them?
I imagine when investors or execs give rides, they probably don't generally reveal their affiliation, since that may skew the experience and the feedback. Though on the other hand, I suppose it could be helpful to say "I work at/with Lyft, how do you like the app?"
The same happened to me when I took an Uber. The driver disclosed it immediately that he is an investor after we started driving. I think it makes sense because it allowed him to ask very specific questions without it being awkward.
Airbnb does this too. They give all of their employees an annual stipend to travel via Airbnb, so that they can regularly experience the app from the perspective of a guest. I think they also provide benefits for employees who host guests, for the same reasons.
When you think about it, it seems so obvious that companies should do things like these, yet it still seems so rare. It's easy to fall out of touch with your users and product.
Wouldn't the equivalent be incentivizing employees to rent out their own place on Airbnb? I'm sure that plenty of Uber and Lyft employees are customers without it being a Big Deal.
Yes, that'd be the equivalent to this specific situation. Airbnb incentivizes that as well. I don't remember the exact details, but in an interview with one of the founders, he said they have a program for employees who host Airbnb guests.
I have some friends at AirBnB and I've heard it's much more modest, like $300/yr or something. My experience with these type of "dogfooding" credits is that companies are much more generous when they're smaller, so it could be that older employees get more than newer employees.
That makes sense. $300 is enough to cover Airbnb housing costs for at least a few days in pretty much any location around the world, so they probably still get value out of it.
Still does. I grew up near Oak Brook, IL, which used to have the corporate hq of McDonalds, and still has Hamburger University.
I believe every franchisee is required to attend Hamburger U. There are a number of corporate-owned stores in the area where the a lot of the staff is white-collar professionals in training. Those stores are always amazing.
In general stores in the Chicagoland region are way better than stores elsewhere, and I think part of this is due to the fact that corporate sends managers around for training here. I didn't understand the "mcflurry machine is broken" meme until I took a road trip. I had a number of horrible experiences, including a 20 minute wait for a mcflurry that ended up having more ice cream on the outside of the cup than the inside.
This really makes me curious about what is served at the lunch counter at McDonald's corporate headquarters.
I used to work at the corporate headquarters of a company that owns several chain restaurants. The cafeteria there didn't have any of the chain food dishes, but was very high quality as far as office cafeterias go. They had a test kitchen there also and sometimes they'd give out free meals of the stuff they were testing.
There is a free McDonald’s restaurant in the corporate headquarters, but there’s also a deli counter which has fresh sandwiches, as well as a counter for other hot meals (chilli, chicken curry (UK) etc.
Back around 2009 or 2010, my roommates and I put a spare room in our Berkeley place on Airbnb. One night the Airbnb CEO booked the room and came over for the night. Sounds like the company had grown large enough that the co-founders had to move out of their apartment-turned-office. The CEO had not yet gotten his own apartment. He instead would stay each night in a different Airbnb rental and go back to SF in the morning for work.
As a former engineer at Lyft, looks like my RSUs would be worth ~2x my salary per year. Typical RSU grants are 25% of your salary per year, so those Lyft RSUs would have been a good return. But that's at a $18-25B valuation. I think $15B is more realistic given the losses and most recent round of funding. Lyft is in a tough industry. Kudos to Logan Green for getting this far. Good to see a UCSB alumn do well.
I don't see how these companies will stop losing $1b+ a year each year. The public markets will not be too kind.
The end game was supposed to be autonomous taxis (cutting the driver out). I don't see how that's going to happen before they run out of money unless they 1) significantly raise prices or 2) take increasingly bigger cuts from drivers.
Right? Even if autonomous taxis _is_ their endgame, why couldn't companies that actually produce the cars do it cheaper? Almost all of them are heavily investing in it right now, some are even partnering up with companies that know how to do a lot of it.
I don't see how this works out for Lyft or Uber. To me it just looks like they'll both eventually run out of money and get squashed. Maybe I'm missing something?
Auto companies are not service companies, those are very different things.
That said, given the dynamism of markets, there's nothing to indicate that Lyft/Uber will have any huge advantage when the time comes.
But this is a game of musical chairs - early investors need to create the biggest, most miraculous but 'believable' story so they can pass the bag onto retail investors long enough to cash out.
If retail investors were able to do their homework, or rather, if their advisors at Morgan Stanley etc. were to do their jobs, I think that they'd see there is far more risk in these things than the valuations imply.
The problem is of course is that Morgan Stanley private wealth managers, managing for all those doctors, dentists, lawyers etc. only make money if there is buying action. And the emotional excitement of 'getting in on an IPO' is just too much to ignore.
The 'bragging rights' value of your dentist in Akron Ohio being able to tell to his buddies on the golf course that 'he has an 'in' on the Lyft IPO' (not really of course, he's at the tail end), is just worth more than a scrutinized deal.
Also - notice the PR/branding for Lyft, it's so funny, like the opposite of Uber - and yet they are for all intents and purposes the very same thing.
My thoughts exactly. My understanding is that Tesla plans on including a clause to prevent their autonomous cars from being used on other ridesharing platforms and simultaneously launching their own service.
I believe Lyft has significant financial ties with GM, who has Cruise, so maybe they'll be able to navigate it from a partnership angle.
Why couldn't companies that actually produce cars just rent them out? I'd really like to have a mono-brand short-term rentals, but they just don't exist. Why is the case different for taxis?
Currently renting cars could be seen as a distraction from car makers core business. The idea is self driving taxis becoming so cheap owning a car will be uneconomical. In other words they drastically have to revamp their sales product anyway. Worst case they'll be a the whim of very few big ai taxi companies... so cutting them out in the first place seems quite realistic.
FYI, Toyota bought a 500m stake in Uber. Some of these car companies just consider the internal combustion engine to be their core competency; then just outsource everything else. Usually to India, Japan or H1B bodyshops.
> I don't see how these companies will stop losing $1b+ a year each year. The public markets will not be too kind.
Are you saying taxis can't exist?
As far as I know, any taxi dispatcher take a similar cut (30%) as them and their cost seems way higher (no automation at all, require people on phone, etc..).
Theses loses are either because they are considered unlawful somewhere (I never heard of this issue with Lyft but I guess that's may be happening) and have to fight for it, or because they are trying to expands. If they stop both of theses (operating everywhere they are considered unlawful and stopping to expands) then their cost remaining are pretty similar to any Taxi dispatcher but they require much less staff.
The difference is taxi companies are profitable (or at least break even) -- and by virtue of necessity. There's no nationwide taxi company. Each tends to be local to their municipality. As such they can't absorb big losses and aren't subsidized by VCs or public markets. Taxis charge more than the service costs to deliver, Lyft and Uber don't.
Lyft and Uber have higher cost basis than taxi companies because they don't leverage economies of scale of car ownership and insurance via shared fleet as taxis do. Then they also charge less to riders. There's also no guarantee people would continue to use Lyft or Uber if they raised their prices to above the cost to provide the service, particularly when that number is actually higher than a taxi.
To my knowledge, Uber has a -61% profit margin. You give them $10 and they spend $16 to provide you the service.
Right, but the market for taxi dispatchers is a much smaller market than the market for taxis. Riders aren't the customer, drivers are. And price in this competitive market will tend towards a fixed monthly subscription cost, not a % cut of their rides.
The public will eat this stuff up if there's huge YoY revenue growth like they've shown in the S1. You see this all the time with public SaaS companies. Sure, the losses also increase... but nobody seems to care.
Nobody seems to care...until the company literally runs out of money. The difference between a SaaS and a Lyft is that Lyft has huge operating expenses. Burn rate is order of magnitude higher.
"Running out of money" is a long way in the future once your public. Most SaaS companies have huge operating expenses that exceed their revenue, most of it going towards marketing. Take a look at HUBS, NOW, WDAY... The list goes on. Lyft has huge expenses, but they also have huge revenue.
SaaS companies are basically zero marginal cost businesses. Uber currently operates with a -61% margin. Their marginal costs substantially exceed their marginal revenues.
But why do these companies have such huge operating expenses? It's a phone app for goodness sake. They're not fronting or maintaining the cars, insuring the drivers, paying any pensions or benefits to their workforce, etc.
These are just pyramid schemes disguised as companies.
Are you kidding? The driver acquisition costs are extremely high. Driver turnover is high. There's much higher support costs on both driver and rider side than a typical "pure" software co.
It's only high because they are in a money burning contest with a swath of other VC funded gig companies. There's nothing inherent about their business model that requires extremely high driver acquisition costs.
What do you mean "nothing inherent." Turnover is high because pay is low, so they need to constantly recruit new drivers via signup bonuses that pad their earnings for the first X months. If they fail to attract drivers then their growth will tank because supply will not keep up with demand. Support needs are naturally high and things go wrong all the time because you're dealing with real people in the physical world - it's not just some bugs here or there on a computer screen.
Companies like Lyft/Uber also have a much higher % of their full time staff in "ops" roles that are driver-facing (support, onboarding, offboarding, marketing, acquisition, etc.)
So long as their business is extracting maximal fees from each fare (thus keeping driver pay low) this cycle will go on as long as it can, and acquisition costs will continue to be high.
There's nothing fundamental about a ride share company that requires high driver acquisition costs. They are a result of a bunch of companies trying massively grow in the same space. Once Lyft stops trying to grow so rapidly and the industry settles they will not have to spend as much on driver acquisition. Indeed, it's already happening as their cost of advertising as a percentage of revenue is dropping dramatically.
> Support needs are naturally high and things go wrong all the time because you're dealing with real people in the physical world - it's not just some bugs here or there on a computer screen.
Why do you think support needs are naturally high? Higher than say what Ebay provides to sellers or what Dropbox provides to their enterprise customers?
>Companies like Lyft/Uber also have a much higher % of their full time staff in "ops" roles that are driver-facing (support, onboarding, offboarding, marketing, acquisition, etc.)
Higher than who? And what are you basing that on?
>So long as their business is extracting maximal fees from each fare (thus keeping driver pay low) this cycle will go on as long as it can, and acquisition costs will continue to be high.
If it does, that's only because it's more profitable for Lyft to cycle through drivers than pay more to retain them.
Lyft doesn't provide any services. They connect riders with drivers and provide the technology to make that work. Whether or not that merits a SaaS label isn't really the point. The point is that Lyft has the same cost centers as SaaS companies.
So that comes out to be about 8x multiplier. If that's something given 8 years ago then you would just break even if you had gotten a 25% raise in your first year. Of course you might have gotten more raises through out those 8 years s you may have come out ahead.
This isn't accurate. He has another ~6M shares of class B, look at footnote (1) for his holdings:
"Consists of (i) 4,663,809 shares of Class B common stock held by El Trust dated August 3, 2015, for which Mr. Green serves as trustee, (ii) 675,564 shares of Class B common stock held by The Green 2014 Irrevocable Trust dated June 12, 2014, for which Mr. Zimmer serves as trustee, (iii) 360,979 shares of Class B common stock held by The Logan Green 2016 Annuity Trust, for which Mr. Green serves as trustee, (iv) 360,979 shares of Class B common stock held by The Eva Green 2016 Annuity Trust, for which Mr. Green’s spouse serves as trustee, (v) shares of Class B common stock issued pursuant to the Founder Option Net Exercises and (vi) 1,180,329 shares of Class A common stock underlying RSUs for which the time-based vesting condition would be satisfied within 60 days of December 31, 2018 and assuming the satisfaction of the performance-based vesting condition. Subsequent to December 31, 2018, a portion of the shares described in this footnote were transferred between the trusts described in this footnote for estate planning purposes."
This is incorrect. The S-1 shows asterisks for the co-founders and are likely just placeholders. In the previous S-1, it showed that they owned roughly 7% together.
This is something that always strikes me about the amount of money swilling around in tech. $90M is an absurdly huge amount of money. By absolutely any outside objective measure of work put in to payoff it is off the scale. To look at this as the founders having lost out is almost comical.
That's why I said outside objective measure. By any objective standard 90M is an insane amount of money for one person to have. One billion is so far off the scale it is impossible to describe.
I’m making the point that you have two cos whose main US product is virtually indistinguishable from one another. The founder from one became a multi billionaire and the others are 1/20th of the way of becoming one.
But you're injecting that point into a discussion about a different point as though it provides more insight. It's not particularly interesting; we get it, they're fabulously wealthy. But no one contested that, which is why several people are trying to explain that it's not what they were talking about.
An S-1 filing encourages relative financial comparisons by design and intention. It's not surprising that Lyft's founders are extremely well off now. What could be surprising is the degree of dilution they experienced. Those kinds of financial technicalities require us to engage in discussion that treats objectively fantastic returns in terms of relativities.
We're about four replies deep into a conversation chain by now. I would have imagined that if people didn't want to talk about my original observation they would have simply not replied to it! "Interesting" is an obviously subjective term, I find the relative measures that Silicon Valley apples to wealth absolutely fascinating.
If you had a billion dollars, would you immediately give away $900mm, since you appear to be arguing the difference is meaningless? Just because they're both a lot of money doesn't mean it's not important to the people who have said money.
On the other hand, the baseball player will create much more than that in value while Lyft has lost billions of dollars. If someone here is underpaid it's not the Lyft founders.
Lyft has enriched its investors far more than any baseball player could even dream of. You're just looking at accounting losses. But when this IPOs, early stage investors will have all made billions.
The current price is the market's expectation of value throughout the existence of the company. The market believes that these past cash flows are not indicative of the company's future ability to create accounting value. And you may choose to value a growth company by its historical cash flows, but the market doesn't.
Drawing viewers' eyes to lucrative television timeslots for advertisers, selling tickets to local stadiums, and selling merchandise such as jerseys and figures.
Those are all wealth transfers, though, not really value creation. The only value creation by the baseball players is the entertainment provided (which is definitely not nothing).
The baseball player brought joy to other humans, which is the whole bottom-line point of the entire economic system.
Lyft has burned piles of investor cash to give people artificially cheap taxi rides; the wealth-transfer is zero-sum and it's actually worse than that because their dumping distorts the real transport market (and exacerbates the negative externalities of cars). You could argue they've done some genuine value creation by being a more efficient taxi dispatcher, but if there was any substance to that then they'd have a profitable business.
There is always a greater goal to achieve that requires more money: 90M to retire? Sure, it’s plenty of money. 90M to do greater and better things? Not so much. Hell, even Bill Gates would gladly take more money that what he already has in order to achieve his foundation goals.
Your point of view is not objective, but subjective to how much money you need to have in order to do the things you are planning to do.
But they did lose out. You're confusing a comment about relativities for a comment about absolutes.
No one is saying they're not going to be well off, or that it wasn't a worthwhile use of their time to build the company. They're just saying the return is smaller than it could have been. Your "objective outside measure" isn't enlightening in that sense, because the point is specifically about relative measures.
Responding to a discussion about funding dilution by saying, "well they're well off anyway!" is kind of odd, because that's not really relevant. Dilution also materially impacts non-founding employees, and small changes in dilution could have outsized impacts on their returns.
It's also comparable to negotiating with a company who tells you that you're still getting a lot of money "by any objective measure" even if they won't meet your ask, because their offer is higher than the median wage for your locale. Yeah, sure, but that's a pretty empty observation isn't it?
The founders (theoretically) created $20B in value and you think $90M is sufficient compensation?
$90M is definitely enough to be more than comfortable the rest of your life. But a $5B payout would have meant they could start a VC firm, invest in the next several generations of startups, partially self-fund something ambitious like a Space-X, start funded non-profits, etc.
> The founders (theoretically) created $20B in value and you think $90M is sufficient compensation?
...yes? I don't really see what's so absurd about that idea.
It also seems more than a little disingenuous to suggest that the founders were the only ones that created that $20B in value. They didn't single handedly create the apps, the marketing platform, drive the cars, etc. etc.
> In 2009, Sequoia Capital led the $2 million investment round into an entity of Y Combinator which would allow the company to invest in approximately 60 companies a year as opposed to their previous 40 companies a year. The following year, Sequoia led a $8.25 million funding round for Y Combinator to further increase the number of startups the company could fund.
I’m as pro founder as it gets but they alone did not create that value. Tons of employees, and the investors who put up the money, helped along the way.
Elon invested more than $100M in SpaceX and they came very close to death before they finally succeeded with their last rocket. So it's not clear that you can do all of those and more, easily, with $90M.
You must be joking. $90m, even in the valley is massive, generational wealth. It is the top 1% of the 1% if not higher. You could buy some of the largest mansions in America. It puts you around the wealthiest 30,000 people in America.
Idk. I’m very bearish on Lyft. It’s a pure bet on US ridesharing. They didn’t expand internationally (now those markets are saturated) and didn’t get into delivery (Uber Eats alone is worth 5-7b).
Do you think ridesharing is just going to die as a product or do you think it will become easily commoditized and all margins will essentially disappear? I think it's going to be around for at least another 10-15 years, and considering how much money uber and lyft had to burn to get to where they are now, I feel like it will be quite hard for competitors to capture relevant amounts of market share in the US
The business is clearly not sustainable..it’s not that there’s changes in demand but the company will literally run out of money in a very short amount of time unless something drastically changes with their business model
Unfortunately that's not what matters to Wall Street or VCs. They could arguably have scaled slower and made sure each market was profitable, but since everyone wants "growth over everything", they were forced to scale as quickly as possible.
They are actually successful in the minds of VCs because their revenue has been growing.
I know a couple of bootstrapped founders running businesses with $10M+ ARR. They own the business in full. Any exit event would net them the same figure as Lyft's founders.
Makes you wonder if raising money to run a business like Lyft is worth it from a personal financial perspective. The bootstrapped founders I mentioned are extremely satisfied with no outside interference or investors breathing down their necks
But then again, not everyone can build a $10M ARR business
I mean if I went all in on crypto from the start and sold around end of 2017/beginning 2018 I would have been a multi billionaire. But that's not the point. Similarly, if I "only" bought far OTM option on FANG between 2010 and 2017 but not march 2014-2015 I would have also been a multibillionaire. Heck, I could have went long volatility at high leverage in February 2018 and be a good way to a billionaire. The point is, of course, if you make "good choices" one can be a multibillionaire in many ways. Alas, I am not a multibillionaire because "good choices" are good mostly in retrospect.
The S-1 notes a lot of RSUs and options outstanding, that I assume are owned by the founders (and employees). So I assume when those kick in it'll be more than that.
Also suspect there has been some secondaries where early investors, founders and early employees have sold some of their shares to late stage investors.
> Secondary sales is existing shares changing hands, there's no dilution
Sorry for being unclear. I was positing an alternative mechanism, apart from dilution, through which the founders could have ended up with a small share of the company.
Not as bad as snap but what could they possibly be spending $100 million a year on?