I appreciate the difficulties of getting an app to fit in a fixed space having worked on a point of sale system that had to run on a MIPS device with 32MB of Flash, of which half was occupied by Windows CE. It's not really clear to me why Uber's app needs to be so large and complicated other than the availability of a large amount of money, high-powered engineers, and shiny new unproven technologies.
(The compact app fit in about eight megs in the end, plus another meg or so for the file which defined all the screens; it was written in the least trendy, unsexiest framework known to man, Windows MFC in C++)
The “there are only a few screens” is not true. The app works in 60+ countries, with features shipped in the app that often for a country, and - in rare cases - a city.
The app has thousands of scenarios. It speaks to good design that each user thinks the user is there to support their 5 use cases, not showing all the other use cases (that are often regional or just not relevant to the type if user - like business traveler use cases).
Uber builds and experiments with custom features all the time. An experimental screen built for London, UK would be part of the app. Multiply this by the 40-50 product teams building various features and experiments outside the core flows you are talking about (which core flows are slightly different per region as well).
I worked on payments, and this is what screens and components are in the Uber app:
- Credit cards (yes, this is only a a few screens)
- Apple Pay / Google Pay on respective platforms
- PayPal (SDK)
- Venmo (SDK)
- PayTM (15+ screens)
- Special screens for India credit cards and 2FA, EU credit cards and SCA, Brazil combo cards and custom logic
- Cash (several touch points)
- AMEX rewards and other credit card rewards (several screens)
- Uber credits & top-ups (several screens)
- UPI SDK (India)
- We used to have Campus Cards (10 screens), Airtel Money (5), Alipay (a few more), Google Wallet (a few) and I other payment methods I forget about. All with native screens. Still with me? This was just payments. The part where most people assume “oh, it’s just a credit card screen”. Or people in India assume “oh it’s just UPI and PayTM”. Or people in Mexico “oh, it’s just cash”. And so on.
Then you have other features that have their own business logic and similar depths behind the scenes when you need to make them work for 60 countries:
- Airport pickup (lots of specific rules per region)
- Scheduled rides
- Commmuter card functionality
- Product types (there are SO many of these with special UI, from disabled vehicles, vans, mass transport in a few regions etc)
- Uber for Business (LOTS of touchpoints)
- On-trip experience business logic
- Pickup special cases
- Safety toolkit (have you seen it? Very neat features!)
- Custom fraud features for certain regions
- Customer support flows
- Regional business logic: growth features for the like of India, Brazil and other regions.
- Uber Eats touchpoints
- Uber Family
- Jump / Lime integrations (you can get bikes / scooters through the app)
- Transit functionality (seen it?)
- A bunch of others I won’t know about.
Much of the app “bloat” has to do with how business logic and screens need to be bundled in the binary, even if they are for another region. E.g. the UPI and PayTM SDKs were part of the app, despite only being used for India. Uber Transit was in a city or two when it launched, but it also shipped worldwide.
And then you have the binary size bloat with Swift that OP takes about.
Like in the app, "payments" sounds like it would be a simple Stripe integration but it isn't. For one, that's way too expensive at Uber's scale. They build their own payment integrations to save fractions of a point that adds up to tens of millions of dollars in saved expenses.
There's real time systems to facilitate the trip, then the order processing systems to process that raw data, hydrate it, store it in multiple data warehouses, and publish it to Kafka to be consumed by literally hundreds of other services. The main money pipeline involved many services covering things such as rider incentives, driver incentives, rider fraud, driver fraud, tax calculations, rider payments across all of those aforementioned payment methods, receipts, invoices, driver disbursements, rider support, driver support, and more.
Then take in to account that almost all of those services were written for a world designed only for taking passengers from point A to point B with Uber Black but have huge added bloat to accommodate:
* Uber X
* Uber Pool
* Uber Eats
* Uber Freight
* Uber Moto
and proof of concept work for all sorts of new business ventures like Uber Elevate and ATG. All of that cruft is expensive to maintain and all of those actual product lines started as experiments.
When I left in 2019, there were two versions of the vast majority of those systems in production as all were being rewritten to handle the significant changes to Uber's business model from inception to now.
Additional edit: These services also require massive numbers of server hosts. All of the old stuff was written in Python, all the new stuff is in Go. That alone is saving Uber tens of millions per year just in hosting costs.
Uber PMs and engineers should be super proud. The fact that each user thinks that their one use-case is "all the app does" is fucking brilliant product design.
I've had over a thousand Uber rides (used it to commute) in 10+ countries and it's obvious there the way it seamlessly adjusts to each region but even otherwise I think I could have guessed just from the difference between SF and Seattle. There was a sort of joy of opening the app in a new region and finding unique options. Some sort of (maybe intentional) superstar-level product design.
Rickshaws in Seattle? Cash payment in India? Uber Taxi in SF? Sweet!
Alternatively: The fact that they try to bundle all the world's use cases in some 100+ MB monstrosity of a client app is a tragicomic commentary on the misaligned incentives in the tech world's VC-driven wannabe-monopolies.
What happens when a user who travels unexpectedly discovers they need to download a new version of the app for where they are now? And what if they're on a low-bandwidth or capped cell plan and can't download a 100MB app easily, where 3/4 of the data is duplicates of stuff in their other 5 copies of the app?
What happens when a user tries to book an Uber from one area with a special app version to another area with a different special app version? This could easily generate tens of thousands of permutations.
What happens if the user is physically in one country right now, but is a national of another country and wants to pay for their Uber in their usual currency and payment method?
If the user ends up needing to have 20 copies of the app for various locations, how well will they sync ride history, changes in payment methods, and other such info?
I don't even work for Uber, and I thought of all of these complications in 10 minutes. The real Uber probably has 100x more complications that would arise from splitting location-specific functionality into location-specific apps.
It's not about creating 10,000 different permutations of an app, but downloading/installing content on demand.
If we are afraid that there might be problem that user later on has not enough bandwidth or issues like that, we'll start immediately downloading/installing after
initial bundle in order we think this user might need them.
This means much faster initial usage, much less bandwidth used in total.
Same with developing experience. You shouldn't have to compile code that is not important for your development exp. This way compiling wouldn't take much time, dev productivity would be increased tremendously.
For Android purposes: maybe it's allowed in the Play store? I'm not sure tbh. There are certainly other stores with other rules (and Uber is in many of them too), or they could distribute it themselves (but we can see how well that has worked for e.g. Fortnite), but you still need to handle any scenarios where it's required.
In this case it could easily be reused within both Android and iOS without having to create duplicate business logic which I think already is a nightmare.
Airports logic I think should still be something that is defined using JSON/XML or w/e and algorithms should handle parsing that.
Or come on, there must be a way to get airports logic in an easier way. Does user have to update the app every-time when there's some minor airport update?
If it does, say you made a generic shim, you're effectively downloading and executing dynamic code. Maybe not by a technical definition, but that's not the point of Apple's rule, nor will they care about technical arguments when they reject it. There are plenty of examples of this happening in practice, I'm not talking hypothetically - it's why they have a carve-out explicitly for education apps. Even if you don't do stuff like that, and stay entirely in the webview, PWAs (unless it's actually being run in the browser, of course) are not allowed to make significant changes without going back through review.
edit: you can absolutely shrink your binary size with a PWA or similar though, yeah. There will definitely be other tradeoffs you have to make though, I'm not deeply familiar with those. Performance and integration with native libraries at the very least (e.g. map rendering).
Granted, you can/should strongly isolate them so it's infeasible for one module to interfere with another. But as always, YMMV in practice. And good luck testing them all either way.
Speaking of the defaults, the auto-suggests these guys have are fucking ridiculous too. Both Lyft and Uber. It's 4 AM in the morning and I've opened the app at home: offers me a ride to SFO. It's 11 PM in the evening on a Friday/Saturday and it offers me a ride to my friend's place. It's midnight and I'm at that friend's? They offer me a ride to my favourite club. I am one-click touching everywhere. These guys are good.
Superb user functionality.
On the other hand, the not so skilled ones...
* Uber being a simple CRUD app
* Uber having no moat because it can easily be disrupted by those able to build the app over a weekend, because Uber is a simple CRUD app
* Uber continuing to operate solely because VC's currently subsidize every single ride
Free space in the middle: drivers have no clue as to their costs and are only signing up because they're too naive to realize they're losing money.
Uber works relatively well in every country you go to, including at one point, China. Not many apps have ever accomplished that.
Uber made the right choice.
I'm from EU country and it was so convenient to be able to use Uber in NYC the same way as I would in my home town.
Same with dev experience. You shouldn't have to compile things you are not using to dev.
This would give pretty much infinite scaling.
I do think Uber requires tons of engineers, but I disagree that all of this has to be downloaded immediately or bundle size has to be this big.
All of what can be logic wise should be server side. For instance Airport logic seems like one.
Pretty bad UX, it's clear it wasn't designed for this use case, you have to go into Settings, but it works if needed.
But basically the only time I use Uber is when I travel to another region. I don't want 30 different Uber apps on my phone...
At this point it's really hard to believe the argument that "but our app is more complex that you could possibly understand" is valid. Compare your list with a Linux distribution from a handful of years ago: do you have more "screens" than that? Can you do more?
It's a relatively well known fact that multi-team corporate projects have tech debt. There was a thread recently about Google Cloud being slow due to too-many-cooks-in-the-kitchen syndrome, and many ex-amazon folks say AWS UI is notoriously difficult to refactor for similar reasons.
And there are cases, even in the Uber app, where thought has been put into optimizing both for upfront download size and internal complexity (the legal pages, for example)
Something that is worth considering is that a lot of what was said about the app being complicated is that it is also _dynamically_ complicated. For example, in the driver app, a feature was added to ensure drivers are wearing face coverings. The backend for that does AI-based fraud detection to prevent shenanigans like photos of photos. The picture upload flow is just one small part of a much more complex project that had to be rushed out because the covid pandemic happened. There are a bunch of similarly large impact things that touch several of the Uber apps simultaneously, often in non trivial ways and with tight timelines (e.g. features for compliance with specific legal requirements)
The reason I think Linux seems like a bad comparison is that if Linux contributors don't want to support some random driver for 3 years, that might be annoying but is generally just the way it is. Uber can't really afford to not do some of the complicated things it does in a timely manner.
I understand that Linux doesn't necessarily always have the product marketing deadlines or whatnot that Uber might have, but it's saddled with legacy code too. When things have security issues, or the API is just awful, at some point someone needs to fix it. Especially if your business is being decimated by a bug or you are lacking in APIs that meet your usecase. I don't think it's as poor as a comparison as you might thing it is.
Note how in this story the Android and iOS teams started a big rewrite at the same time, but the story consists exclusively of fuckups on the Apple size. Apple impose an arbitrary cellular download limit. Apple designed a language that can't handle more than 9 dynamic libraries before it hits scaling issues. Apple prevent you from using interpreters or other techniques to change the app after it's installed, unless you use a WebView in which case it's OK for mysterious reasons. Apple don't help even when major brands are weeks away from wrecking their business by hitting these limits. Apple Apple Apple. Meanwhile the Android guys are sitting pretty with Java and now a migration to Kotlin, which is (a) a much better language than Swift and (b) is dogfooded at large scale on the IntelliJ project so you know it scales (had no scaling problems with Kotlin and I've been using it for the last five years).
Apparently all this location-specific business logic has to be in the app itself and is not downloadable on demand.
Edit: https://news.ycombinator.com/item?id=25376346 goes into more details.
It doesn't have nearly as much functionality as gregdoesit mentioned, but it's enough to service the use case of getting you from point A to point B when you're stuck w/ crappy connectivity.
I've worked on similar problems where "everything is a special case" in some vein and it takes a few very clever minds to construct the algorithms and datastructures that can come to a good solution. But that's just it, it's the work of a few engineers and not thousands writing if-else blocks for each airport.
I still don't understand what Uber does that requires so many people getting paid so much money, the only thing that makes sense to me is they don't want them working for Lyft. It would be helpful for people to shed more light on how engineering resources are spent and what particular systems require so many hands to realize, but that's too valuable information to leak I'd imagine.
Now create a driver profile and do the same thing.
Now create the multitude of other special profiles they have and do the same.
Now do the same for Uber Eats.
Now do the same for Eats but using the restaurants version of the app.
And the couriers.
Now get in on Uber Freight (and any other projects they have) and do all of the same.
Now do all of that again for their other platforms (iOS, Watch OS, Wear OS, Android, Web)
Now access their internal tools that make all of this possible and do the same.
Now think about localizing all of that. Handling special cases for different places/languages/legalities. Think of all the different payment integrations. Security issues. Infrastructure. Technical debt. AB testing. Metrics. Dev tooling.
And I've only begun to scratch the surface here. The point is there is a lot more than meets the eye, especially when operating on a global scale.
Or am I crazy?
My point is that there is 100s of screens/features in the Uber app that you don't even know exist. And you have those same 100s of screens across their other apps. Multiplied across all of the different platforms. Plus any internal tools they use.
I'm curious, have you worked at a software company of Uber's scale? Not trying to be a dick here, genuinely curious because I hear these types of comments often here on HN and I wonder how many of those commenting have seen the scale from within.
"100s of screens" is as meaningless to me as "thousands of lines of code" and wholly uninteresting because it seems like the front end is the least significant reason why Uber requires so many people to operate. That's why I'm interested in seeing how many people are assigned to projects and what they actually spend their time doing, because at scale you begin to experience inefficiencies due to scale for bad reasons if your incentives aren't aligned. From the original twitter thread that sounds like it was rampant at Uber.
Not sure how long you've been around, but "Uber scale" used to be a joke a few years ago because no one could understand why they would reinvent so many wheels because "they don't work at Uber scale." The joke being that Uber didn't need to be Uber scale to begin with, they just had investor money to burn.
1. The amount of engineers scales very badly with respect to application complexity. I.e. you may need 100 or 1000 engineers to do something that "looks" about 2x as complex as something 10 engineers can do, and
2. The more complex solution tends to win in the market because on the one hand people value the application handling edge cases better, having shiny graphics, having slightly better UX, etc. and on the other hand you redeem the cost of paying those 1000 engineers thanks to software being free to copy and network effects / monopoly.
You're not answering the question except as a tautology:
- Q: Why is Uber's engineering team so ridiculous and bloated?
- A: Because they need a ridiculous bloated team to make this ridiculous bloated app.
Companies grow into new products, services, markets, etc.
Microsoft and Amazon have multiple profitable lines of business.
Is Uber more like Google or more like Amazon? Is "Uber Eats" (underpaid contractors drive around takeaway dinners) distinct from "Uber" (underpaid contractors drive around people), or "Uber Black" (underpaid contractors with unwise car leases drive people around)?
It's not like people would go and visit a page with just a list of ads; no matter how little revenue the other products around ads bring in, they bring eyeballs to the ads, and serve as a moat around ads.
It's not like people would go and visit a page with just a bunch of ads; no matter how little revenue the other Alphabet products bring in, they bring eyeballs to the ads, and serve as a moat around ads.
Having a large number of engineers is fantastic for making the company valued higher.
Directors love having more engineers under them.
Overall I've been highly speedy with 4 engineers, and slowly adding a 5th. Sure there are limits, but I think limits makes us think about being more efficient with how we build. It is an entirely different mindset.
Once you decide to "go big" you end up in this situation where you usually literally throw a dozen engineers at the problem and the problem goes away. I've seen designers spend 6 months on a _single page_. Because they can. and they get paid crazy money for it.
With an operating system UI, the developer may reasonably know what the UI will look like, what features are present, where they are placed. With Uber, a single person may not know all the rules for a single feature, and the rules change frequently, sometimes in the most obtuse ways. Like when San Francisco decided to ban cars from one of its most important artery streets earlier this year (market st). Now not only you need to indicate to a rider that this is a thing, your UI also needs to to provide walking instructions to the nearest designated pickup spot, and the routing algorithm needs to not put drivers on market st. Oh, and don't forget Hebrew speaking tourists read right-to-left and that food delivery bicycles _are_ still allowed on market st. That's just one change in one city.
Consider that just about anything might change anywhere, any time. From how earning breakdowns are displayed for drivers in Sao Paulo (this differs by jurisdiction), to where the pickup locations are in Pearson airport in Toronto (airport rules change all the time), to what promos apply in Mexico City this week (which depends on marketing and finance analysis), to whether surges should be disabled in response to a terrorist attack, to what documents a driver signup rep should be scanning into the system as local regulations change.
Now build all the infrastructure required considering that the people on the ground reporting their local changes are not programmers, and conversely many of these changes may require making changes to the UIs that these people use, how the new data is aggregated, etc. There are literally thousands of greenlight hub / customer support employees worldwide using these internal tools, many don't speak english and timezones are all over the place. The nature of problems are often fuzzy, hyperlocal-specific (e.g. certain forms of fraud), or extremely meticulous (e.g. legalese, etc).
The trouble of scale comes with having to deal with numerous nebulously defined things from numerous stakeholders and having to distill which engineering teams are even responsible for which aspects of which thing. As the number of stakeholders grow, so does the risk of miscommunication, unclear chain of responsibility and other well-documented large organizations challenges.
I don't reject that there are troubles in concerting real world data into a representation for the constraint solver. But this isn't untrodden ground, and there's no reason a front end developer should need to care about it. The problems you're describing are far behind the UI; and if they're not your organization has immense problems.
Well, yes and no. You're right that trying to express every nook and cranny of the business rules as solely a UI problem is a wrong way of trying to understand the complexity. But a frontend developer does need to care about how to surface whatever complexity needs to be surfaced, and likewise, backend developers need to know how to model the complexity, and there needs to be some level of orchestration between all the moving parts. Remember, Uber is built on a microservice architecture, and many things that would be arbitrary hardcoded values in projects elsewhere (e.g. pricing) are instead tied to complex dynamic systems in Uber.
In the pricing example, when Uber added the option for drivers to set their own multipliers, it wasn't just a new slider in the driver app. It also affected the pricing AI model, the matching algorithm, how and when prices are displayed in the rider app, geofencing rules, etc etc.
When new regulations mandate that every commit that makes production changes in your city must have an appropriate audit trail, when you need to deploy a cheat-proof face mask detection AI model in response to a global pandemic, when you need to figure out how to make customer support cost $1 per engagement instead of $10, when you need to figure out how to make your network protocol faster in Bangalore, when you need to figure out how to make GPS more accurate in Manhattan, or when any of hundreds of fuzzy problems need solving, then saying that variations of all of these problems have been seen elsewhere misses the point that going from problem statement to deployed solution isn't always a trivial task.
The original question was why so many people work at Uber. The answer is that there are a lot more fuzzy problems than meets the eye.
I never work for Uber but I can easily imagine the following problems:
* Map data is not a solved problem. Tons of inaccurate data especially on non-english speaking countries.
* As someone mentioned about payments, different country requires different payment platform integrations.
* Payment is just one aspect of money, you need to build a treasure-like logic to run a marketplace between drivers and passengers.
* The matchmaking algorithm itself is interesting enough problem. It has to be efficient for the company to make the most amount of money.
* Tons of internal data are produced and required to be understood. This requires ML/data analytic talents.
Most of what you mentioned should be server side.
I mean, you could push some of that to the client, but I'd understand why your app is gigantic and doesn't do app-ly job very well with all the background processing. And I'd be surprised the app hasn't been hacked to get at the analytics data, if not the payments.
> The app looks simple from the user perspective. But on the global scale the business rules are insanely complicated. Every region has custom rules/regulations. Different products have different workflows. Some regions have cash payments. Every airport has different pick rules...
A more positive take: if you look at the specifics of how many regions Uber is (was) in, and that each region has cities each of which have wildly different regulatory environments, you can imagine the permutations of complexity they had to deal with. Then you add in growing internal business experiments like Eats or mobility (Jump) and it's easy to see how this balloons out of control pretty fast.
It justifies the employment of 400 iOS engineers, it also lets you book taxi rides.
Apparently with Uber it’s the infrastructure design and back-end algorithms for trip routing and driver management that are “so great.” The SV hype machine helps a lot too.
They also have a huge need for speed. Shipping faster and winning a market is worth 10s of million in annual revenue and billions in valuation.
A lot of this is resume driven development as well. Solving a hard problem, even if it's an invented one, looks good on your resume. So does using the latest and greatest tech stack instead of a boring but functional one.
Problem here with Swift came down to
- Apple not dogfooding its own tech . Hell they did not even help us or other major companies like AirBnB and LihnkedIn (We formed a workforce on this problem with them which then forced the download limit to be upped by apple)
- Engineers deciding to adopt a language which they thought was great (and turned out not to be at the time) because they did not do the right analysis upfront for such a significant project. 'Wow look at this shiny new cool tool, lets use it' (Oversimpiification, swift was more pleasant to write with and you could be more productive which was no doubt a factor here)
Also, while management at the time didnt care for the remote offices much, I think the AMS team played a critical role in the success here. With the recent layoffs that occurred, I heard that they were fired.
Another fun story, we were trying to bring down our build times on our CI builds which were 1hr+ while battling macstadium and their crappy infrastructure with tens/hundreds of patches incoming each day.
Airbnb is 75% executables, 9.5% assets and 8% localizations. This is pretty different from Uber which is 60.7% executables, 26% localizations and only 3.8% assets. While the Uber executables are large, ~10mb more than Airbnb (note: this is only the Uber rider app, while Airbnb has host+guest in one app), their localizations seem to be driving a lot of the app’s 300+mb install size.
Looking at how the localizations are laid out in the app bundle, there are thousands of *.strings files in separate bundles, seems to be one for each feature. Many of these files have comments to provide context for the strings. An example from the app is: "Various screens show this as a title in a snackbar message when there is no network connectivity available". Just stripping these comments out of the localization files would save about 17.6mb of the install size. Another side effect of splitting localizations into so many files is there are over 23k duplicated strings throughout these bundles.
While the Swift code size is part of the problem with Uber’s app size, it's not the only way to free up space. There's a lot of work that goes into bringing a large scale iOS app’s size down, and not many tools to help. I left my job at Airbnb recently to work on making it easier to visualize, monitor and reduce app size, please email me (noah [at] emergetools.com) if you might be interested!
Lately Apple has been featuring size more prominently on the App Store, and it appears to be displaying the install size. This is also what users see when they look at which apps are taking up storage on their device in Settings. I think it’s important to reduce both, so users see a good number and downloads are fast/efficient. It is a very welcome change that binary encryption doesn’t negate all the gains from compression anymore. I had noticed it in the numbers but looked around for an official statement and didn’t find any, if you happen to know where this was announced I’d appreciate a link!
The Amsterdam team 100% played a critical role in solving the issue (at least until apple bumped the limit). No way we would have stayed under it without you all.
The OP indicates that a rewrite was necessary regardless. Even if Swift was not yet mature, do you think it was more worthwhile than rewriting in ObjC? Given that the goal was to "sustain mobile development at Uber for the next 5 years".
Secondly, I'm not a mobile developer so I don't have much context, and I'm wondering about your outlook on iOS development as a whole. Swift obviously had major problems but has supposedly gotten much better. You mention MacStadium's issues, but Apple build servers seem to be making progress with the recent AWS EC2 Mac reveal.
Is the iOS landscape turning a corner, or do you expect a new generation of bad developer experience?
I love Swift. But I've been adopting it in bits and pieces over the years. It's not just that the language has changed significantly, you also have to get used to the idioms. E.g. my early Swift code often featured Pyramid of Doom patterns, and both language innovations and better familiarity with the idioms have helped a lot with this.
Apple should have never pushed Swift as hard as it did at the time. Between the constant API changes and issues like requiring dynamic framework linkage and copying the whole runtime into every app, it was absolutely not production ready.
We didn't do a simple build scalability test until we were well into the project. If we did it would of revealed swift's build problems to us. Our swift experience is what slowed down our kotlin migration significantly. Today Uber android is still a majority Java app.
Even very recently the experience of working in swift compared to the old obj-c code base isn't as good. To this day there are debugger issues, xcode responsiveness issues and a slower build time.
Hiring on the other hand would be harder now, because it's getting hard to find people who know Objective-C and it's surface ugliness scares new people away. Swift is a decent language if you don't have to have a very large code base, so most people rightfully so don't have that much Objective-C, and if you were to work in Obj-C, you might as well go work at Facebook then.
I havent dabbled in mobile developer platform tooling for the last few years now so I cannot comment on the iOS side, all i remember is that back then it was a shitshow :) Mobile is still playing catch up to how backend infra and tooling operates, I suspect its better but probably still got a long way to go.
I hate objective C with a passion. It's like some archaic language of a bygone era forced to run on a cell phone
I can't wrap my head around things like pointers or memory management
With swift & obj-c your coding against the same library APIs, so in many ways it's a very similar level of programming. In many ways swift is far more complicated than Obj-C. With Obj-C iOS apps your not really writing that much C vs. the pseudo-smalltalk attachment to it.
> So said brilliant engineer in Amsterdam, built an annealing algorithm in the release build to reorder the optimization passes in such a way to as minimize size. This shaved a whooping 11 mbs off the total machine code size and bought us enough runway to keep development going.
> This terrified the Swift compiler engineers, they were worried that untested complier pass orders would expose untested bugs
This would scare me too.
> I had privately had the “we need to stop” conversation with my director. He told me that if this project fails he might as well pack his bags. The same was true for his boss all the way up to the VP. There was no way out.
This brings up an interesting point. The author attributes some decisions in the thread to the "sunk cost fallacy". The business costs were sunk, but if the employees and management are too afraid for their job to make the "cut your losses" decision, the costs are not really sunk, at least not in the minds of the right people.
> A bunch of people got promoted. We all breathed a sigh of relief. The 90 work weeks stopped for a few weeks.
When you move up, you move on; issues from then on are the next guy's problem. Exec dynamics are often "gather successes to me and push off failures to others" - sounds like some exec won.
> This would scare me too.
For what it's worth a few (10ish) years ago we did this with the J9 compiler at IBM (technically just the backed that was shared between Java, C++, etc). Same idea with simulated annealing. It did end up finding some really strange bugs, but they were bugs that were present regardless, we just hadn't found the right test case for it. We did end up getting some performance out of it, but not nearly as much as you'd expect. It mostly boiled down to running a few extra constant folding passes.
There's some upsides to that, but basically even though you're a W-2 employee, you're effectively a consultant in that arrangement. Your job is not secure.
This is a damn cool use of simulated annealing.
FWIW, I made it clear that this was also a great way to break the app :)
The story arc is almost always the same: starts out very promising and moves quickly, then at some point the wheels come off but everyone keeps going.
Anyway, Joel Spolsky nailed it 20 years ago: https://www.joelonsoftware.com/2000/04/06/things-you-should-...
Rewriting is neither a panacea nor a problem if you completely understand what you started with and what you will wind up with and be brutally honest. Trying to combine a rewrite with major feature changes is extremely hard to pull off (I've done it a couple times with iOS apps, not fun, but ours came out well) and adding a new language at the same time is probably beyond any team's ability. A rewrite for good reason can be done, but not with 100's of engineers and major features changes, all done in a massive hurry; that's a recipe for disaster.
For example, I work in embedded and I am proposed with a new "X brand" Chinese Modem to integrate in our devices. I don't recommend it but since it costs less I'm pushed to evaluate it. I am handed 3 samples and they all work fine, so management convinced themselves that Brand X from China is the way to go.
Once in production the modems started to fail at a 10% rate (10 devices over 100 won't work). And that is a problem you only see only when manufacturing a certain quantity of devices.
Now there are delays, an ongoing battle with the manufacturer of modems and a unhappy customer (that's why I didn't recommended it).
Also the article was published just a year before Microsoft pulled off what's maybe the most successful large rewrite and customer migration in software history: Windows/DOS to Windows NT (XP was where it became complete).
I've always wished for Joel's take on NT, especially since he was at Microsoft previously and it happened just after this article was posted. What's his opinion on its success when so many weren't?
It's a completely different system, that happens to support the same API/ABI, backed by long experience at Microsoft at making things backwards compatible (it's probably the biggest source of Microsoft's success).
Microsoft also put ~7 years into preparing the transition piece by piece, with probably the biggest impact point being Windows 2000 that provided significant host of features that compelled big (and loud in complaints) clients over, pulling a lot of software vendors along.
What else you would call it?
A full rewrite is not a Ship of Theseus thing. You're just going to reverse-engineer your own thing in another thing. Unless it is unusable (quality-wise or maybe the only current version was written in APL for Solaris, who knows) there's no need for that.
Can Swift talk to ObjC? Great. Then do that. Attack the critical areas first
Yes sometimes a whole new architecture is needed, but then maybe you needed to reevaluate your earlier procedures and/or hires.
Seems less like they weren't offering answers, so much as the answer wasn't "but we'll do some stuff and fix Swift!"
Though the 6 library limit and how that wasn't a gigantic red flag on the language is baffling.
Don't ask why, just do or die.
But yeah, I would have been pissed off if I was put in this situation by myopic upper management.
Maybe as a junior engineer, or an intermediate one. But not beyond that. Seniority isn't just being right more often.
I couldn't find that specific quote, though, and being 25 years back it could be a false memory of who said it (I did read it somewhere, if you were on the team at the time, please speak up!). Here are a couple of illustrations.
I used to be a game developer at EA. In the middle of one particularly nightmarish 60-hour-weeks-for-months crunch, a twenty-something artist with no known pre-existing conditions spontaneously died in his sleep.
We had one person go to the hospital with tachycardia and 3 people who were vomiting. There were only 10 people on the team.
As an engineer, I understand I'm a vehicle for success at a startup. Even if I ultimately get burnt out and quit. If I make important changes, that maybe only I could have thought of before I quit, it probably becomes a good investment to hire me.
Uber managed to hire enough of the "right" vehicles for success to get to the point where they shipped a platform rewrite at massive scale. By all high level objective metrics, this is probably success story.
The one metric I think could derail the projects success isn't really documented--what was the human cost to this effort? I bet you looking at a graph of turnover at Uber you could probably identify exactly when this project happened. I'm curious if the engineering turnover from such a massive effort was enough to offset the benefit.
This is one of the main reasons why I have no intention of going back to BigTech any time soon. I am sure the author is an extremely capable engineer but he is obviously missing the elephant in the room: that the root problem is app growth and feature creep caused by uncontrolled growth of the eng. workforce (who are all incentivized to ship their code because it's good for promotion). As if app growth was some kind of god-given fate that can't be changed! But instead of fighting the real (organizational, political) issues within the company, they blame it on technical issues (Swift is not mature! 100 MB app size limit is too low! etc.). Incredible.
The larger community benefited from our learnings.
No. The larger community would have benefited if you had written a story about how someone in the org. stood up to management, fought the feature creep, the crazy app growth and doubling eng. each year, and brought the whole project back to sanity. But that didn't happen.
So my advice. Everything in Computer Science is a trade off. There is no universally superior language. Whatever you do, understand what the tradeoff are why you are making them. Don’t let it descend into a political war between opinionated factions.
No disrespect, but no choice / tradeoff in the world would have led to a satisfactory outcome if you have hundreds of engineers burying this app under an avalanche of code day after day. The tools / tech are alright. The company is not.
In aggregate, the goal is obviously that it leads to the company making more money than was lost by paying all those engineers to move mountains to get around all the issues they hit up against. In the case of Uber, now a $96 billion public company and worth nearly double their 2016 valuation, it's a little ridiculous to call it a failure.
You read this story and see an unmitigated disaster, but it's mostly just a picture of things working the way the company wants them to. The goal is to grow the company, not to have a nice, conceptually clean, tech-debt free codebase just for the sake of it.
But, essentially, your final paragraph's point is the bottom line. This was not fundamentally a tech problem, it was a too-many-cooks-in-the-kitchen problem. That's a business decision.
There's also a bunch of big articles on VPs quitting during this time. It was peak Uber controversy with the sexual harassment scandals, Waymo stolen IP, TK resigning, Fowler, spying, backdoors, etc. Reading the headlines in the algolia search is like a telenovela.
Reflecting on one very, very strange year at Uber (Susan Fowler) - https://news.ycombinator.com/item?id=13682022
I am an Uber survivor - https://medium.com/@amyvertino/my-name-is-not-amy-i-am-an-ub...
Uber Founder Travis Kalanick Resigns as C.E.O. - https://news.ycombinator.com/item?id=14600873
How Uber Used Secret “Greyball” Tool to Deceive Authorities Worldwide - https://news.ycombinator.com/item?id=13785564
Suicide of an Uber engineer: Widow blames job stress - https://news.ycombinator.com/item?id=14200486
Uber's Design Meltdown - https://news.ycombinator.com/item?id=11109272
Uber employees used the platform to stalk celebrities and their exes - https://news.ycombinator.com/item?id=13163895
I realize this isn't a small ask. I've worked in BigTech myself where the reward / risk ratio of trying to actually solve problems at the root is basically 0 if you're not SVP or higher. Anyway, I enjoyed the tweets nevertheless. Good luck.
I really like this point.
One thing that could happen when making a decision like ObjC vs Swift is that the decision makers can think they have performed a sober risk/benefit analysis, but failed to dig deep enough to uncover show stoppers. So they may have looked at developer productivity with Swift and took a cursory look at build times for a small pilot, but failed to look deep enough to find some of the crippling shortcomings.
Related to the first point, a lot of the time with a new technology, the show stopper bugs/shortcomings/issues haven't been discovered yet. If you're choosing a battle-tested technology, most of the nasty edge cases and limitations have been discovered at some point. But with something new, there's a lot more uncertainty with what you'll run into once you put it into production.
That risk of surprises is higher with some new tools and technologies than others. But it's a big wildcard when you're betting your billion dollar business on it. Maybe that bet pays off, but you need to do some sober analysis on that risk of unknown unknowns and have some idea of how and when you'd make the decision to cut your losses.
But, there is a lot of space nowhere near that trade-off optimizing boundary. You can absolutely make languages worse in ways that are strictly negative for all users. For historical reasons, all languages contain some amount of that dead weight. So the absense of a perfect language does not imply that all existing languages are right at the edge of the optimizing boundary.
It hints at a rule-of-thumb I use: try to wait making an important technical choice until you can explain a pro & con of each option (or, similarly, explain when you'd use each one instead of the other).
Speaking for myself, I didn’t add any Swift to any codebase I worked on until after 3.0 shipped because it was pretty clear that the language just wasn’t quite ready yet.
I’m genuinely surprised no one ever thought to just email Chris Lattner or invite him out for a drink to say ‘hey, confidentially between you and me we’re thinking about rewriting our app in your new language. Are there any other apps with this order of magnitude of LOC written in Swift yet?’
Sure, if they exist. I'm not talking about the world as it is today in December 2020 with Swift 5.3. I'm talking about 2016.
Uber's culture of "letting builders build" is fine, as long as teams have the ability to change things early and quickly. I think top-down decision making and monumental bloating of eng teams made things very difficult over time.
When there's no room to come up for air and look at a project from a perspective other than "this is super broken and way behind schedule" my brain just fixates on the micro-steps that need to be done to get it across the finish line (whether or not the finish line is realistically anywhere within sight). It's work being done, but none of the exhausted people working on a project like that have any energy left to consider whether it's the right work.
> I had privately had the “we need to stop” conversation with my
director. He told me that if this project fails he might as well pack
his bags. The same was true for his boss all the way up to the VP. There
was no way out.
> With only a week left we decided eat the 8 figures and drop support for iOS 8.
I'll say this: If I were an Uber investor I would be pretty angry. And the story
just gets worse from there.
This is a story of gross incompetence. It's not an engineering disaster,
it's a management disaster. This is classic "inmates running the
asylum". ( https://www.goodreads.com/book/show/44098.The_Inmates_Are_Ru... )
This is the story of some hackers who got in good at a big company with
lots of money and sturm und drang so they had no adult supervision
while they partied while pretending to be actual engineers. (That they
were pretending to themselves too doesn't make it any better.)
The writing was on the wall, but rather than own up and take
responsibility they double down and keep messing things up for months
> This is when the real brilliant engineers started to shine.
This is when they get pulled from other tasks to clean up your mess!
(cf. Broken window fallacy.) Those real brilliant engineers should be
doing other better things than dark rituals to appease the Swift compiler
Look, we all make mistakes. There's a blog post from an engineer at a
company I used to work for complaining about the pointless middleware I
wrote years earlier. (I feel ya bro, I told them at the time that it was
pointless.) Here's the thing: I got out of that place when I realized
that the folks running it weren't really competent. (And, sure enough,
they got pushed out by the investors and now the company is a "Wallstreet
darling". So... Yay?)
I've burned enough investor money in my career that I don't think it's
funny anymore. (I never really did think it was funny, I don't like
waste, but I was young. REDACTED was only my second official job, and in
hindsight it was really crazy. I mean, they built a skating half-pipe in
the office! At the time, everybody thought that was real cool.)
And their Android app patching Dalvik at runtime to deal with the huge number of Java methods:
And they're not the only ones known for bloated apps, either. Take Amazon's apps for instance:
Which partly implies that the only solution is to not grow to that scale. Keep a smaller, more skilled team instead. In most cases I think that'd be better, and I think most people (and companies) would agree (if only they could hire those skilled teams reliably)...
... but sometimes throwing more people at a problem really is the best (or only) option, e.g. when you have to deal with a large volume of ridiculous external constraints. For example: my first job was at a company that aggregated loan providers, which entailed consuming hundreds of random PDFs per day and dozens of weird RPC calls. We had dozens if not hundreds of hand-crafted bits of automation to detect when things changed and handle the new format. Many APIs had hand-rolled XML parsers and producers... because the company we were calling would choke if your attributes weren't in a specific order, or if X wasn't within a specific number of bytes from the start of the request, or they returned invalid XML. It was absolutely ridiculous, but there's no way a bunch of banks were going to fix their APIs for us, so there's not much of an option but to do it by hand.
It was also, coincidentally, the company where I most respected the key technical decision-makers, because they had the will to discontinue a product that a few customers already were using, because it wasn't meeting its criteria.
One was attaching the code signatures (which gets attributed to the dynamic linker because it is synchronously stalling waiting for the the validation to occur).
The other was that reactive patterns tend to cause A LOT more classes and methods which result in a lot more fixups. When I am debugging apps I can often tell if they use react based one the ration of Objc/Swift metadata to __TEXT in the binary. That ended up causing dyld to stall waiting for page ins (which again are not really the linker but get attributed to it).
IOW, many of those clever patterns that make doing rapid development easier are really just a way to transfer work from the developer to the runtime, and it has a cost. Swift and SwiftUI are actually have some very clever ways to mitigate this, but reactive Swift libraries int he Swift 2.0 timeframe certainly did not.
I really loved it and continue to from v3 onwards though.
That’s the problem. Apple knows by now that their developer community laps up absolutely everything they put out and regularly gets burned. (I’m old enough to have rewritten code for Cocoa Bindings in 2004. It’s the next big thing! And garbage collection in 2008. It will fix everything! And GCD... and...)
For such an enormous change as Swift, they should have switched off the reality distortion field, and gone with a more sober, more academic approach.
Here's one account: https://tclementdev.com/posts/what_went_wrong_with_the_libdi...
Also most of the performance issues are with excess parallelism (runnable threads), not concurrency (hiding IO latencies).
LinkedIn adopted Swift back in 1.2. I would love to read some war stories of how things there have progressed over the years.
Source: https://www.youtube.com/watch?v=X9waDi787uo (14:15-15:30, 39:26-40:37)
At an interview with them a few years ago, I was told they had been moving towards rewriting the app in Objective-C.
They should have solved the architectural problem incrementally without adding a 1.0 language rewrite to the mix and then slowly rebuilt the app using Swift.
This is so obvious to anyone with experience it seems pretty evident that the decision making process there at the time was deeply flawed.
So easy with the absolutes :-)
- Performance (which was a major driver) improved 100x-1000x.
- Reliability (other major driver) improved ~100x
- Codebase reduced, features increased, habitability dramatically improved
- Resource usage redcued ~10-20x
What would you consider a success?
I'm following George Hotz's tinygrad that is a CPU+GPU deep learning framework under 1000 lines of code with great interest where all engineers are trying to shave lines of code while maintaining readable code (it's like a game when you set rules):
Here's the GPU ops part:
It's interesting to me that the author feels that things were great with Uber in 2016 pre-Trump and that Trump's election was the catalyst for the negative sentiment. Susan Fowler's famous post was early 2017, and maybe _posting_ it was catalyzed by the issues surrounding Trump, but clearly the serious cultural problems she described at Uber had been going on for some time before that.
I'm trying to figure out what to make about a lot of these mis-steps. On the one hand, many of them seem preventable: you could imagine identifying ahead of time that load time and binary size might be risks for the new app and scale-testing these very early in the process to de-risk it. You could imagine testing the new app long enough (and widely enough) to discover the location issues (drivers going to the wrong blocks, etc.). You could imagine foreseeing users being upset about suddenly needing their location data when they're not using the app. Obviously hindsight is 20/20, but is there also a pattern (that we see all the time, especially in Silicon Valley it seems) of not really thinking through the consequences of a lot of decisions, especially where things like user privacy are concerned?
For the other half of your comment: I personally think many companies have issues focusing on non-technical problems. From the thread, the issues were clearly that they were writing far too much code, it seems like they were not testing enough, they lacked a person who would tell them how their changes would be perceived by the public. This is not unique to Uber in any way, these problems show up to various degrees across companies such as Apple, Google, Facebook, Microsoft and others. And you can see the solutions: while highly technical and "awesome", they are obvious band-aids. It's just that it's much easier to slap on technical band-aids than fix the root, non-technical problem.
> I personally think many companies have issues focusing on non-technical problems...This is not unique to Uber in any way, these problems show up to various degrees across companies such as Apple, Google, Facebook, Microsoft and others.
I agree with all that. To me, this is about taking responsibility. Our industry has shown a pattern of thinking poorly about non-technical problems while rolling out changes that affect millions of people. That's part of why so many people are angry at the tech industry -- and for good reason.
The thread is thoughtful in a lot of ways, but I get a whiff of not-taking-responsibility through some of it. The first example struck me for whatever reason. I get that people in 2016 didn't necessarily know what was going on or how bad it was. But I would expect that in 2020, we'd look back at 2016 as the time when serious issues beneath the surface were about to explode as a result of choices made at Uber, not some golden time before external forces made life hard for Uber. I don't mean to be too hard on the author or even Uber (and after all, this was a casual tweet thread). I think it's a widespread challenge for our culture in the tech industry.
Admittedly the things I've worked on has been games and CAD systems so the internal data models and simulation are a lot more complicated than the UI, but I think that holds for a lot of apps.