Hacker News new | past | comments | ask | show | jobs | submit login

This is hardly surprising at all. There used to be lots of complaints, both internally and publicly, about how bloated and ineffective Uber's engineering organization was. Many engineers were bitter about many overlapping projects in Uber. It's also funny that Uber engineers used many bogus reasons to implement NIH projects. A typical example is that Uber had a team to implement their own Bazel-like system because "Bazel (or any other tool) does not scale in Uber-scale". Same went for their own GPU database, their own SQS, their own key-value store, their own resource scheduler, their own datacenter, and their own deployment system. The end result was that almost everything sucked. Engineers suffered productivity loss for years. I kid you not that Uber's docker system didn't support persistent volume for more than three years, so anyone who wanted to build a persistent system like Elasticsearch had to jump hoops and spent weeks even just to get machines. And the sad fact is, Uber did not have a "Uber scale". The traffic and complexity it had to handle was far less than other internet companies.

I wish they didn't hire so irresponsibly back in 2015 and 2016.

I worked at Uber from 2014-2018.

Can confirm this. It was a very poorly managed org because it was extremely grassroots driven—leadership was underempowered to say no to front line teams or hold groups accountable.

The overlaps and poorly considered projects described above are not an exaggeration. Many of them were designed for building the lead engineer's open source brand, not for company needs. Half of the projects described above have since been canceled.


- Hyperbahn

- Cherami

- Schemaless

- Peloton

- uDeploy

- m3db

- piper


Contrary to their glossy open source and tech blog presentations, these projects were highly contentious internally and widely viewed as inferior to industry counterparts. Each of these projects had 5-20 engineers working on them for over a year; many of them are being EOLed or phased out currently. And this is just the list that I can publicly talk about. This development came at extraordinary cost for the org and, in the case of the people who were laid off, cost to people's careers.

Ok, innocent question, having never worked in a giant tech company with a viable, popular product -- how do they deploy the massive amount of available engineering hours?

Do teams just work on random projects like the above while other people fix bugs? Is it like a million mini startups internally all pitching an idea to the CEO all the time?

Or do you need 90% of people focused on how to develop your product at first, then suddenly it flips and you need 90% of your engineers to balance enormous amounts of network traffic and data storage?

A guess, from other companies: a manager is given some engineers, they ask themselves "What should we create?", they come up with an idea, make up a random business justification, POC it to upper management, get approval and headcount, and spend a year or two trying to prove it was worthwhile. Best case it works and gets used by other internal teams for 2-5 years, worst case the team gets broken apart and the project abandoned. (Mind you, nobody gets laid off, they just find other reasons to keep paying people who aren't making anything of value) If you have ever heard about it publicly, it was probably someone's pet project and they wanted their ego stroked, or to "get a win" and a promotion.

Many of the projects are just a response to some other internal problem, like "the teams aren't using a shared platform", so they build a platform for the platforms. Persistent problems get ignored because nobody wants to rock the boat. The joys of working in a large corporation...

I saw this at a decades-old retail company. New CTO brought a Silicon Valley perspectives to the company which involved hiring a load specialist engineers into newly created departments without any explicit business justification. Literally said that once they were hired they'd come up with useful projects. He also wanted to start exporting our in-house software as standalone products even when we were still running transactions through a mainframe. CEO bought it all hook, line and sinker.

> Literally said that once they were hired they'd come up with useful projects.

This is a great way to have engineers burn time on projects they think are interesting rather than projects that will actually have good ROI.

How do you balance avoiding this with wanting engineers who have ideas and agency?

Bring engineers in to solve problems you know need to get solved. As they learn the system, its debt, the business, etc., listen to their ideas. For ones that sound good, give them the resources to develop a POC. For the POCs that work, give them the resources to take it all the way.

Larger companies with substantial profits can hire engineers explicitly to come up with POCs one after another.

The idea and proof of concept often come before creating a new team.

In general, initiatives tend to be top down while tasks tend to be bottom up. Politics between the "managers" and the "doers" happens at the interface of those two. What actually gets done and delivered ends up being some function of the top down, bottom up, modified by the political battle.

All of those are plausible but typically you have core teams built around product lines, teams that build shared components. maybe have infrastructure dev and devops too. maybe a database or datawarehouse team and data analytics/ stats team etc.. a cyber security team. all of these will have some team charter for where they fit into the business and there may or may not be shared resources technical and otherwise. These are typical in post market fit companies that are now “giant”. But yes, first you need the product, everything comes from there.

> This development came at extraordinary cost for the org and, in the case of the people who were laid off, cost to people's careers.

Except the people who were laid off, if they have been there a while, got much deep knowledge than if they just glued together a bunch of AWS stuff.

Gluing together AWS stuff can get you a good job, but it's not the same as solving these sort of problems.

I'd rather spend 3 years working on this and get laid off than the way I spent my last 3. Guarantee I'd be in a much better spot in the job market too.

Instead I'm trying to figure out how the fuck I get my career back on track because I'm headed towards the sort of job I will hate every single day.

> but it's not the same as solving these sort of problems.

But that's just it: the problems were not solved. Personally I would rather devote years of my career building something that actually works by standing on the shoulders of giants, than going full NIH and wasting those years on a clunky POS that will be mothballed before the decade is out.

But hey, I chose science.

And there, ladies and gentlemen, is the source of the problem: People who think, Solving a general-purpose problem that has already been solved by somebody else will do more for me personally, than using somebody else's solution to solve the problems my employer actually has.

I remember doing that too, once upon a time. I invented a templating system based on Lisp for building web apps, back in the days when there were servlets, but no Java Server Pages (or Handlebars, or jsx, or anything we have today).

The templating system was wonderful, and it did improve developer productivity for our group, but there is NO WAY WHATSOEVER that it paid back the time I put into it.

The business would have been much better off if we'd simply used what was available and spent the majority of our time solving business problems, not developer productivity problems.

Furthermore, my templating system didn't wind up being that special. You know why? Because I wasn't really solving a business problem with it, so I had no constraints upon my design. Without constraints, such things meander from shiny bauble to shiny bauble according to the engineer's whims and fancies.

When you're solving an actual problem, you have constraints in time, resources, and solution. This forces you to create actual value. That makes what you build actually useful. But when you're building a thing because you want to play with the thing, you have no constraints, and rather than freeing you to create something amazing, this simply sucks you into making something that is worthless for anything except résumé-building.

"Ah, but my résumé," says the narcissist who feels no obligation to create value for their employer. I have bad news, and I am speaking from experience. If you build a bunch of things with way-cool tech that created no value, you become an ignorant person's idea of a smart developer.

Sure, you can get hired by people who don't really understand how to manage engineering, and they will throw money at you, but their ignorance combined with your self-serving ways will result in another disaster. Lather, rinse, repeat.

Good engineering cultures will ask you about your projects, and they will ferret out the fact that your projects did not add value, and did not "move the needle." I am not hypothesizing this, I have gone to interviews and had the interviewers grill me on my vanity efforts.

You cannot pull the wool over the eyes of anyone who has been through a cycle like this and is determined to never have it happen again. You can put lipstick on a pig, but there's no way to hide the fact that all projects like this wind up becoming ad-hoc, informally-specified, bug-ridden, slow implementations of half of the thing you should have used instead of building your own.

Now all is not doom and gloom. For everyone who has done this, either deliberately or by being dragged into one of these white elephant projects, there is a way to supercharge your career going forward.

Instead of boasting about your ability to write Lisp-based templating systems, or synchronization protocols based on Operational Transformations, or whatever it is you built, try confessing to it.

Point out the engineering experience, but then be up front that the project was a failure/bad idea. Sure, point out that you learned some great tech. But spend much more time talking about how much you learned about failure modes for projects. Talk about the scars.

Companied worth working for are eager to hire people who know better. Position yourself as someone who has learned these lessons the hard way.

> Companied worth working for are eager to hire people who know better. Position yourself as someone who has learned these lessons the hard way.

Learning those lessons the hard way still requires doing those projects, which means you learn how the tech works, which means I still feel 100% justified in my previous statements.

I want to work on a project like this that actually has lasting value to the company. Actually does something novel enough that it's worth doing.

How do I get there? Is the only way to win the lottery and get placed there as a new grad/junior engineer?

Cause that seems like what you're saying.

My actual advice--for whatever it's worth--is that at the outset of your career, it's better to chop wood and carry water on successful projects in a good engineering culture, than to collect buzzwords working on projects doomed to fail in a weak engineering culture.*

I know it's very hard to get jobs in established good companies, but there are also quality startups, founded by people who have these kinds of scars already and won't go down the path of building things nobody needs.

Here's a specific tip:

When interviewing, there's a point where you get to ask them questions. Great! Ask about projects like these. Is the company working on any of its own tech that could otherwise be rented from Amazon or is available as open-source? Why? What is the expected benefit? If not, how do they manage development to stay focused on projects that deliver value?

By asking these questions you will identify engineering groups that are strong or weak, and you will also send them a strong signal that you care about success. Good companies will pick up on that signal and be more inclined to hire you.

JM2C. My advice plus $3.00 will get you a latte at Starbucks.


* This is not a dichotomy, of course.

Someone had to build the "thing you should have used instead of rolling your own".

Most of the things we should use instead of building our own are either currently someone's shiny, golden side project with no business case, or else started that way.

I wonder how conscious the decision is at a management level to tie up engineers with expertise in a particular domain just to keep them away from competitors.

I've seen a lot of vanity projects over the years, and often it feels like this, or a form of in-house retirement. That guy has done so much for us that now we just let him do whatever as long as HR is okay with it.

Can confirm that your confirmation of M3DB being cancelled is incorrect and is actively being worked on at Uber and elsewhere.

I do agree there were definitely projects that were overly ambitious and optimistic that likely that should have been better planned. I think you are weighted firmly at one end of the spectrum however. I have some obvious bias being involved in one of the projects you outlined, but I still think you are blindly covering a swathe of projects in one go without any nuance at all.

I think M3 is a necessary project for Uber. Open Source solutions like Graphite is really not designed for Uber's scale in this particular case. And yes,

What did M3 offer that existing projects at the time such as Prometheus and OpenTSDB did not?

The project page states "M3, a metrics platform, and M3DB, a distributed time series database, were developed at Uber out of necessity. After using what was available as open source and finding we were unable to use them at our scale due to issues with their reliability, cost and operationally intensive nature we built our own metrics platform piece by piece."[1]

[1] https://www.m3db.io/

Last I checked, M3 is open source.

Just curious, do you have a good idea of how many of these are still being actively developed? I remember seeing stuff like Ringpop and Hyperbahn get moved to the uber-archive github account.

I'm especially curious about Schemaless, since it seemed like at one point every company tried to build a NoSQL KV store on top of sharded MySQL, realized how janky it is, and are currently decommissioning it.

Cadence (cadenceworkflow.io) is a project originated in Uber that is worth mentioning. It's a workflow engine but with very unique capabilities. It has a lot of potential. I've heard of many companies using it, e.g. HashiCorp, Airbnb or Banzai Cloud.

The development team is pretty active. They've been doing an important effort to gain community adoption, e.g. persistency with MySQL, Postgre in the works, Prometheus style metrics, a proposal process or design docs. Very interesting.

uDeploy and m3 are still widely used internally. Hyperbahn is dead AFAIK. Not sure about the rest.

> in the case of the people who were laid off, cost to people's careers.

I fail to see how.

Considering Uber is reinventing the wheel a lot, the people who were laid off where probably working on some interesting problems. Uber is a fairly well known brand. Unless they are incredibly inapt at selling themselves, their career should do fine.

It is better when you can decide, yourself, to exit to another company, rather than being pushed out and then having to find a new job. Certainly, a competent engineer SV will land on their feet, but you lose significant negotiating power when you're not actively employed.

Unless you've got plenty of money and are funemployed and wealthy enough to turn down offers and move on if they are unsatisfactory.

You're only in a bad bargaining position if you need a job to pay your bills.

The key in the discussion with the recruiter is not to say that you are unemployed but to say that you're taking time for yourself and looking for opportunities that meet your professional desires and baseline financial expectations. The key is to make it clear that you have no financial needs, just financial expectations.

Be prepared to use the Noel Smith-Wenkle salary negotiation method.

>you lose significant negotiating power when you're not actively employed.

I took six months off after a large liquidity event and then went back into the job market. I don't think I lost any negotiating power - the trick was to make sure I times the interviews so that I'd have several offers at the same time and then could use the fact that company X was offering a certain amount to ensure I got the job at the company I liked the most.

I do just want to say one thing: most tech investors now think we are in a bubble. A downturn is likely at some point. Now would be a very good time to ensure you have six months of expenses in a savings account so you don't have to take the first job you get offered - I had to do that in 2008 after the financial crisis hit and my startup died and it wasn't a fun time.

To be completely honest, the parent poster is referring to the common experience of normal people. If you were in the position to take months off after a large liquidity event, that experience doesn't apply to you.

There is a similar effect people have when they reach a certain level of financial independence - not only are offers better, but negotiation is much easier. Companies react to confidence (whatever the source) the same way individuals do.

Probably not too many avg Joes doing skunkwerks at Uber and getting laid off..

Every engineer on HN should be earning enough that they can put away 6+ months of expenses just in case.

Do you think top down strategy and control is important for large organizations? Most people idealize organizations that are grassroots drive so it's interesting to see how it didn't work in this case.

Also it sounds like Uber needs to fire it's CTO.

> I worked at Uber from 2014-2018.

Seems like you probably got a lot of shares and were lucky. Now that they IPO'd, do you consider that you came out ahead financially (compared to working elsewhere)?

Just a note: Uber's valuation was higher in 2015 than it is right now...

No inside info on Uber, but it is common for options at private companies to have strike prices considerably below reported valuations, which is proper (and adjudicated by a third party) considering the reported private valuations usually based on preferred shares that come with extra rights. That said, I dunno what year Uber switched to RSU’s, which would make this moot.

It was $40b at start of 2015 and was rumored to be at $50b mid 2015. It's $53b currently.

That said with dilution, it's likely 2015 share price is less than it is today.

What's the point of a value if you can't realize it in any way ?

Well Uber IPO'd this year in May, so given that he/she left in 2018, I would count the parent poster as unlucky if they forfeited those shares after leaving.

Did they forfeit them? Or had they vested and in anticipation of IPO likely exercised either all or nearly all of their options and then sold post-IPO?

Before the Susan fowler thing they had a rule that you had 30 days to exercise your options after you leave or separate as they called it. Since the company's value went up so much most people couldn't afford to pay the tax and because it was private you couldn't sell it (employee lock out is in November). Lots of people were handcuffed because of this, and I know a few that walked away. After the Susan Fowler thing they changed the option thing to like 7 years.

I think they moved to RSU's in mid 2015 or 16.

With 4 years of tenure they should have vested much of their stock, but I'm not sure when Uber started doing grants rather than options. If they had options they may not have had the cash to exercise.

At least these projects were engineer driven and open source. It's not really unusual for projects to be abandoned or fail based off of some faulty forecasting by sales or product.

Don't forget Slack and HipChat.

Lol, people keep taking about Uber building internal products because 3rd party stuff can't "Uber" scale, I think the real drive to build internal apps is to save on licensing costs.

Take chat, hipcat is like 5-10 bucks per month per user. Uber has about 20k FTEs, 40k contractors all use chat. That's 3 to 600k per month just on chat.

Don't forget scale is expensive.

I can't imagine how much that many slack licenses would cost.

I mean, they’d probably get a good discount.

Yeah but you dont need all those engineers if you're not re inventing every wheel.

what replaced Schemaless?

The person is just listing tools which Uber could have not implemented because something similar exists. Schemaless was widely criticized, yet I don't think the public understands all the use-cases.

That said, Schemaless isn't going anywhere. Cassandra doesn't cover all the use-cases.

> Uber engineers used many bogus reasons to implement NIH projects

In my experience this is one of the most toxic diseases an engineering org can catch. It often comes from smart, technically capable engineers who are more concerned with impressing their peers than growing the company. They come up with a BS reason to re-implement some rock solid 3rd party tech, spend ages doing it, only to leave the company as it starts to comes crumbling down. At that point other engineers have to put in massive effort to rip out their buggy timesink, and replace it with the open source library/database/protocol/whatever that was the obvious choice from the start.

If people like this exist in your org, you either need to convince them of the error of their ways, or let them go. The damage they cause when left unchecked is just too great.

>It often comes from smart, technically capable engineers who are more concerned with impressing their peers than growing the company.

Well, if the company's business is connecting drivers with riders, there isn't much engineering involved in growing the company. Pricing, handling investors, PR/advertising and working with municipal governments does affect your growth. The exact details on how you store the destination's GPS coordinates in the database don't really matter. A good product manager who understands usability and can communicate their ideas to the management can add some value, but having a 20% lower price than the competitors do will make 90% of the users ignore any usability hiccups.

Hiring thousands of engineers to maintain a process that only needs a handful is perfectly reasonable, since the investors don't understand tech and generally like the head count. So this is critical for raising VC capital needed to subsidize the rides, that in turns gives great growth numbers.

A typical developer doesn't understand these mechanics. And even if they did, the only alternative to delivering useless unneeded results would be to resign, except good luck finding a competitively paying non-bullshit job in the current market.

This doesn’t match my experience at all. Every tech company that I or my close friends have worked for has huge backlogs of features they want to implement, and huge amounts of tech debt they’d like to eliminate. They have to prioritize ruthlessly to select a small number of projects that they actually have the resources to complete. Reimplementing rock solid 3rd party tech just takes away from this, AND introduces huge maintenance burdens.

Out of curiosity, is there anything in common about the structure of those companies? Small bootstrapped businesses? Mid-size profitable enterprise? VC-backed startups? Large publicly traded companies?

Mostly VC backed ex-startups (now 10+ yrs old, 100s to 1000s of employees). Though I’ve also got a few friends at a bootstrapped company that never took VC money (~15 years old, but still small, ~30 employees), as well as a number of friends at Amazon (working on different RDS teams).

All have had the same experience re: there being more feature work and tech debt elimination than they have the resources to complete.

FWIW, I used to work at a company with a severe case of NIH syndrome, but it worked out well. From what I could tell, the difference was all about mentality: This company did it, not because of weak management, but because of strong management that was trying to push a culture of KISS and technical ownership.

So, whenever a technical team identified a need for X, there was a lot of time putting into understanding the problem, understanding the company's real needs, and figuring out the simplest thing that would get the job done. If there was an off-the-shelf product that did one thing and did it well (e.g., protocol buffers for serializing datagrams), that would get chosen. If everything out there was a confusing pile of feature creep that was trying to be all things to all people (e.g., configuration management), an in-house product would get launched.

TBH, the tech stack was a joy to work on. Especially that home-grown configuration management system. So simple, so effective.

I've never worked for Uber or anything, but I've attended a few lectures and conference sessions where Uber engineers presented things they worked on, and my sense was that that wasn't at all what was happening there. Most the projects I saw seemed to be motivated by resume-driven development. Which is absolutely the way to go if you want to end up with solutions that are complicated, hard to manage, and half-implemented.

Are any of these projects open source, and if so can you point me toward some of them?

Nope. They had no particular business interest in doing so, and, from a technical perspective, doing so would have created a conflict between the needs of the community and the needs of the company. I'm not sure a lot of it would have flown, anyway, due to going against the grain of modern software development.

For example, automated package management was banned at the company level. Instead there was a central repository of approved packages that everyone used. That's actually another great example of figuring out the simplest solution that would work: It ensured internal consistency in a way that more-or-less eliminated a problem that is more commonly solved with baroque solutions like containerization. It gave tech ops a really, really easy place to go look when they needed to do security audits on our dependencies or for pushing security updates. And it made it easy for legal to keep an eye on compliance. Stuff like that. Honestly, I thought it worked really, really, amazingly well. And I'd have never thought of it, because it flies in the face of 20 years' worth of the entire software field pulling things in in a completely different, much fancier and more complicated and bell-y and whistle-y, direction.

Also, most of what was going on was so simple you wouldn't even be able to turn it into an open source project. Like, I know that the official wisdom is that you're not supposed to route your event streaming system through a database, but wow, assuming you're willing to shell out for a commercial RDBMS, it's amazing how far you can scale an event streaming mechanism that routes everything through database tables. And don't even get me started on how easy it was to support in production. Sometimes I have to remind myself, open source is only free when your programmers aren't drawing a paycheck.

> For example, automated package management was banned at the company level. Instead there was a central repository of approved packages that everyone used.

Do you mean they had a private dpkg/yum repository? That is a well supported and somewhat common setup, and counts as automated package management (your application gets packaged up as a .deb or .rpm with the necessary dependency and installation info).

> It ensured internal consistency in a way that more-or-less eliminated a problem that is more commonly solved with baroque solutions like containerization.

I worked at a company with that kind of build/deployment system. About the time of the start of Docker mania, a corporate merger happened, and the acquiring company, in addition to buying the company I worked for, also bought into the hype, and insisted the build and deployment process go through Docker. Early days of shitty Docker tooling meant a ridiculous amount of time spent by one of my colleagues getting the system working, with the result that build and deploy times went up an order of magnitude, and debugging got a lot harder. Containers are not a substitute for package managers or automated provisioning.

> Do you mean they had a private dpkg/yum repository?

Two things:

All libraries were collected into a single repository in source control that developers would clone onto their machines and use. These were the same libraries that were installed on the production servers. Very consistently, modulo major platform migrations and the fact of running multiple server OSes in production.

Binaries were named and versioned tarballs that were deployed by just unzipping them on the target server. That was all that was necessary, regardless of server OS, because of the abovementioned consistency in what shared libraries were installed on the servers.

With those standards in place, CI, CD, configuration management, all those sorts of things became just wildly simple.

It's a bit like Bazel's golden handcuffs: If you're willing to let go of the basket case of flexibility that C and C++ have traditionally offered, Bazel can deliver you a build system that's just a dream to work with compared to old friends like autotools and CMake.

Name this unicorn?

A couple news stories back, it came out how much money Uber was spending on Eats and it was simply staggering.

My company has similar stories. When you look at the amount of requests per second we're actually dealing with, there are plenty of other companies doing something far less "sophisticated" and handling more traffic.

Keep it Simple, Stupid.

Allowing for heterogeneity is a blindspot for a lot of wannabe architects. We have a system where most of the code is dominated by 2 platforms and there is no way to democratically add/test/demo new routes that don't go through one of those two. And every change has to be able to deal with production traffic levels from word one. Similar with deployment management.

In this sort of climate, there is no incentive to ever go back and revisit NIH vs off-the-shelf tools that has now outstripped the functionality of the in-house tool. When the friction is so high it's a non-starter. And when some of those NIH projects started in ignorance (ie, a tool existed and was already mature enough to start using, but the author didn't find it) it represents extra frustration.

At some point your engineers figure out nothing they are working on will present well on their resume, and they either get burnt out or flea.

>At some point your engineers figure out nothing they are >working on will present well on their resume, and they >either get burnt out or flea.

Shouldn't the fact that they were part of standing up solid reasonable solutions reflect well on their resume, to reasonable hiring managers elsewhere?

> A typical example is that Uber had a team to implement their own Bazel-like system because "Bazel (or any other tool) does not scale in Uber-scale"

Citation needed? AFAIK Uber was using Facebook's Buck and decided to move to Bazel for stacks where Buck is admittedly not good at (e.g. Go). I'm not aware of any org within Uber reinventing Bazel. The web org might be closest to what you're claiming, but even there, it merely wraps over Bazel to bridge over its shortcomings (i.e. package management is typically done "outside" of Bazel, via yarn/npm in web, as seen in the official rules_nodejs and other 3rd party spin-off rulesets)

This was under consideration at one time by the Seattle build team and was a recurring point of discussion during infra summits. Unclear if we actually broke ground.

Just saw your other comment. Not sure exactly when you left in 2018, but around that time, there was an internal summit involving various platform/infra orgs that decided to invest in Bazel as a company-wide standard. Java is still on Buck due to some technical reasons, but several orgs are either moving towards or already fully on Bazel.

And the team publicly defended their consideration in a company all-hands, IIRC.

I can understand Uber wanting to build other tools in house because the existing open source ones might not scale in Uber-scale. But Bazel? That's just absurd.

It's a self-sustaining reaction in any organisation left unchecked: Manager has an incentive to hire below him, create projects to justify jobs, hire more, create more projects, etc.

There needs to be a way to reward managers for running teams/organizations that have a big impact with low headcount. After a certain point, in most organizations, the management promotion game becomes almost entirely about headcount underneath you. If you're a middle manager, this is not just true for you, but also for your boss, who is then aligned to help you get more people under you, since those people are also under them indirectly.

> the management promotion game becomes almost entirely about headcount underneath you. If you're a middle manager, this is not just true for you, but also for your boss, who is then aligned to help you get more people under you, since those people are also under them indirectly.

Yes, that's exactly how it works. It helps you move from manager to senior manager. It helps your boss move from senior manager to director, etc.

Growing your team is management 101 for anyone with a minimum politically astute. So if the message you get "from above" is that the budget is unlimited (which honestly seems to have been the case at Uber) then you dream up any justification to increase your headcount.

One of my companies associates. Said things for him got a lot different after he managed a group of 500 engineers. Going from 5 to 50 didn't change things much. But 50 to 500 was night and day.

At my (large but not tech) employer, some mgr/execs hire reams of contractors to pad the headcount, then have to beg/borrow/steal budget to keep them around until they get promoted.

Contractors typically come from a different budget than headcount. So when you can't hire employees, you hire contractors, and then use them to justify your next headcount request. Rinse and repeat.

It seems like this only happens with office staff. In factories where output is very directly measurable, headcount is expected grow slower than or drop compared to production.

> Bazel (or any other tool) does not scale in Uber-scale

This is hilarious, given they Bazel is the public version of Google-internal Blaze

Not Uber related, but as soon as ERP systems or warehouse management or logistics and supply chain management are concerned I sometimes have the impression that a lot of "start-ups" are falling in the same trap. There is simply no way to beat the likes of SAP, Microsoft or even Oracle in the sheer man hours, resources and research spent on their systems. And yes, they are cumbersome and UX sicks. But then they are extremely powerful out of the box, and completely auditable. Maybe it was necessary back when Amazon started due to limitations with webshop integration (or existaznce as far that is concerned). Nowadays not so much.

I can definitely speak to that (having done multiple go-lives for third-party ERPs and WMSes and such for my current employer). There are definitely tradeoffs between going with something off-the-shelf v. developing in-house v. something in-between (e.g. a system that's ostensibly "off-the-shelf" but expected to be heavily customized), and the vast majority of the time you're gonna be best served by something closer to the "off-the-shelf" end of that spectrum, no matter how much you think your "special" business needs warrant it.

That said, a lot of these sorts of systems are notorious for being pretty awful, too. Yes, there are many millions of man-hours poured into them, but that doesn't mean they're good; those man-hours could very well be spent on trying to implement every bell and whistle under the sun while entirely neglecting bug fixes, for example (and - speaking from experience - that is a disturbingly-common example). Also, having a bad UX can (and often is) a fatal problem for such a system (sometimes even in a tragically-literal sense, e.g. with EMR/EHR systems and other medical software); the whole point of business software is to optimize user productivity, so the last thing you want is to make the user's life harder.

So yeah, tradeoffs. If you're gonna fall into the DIY trap, the key would be to not try to replace the entire ERP or entire WMS in one go, but rather to build smaller tools that optimize specific business processes, and make sure those tools stay small and maintainable; any expansion of features should happen through interoperability between tools rather than trying to build a giant bloated tool. If you're thinking "wow, that sounds like the Unix philosophy" and/or "wow, that sounds like microservices", that thought ain't exactly a coincidence.

> completely auditable

The biggest thing that people who try to roll their own fail on is this right here.

Auditable includes financial auditing. It should be possible and simple to reconstruct the state of a customer account at any point in time. What products they had, what discounts and promotions were in place, how they were purchased and what was owed at that exact point in time.

It is scary how many systems cannot be audited this way because the engineers and business owners had no understanding of basic accounting concepts like double entry book keeping.

Systems that are not designed to be easy to financially audit are either going to overbill or underbill customers. It isn’t a matter of if, but when...

Curious, can you elaborate on what “completely auditable” means?

In short: the system should log every operation performed within, and that log should be readily accessible to (authorized) end-users.

For example: let's say you have a warehouse management system, which tracks inventory movements in a warehouse / fulfillment center that ships, say, toothbrushes to customers. Multiple customers call in with complaints about getting blue toothbrushes when they ordered red toothbrushes. Seeing as how this is clearly not an isolated incident (multiple people are upset about not getting the right toothbrush!), it's up to you to find out where in the whole lifecycle of that toothbrush in your warehouse - all the way from when it was received to when it was shipped - that error might be happening. If your warehouse management system is completely auditable, then you'll be able query the transaction log (given perhaps the affected order and item numbers) to work backwards to investigate where the fault occurred, be it in shipping, picking, replenishment, receiving, the vendor, or what have you.

The "incumbent" off-the-shelf ERP and WMS vendors tend to support this sort of comprehensive audit trail much more thoroughly than those writing new ones (whether as an in-house project or as a startup attempting to disrupt the space), simply because those incumbents have already had their hard-earned lessons on why literally every transaction must be thoroughly logged (whereas the newcomers and DIYers typically have not, or are in the process of learning as they encounter more and more situations where that audit trail is necessary).

That. Also the financial aspect of any transaction is important. Just this week I had a chat concerning point-of-sales systems with an accountant in Germany. Tax authorities require that any record has to be impossible to alter afterwards. Same goes for ERP systems, every transaction has to be able to be backtracked, every change has to the books has to be recorded.

Any system, e.g. SAP, that has a proofen track record in that, that is SOX compliant, is a huge benefit in that regard. Also for the company itself, IMHO, as the company is forced to work in compliant ways that just make life a lot easier.

where I am, it means “so-and-so took such-and-such an action with these permissions at this time” which can be correlated with things like “this was the exact configuration of the entire system at the time” with reliability and immutability, across multiple versions.

On the other hand this probably is a huge advantage that Amazon has.

Bazel is a ground-up rewrite. Its not related to Blaze apart from naming. There was a skylark project aiming to combine the two over time, not sure how the progress is going on for it.

Bazel is the open-source name of Blaze. It's the same codebase. Blaze has a few Google-specific modules (e.g. integration with internal tools, some legacy Google-specific rules).

Skylark (renamed Starlark) is the extension mechanism. It's exactly the same in Blaze and Bazel.

Source: I work on Blaze/Bazel, and I'm the lead for Starlark.

Could you talk about why Blaze was renamed? I'm guessing it was for SEO reasons so you actually got correct responses outside of moma or whatever it's called.

For legal/trademark reasons.

> Its not related to Blaze apart from naming.

And functionality (and architecture?); I started my career at Google, and am now at a mid-size place that uses Bazel and it's almost identical from an interface perspective

"Blaze" is littered all around the code base [0].

[0] - https://github.com/bazelbuild/bazel/search?q=blaze&unscoped_...

The lays off this wave is mostly operations, marketing, and other areas. Not in core engineering.

> their own key-value store

It seems like everyone does that.

and it makes sense cause if given any chance to do it I'd jump on it immediately.

Not in 2015/2016 when so many choices were already available, and certainly not in an org where MySQL would just be more than enough to meet Uber's need. Besides, Uber built their version on top of MySQL following pretty much FriendFeed's design: https://backchannel.org/blog/friendfeed-schemaless-mysql. A strange choice by itself given that better solutions had popped up left and right during the 6 years since the blog was published.

Uber switch to Schemaless from a Postgres cluster, claiming that Postgres "does not scale for Uber". I see irony in that piece of Uber history. Particularly, Uber always claimed that some OS solution "had serious reliability issues". Cassandra in 2014 worked for companies that had orders of magnitude of traffic, yet Uber found it unusable. Similarly, Uber switched from MySQL to Postgres and then from Postgres to MySQL. Each time Uber published an article to claim engineering victory. The truth is that any of the three systems would be perfectly fine for Uber. Oh, didn't someone also claim that Elaticsearch crashed under Uber's load" and JVM-based systems were not usable for Uber's scale? " before Uber ended up adopting Elasticsearch?

What really happened was that some teams in Uber simply didn't want to dive deep and produce objective tech assessment.

I meant it makes sense that people keep building them needlessly because I would just at the opportunity even if I felt like it was totally pointless.

KV store as a concept seems to be a particularly Attractive Nuisance. It sucks people in like a pool without a fence around it.

The Dunning-Kruger quotient is especially high.

What I've always wondered is, why do people get so excited about working on ideas that they themselves characterize as 'simple'? The level of excitement around these sorts of projects does not match the level of rhetoric. You put the second rate people on the 'simple' stuff. You save some of the meaty bits for yourself.

Which means that in your head, you know that KV stores are actually pretty complicated to get right. You either aren't admitting that to yourself or to the org or both. It just screams cognitive dissonance.

Because if you were brutally honest and up front about it being complicated, you'd select a good one and have your fun elsewhere, say in using or scaling the KV system instead of building it.

two words - job security

I've been in many orgs where engineers like to reinvent the wheel. Just so they can claim ownership over more components, add polish to resumes ("Led project to rewrite X to scale", "built X from scratch") and makes them look busy. When it comes to perf reviews, or when it comes to job cuts, they'd likely survive given they know more / built those components.

What I would love to know is how many people built these homebrewed systems at Google in 2007, then jumped to Facebook and built them again in 2010, and on to Uber to do it yet again in 2013.

I agree with how engineering was disorganized at Uber but it makes sense to me to not trust your competitors with your infrastructure. Some projects worked well, Schemaless is still going strong and I'd take it any day over DynamoDB or Cassandra.

No, no, Uber employed all those developers because every one of them's work was crucial to the success of the company! Dan Luu wrote a famous essay about it, and he can hardly be wrong, can he?!?


What are NIH projects?

"Not invented here"

Not invented here


national institute of health, aka healthcare

What's NIH?

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact