Hacker News new | past | comments | ask | show | jobs | submit login
Why's that company so big? I could do that in a weekend (danluu.com)
712 points by pyb on Oct 3, 2016 | hide | past | favorite | 423 comments



I often hear this logic from inexperienced engineers. My answer to them is: Then why don't you do it? <grin>

Recently a junior engineer said this to me about Uber, and I helped him to understand - His logic was "it's just a taxi app".

There were loads of Taxi apps before Uber and loads of them after Uber. Yet none of them were uber?

Sure that a medium complexity CRUD app can do the pickup and scheduling, and spit out some json to the app.

But what if you have 100,000 journeys taking place simultaneously, and you must track their positions so that you can render the journey map on the invoice?

Can you do live traffic planning and re-route around the busy traffic on the fly?

Does your weekend include the full billing system? This is a mandatory requirement for a business.

Can you spot busy periods and increase the fares according to the laws of supply and demand? All online in realtime? Can you continuously A/B test this to optimise profits?

Can you deal with the fact that maps and roads are always changing?

All of these and more are the things that make Uber, Uber, otherwise you're just one of those shitty taxi apps that failed.

Still think you can build it in a weekend?


I agree with everything you say, but that response, like the junior engineer, focusses on the technology. So give people six months not a weekend -- there's a good chance they still won't be Uber.

The success of many companies, and probably all of the unicorns, has nothing to do with technology. The tech is necessary, of course, but so are desks and an accounting department.

Internalizing that has been difficult for me as an engineer.


At a previous job, we had serious development bandwidth issues. The company was trying to solve that by hiring more developers, and all it was doing was making things worse.

The root cause of the development bandwidth problem was that we wasted vast amounts of time and energy on building misfeatures. And we built so many misfeatures because we had way too much development capacity in relation to our product planning capacity. So the BAs didn't have enough time to fully develop ideas before sending them to engineering, and were instead stuck throwing things over the wall when they were still half baked. All these half-baked ideas took time to build, and then they took time to fix after they bombed on the market, and they slowed everything else down since every new feature is something you have to ensure still works when you make a change in that part of the software. Day in, day out, we were taking on technical debt to cover our design debt bills.

So then we tried Agile, and man oh man did that make things worse. 'Cuz now instead of just lobbing things over the wall, the BAs had to be actively involved in every sprint. There goes any opportunity to sit down and concentrate, because now you've got a nattering horde of 20 developers all trying to get clarification on last week's decisions, which leaves less time to make this week's decisions, which means next sprint things will be even more sketchy and ill-defined.

The correct solution would have been to hire more business analysts, and they key thing we needed from those business analysts wasn't instructions on what to build. It was instructions on what _not_ to build.


I know I've seen this pattern in person, but something about the way you wrote it makes me want to go back and think about some of my other projects before I could identify that was the problem.

I know one project in particular we did a release about every 7 months. Every release cycle they estimated 2 weeks for requirements, every cycle took a month, and yet the next cycle you'd see a 2 week estimate on the roadmap.

As for switching to Agile, I know some people would tell you that Kanban would identify this problem for you by around week 8. Did you guys ever try having development liaisons help them draft requirements that were precise from an engineering perspective? Sometimes that initial meeting works better when there are only three people in the room, instead of a bunch of exasperated people being condescending to you again while you're trying to do a good job. adjusts collar nervously I'm uh... asking for a friend.


"...precise from an engineering perspective?", this (in my experience[0]) doesn't work. Because requirements are messy and always change. As they say, " No plan survives contact with the enemy." and "No requirement survives contact with implementation." The reality is that you need to come up with a timeline that accounts for this problem (say 8 weeks) and only scope the release to fill 50% (4 weeks). An 8 week release cycle may even be on the long end of a release cycle.

Why 50% of a release's entire timeline? Because you're going to have overruns, errors, re-dos, and any number of random unplanned elements creep into the scope of your release. And the reality (I may get blasted for this blasphemy) scope creep is fine. As long as it pertains to the originally scoped requirements and is clearly adding value. "A delayed game is eventually good, but a rushed game is forever bad.", Shigeru Miyamoto. The important part of that quote is "delayed", you still need to make sure you release otherwise you're just spinning and wasting company time and money.

And whatever you do, don't try and "fill in weeks". If the release is ahead of schedule, let it be. Let it finish weeks early if it has to. Then, rinse and repeat. Admittedly this type of release process can be hard to sell because there's usually a negative emotional response to the idea of only scoping 50% of the total allowed release cycle and that somehow the other 50% is waste. It isn't, it's there to guarantee that the items scoped initially actually get done on time. That way QA can get to them in a reasonable amount of time, give feedback, and (during the second half of the release) engineering can fix the issues raised by QA. Business development can also rest comfortably knowing that the features scoped into the first half of the release are actually going to come out on time.

I know it sounds like "You're just adding slack." But it's intended to be more than that. If you see teams attempting to squeeze more items into the 50% release you know there's a prioritization problem within those teams. Which can be remedied by either: 1) reducing the overall size of the release cycle further (eg 8 to 4 weeks with 2 weeks as the 50% mark) to provide more flexibility into what goes into the product in a shorter time frame. 2) Helping those teams clarify what the high-level product direction and priorities should be.

Hopefully this 15 minute rambling comment is valuable to you. If not, I apologize.

[0]The mistake I made and the advice I can personally give, don't focus on up front high-accuracy estimates and designs. Come up with a reasonable design, your best guess estimate, and then implement as furiously as possible (coding best practices should still be enforced). You'll know much earlier in the release cycle what the real timeline for your feature will be than if you'd tried to spend all that time up front.


> The company was trying to solve that by hiring more developers, and all it was doing was making things worse.

You don't mess with Brook's law!


I second this. Additionally, get a grip on your queues!

I have yet to find a better text on how to properly manage software projects than, "The Principles of Product Development Flow: Second Generation Lean Product Development"[0].

[0] https://www.amazon.com/Principles-Product-Development-Flow-G...


> The success of many companies, and probably all of the unicorns, has nothing to do with technology.

This is 100% true and unfortunately gets distorted from the financial backers who pump them up for unicorn status. Quite a dichotomy:

"Software is eating the world!"

"The hard thing isn’t setting up an organizational chart. The hard thing is getting people to communicate within the organization that you just designed."


Of course software is eating the world. Wheels ate the world. Fire ate the world. Written language ate the world. Plumbing ate the world. Electricity ate the world. Radio communications ate the world.

Not every project or human endeavor that involved wheels, fire, written language, plumbing, electricity, or radio ate the world, but every project that did not involve them eventually got eaten, for the simple reason that you needed to keep up with technology as a minimum. But beyond that, projects succeed or fail for the same reason they always have: understanding the actual problem you're trying to solve.

Uber's success is not due to software. Uber's success is due to its understanding of the transportation market and how to play the regulatory game to its advantage while never getting permanently in trouble. Uber's ability to deliver on the huge market opportunity they saw—i.e., its lack of failure—is due to software. We quite reasonably call this "success" when talking about technology, but if we're talking about companies succeeding or failing, that's not a discussion about technology.


> The success of many companies, and probably all of the unicorns, has nothing to do with technology.

So why not look at failure then?


Its harder to learn from failures than to learn from successes.

Success has the false-positive issue. Failures have the false-negative issue.

Cargo-culting false-positives is going to be less harmful than cargo-culting false-negatives. (X Company didn't use stack-ranking and then failed. Clearly stack-ranking is bad for companies)


Maybe. Just to muddy things even more, some successes are failures.

Have you ever met a billionaire who said, "I was lucky" and left it at that?


Yeah. At least with a success you have a data point that it could be successful - with a failure, you don't even know that.


It's not the technology success/failure axis in question, but how much weight to give it relative to other factors.

AKA in predicting overall success would a team with low tech expertise, but high sales expertise be have a better chance?


I know - but I was talking to a junior engineer, and I was concentrating on the technical aspects to inspire him.

I expected him to go off and actually attempt building it, that way he would get some more experience.

And he did! He went off that weekend and wrote a proof of concept route planner, it was great to see him learn something new and complex


many a junior engineer who persist on the idea its easy, can be done quicker, cheaper, and such, also rarely have the needed dedication to see it through especially when it takes longer than expected, missing requirements are identified, and such.

I have seen multi million dollar projects hit the same snags, horribly under scoped all because Mr Easy didn't think about asking all of was needed. They saw the tree and missed the forest


Proper project scoping (or to rephrase: identifying the true core business needs) is truly one of the hardest problems in tech. Over-scoping makes you end up with a ponderous system that doesn't serve core business needs very well, while underscoping can lock you into poor initial decisions that cripple your ability to expand into other related functionality.

What's the expression? "Plan your system for 100x growth, but build your system for 10x growth". Something like that except for functionality rather than scalability.

That and supporting the system. Junior engineers tend to think about building the tech, which is honestly the easy part. Keeping it working until the last of your customers are finally ready to upgrade 10 years later is harder than junior engineers tend to think.


Ha!even this sounds like a mistake to me. How about "Make the simplest thing work for one person.". Make sure they get some value, and then iterate and repeat. Once you've clearly solved the problem at hand then start to scale, optimise etc That is of course all after you've presold the solution.

I know it's not always easy to do this, but that really is part of the challenge. There's still too many deathmarch projects lead by people trying to do things 'the right way'. If you're working on a new project for the love of science, please make sure you're releasing something valuable every single week if not every one or two days!


It has a lot to do with the technology. It's not the only or even the most important factor, but it's still one of the most important.

I think a big part of the disconnect being discussed originates in the myth that CRUD-based apps are fundamentally easy. This simply isn't true. The challenges of building a modern, complex, scalable CRUD app with good UX and a maintainable code base are not 'less than' challenges rooted in algorithms or bit bashing. Doing it well enough and fast enough to be competitive in the market is HARD.


Sometimes it is the tech... just not in obvious ways. You have to look beyond the basic spec.

Uber and Lyft are not just taxi apps. They are massively polished. When I use them I am astounded at how good they've gotten the UX and then I shudder at how hard that must have been across two mobile platforms and in the presence of very unreliable networks. I've ordered an Uber with no cell Internet using barely usable coffee shop wifi from a block away.

Then you have scaling, which brings up or creates issues you just do not see at smaller scales. Big O type considerations and weird edge cases that only come up once every billion transactions start mattering.

Then there is all the big data stuff. These services do tons of route optimization and analytics.

So yeah marketing, sales, fund raising, management, etc. also matter but tech often matters as much or more. It's just that junior engineers usually only see the tops of icebergs. This is why despite decades of pure biz people saying "tech doesn't matter" you still rarely see founding teams succeed without at least some tech founders but you do occasionally see even lone tech founders make it. (But those tech founders do have to know or be willing to learn the non tech stuff!)


> The tech is necessary, of course, but so are desks and an accounting department.

> Internalizing that has been difficult for me as an engineer.

After I got over that particular hurdle, I would still think:

Ok, so "build it and they will come" is not a thing, got it. But still, engineering is most important by far. Without sales, support, marketing, or legal, you still have a product, just one that is adopted slower. Without engineering you don't even have a product to sell in the first place.

And that's true. But in the end it's just splitting hairs. It doesn't matter. Over the longer term, no successful, scalable business is ever going to exist solely as an engineering team. Every job function has a purpose[0] and gets you where you want to go faster; in some cases, lack of certain non-engineering job functions will make it impossible to even do certain things.

So sure, you can bootstrap a product with just a few engineers and nothing else, but unless you're ok with it being a niche-market product that you spend all your waking hours working on, an engineering-only org is never going to cut it.

[0] With the obvious exception of those people hired to satisfy the latest management fad du jour, or to give the CEO's deadbeat nephew something to do with his time.


> The success of many companies, and probably all of the unicorns, has nothing to do with technology.

Execs love to think this but it's just not true. Top heavy companies waste money. It's true that tech of not the only ingredient but for a tech company it's critical. Good tech is not a commodity. Top-tier engineers are worth as much as a good CEO. That is if you want to remain lean.


> Internalizing that has been difficult for me as an engineer.

I suspect this is because everyone wants to not only feel as part of something but they want to make an important contribution. I venture to guess accountants at Google do not feel they make a difference to search. The culmination of parts is what makes the company whole and all those parts are required.

Its hard to accept you are not as important as you wish you were. Its a great step in the maturation process of one's professional career though.


> The success of many companies, and probably all of the unicorns, has nothing to do with technology

Have you considered broadening your personal definition of technology past computing and engineering?

From Wikipedia:

> Technology can be the knowledge of techniques, processes, etc. or it can be embedded in machines, computers, devices and factories, which can be operated by individuals without detailed knowledge of the workings of such things.

Technology is the creation of systems. Those systems run as much on the users of that technology, and the members of the organisation supporting it, as they do inside of microprocessors.


> Internalizing that has been difficult for me as an engineer.

Technology exists largely to accelerate the pace of development and of course, it's never a sole driver of any business. There is no reason why you can't create a small-scale uber with call centres and excel spreadsheets but I doubt it would reach the scale and impact of what Uber is today. Granted that, it's hundred times easier to figure out to technology than business, I won't trivialise role of engineering because it's something allows a business to near its global maximum.


I thought like those junior devs for a while, too, and I agree to your sentiment.

The feeling that the CEO I was working with was doing mundane tasks was astonishingly - he even said that he could to his whole job using his smartphone.

But after trying to do the business stuff myself I realized that no one in all of this process is irreplaceable - finding a good software developer may not be easy, but it's also not easy to find a good CEO. This is important to realize that software developers shouldn't see themselves as some type of elite.

The main problem is that junior devs don't realize that 90% of business is based on networking and being friends with people who are in middle management and want to increase their reputation in their big corp. You don't get big clients because you're a genius hacker, most of the time those B2B applications are simple CRUD apps. You won't get that $500k because you simply didn't play golf with the key partners. I'm not even exaggerating, I've seen that the best deals are made in ones "spare time".

Maybe startups are another breed than the software companies I know, but don't drink the cool aid and don't assume that technology is your main differentiator. It's working with people all along.

I think it isn't even about scaling high-performance systems, it even starts with the mindset of those who think that software developers are entitled in some way, usually junior devs and people who never did any real business work.


>The success of many companies, and probably all of the unicorns, has nothing to do with technology.

Most of the unicorns are founded with engineering talent coming straight from Stanford, MIT, Harvard, UCB, Waterloo etc. you name it.

Making a company successful is as much about economics of scale than raw engineering and of course, both are closely related in this day and age. Engineering shapes what and how your products feels like in real-time (= understand very short code-to-production lifecycle).

It might not be politically correct to say but I am not here to make you feel better. Anyone who has actually been in a team at a company that has exponentially grown will tell you the same: your chances to ever survive that "first wave" of users is directly proportional to your ability to hire the best, which of course is a lot easier when they are your current or former classmates.

That's not to say there aren't companies founded by people not going to the schools mentioned above or that engineering talent is only available at these places. It is obviously not the case.

However, the density of talent at top-schools is absurdly high. And this is a game changer for most early stage companies and even more so for soon-to-be-unicorns.


Probably why a lot of professors say that University is often about networking opportunities just as much as it is about learning.

But aside from knowing talented people, it takes visionaries to generate fantastic outcomes. Walt Disney for example stopped drawing cartoons himself early on because he had better artists around him. However, he had the vision that ultimately brought the best out of each artist.


I'm not sure you agree with the parent if the parent is saying "no, it's not possible, because the tech is so hard", and you're saying, "it's not an issue of technology, but the business side".


As an inexperienced engineer I did the same thing. My punishment for my hubris was to have bizdevs who felt the same way about us.

The weird thing is though, I still actively encourage new devs to make this mistake in certain circumstances. For one thing, telling them no isn't going to help, we have no shared context, and the math of their productivity versus mine works against me spending too much time convincing them.

If the problem has a logistical or bureaucratic component, once in a while the stars align and they manage to pull off something the department has been trying to do for years but always runs into a wall. Some people in an organization will assume something is true if they hear it from enough independent parties. I know if three people ask me the same question about a block of our code, I start looking at that code suspiciously. Clearly something is wrong with it. Sending the 5th guy to ask some IT guy or product manager he same question, especially in the way a new person tends to formulate questions, sometimes knocks the cobwebs off and the person decides to change their behavior or volunteer to do something they have resisted doing before.

A week of junior dev time for a 5% chance to clear out a bag of tech debt is a good investment IMO. And if it doesn't work they learn something about why things are the way they are.

I had a senior coworker who used to laugh and say "You're new here aren't you?" and I started giving him grief for doing it. Sometimes the new guy is the only one who can get things changed.


Great insight there.

Additionally, the new guy will be much happier when he's treated the way you suggest. I know I would have appreciated that instead of the bs I got at my first proper job in the beginning.


I always try to point out to the junior folks when a thing is janky and broken for "historical" reasons rather than technical reasons, in the hope that their fresh eyes on the problem will find a better solution. Sometimes we're still just coasting on the best solution we could come up with 2 years ago, and in those cases it's likely that we could do better if we re-think the problem from scratch.


And even that is only a small part of what makes Uber, Uber. If you want to actually compete with Uber, then once you've gotten all those minor details worked out, you're going to have to tackle the real problem which is dealing with 100's of different regulatory frameworks for offering taxi services, being a taxi driver (or a not-taxi driver as the case may be), hiring of private contractors and two dozen or so other entirely non-technical things neither you nor I have considered.


>> dealing with 100's of different regulatory frameworks

Isn't ignoring regulations the most often quoted reason for Uber's success?


Having an effective strategy to skirt regulations probably requires just as much knowledge of the regs as a company who complies with them.


I don't think that's really true in this case. In the few cities I know any details for, the Uber answer was, "we'll just do it until the city sues us". That doesn't really take any special knowledge, just preparedness with the lawyers.


> Having an effective strategy to skirt regulations probably requires just as much knowledge of the regs as a company who complies with them.

Not really. The "effective strategy to skirt regulations" seems to be "when someone makes an issue out of it, we'll burn investor money on PR and lobbying against the regulations, and on paying any penalties that get assessed for violations".

For that, you don't need much advance study of the regulations, just a deep supply of investor funds.


They don't "ignore" regulations. They violate regulations. They do so strategically or they would have been shut down long ago.

For example, if public opinion weren't on their side, perhaps due to, say, Uber rides being as dangerous as they're sometimes claimed to be rather than as safe as statistics show them to be, some of those "little" city lawsuits could have blown up much worse.


Unfairly, IMHO. It certainly helped, but they did so much, and better, than the other apps and they made legit, value-improving changes to the ride-hailing process: actually acting on feedback, dynamically setting prices, coordinating a movement away from tipping, applying modern IT.

Remember, their core product was not violating regulations: it was filling empty spots for legit black limo drivers, and was more expensive than taxis but still competitive because of the quality of service.


Ignoring regulations is easy.

Getting regulations to ignore you is the hard part.


That and the economic recession where lots of people who had vehicles were now trying to make ends meet. Also, they paid their drivers better initially until they build up their driver network. Of course leveraging the media helped and a symphony of other things all pushed them to where they are now.


That's what the problem is. You can invent something in a weekend; the red tape takes longer. I think that still counts as "built in a weekend."


Your comment makes no sense. Invent is a pretty meaningless word in this case. Invent doesn't mean 'make a product capable in working in the real world where it has to follow laws, physical limitations, and deal with stupid people'.

If you invent an app in a few hours, then get fined out of existence your app sucks.

If you invent an app in a few hours, then it crashes under high load, your app sucks.

If you invent an app in a few hours and your code sucks and you can't adapt it later, your app sucks.


> If you invent an app in a few hours, then it crashes under high load, your app sucks.

Yes

> If you invent an app in a few hours, then get fined out of existence your app sucks.

No


>> If you invent an app in a few hours, then get fined out of existence your app sucks.

>No

Sorry, but back here in reality apps must comply to regulations. It's your job as a voter to make sure your regulations comply with reality.


For one, not everyone cares about the rules. For two, rules rarely apply to all people all the time. For three, this is external to the apps and you can hardly judge the app for what arbitrary regulations their location has at that time.


Caring or not caring about the rules is irrelevant if those rules make it impossible for you to operate.

Uber has famously skirted rules and has gotten away with it largely because they've been able to throw VC money at PR, lobbying, and paying fines. Unless you happen to be similarly lucky funding-wise, your first round or two of fines for breaking rules can easily shut you down permanently.

Maybe it's a bit hyperbolic to say that this would mean "your app sucks", but the end result is that you built something that no one can use and you can't make money of of. That sucks.


Apps that stop working because the company folds definitely suck.


Something I find really interesting is that Uber drivers in the US are ordinary people. In the UK, they are straight up taxi drivers. It's the same brand for two totally different implementations of the idea.


You're going to talk about Uber? Watch this video: [1]. He talks about every regret regarding their infrastructure. At first you kind of nod your head along with "yeah, it's not that easy", but as he keeps talking you can't help but wonder why the company has absolutely no internal technical direction whatsoever.

They employ over 2000 engineers (just devs, excluding support staff, etc.). They use micro-services designed to the point where teams can't reuse pre-existing services. They are rewriting the same functionality over and over, and over and over again, as separate teams don't even have a clue what has already been written. Pick a piece of functionality you need. There are straight up 20 different APIs that do the exact same thing, but good luck figuring out which one you should be using - oh wait, why bother when you can just write the 21st version of the same thing. They are at the point where they have no clue what their repositories contain. In the video, he can't even provide an accurate number of deployed micro-services. THEY CANNOT EVEN ASSESS WHAT THEY ARE RUNNING IN PRODUCTION. They're literally drowning, and cannot recover.

The point of the article is wrong. It's not about what can be done in one weekend, it's what can be done in 12-24 months by a competent team. Yes, Uber operates internationally which comes with a very crazy set of challenges. 2000 engineers worth of challenge? Not even remotely close. You should need fewer than 100 people who really know what they're doing, and another 300-400 people who can follow the lead of the first 100. If you need more than that for a single product (ie: Google does not apply as they have 100+ full products), you are doing things wrong.

If you're going to provide the anti-thesis, don't use Uber as your example. They don't even understand how their business is managing to remain afloat from a technical perspective.

[1] https://www.youtube.com/watch?v=kb-m2fasdDY


I'm pretty sure Uber uses Google Maps in for a lot of it's mapping and navigation [1]. I'm not saying Uber adds no value - payment processing isn't easy, but it's not solving any new mapping problems that Google hasn't already solved. Its main innovation has been to ignore existing taxi laws by classifying itself as "ride-sharing".

[1] http://www.zdnet.com/article/uber-to-invest-500m-in-mapping-...


I doubt Uber would have gotten started without leaning on Google for both mapping and directions. This benefited them both as Google incorporated Uber back into it's Maps product.


Uber initially (and still does) rely Google Maps for a lot, but they have an internal team working on their own mapping, and have spent a lot of money acquiring map-related tech.


Now. But they initially bootstrapped it with Google Maps, which was sensible to do for their initial MVP, which is all an "I could do that in a weekend" project is.


They also use a home grown solution for routing: https://eng.uber.com/engineering-an-efficient-route/


Can you provide a source for the claim that Uber violates taxi laws?

As far as I'm aware, Uber doesn't violate taxi laws. Uber started out providing "black car" services which are not classified or regulated as taxis. Never have been, still aren't. Black car services are ones where you schedule a trip in advance, and there's no "hailing on the street". Being able to hail taxis is what makes them taxis and subject to regulation.

A lot of the drivers who first began working for Uber were existing black car drivers, and who operated their own black car business on the side. Many still do. Uber didn't come along and make this existing class of ride service illegal.

The only vehicles subject to taxi regulation are ones that offer rides to people on the street without prior arrangement, as far as I'm aware.


Non-hailed cabs are regulated in many cities as well. E.g. in NYC the TLC regulates black cars as well as yellow cabs.

https://www1.nyc.gov/nycbusiness/description/livery-car-base...



I was offered to interview at a company, they wanted me to do a coding "pre-screening" which consisted of creating a full-stack, and fully operational registration website from scratch.

Maybe I'm a super slow architect and developer, but I'd estimate that to be a greater than 8 hour time commitment.

I assume that the org's leadership acted in good faith, that someone honestly think's a non-slapdash solution would be a couple hours of work. It boggles my mind.

And yes, I declined their invitation to pre-screen and thus did not complete the interview process.


Depends which framework you pick for the solution. Using Django what you propose is doable within an hour.


With something like cookiecutter-django a registration site takes about 5 minutes.

https://github.com/pydanny/cookiecutter-django


a nodejs solution was the requirement, but maybe if the org's leadership was comming from a Python/php background they might assume using a scafolding was a given. I don't know anything like django for nodejs though!


Express + Passport can easily be done in less than an hour. It's super simple to setup. I've got my default template for every site I whip up. Now writing that from scratch is a completely different story. I doubt any of their current engineers could just put that together in a day.


isn't express.js kind of like a django for node?


No, Express is much closer to Flask than Django. One of my biggest gripes with Node is that there's really no equivalent to Django/Rails in terms of developer productivity and best practices.


That's what gives Express its appeal. Rather than a walled in environment, I can pick and choose the pieces that I need. Much more flexible if you want to move over to a different library within Express.


I understand that's why people think Express is great, but in practice you end up rebuilding a lot of the same functionality. It can work okay in a microservices architecture, but in the end you have to reinvent a lot of the wheel.

There's also a common misconception that Django is a "walled in" environment. It provides sensible default utilities for everything, but you actually don't need to use them at all. For example, you can bring anything from your own template engine to your own ORM if you want to.


Not sure what wheels I'm reinventing, I just choose to include the libraries that are correct for the application.

If I don't use half the stuff Django is offering me, only to bring in my own. Aren't we back to the same place?


The difference is that the Django ecosystem really focuses on developer productivity and has strong best practices. Look at 5 different Express apps and you'll find 5 different ways of structuring models, controllers, and routes.

Plus the fact that Django has some suggested defaults allows you to be very productive with them. For example, I really haven't found any equivalent to DRF for Express.

I want to make it clear I'm not some Node hater. Probably half my projects are still in Node, but I do know I'm much high productivity in Django.


So really all we're seeking here is a standard for application layout. I really really dislike being forced into some heavy handed framework, not that Django is that, but it's on the topic of discussion.


Ember? Sails?


Ember is not a server-side framework.

Sails is leagues behind Django in terms of everything from tooling to quality.


I thought Ember had isomorphism now...


Server-side rendering ("isomorphism") does not make a framework into a server-side framework. Ember is not designed to be a backend for your application.



Sorry, I'm not sure how that's relevant to my comment? I personally love the diversity of the JS ecosystem, but that doesn't have any bearing on the fact that Ember is simply not a backend framework.


Meteor


Meteor is not at all equivalent to Django. I've used both, and they're leagues apart. Meteor still can't even use SQL.

If you choose Meteor, it's going to dictate your entire stack from database to frontend framework. That's fine if what you really want is an all-in-one solution but it's definitely not a Django equivalent.


Just to confirm, you meant just register -> store to database + forgot password and validation right? (Greater than 8 hour time commitment do sound about right, but I'm just curious to make sure I read it correctly).


The instructions did not give an upper bound in terms of functional requirements, though the lower bound of "realistic use" was given!

So yes, I would assume that includes things like full email validation round-trip too.

Even just slopping together an in-memory, no-email-sent deployed solution is at least 3 or 4 hours in my book, which seems way excessive for a pre-screening question!


And you had to do that from scratch? No libraries allowed? That does seem rough. With libraries allowed you could probably spin something up with rails + scaffolding + devise though.


maybe libraries are allowed, but I don't know any that can really turn this into a 30 minute project. And even researching libraries is hours of work :) If you have any suggestions for this on nodejs, I'll be happy to add it to my interviewing toolbelt :D


Sorry for the late reply, Just noticed your comment. I don't know a way to do it in node. If you have setup devise a few times recently you could possibly get it done with ROR + the devise gem + scaffolding. This is a pretty terrible interview question though, as testing if someone has practiced a canned solution ( which is really all you could do in that time) doesn't seem that great, and actually taking the time to make your own solution would take much longer and an 8 hour unpaid interview question also seems bad </rambling>.


Cookie cutter Django. 30 mins seems on the high side.


Even if that were allowed as a solution, it is still a terrible prescreen interview task.

Depending upon whether or not you can use pre-existing frameworks/libraries they are either asking you to: 1) do something which takes so much work that it is completely unreasonable as a prescreen (which will filter out good programmers, who won't deal with that shit), or 2) they are asking you to basically edit some templates on top of some off-the-shelf solution (which will not filter out bad programmers).


I don't disagree at all, was just poining out it could be done :)

Coding to even talk to someone is a terrible idea.


What about user editable profiles with the associated valudations on email change ;)


I think you're making a related mistake, which is to overestimate the importance of the tech to the business.

When Uber first released, my guess is they probably couldn't do most of what you list above. But it didn't matter, because they had one compelling feature: they routed around regulation. This allowed them to pay drivers more and charge riders less than traditional taxis, while taking a cut for themselves. This is a clearly profitable position to be in, which means that they have the money to bring in capable business analysts and programmers, and the rest of what you say above will happen naturally.

The taxi apps before Uber didn't recognize that the killer feature was routing around regulation, and the taxi apps after that didn't do it first.

Sure, I can't write Uber in its current state in a weekend, but I could absolutely write Uber's minimum viable product in a few days. The problem is having the right combination of a good idea, the correct understanding of why it's a good idea, and the expertise to execute it (not just technical expertise).


You couldn't build the Uber of today without significant resources, however you could indeed build the version of Uber that made them popular and got them access to those resources in a weekend. Uber started out as a black car app. They got lucky, went viral, and the founders were well connected enough to get funding to build out much of the advanced functionality you're referring to after the app already became popular.

It isn't the advanced functionality and scalability you're talking about here (that was built in reaction to its popularity) that made it popular. It was their MVP.


To me the question isn't "Think you can build it in a weekend?"

You mentioned 5 product "concerns" that Uber must contend with to make a point. No doubt if we brainstormed more we could come up with perhaps 10 or 20 more such "concerns".

Now let's be very generous and assign 10 engineers to each concern.

I still come up with a maximum of 200 engineers for Uber.

But Uber has 2000 engineers.

I don't see any creative ideating that would justify that many engineers _as necessary for the top or bottom line operation of Uber_.

They are there for other reasons but not because Uber needs that many engineers to operate.


> You mentioned 5 product "concerns" that Uber must contend with to make a point. No doubt if we brainstormed more we could come up with perhaps 10 or 20 more such "concerns".

> Now let's be very generous and assign 10 engineers to each concern.

> I still come up with a maximum of 200 engineers for Uber. But Uber has 2000 engineers

There's two ways to interpret this: 1) Uber is overstaffed by an order of magnitude. 2) You are underestimating the complexity of Uber by an order of magnitude.

Uber might be overstaffed, but I suspect the answer is mostly that you are drastically underestimating and don't see the complexities of running a large-scale service in multiple countries, dealing with regulations, worldwide scalability, time and cost estimates, surge pricing, tracking and mapping, payroll, billing, payments, apps, internal IT, etc., etc., etc. There aren't 10-20 "concerns". There are hundreds.

I bet they have at a hundred engineers just working on billing and they're all overworked.

> I don't see any creative ideating that would justify that many engineers

Because most of Uber's engineers aren't there to build "creative" solutions. They're there to enable the business.


I agree with your two interpretations. But to clarify that is my position: I believe Uber is overstaffed by an order of magnitude.

It seems after reading through this thread that I am definitely in the minority on that opinion, but I'm not sure if I understand the position there.

How many engineers would Uber need to have before one would say "okay, that's too many". 3000? 5000?

Uber can afford 2000 engineers. That doesn't mean they are anywhere near good at utilizing them, which seems to be the implicit assumption in many comments on this thread.

Google has in the tens of thousands of engineers, which they can afford to have because of Search. I highly doubt they are anywhere near reasonably utilizing those engineers to top line or bottom line product value.


> How many engineers would Uber need to have before one would say "okay, that's too many". 3000? 5000?

However many it takes before the cost of an engineer is higher than the value added. I have no idea where that line is for Uber. Maybe it's 10K. Maybe it was back when they hit 50.

> Uber can afford 2000 engineers. That doesn't mean they are anywhere near good at utilizing them, which seems to be the implicit assumption in many comments on this thread.

This might be true but it doesn't support your point. If Uber is bad at utilizing engineers, that doesn't mean they should employ fewer. In all likelihood, they'd be about as bad at utilizing their workforce if they cut it in half.


Netflix is another example. A couple of thousand programmers and tens of thousands of VMs to build a top drawer e-commerce site. I built a drawer-below-top e-commerce site with twelve people in a year, and a third of them were muppets.

I interviewed at another well-capitalised streaming video place, and they also seemed overstaffed. They had a permission service that worked out what people could watch, based on purchases, subscriptions, and bundles. Something that's basically one SQL query. They had something like six people working on it full time.


Is it not a problem to people that this many people would be required just for billing? I don't want to grossly oversimplify but at some point people need to start getting judgemental and saying, "no, we have created something too complex. We have to simplify."


You clearly have never worked anywhere close to payments :)

Kidding aside, the regulations around anything that handles money are quite complex, and the liability issues mean that you'll spend a lot of time on getting it right, too.

In theory, payment is super-easy. In practice, any payments solution that is not just a toy eats engineers for breakfast. Stripe is probably one of the leanest ones - and they have 400+ engineers, lean on existing infrastructure, and cover only 25 countries.


>> I still come up with a maximum of 200 engineers for Uber. But Uber has 2000 engineers

> There's two ways to interpret this: 1) Uber is overstaffed by an order of magnitude. 2) You are underestimating the complexity of Uber by an order of magnitude.

The third interpretation is that we have no idea what else Uber is working on besides the operations that are visible to us as consumers. Uber can afford to pay 100s of engineers to build out a few technical concepts/prototypes that will never see the light of day. They basically need to do this, because their unicorn valuation isn't based on chasing the vanishing margins of the taxi industry, its based on owning an entire market that won't exist until they create it.

I suspect its a mix of all three.


Exactly, Uber needs to scale along axes in a way most traditional web companies do not.

The axes have been pointed out are heterogeneous city/state/nation regulations, labor law, naviagation/connectivity issues, etc. And onto each of these diverse cases, there needs to be a way to analyze results consistently and deploy business objectives from a centralized corporate strategy.

As you point out: to justify its unicorn status, Uber needs to prove that it can succeed not just in competing with taxis in the major markets where it already exists, but that it's technology platform can be deployed to the edge cases, and the areas where it hasn't.

So what the extra engineers are doing finding a way to acheive user reliability under the reality of accelerating changes to requirements and sharding of policies for separate operational units that will result from forced-growth into non-optimal markets.


To repeat myself from an earlier comment. How does one even arrive at the number 200 based on the complexity of the problem? Secondly not all 2000 will be working on their core app. There will be solutions required at each level. For example in some countries cab services are provided by larger contractors employing drivers and not individual drivers themselves. So Uber needs to have a billing system which collates data from their individual drivers and then calculates the payout to the company. This requires a separate set of engineers, not working on the core app. And then there are HR systems, Finance Systems, every bit they are not contracting to an Indian Service company has to be handled by an "engineer".


Okay. Then how many engineers would Uber need to have for you to say "that's too many"?


I don't really know. What I do know is people seem to misunderstand engineering team size as a simple sum of people working on the Uber app. It is not only the android/ios developer or the full stack developer who is an engineer in Uber. An IT administrator who helps people, technical or non-technical, with their work laptops also qualifies as an engineering team staff to the outside world.


What makes it your or my place to decide that? They're a private company, and they have plenty of profit motive for efficiency. If they didn't think they were getting value from all of those engineers, then they would fire them and save themselves some money. Or are we going to argue that corporate executives aren't greedy enough?


And my answer to your question is: your reply is pretty far off base. Quite simply: when Uber first launch, Uber was not the Uber you are describing. Telling an engineer they need all these things from the start is completely wrong.

Yes, competing with Uber now, with "this can be done in a weekend", is naive. But that's not the point.

The point is, that there are plenty of problems you can make a lot of money on, with a timeline anywhere from a week to a year. Some will be sustainable, and some will be simple cash grabs. The hardest part is identifying not just what, but when. If you have enough connections in the industry, it can be much easier to predict this than if you are on the outside of things.

I have built plenty of things people would say "I could do it in a weekend". And yes, a weekend is a little short, but many of them took just a week, at least initially. The answer is not "Uber does all these things, you need to too." The answer is, "you need to predict the next Uber, so you don't have to do all the things Uber currently does."


Fully agreed.

However, remember l that Uber (probably) started without any of those complexities. Probably just on a weekend.

So I think the absolutely incorrect "I can build this in a weekend" could be rephrased as "An absolute MVP (Or a proof of concept) like this product could be built in a weekend".

This would make sense, right?


"An absolute MVP (Or a proof of concept) like this product could be built in a weekend".

Maybe, theoretically, in some cases. But even so that's very different from the original "I could build that..." bloviation.


I'm pretty sure uber doesn't do any of that locations or traffic routing stuff you mentioned. When I take uber the driver just has their favorite app if Waze, Google or Apple Maps for the actual routing or directions.


Or the passenger in the case of Uber Pool. I couldn't believe what Uber expected of a driver when a new Pool request came up. They had to hit a button, look at the screen, rethink how they were going to go somewhere. Oh and then worry if the requester will cancel.

That happened on my last trip. I felt bad for the driver so I helped him out finding the best way to pick up a passenger after the first two cancelled.

I think Pool is a good thing but not in rush hour traffic where going slightly off the route adds 10 minutes to a trip minimum.


It has to for the time estimates and finding the 'closest' driver to you.


For me the time estimates are always super incorrect and seem only based on the distance.


I use to live next to a highway with the closets exit being a few miles away and I would constantly get drivers on the highway as my closest driver. I always had to cancel since they were actually ~15 minutes away, not 3-5.


I have found that Uber's estimates get better with time. When Uber was newer in my city, it used to give ridiculous underestimates. They've become far more realistic now.


I think the critic has the benefit of hindsight which may translate to 1000s or hours saved in DEV. E.g. Copying Uber verbatim versus all the experimentation to get there.

Hence it's not uncommon to see clones of things (e.g. Dropbox, slack, etc) developed at fractional cost.

Your junior may be onto something , get shares in his. Ew startup quick !


I think you can ask a simpler question: how do you get traction, drivers and passengers included? You application will be invisible, nobody will see it.


"If you guys were the inventors of Facebook, you'd have invented Facebook." The Social Network.


Even more than that: code is only a tiny part of a tech business, even one that has code as its most critical foundation.

A classic example is banking or money transfers (like paypal). So many people think that financial services are nothing more than moving bits around, which seems like an easy thing to code. Aside from the fact that it's not as easy to code as one might imagine, that part is maybe only 1-5% of the business. Paypal got where they are not because of the quality of their code or their operations but because of their relationships. They established relationships with banks, with regulatory agencies, with governments, and they spent years and years doing these things. They signed contracts, they established trust, they created systems for interfacing with many different banks and governments (not just tech systems, but policies and personnel). The result is that paypal is now one of the easiest ways to move money around the world with minimal friction. Cloning paypal's tech wouldn't enable you to clone their success, for those reasons.

Same thing with Amazon. Amazon's user facing software is only a fraction of their operations, their main competitive advantages are selection, price, fast fulfillment, and customer service.


>Yet none of them were uber?

You underestimate the amount of luck involved in becoming Uber. It's not all luck, but luck is a major factor.


Consider this then: If you could build an app as polished as Uber's, there are hundreds of minicab companies in the UK alone, most of which have apps massively inferior to Uber yet has to compete with them, and that are thus a market for better apps.

There probably is a lot of luck involved in becoming Uber, but if you can make an app that good in a short amount of time, then the opportunity cost in doing that and sending a demo round to a few dozen minicab companies is very, very low.

Yet these companies are still stuck with horrible apps, because in reality it takes a lot of time to develop a good one, even if a prototype is fairly trivial.


After several startup failures, the number one lesson I have learned is that you better have the right team who can execute. Uber has the right team http://www.businessinsider.com/the-power-players-of-uber-201...


I think a related thing people say (that makes me cringe) is that xyz is "just a CRUD app", quite dismissively. You could argue that the vast majority of applications are "just CRUD apps"! Once you get up to a certain scale, even "basic CRUD apps" are complicated.


A lot of the work is convincing people:

1. Convince customers to download the app, give their credit card, and then trust their rides and selves to a driver

2. Convince drivers to clean up their cars, download the driver app, and turn it on, and wait for customers to ask them for rides


The hardest parts of starting and running a business generally have nothing to do with tech. That is even true for most software businesses. This is what most developers who have not started or ran a business don't truly understand.


> Can you spot busy periods and increase the fares according to the laws of supply and demand?

And beyond the technical issues: Can you effectively deal with the negative PR that results when you increase fares during a natural disaster or terrorist attack?

Uber isn't just an app, it's also a company and a brand and a service. It's something people understand, it's something that's popular, it's something that people rely on.

So sure, let's say a really good dev could crank out a software engine that does everything Uber does in a month. Now what?


You seem to be implying that really good devs are just lying around not being used whereas great PR guys and execs are the real value. I don't buy it.


I'm not really sure how you got that from the words I wrote. There's tons of work in addition to the tech which leads to a success story like Uber. That doesn't imply that the tech doesn't matter, just that other stuff does also.


The evidence of PR being effective is right there if you think about it.


The evidence of good tech being important is at your finger tips.


Well, with 2000 engineers I could do it in a weekend :)

Anyway, here's a related link: https://www.quora.com/Why-do-AirBnb-and-Uber-need-so-many-en...

And here another: https://news.ycombinator.com/item?id=12597232


You couldn't.

No way you can delegate work to 2k engineers and tell them what to do for next 2 days. Physically impossible.


Not my problem - I'm their manager. If I have 2000 engineers I just make one of them Lead Dev and then shout at him when he hasn't been able to deliver after I gave him 1999 software engineers. What an idiot.


So Dilbert got promoted then demoted...


Yes, Amdahl's law.

I'd probably just use the 10 most productive engineers, and tell the others to prepare for testing :)


I don't want to sound bad, but again you couldn't do it.

Say you have 2k engineers for next two days. Where do you begin?


You are correct, but are taking these guys too seriously. They know about the mythical man month and are just riffing, for better or worse.


Describe the app generally, and leave it to them to form their own teams and solutions. Hackthon style, and hackathons at least usually have functional prototypes by their end.


Nope. Nine women can't make a baby in one month.


Just use one "10x woman".


Unless you pipeline them.


While I agree 100%, I think the answer is more satisfying when viewed a different perspective.

It's not that Uber needs all those engineers. It's just a question of marginal cost vs marginal return. Uber made 1.5 billion in revenue in 2015. An engineer costs roughly $150k / yr. An additional 100 engineers only have to make enough small improvements to increase revenue by 1% to pay for themselves.


> An engineer costs roughly $150k / yr

This number is far too low. Uber is headquartered in San Francisco.


ALso, they are literally trying to solve a problem similar to the travelling salesman problem when you consider Uber Pool scheduling, Uber Eats, etc.


True, but they don't have to find the optimal solution, or even anything close to it. They just have to find a solution that's slightly better than the next-best solution, which in most cases is very far from optimal.


And then you have to get a million people to say, "I'm just going to Uber there." Except replace Uber with the name of your new app.


Also, all the edge cases which become visible only after real users start pounding the service exposing all "this should never happen" conditions.


Data federation is the key to complex data. In a month or two, the first cut anyway, I doubt it could be done in a week-end.


And on top of all of that, can you get 100K people to use your app?!


Oh yeah, infrastructure & marketing ;)


Thanks for the nice specifications!


Uber didn't start with all of that either.

The MVP can be built in a weekend though.


Nope, that is the point.


Also, it's often the case that it's better to buy a tool and integrate it into the company than build from scratch. It's especially true with a small team. It's not the most romantic idea and sometimes engineers feel that they could build the solution themselves in a weekend.


I think you chose a bad example. Imho Uber has waaay to many engineers for what they are doing: to the point where they need to invent work for themselves with no business value.


I recently had a discussion about how hard it would be to have a system inside a bus to announce the next stop. Sounds like a weekend project right?

Think a bit further:

- Hardware needs to be resistant to harsh diesel engine vibrations

- 3G/4G connectivity

- Need a mobile data contract with local ISP

- Software needs to be able to handle network disconnections

- GPS needs to be able to pin-point at which stop the bus is currently stopped, even with bad GPS coverage in larger cities

- If the bus skips a stop because of a detour, the software should be able to detect it and announce the next stop

- The announcement should be bi-lingual for Airport busses

- The announcement should work for people with hearing aids

- Server needs to know all bus-stops

- Server needs to know different stop types, such as bus terminals, intersection stops, regular stops, hand-over stops, virtual stops

- Server needs to be able both work of a yearly bus schedule and real-time update

- Server needs to output in an understandable JSON/XML because the government subsidies demand an open-data format

- Server needs to publish data to an open-data server because of the subsidies

- Because the bus-company is sponsored by European Union money the control interface should be translated to German/French/English/Spanish

And now this weekend project takes a team of 10 engineers working for a year.


All London buses do this. I don't think you need a server, 3G/4G connectivity or even GPS really. You might be over thinking it.


On the other hand, the overthinking in question (which is probably true for bare-bones next stop indication) provides the base for my personal bus bug: know if/when a bus is reaching a stop from outside the bus in question, especially with geofencing, in order to know whether the bus I want to take is running early/late/not.


This is available for public buses in NYC. The city added GPS to their buses and then the ability to query distance from current stop via SMS.

Then they opened their APIs, and now I have a half dozen apps to choose from that will show the approximate time until the next bus from a particular line will reach any given stop.

(Not always 100% accurate, but useful enough to know when to wait indoors for 5 min if it's raining, or pick another line/choose to walk because of delays.)


They actually do this as well (and many have live updating signs), so the buses do have GPS linked to a server of some kind. It's normally absolutely fine, but on the couple of occasions for me where it broke it seems that the server doesn't conduct the bus in any way and kind of hopes the bus is on the right route.

That's all speculation though.


The TFL system will remove a bus if it doesn't get an update often enough that includes actual progress on the route. This is in fact the opposite problem of what you mention, in that they don't do a very good job of predicting based on past performance. E.g. I used to take a specific bus to pick my son up from nursery that always would "disappear" for a certain number of minutes at a certain part of the journey because a very short stretch of the route was always extremely congested at that time of day.

It's a tricky one, because they have big concerns about over-promising (e.g. the bus could be stuck behind an accident instead, in which case guessing that it's 5 minutes away because of past behaviour will really annoy people) so in some situations they err on the side of caution.

But they are unable to get rid of all of the corner cases. E.g. a bus that moves rapidly for a short stretch, then gets "stuck", will often appear, disappear, appear, disappear, with much less progress in time estimates than elapsed time should indicate.

The thing is, the team working on it has a huge amount of gathered info on how to give these predictions in a way that tries to minimise surprise and avoid making promises that cause problems for people. E.g. often it's better to take a bus off the list so people who are in a hurry will grab a taxi, rather than "promise" the bus will be there in just another few minutes. They still get it wrong too often, but I really don't envy them trying to get the right balance between promising too much and too little.


> They actually do this as well (and many have live updating signs)

I don't know about London, I've seen some systems where the stops had "live-updating signs" but only provided the time of the next official passage. And while I wasn't clear, my bug is not just stop signage but phone access as well (that's most useful when e.g. it's raining cats and dogs and you want to avoid waiting in the rain at uncovered bus stops)


There's a standard by Google for transit timings, and Translink, the public transport authority in my state, outputs these times. They're live.

Apps like 'Transit' also (accurately) show the current GPS location of the bus as it approaches, and it updated every minute (I believe)


In London you can go to the TFL website and see live updating times for when the bus will get to your stop: https://tfl.gov.uk/travel-information/stations-stops-and-pie...


Oslo, Norway has realtime data available for all stops as well :)

http://reisapi.ruter.no/Help/Api/GET-StopVisit-GetDepartures...

>Returns a List of StopVisits (departures) from a Stop. If no time parameter is supplied, departures in realtime will be returned.


OK, that's really cool. Definitely means they have "a server, 3G/4G connectivity [and] GPS" though.


Not necessarily. I think some of the systems use radio links to a central location and transponders at fixed points on the route to track the buses.


There is an API for London: https://api.tfl.gov.uk/


Halifax, NS, was doing that in the mid-'80s without the benefit of GPS or handy mobile apps. (Every bus stop/route had a phone extension, and major stops/terminals had talk boxes.) Granted, it wasn't a huge system, but it was hitting rocks with sticks and hollering over tin cans and string compared to what's available as a starting point today, and it worked. The idea that it should be a hard problem today, given GPS, vastly improved wireless communications and eleventy-seven different ways of figuring out where a user is (if entering a stop code seems like it might be too much work) is just plain silly. Or we've significantly redefined what "hard" means.


Or expectations have increased, both in terms of accuracy and in keeping costs low.

Consider that e.g. in London the old route indicators on the buses used odometers. Which worked for telling you were along the route a bus was. Most of the time. But didn't tell you how long it takes until the bus reaches the next stop other than on average, as it would tell you nothing about traffic etc.

Today London buses don't use just GPS to replace it either, because that's nowhere near good enough. See the "Tracking" section here [1].

https://en.wikipedia.org/wiki/IBus_(London)


When I said "worked", I meant worked, not approximated working or came lose enough not to have caused too many upsets. If expectations have grown to be higher than plus-ten-second accuracy (early wasn't tolerated), then we collectively need some sort of counselling/therapy.


You don't get that kind of accuracy in any system that has to deal with congested roads, or any dense major city. E.g. there's plenty of bus-stops near me where stops on the same route involves going around corners and traffic machines and stopping close enough that GPS would be insufficient to be guaranteed to tell the stops apart if you don't get a good enough fix on additional satellites quickly enough.

Even with all the extra data, and a lot number crunching, in London it's a challenge to get it to withing a few minutes during rush hour.

So, sure, if you're dealing with simple, small systems with little congestion, you can simplify.


>know if/when a bus is reaching a stop from outside the bus in question, especially with geofencing, in order to know whether the bus I want to take is running early/late/not.

We had that, through GPS + GPRS (or SMS?), back in 2001-2 in a small city I was living in Europe. Complete with full (flash based) website map of where each bus is.

I used it to get out of the tech park/institute I worked at and into the bus stop outside at the very last minute (since bus timetables where completely unreliable).


GPS is insufficient to give accurate enough estimates these days. It may or may not work depend on your local routes and stops, but it's too imprecise for a lot of locations and traffic conditions to give decent precision. It's good as a "first approximation" and if you don't have anything better. Most systems today end up integrating data from multiple different sensors to improve on gps.


This is the only thing I want. Displaying a bus timetable can be done just fine with laminated paper, and I'm not convinced that the ability to update (with cancellations, route changes, etc) justifies a tech solution.

Displaying the actual bus status, though, is huge progress.


Many providers have an API for this. I used one in my city to build a very simple display that shows you bus times (and I put it in a toy bus):

https://i.imgur.com/LjV7MtB.jpg


Sydney even uses the electronic ticketing system to provide an estimate on the capacity of the bus.


Thats clever. Most public bus systems I know in the US still see frequent cash payments onboard (the elderly, mostly). Is this not the case in Sydney, or is the system cashless?


Over the past few years they've rolled out a new electronic ticketing system which is used exclusively.

On some bus routes (and/or outside of peek times) you can pay a cash fare on the buss, but even that is still a single-use paper RFID card that should be tapped on and off. These sales are in the extreme minority though - the elderly require an electronic card to get their discount IIRC.


Really?! Where can I see this? On an app? I admittedly only use Google Maps when I'm in Sydney.


TripView has it.


I work part time at the company that manages the new style signs for bus stations (EDIT: in London, to be clear), so we've gotten to see a bit of the size of the operation TFL has to deal with gathering and processing the bus information. He might have some details wrong, but he's certainly not gotten the complexity level wrong.


what about the bus stop broadcasting a signal with the id, name etc? the system is activated by the proximity..so you can have smthing like approaching station A, leaving station A.

it is scalable because the bus-stop transmitters are added , moved, removed as needed.

updating the data can also be done remotely


My local bus network use M2M Sim's running on the Vodafone network but its basically a virtual private mobile network running on top of Vodafone's network with no connection to the internet.

These connections are in both the busses and the stops. The reason is so the stops can provide real time updates to people at the stops too. If a bus breaks down its no longer tracked on the real time updates so you are not effing and blinding because a the stop said there is a bus due in 5 mins and no bus comes. And if a bus is removed from service unexpected then the information on why can be feed to people waiting for the bus. People are less likely to be pissed that a bus didn't come when expected if they are given a valid reason.

Personally I was hoping they used some form of radio link I could of sniffed with a SDR so I could put a live updating sign in my house. I just scrape the data from their live feed they use to update the tracker on their mobile app/website instead.

Broadcasting out a signal from the stops could need a transmitter that may need to be licensed as the signal would be to be picked up quite a distance from the stop. Also not all the stops in my area are these "real time updating" stops as they are normally put into covered stops (EDIT: Bus Shelters... The correct term wouldn't come to mind while I was typing this.) and some of our stops are just poles in the ground with a sign on them. The busses tracking their own position wouldn't need these stops upgrading to give this information to people on board the bus as the covered stops can be quite a distance between each other and removes any issues if a stop get damaged and stops broadcasting data (say a car crashes into it).

The busses tracking themselves also have benefits that don't visibly benefit passengers. The services back office can monitor busses and spot problems on the network as they happen. Unexpected heavy traffic on a route may trigger the company to skip or stagger a bus (one of my local routes in advertised as being "every 5-10 mins". The real time traffic condition information could also be sold on to other companies (not sure on the value of this data esp for busses that run once an hour or even longer).


There are systems that solve Bus shelter problem by piggybacking on FM station RDS

http://www.windytan.com/2013/11/decoding-radio-controlled-bu...


I was surprised my local network didn't do something similar as it seemed like a logical choice. But after some research I found nothing about them using RDS to do this.

I would love to be able to talk to a tech at the travel network about this. My own research came to the suspicion that because where I am in the country they would of had to strike up deals with multiple transmission sites to get the coverage needed.


That's more expensive because the cost (installation and servicing) scales with the number of bus stops.

Also, there are things you want to do that this doesn't cover. For example, announcing the next stop when you pull away from the last stop.


The announcing the next stop could be achieved with a dual radio setup, one high powered and low powered. The high powered could be used to detect an upcoming stop and the low powered could be used to detect proximity to the stop.

(Lets ignore the security issues that have been shown with amplifying the low powered radio in cars for a second) It's a method used in keyless entry in cars. A higher powered radio is used for button press on the cars key so you can unlock the car/open the boot/sound the horn to locate the car at a distance but keyless unlock should only work at a much closer distance.

Not saying its the correct way to do it (my local bus network doesn't do this) but stop proximity can be worked into their idea.


German bus drivers have it as well. The routes are lists of stop names and their change is triggered by opening the doors. Can be adjusted by the driver if stops are passed, etc.


Many cities also have an app that displays how many minutes the bus is expected to run late, based on when it passed the previous stop.


Runs on some windows if I remember correctly and routes are probably hardcoded along with the data for the segment display outside above the windscreen.


They use GPS receivers on the bus, which calculates the position then transmits back using GPRS for central reporting https://en.wikipedia.org/wiki/IBus_(London)


Same in Munich. I think those work offline, also the driver can pick the next stop.

I guess it is easier in the long run to have all routes mapped and instead of GPS you use something like doors open / doors close as a trigger to jump to the next entry.


you need it. what if you want to connect to something and make some special announces.


What special announces? Either the bus driver does it or it's one that's used frequently (like 'Please move down inside the bus'), in which case there is a button for it.


The GPS and mobile data are overthinking it. The routes are known so simply trigger on door open/close (with some logic for denounce/etc) and cycle through.

For maximal accuracy simply have an electronic beacon of any sort broadcasting an ID that corresponds to the stop and you've got it solved.

And the entire data set for the route should be able to live offline onboard the bus. Changing bus/numbers routes should include the step for updating route data.


Except that quickly becomes inaccurate if the bus driver decides to re-open the doors for an incoming passenger, or due to traffic conditions, opens the bus doors further from the stop than the beacon reaches. Or skips a stop due to road conditions.


1. Set a minimum time until next stop can be announced 2. Add a button to skip a stop


And then when the driver accidentally hits the button and needs to go back?

Notice how we're now adding more complexity now anyway though...


> Set a minimum time until next stop can be announced

This feels like a dangerous hack that will come back to bite you in any number of unexpected scenarios that probably occur frequently during an average bus driver's day.


Consider a bus stuck in traffic. It is not unusual, at least in London, for a driver to open/close the door multiple times between stops when traffic moves slowly enough, often with longer intervals in between than there usually would be between stops.


RFID like chips on bus stops too, maybe?


While active RFID is possible, other radio tech designed to work as beacons is more likely


Bluetooth beacon (estimote?) at stops would help.


This has all kind of usability issues (now you're expecting the driver to pay attention to and correct mistakes), while providing less accurate data (buss pulls away from stop, gets stuck in traffic; estimates at next stop are based on the bus having pulled away and average times; customers gets angry) and less ability to do traffic management (e.g. spacing out buses by slowing down later buses slightly) without having drivers guessing and reporting data back.

> For maximal accuracy simply have an electronic beacon of any sort broadcasting an ID that corresponds to the stop and you've got it solved.

That's not remotely accurate for the reason above.

> And the entire data set for the route should be able to live offline onboard the bus. Changing bus/numbers routes should include the step for updating route data.

It can. E.g. the old London system used odometers. But that was sufficient only for showing "next stop ..." and for reporting very rough estimates. Today it uses odometers, gps, rate gyros and turn rate sensors to be able to more accurately position the buses along a route, and they still regularly "give up" and take a bus off the schedule when traffic conditions means the uncertainty is too high.


For an robust system I would use multiple data streams including bust status data/beacons/GPS. I've seen busses with offset stations.

3G is necessary for live ETA, if you want to get fancy. All these complications, in my view, are worth it if you want a system that works 24/7 and serves thousands of people daily. The HW and SW dev costs are acceptable, but it does need a decent team, depending on what the starting point is and the ability of the team members.


> For maximal accuracy simply have an electronic beacon of any sort broadcasting an ID that corresponds to the stop and you've got it solved.

Yeah, and now we need to install and manage thousands of beacons in bus stops in pretty hospitable conditions.

Of course, GPS isnt perfect either. It tends to fail in CBD areas.


changing bus/numbers routes manually is not feasible. in complex networks this happens daily a hundred times.


I've worked in transit. This is not only feasible, it is how it is done.


cannot see why you would want to do it manually. it is done automatically in munchen.


I'm confused, are you suggesting there are places where bus routes change hundreds of times a day?

I can't imagine many reasons a route would have to change on short notice (accident blocking a road?), and in those situations the announcements not quite being right would not be the end of the world.


Buses are often assigned to a route and then leave that route unexpectedly in the middle of the day to service another route (accident, other bus broke down, traffic, subway shutdown etc) , often not starting at the beginning of that route.


Then the bus changes its route number - certainly in the UK I see the drivers punching in the route details when they take over a bus.


Oh right, that makes more sense. I guess you could just keep a record of all the different routes in every bus though? And skip to the right spot manually as needed.


One of my "fights" at work is prototypes. A prototype can be build very quickly (cheaply) and can exhibit many problems that may be overseen by the "heavy design process". A prototype is a marvelous way to get feedback from users very early in the projects.


The only downside to prototypes is that then the managers don't understand why the actual project takes so much longer and end up asking to just go with the prototype for production.


Get better managers.

I've never had this problem.


>- Hardware needs to be resistant to harsh diesel engine vibrations

That's a non-issue.

>- 3G/4G connectivity, need a mobile data contract with local ISP

This is a duh!

>- The announcement should be bi-lingual for Airport busses

It can always not be. How would that be worse than NOT having the announcement at all? Besides what would the other language be? French, German? That would still be useless to 90% of foreign visitors...


> Besides what would the other language be? French, German?

I guess the names of the stops are untranslatable anyway :)


Canada. Here it would need to be bilingual by law.


umm I travel in GO and DRT everyday. Neither have bilingual announcement. Same with Mi-way.


Sorry, meant Montréal.


This exists in offline versions.

I think it just counts the number of stops and has some input from the driver


But, adding the wireless connections and the servers and the technical staff to code and maintain, would save the driver the labor of pressing the button. That's what technology is for.


KISS applies here. By keeping the 'driver presses a button' interaction, the complexity is reduced by at least a factor of 5, and the TCO ends up being much less.

Sometimes, a doorbell just needs to be a doorbell.


I am so glad you said this. With the boom of the tech industry, it feels like a lot of stuff is getting over engineered these days. Instead of products that focus on something that does it really well, it does its job mostly alright with half-broken bells and whistles.


Sure but if I add all the above complications then I get automated bus tracking system where everyone on their phone can see where the bus is and when it's arriving, I can track and automatically detect issues with traffic, with the bus, with the driver.

Sure you can do the simple announce but the even easier one is to just have the driver talk over a mic, he needs one anyway. There I one upp'd you on KISS. But there is a reason not to KISS.


You can publish wireless / Internet data on where the bus is, but keep the driver pressing buttons as the source of the data.


How often do you want the driver to press buttons? And how do you mark the intermediate positions in between bus stops? Because on any remotely complicated high congestion route, you will not get reasonable estimates if you only update at bus stops.


That's how it works in our (German) city. It works great.


I think you missed the implied /s


I think the GP was being sarcastic ;)


Except asking a bus driver to keep track of button pushing is distracting and could endanger the lives of the passengers or people outside. The stop needs to be announced 30-60 seconds before they arrive at the stop so then is driver is still actively driving. This is something the drivers union would be all over, it's not as straightforward as you assume.


The stop needs to be announced 30-60 seconds before they arrive at the stop

Why? You can announce the next stop a few seconds after departing from the current stop, and then again at the opening of the doors.


If there's a long gap between the stops, people may forget what's the next stop. Announcing at the opening may be too late or the stop has to be long enough. Even for slow passengers in a crowded bus. Announcing a minute before the stop is a good heads-up that bus is approaching the stop and people should prepare.


Then make the delay for the announcement of the next stop adjustable, configured according to the timetable.


but the driver forgets to press the button and then people are affected. ah, was that my stop, why wasn't it announced?


That's why bus drivers get training. And if they don't perform as expected, it's not that hard to hire a better one.


A skilled engineer weighs the costs and benefits of a system before architecting it.

Is it more economical to pay for constant cellular connectivity across 1,000 buses, support IT staff for when the system fails or needs upgrades, etc. or to have the guy you're already paying $15/hr press a button after each stop?


At least on most UK buses, the driver already needs to tell the computer where on the route the bus is so that it can issue the correct tickets to customers. This has been the case for decades.


It could just use the open the doors button with some hysteresis for opening/closing doors few times in a row in some short time.


Check out APRS[0]. Solves most of the above already by centralizing data-collection. Would just need a beacon on every vehicle. No need for internet connectivity. Receiving stations can be scaled up as required. Logging of course not an issue as can be logged on-board and/or externally.

The on-board display can overlay a map and call the next stop based on proximity. Not a weekend project, I'll admit, but many existing moving parts available off the shelf.

Hams play with this stuff... a lot.

[0] https://en.wikipedia.org/wiki/Automatic_Packet_Reporting_Sys...


I don't think anyone would have assumed that you can build such a system in a weekend.

Half of your points assume that current transportation companies don't have any digital data about bus stops and reroutes.

Anyway I haven't been in a bus that doesn't have such a system for at least for a few years (in western Europe and China).


> Half of your points assume that current transportation companies don't have any digital data about bus stops and reroutes.

The question is what is more work: Using the existing data that is stored in some obscure format for which you first have to write an importer/converter or just recreating the data. Add the fact that the existing data might need some cleanup/additions (e.g. times on timetable are only stored with minute precision and you want second precision for more accuracy).


Alternatively, you could have the driver press a button.


Far more useful is the system which tells you, at the bus stop, what the ETA of the next bus(es) is. And that really does need the whole GPS+data system, along with physically robust signs. But on the other hand, that can still be done by a small company - I know there's one here in Edinburgh but I can neither remember their name nor find them in a search.

You can do quite a lot with a small local company. It's global reach that's expensive, as soon as you start needing any kind of high-touch sales.


You can do this with a RPi and a RFID reader. Every station would have a tag, when bus stops tag is read and your script just announces the station.


You can certainly do it with an RPi - although you might have to keep rebooting it every hour or so. :)

GPS and 3G/4G are trivial add-ons. There are stacks for mobile data.

RFID doesn't have the reliable range, so that won't work as a solution.

It's not really a hard problem for a competent small engineering team with a mix of embedded hardware and software skills. It's not quite a trivial problem, because there are some interesting edge cases, mostly when a bus is diverted. But there is absolutely zero rocket science required.

You could hack something unreliable together in a weekend, but it would take up to six months to get all the bugs out, do the production engineering for a bullet-proof solution, standardise the installation process (if retrofitted), and so on.

Double that if management is poor. :)

Those last parts often get forgotten in software. Hacking something together that kind of works if you don't look at it with a visible frown is hobby coding, not engineering. Engineering is the boring work that happens after that, to build something with a quantifiable many-9s uptime that meets or exceeds a standard spec with comprehensive test cases.


Usually the announcement is not "we have stopped at Foo and Main". The announcement is "Next stop Foo and Main" while you're on the way to it, so you have time to sling your bag back over your shoulder, get your coat back on, and make your way towards the door. Depending on the density of stops along the route this can come anywhere from 1-7 blocks ahead of the stop.

Source: I don't drive,and ride the bus a lot.


You generally want to make the announcement before the bus stops so that people can be ready to get off when the bus stops. Also buses don't always stop at every station on it's route so have to deal with what happens when stop is skipped.


That sounds expensive.


Complicated, but not that bad... just need to optimize for the worst case and the best case will purr along happily.

Some sort of near-field beacon on the stops might be more durable than relying on GPS, but it won't help with missed stops.


This seems like a problem that has been solved many times in the Western world.


Or... microphone and speakers for the driver to announce the stop. If you'd like visual indicator too (accessibility FTW!) plenty of low tech and/or cheap solutions that would meet the requirements.


> Or... microphone and speakers for the driver to announce the stop.

Breaks down quickly if you need bi- or tri-lingual announcements, which is the norm in many popular tourist destinations.


Really? Either it's a location 'Piccadilly Circus' or its a place type 'Airport' - any of those can easily be learnt by the driver. Or writen on a printed sheet that they can easily see. Don't jump to making it more complex than it needs to be.


I see you have never tried it in practice. Train conductors in Austria/Czechia/Germany – which tend to be more trained and less busy than bus drivers – tend to stumble around for half a minute for each unintelligibly badly pronounced sentence. Even the pre-recorded messages in buses are of so bad quality that tourists prefer asking other passengers rather than trying to figure them out.


Laminated card in the cab with phonetic spellings for the other languages.

Aside: I'm impressed with the amount of bike-shedding for this one. Though I'm part of the problem, too :-)


Ha, indeed. Just thought of another - even lower tech - a printed list of all stops, visible to passengers on the bus, then clear signage for each stop. If the bus stops every stop by default, then this is simpler, if not then the passenger needs to be able to see this on the approach, with another time to signal that they want to get off. Can this get even lower tech?


You've just described a cellphone running apps that already exist.


> And now this weekend project takes a team of 10 engineers working for a year.

More like 2 guys working for a month.


I'd say ~ 5 Engineers working for ~ 3.5 months

(I calculated the geometric mean of the numbers both of you have given, in case you're wondering ;) )


Generallly my approach if engineers disagree significantly on a timeline is to multiply the worst case estimate by a large factor, on the assumption that odds are good it means neither one has a clue what the actual complexity is if they don't manage to get to rough agreement...


Absolutely impossible. There is hardware involved.

I'd say something like 3 people for 6 months to 1 year (not all 3 being needed 100% of the time).


You are massively overthinking the problem. The very first system of this type I saw didnt even have a microcontroller. Just an eprom for driving LED array, bunch of TTL logic and two inputs. One linked with 'open doors' switch, the other going to undo switch. And it worked fine for couple of years in a >million city.


But nobody wants that. They want conductors back. This is not a problem with a technological solution.


> They want conductors back

That's just faster horses though.

Conductors aren't a solution to a problem in themselves, they're a means to solve a problem. So what's the actual problem? WHY do they want conductors back?


Move to China, every bus in Shenzhen has a lady conductor.


This is such a straw man conversation. Of course it is a fallacy to think "I could build that in weekend" and translate that to a claim about the inefficiency of a company. I think that's what is being responded to in this thread. But that's not the interesting debate.

A better consideration (imho) is this:

   For a given business/offering/product, what is a likely optimal number of engineers needed to continuously deliver the product's value efficiently? Then: How far is a given company from that optimal number?
This is a hard analysis to do. It is virtually impossible to quantify or put this kind of evaluation in a laboratory. There are too many counterfactuals that would be at play.

We really just have to consult our experience and consider which of these is the more probable:

(1) A business/offering/product team with lots of money is really good at utilizing its hundreds of staff. OR (2) It's not.

I've seen a few (2)s from the inside. I have never seen a (1). Therefore I lean more probabilistically toward the belief that a company is needlessly big.

What surprises me is that that's not where people on this thread are leaning---but has anyone on this thread actually ever seen a (1)??? If not, where is this belief coming from? I've never seen a ghost, so I don't believe in ghosts....


Consider, for the sake of argument, the possibility that perfect efficiency is unachievable. Almost physically impossible. As a company gets larger, certain kinds of inefficiency in hiring and utilization become absolutely inevitable. With that premise in mind, how many employees does a company "need"? If it would need fifty in that universe of perfect inelastic spheres with no friction or similar effects, how many times that does it actually need in the real world of such efficiency-robbing factors?

In my experience, the answer is two at best and more often, around five. The incremental work product from each new hire diminishes more rapidly than many realize, but the incremental revenue might still make such hires worth it. As Dan points out, a few percent increase in efficiency from optimization can be huge. It might take multiple people quite a while to achieve that, and it will still be worth it. There might be another group of several people trying something else that doesn't pan out, but that doesn't mean they were unproductive dead weight either. Bigger companies have to take risks too. They just take them differently, and the cost is more obviously attributed to them instead of the market.

Maybe a single startup could do what Twitter (for example) does, in fairly short order. Even that's unlikely when issues of scale etc. are considered, but that's not even the point. That single startup is not a valid point of comparison, because it doesn't count the cost of failed experiments. Add up all of the startups trying to do what Twitter does, including the failures, and that probably be a better estimate of how big Twitter (Uber, Netflix, whatever) needs to be.


Well, I think one of the best points the article made was essentially: A company will continue hiring more people as long as the extra profit from an engineer outweighs the cost.

This isn't literally true in all cases, but it points out that the business decision of how large to grow is based more on that sort of reasoning than "what's the minimum team that can get X done", and maybe a bit different than the analysis you propose.


There's a naïveté in saying "I could do that in a weekend" as a measure for employee bloat, but that's making a straw man of the question.

It is nearly always more probable that a large company with several hundred or more plus engineers doesn't need that many. That those engineers are there for other reasons -- political reasons or speculating on new products, valuation reasons, etc. -- than for the reason that those engineers are fundamental to the top or bottom line of the company's core product/s.

Uber does not need 2000 engineers. That's a prima facie fact.

I've worked inside several large companies and saw this first hand. Hundreds of engineers across teams doing "stuff", no doubt, but nothing that ultimately or ever tied to the top or bottom line. The only ideas that I could come up with for why these people were there were: perhaps the headcount helped company valuation, perhaps we needed that many benchwarmers for when there was company churn, maybe there is some political reasons a manager wants a sizeable headcount on their team, or maybe it's just leadership ignorance as to how things are getting delivered. But in no way could I see a concrete justification for many teams and many engineers.

What am I missing?


Maybe some of those engineers really were superfluous. Maybe some were contributing to the bottom line in ways you don't understand or appreciate. I've seen plenty of engineers take infrastructure for granted, discount the value of research ("speculating on new products" is important), or even dismiss support as a cost with no revenue upside. Most often, as those people gain experience they also gain a broader view, allowing them to distinguish between complementary roles and truly superfluous personnel.


I searched for GTGACCTTGGGCAAGTTACTTAACCTCTCTGTGCCTCAGTTTCCTCATCTGTAAAATGGGGATAATA and found another post by the same author on another website http://bitfunnel.org/strangeloop/

I've seen some other posts by Dan before but didn't know what kind of things he works on because I generally don't check that because if I did then I'd end up just checking people instead of reading what people were writing. Anyhow, from the article linked above -- for which a video is available at https://www.youtube.com/watch?v=80LKF2qph6I -- it turns out that Dan has done quite a bit of work with implementing search related technology, and I think that was interesting. His contributions to BitFunnel can be seen at https://github.com/BitFunnel/BitFunnel/graphs/contributors


I thought you had accidentally pasted some of your genetic code from your clipboard. future internet problems


The other side of the coin is that big companies do move slow.

There is a lot of communication overheads, a lot of bureaucracy, backwards and cross compatibility concerns, conservatism, wasted duplicate effort, wasted pointless effort, some people who aren't contributing much etc. etc.

This is why tiny startups can sometimes dethrone a large leader. However that's often a multi-man-year effort.

So I'd rephrase the statement: "Why's that company so big? I could do that with 20 brilliant engineers and 18 months". I think that order of magnitude can attack pieces of any established software business from a technical perspective. Then there's obviously the business side.


That would get you your first launch product maybe. Not an actual enterprise product. Sure, people forgive the small startup for their crashes, their unfinished and unpolished app, their weak customer support, their performance hiccups, etc. As long as they bring something that solves an actual new problem that isn't solved by another product easily available. But an enterprise product, no way you will make that in 18 months with your 20 people while keeping up the maintenance burden you introduced in v1 when you took some shortcuts.


Yup. Have never gotten the whole "Airbnb/Uber/<startup>" is just a crud app line of thought.

If they're crud apps then everything else in between is just glue programming.

The author of the post covered some very astute points but one not covered in much detail is the challenge of scale and reliability. Maybe this falls under the bucket of optimization and/or latency but I think this deserves being called out on its own.

Once you have Uber-scale number of users requesting taxis at any point, and have a network of drivers constantly communicating their position with the app, suddenly you no longer have a trivial crud app on your hands...


These companies are not "tech" companies in my book any more than the Sears catalog was a "tech" innovation.

They are normal business that happen to be enabled by technology, but the technological challenges are not new or to be frank uncommonly challenging.

Because we're I assume mostly coders, we think they are differentiated by tech, almost everything else (marketing, sales, customer service) is likely more important.

Bad technology could certainly kill a platform, but excellent tech won't save it, and a mediocre implementation is good enough.


If you think scaling a service like Uber is not challenging, you're vastly underestimating how large they are.

Uber especially has requirements that make it tough to scale correctly. You need to track each Uber session in near real-time, right? That means you simply cannot afford to drop or interrupt those sessions. It's a completely different problem than scaling Instagram or WhatsApp.


Hmmm... nothing like near real-time. Challenging near real-time is sub-nanosecond.

Uber real-time requirement of what < 30s? (for a viable product).

No that doesn't seem like a challenging problem. It seems like a problem that would require almost no throught for a MVP, and then incremental challenges as the problem scales.

It seems like a problem that is so comparatively easy compared to what computers are capable of that I wouldn't even class it as "difficult".

But what's telling about your answer is that your thinking about this technical challenge, this feature as being of primary importance.

Do you think if Uber was 10 times cheaper than a regular taxi real-time tracking would be as significant?

Uber's funding which has enabled them to subsidize their expansion (and rides), advertising and promotion and possibly that they plugged in to a easily accessed communications channel (mobile apps). Were likely all more important than the quality of their implementation.


Probably less than 30s between pings during a ride, but I haven't tested it tbh. I usually use the word real-time when talking about systems that operate in the ns time range, and near real-time when talking about software. Sorry for the confusion.

What I was trying to emphasize is that Uber needs to keep track of these sessions and make sure they're not interrupted, or else they lose customers.

Yes, I know building something that fits these requirements and works most of the time is trivial. But we're talking about a system that must basically work as close to always as possible. More importantly, it must do this at scale, maintaining and keeping track of perhaps millions of sessions at once.

In a nutshell, an Uber MVP is probably not difficult, but the transition from MVP to real-world product is non-trivial in my opinion.

I'm actually planning on setting up an Uber-like service for my home country, which is why I've been giving these issues some thought.


Even 1M simultaneous users reporting every ~30 seconds could average down to ~30k messages per second. And they'd be extremely trivial to segment between servers. You're right, not a weekend hack, but a few weeks of a small team's time isn't out of the question. The technical side is not the challenge here though really - the business side is.


Perhaps you're right when it comes to scaling, but you're assuming that the system was designed correctly in the first place, which would be impressive imo. There is also a lot of failure handling and recovery that needs to be done gracefully, which I think would be complicated for a session-based system.

Again, I'm no expert, just making educated guesses!


I'm not sure why you need near real-time single session for the tracking. A ping from the app every couple seconds for the available driver screen. Then for the route tracking and billing have the same pings include the GPS track that the phone is keeping track of. If the servers lose track of any of the cars the data would be stale until it comes back into connection.

Also Uber is pretty conveniently partitionable. There's a little bit of overlap in some places but that's mainly for the available drivers screen.


I agree with you completely. I think a lot more effort is required once you are running the system at scale. Since I don't have experience with running Uber, I'm simply making an educated guess as to what kind of complexity is involved.


In addition the whole "we code for a better world" is just HR marketing to attract engineers. Sure there are companies that have innovative areas and oppurtunities, but many engineers are just code monkeys fullfilling daily duty without glory.


Exactly. I would be surprised if Uber hired more than 20 developers (which I consider a lot already).


I would extremely impressed if Uber is maintained and developed by only 20 developers.


Care to explain why?


They have an Android app, an iOS app, driver management system, fleet management system, ride tracking system, ride scheduling, navigation system (not sure if this is 3rd party or in-house), payment processing, profile system for both riders and drivers, price calculation system, and probably a bunch of other stuff in the background.

All this must work correctly and reliably in the 400+ cities they're operating in. To me at least, 20 developers is not enough to develop and maintain that many interconnected systems. I would say 100s at the very least.


Don't forget cars that drive themselves


20 people is more than enough to build and support these things...


Are you serious? With many 9s of uptime, low latency requirements and hundreds of millions of dollars in revenue on the line? Uber employs over 2k developers right now.


Take a look at Uber's career's page. Even broken down to USA Engineering on the Engineering team hires, there are dozens and dozens of open positions. It's safe to say Uber thinks it's a > 20 person job.

https://www.uber.com/careers/list/?city=all&country=united-s...


I agree with you and parent that it is (and should) be a > 20 person job for its scale, but the career page is probably not the best example.

Many of those positions are perpetually open or rotated periodically (see: "You're leaking trade secrets" by Michael Schrenk) while the actual position was given to someone internal or someone connected to someone else internal to the firm.

Most of these pages are used for resume harvesting so that [if the firm has resources] an HR or technical recruiter can periodically evaluate the rest of the field. The firms holding out for the "rock star" often are more aggressive in recruiting them (whether at Job fairs or through LinkedIn headhunting).


Just FYI I recently watched a tech talk from some conference where Uber guy claiming they have 2000 developers...


Does that include everybody working on autonomous driving and related 'side projects'?


heres a youtube link for those interested

https://www.youtube.com/watch?v=kb-m2fasdDY


What the hell that's insane...


I'm not sure just because you mentioned 400+ cities they need as many developers as cities they operate in.


Did you read the article? In it, the author discusses the engineering issues that arise from trying to deal tackling problems in different languages. Now, granted, Uber in this particular thread of discussion does not have to deal with tokenization issues of Japanese/Chinese character search strings but I think it is naive to not think that the nuances of individual communities of 400+ cities intra-country, let alone dealing with separate nations, will add significant engineering overhead... (even if the relationship is not linear)


Well, that's the thing - they are just a CRUD app in a lot of ways. Even at "uber scale" it really isn't that bad, it'd be a few incremental challenges but nothing a small team couldn't handle from a technical perspective. The catch though is that on top of that CRUD app is a business which involves some serious legal and marketing divisions to make that app work.

The tech may be simple, but the business side of it certainly isn't. The "I could hack that up in a weekend" attitude is often true - but it only covers the bare basics of the software it doesn't even attempt to address the business, which if often where the real struggle is these days. Especially if your background is development. I think many engineers still say this knowingly though, just to (perhaps incorrectly) scoff at the value of good business people.


I read an article the other day that claimed Uber was beating Lyft because they had a strategy of opening a physical office in each city, having an office for drivers to sign up, and a human being who tailored things for that location. Their rival on the other hand assumed that some programmers in SF could do everything neccessary from a distance. Not sure how true it is, but it seemed an interesting theory.


Most apps are crud at the end of the day. Making them scale is the hard part.


To me this is a form of the Dunning-Kruger effect[0], where only somewhat informed people estimate the cost of a "0.1" (or POC) release without considering the famous "unknown unknowns" which may involve scaling, billing, 3rd party integrations, etc.

[0]: https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect


I find I've had to bar myself from thinking about many of these factors, after I realised how many projects I stop myself from pursuing because I know about the hard bits, and realise how many of my past projects I'd never have taken a shot at if I knew what was coming up and was thinking like that.

It's great to know, but often it's not so great if it leads to inaction...

E.g. if you have a "I could do that in a weekend" impulse, sometimes it's worth following that impulse. If you're right, you've spent a weekend building something potentially great. If you're wrong, you've spent a weekend learning about things you clearly didn't know in advance, and the lessons can often be very different from the objections you'd have throw up if you thought about it without doing, and sometimes what you learn may provide the impulses for another new idea.

But it's hard. E.g. my first impulse for anything that requires collecting money is how painful it is to deal with credit card processing and reporting and VAT and accounting, because I've done big complex billing systems with heavy reporting requirements, and compliance complexity etc.. It's very hard to put that impulse aside, and "just do it" and think about how to solve the payments afterwards. But that's how I did things he first few times I needed to handle payments.


And yet sometimes a small team really could do it. I spent years complaining about MSN/AOL/Skype etc. and talking about how a small team could build a better messenger in a week, but we didn't really believe it. Then Slack actually took that possibility seriously, and look at the results.


Sure, they were a small team in the beginning. Google too... Now they probably have more than 500 employees. (They had 385 in february 2016: http://uk.businessinsider.com/slack-ceo-worried-about-growin...)


Slack also jsut recently hit 3 million users. 100 million people were using AOL Messenger at it's peak, and 12 million were using ICQ when AOL bought Mirabillas.


Whatsapp is a good example too.


They didn't build it in a week.


Yes, but that's something a lot of people are arguing here and I agree. The "I could build in a week" is ridiculous, but "I could what this 1000 engineers company do that with a small team of 10, in a lean way in 18 months" is very frequent in the market.


MSN and AOL come from the 90s, and Skype is from 2003. Slack is from 2013 - the web has more than doubled in age for Slack compared to the others. Computers are ridiculously faster, browsers are more mature, language libraries are more complete.

Slack also took much longer than a week, and to be honest, the actual chat part of slack is pretty meh. Where Slack unbelievably shines is in the buttery-smooth onboarding process and the management around the chat accounts.


git is a great example of this.


Actually it actually wrong to assume somebody could solve a trivial problem in a weekend, however some companies still have way too many people working at it. Consider twitter with their 3860 people. they could probably do it with half of them. especially the technical people are 40% of the company... way too much. i don't think that they need no people but well...


Yeah, this post seems to be addressing an annoying conversational experience, but ignores the other side: the reality, which you describe. I understand that just scaling out their platform is a ton of work that justifies having a lot of engineers. But how does Twitter have so many engineers on staff and yet their users constantly complain that they don't improve even seemingly low hanging fruit? Is ~1,500 people not enough? Is there some lingering technical debt they're dealing with? Or is it just people accumulated over time they value too much to let go?


The sad truth is that in the majority of cases the quality or time taken over the implementation isn't particularly important other factors matter much more:

* building something people actually want. * advertising * engaging with users * aggressively promoting your product. * customer service

It doesn't matter if what you did was hard or not. It matters that you can acquire and retain customers.


> Businesses should keep adding engineers to work on optimization until the cost of adding an engineering equals the revenue gain plus the cost savings at the margin. This is often many more engineers than people realize.

I think this is the key realization, and it's actually a variation on Conway's Law[0]. In a nutshell, some part of BigCorp discovers (or believes) that by hiring a software developer for $X, they can make $Y > $X in return. So they do.

As long as some part of BigCorp is able to do that, you get more software engineers. This continues until you either (a) run out of budget, or (b) run out of opportunities.

Because it's not done "top down", it less efficient than it could be to get all of the features in total. But due to how these engineers are funded internally, it's not possible to do the "top down" approach at all. And that's okay, all that really matters is that $Y > $X.

[0] https://en.wikipedia.org/wiki/Conway%27s_law "Organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations." In this case, it's not the communication structure that causes the perceived bloat, it's the funding mechanism. The basic idea that software organizations reflect some non-software, human structure in the business, holds.


As of 2008, the core search team at Google was just under 100 people.

Then came search customization. Suddenly, no more cached replies.


100 people?! I could do that myself, in a weekend.


Everything seems easier when you're not doing it yourself. It's only a minor variation on the grass being greener. Everything gets way more complicated when you're doing it at scale, with high reliability, for real users, with some hope of making money. Practically nobody adds complexity for its own sake. Usually, if you think something is much more complicated than it needs to be, it's because you don't have any idea what the real requirements and constraints are.


To me this post addresses the same naivety that David Graeber demonstrates when he claims that everyone not involved in hammering out a sheet of steel to make a horseshoe or pulling a plough is engaged in a bullshit job[1].

I think it's not a coincidence, therefore that many of the people I run into who advocate a universal basic income, arguing tooth and nail against a job guarantee (the YC guys no different) is young, tech savvy and overly optimistic about the scope of automation to subsume all human labour within months.

[1] http://strikemag.org/bullshit-jobs/


Graeber's biggest point is that even when he talks to people and asks them if their job is valuable, and they should know better than anyone else, they often say 'no, not really.' I don't think he thinks "anyone not involved in hammering steel" is unnecessary, that's a bit of a strawman. The argument goes that big corporations have a tendency to hire more people than necessary because middle managers are rewarded based on how many people work under them. Or that people looking to keep their jobs, try to make themselves look more busy and valuable than they really are.

People waste a lot of time at offices doing things that aren't work. But our culture demands we be at work 8 hours a day, even if the work only requires 4 hours. Economists in the 1930's thought the world would be so rich by today, that people would only need to work a few hours a week. And the economy has grown vastly since then, and we are much richer. But I don't think most people have the option of working that little, even if they want to.

But that's a different subject entirely from automation. Those jobs can really be valuable and still be vulnerable to automation. AI has improved a lot in the last 5 years. I don't think it's implausible at all to imagine most jobs done now being partially or totally automated.


> Graeber's biggest point is that even when he talks to people and asks them if their job is valuable, and they should know better than anyone else, they often say 'no, not really.'

I don't think that's really true, especially in a big corporation. Transparency of the value pipeline in such institutions isn't great at any level, and big corps (and similar large institution -- government often has the same problem, for instance) have a tendency toward not making even the concept of the value stream that justifies the position known to the people working in it much of the time. Its quite likely that the people working a position don't understand how its supposed to deliver value (and are in a poor position to see the big picture of whether it does), while the people who do know how it is supposed to deliver value don't see the small picture of what actually goes on (and thus are in a poor position to evaluate whether it does deliver the value it is supposed to.)

There's no particular reason to think the people working in the position are in the best position to evaluate value delivery (or that even the people in the best position, whoever they might be, are better at answering the question than a Magic 8-ball.)


There is a great incentive to obfuscate when your "value delivery" is skimming (mutual funds that charge 1.25% management fees for underperforming the S&P), privatization fraud (health insurance in the US, government contractors, and pretty much everything related to real estate since "home appreciation" is now supposed to pay for people's retirements instead of pensions), taking advantage of stupid people (lottery, penny auctions, telemarketing, almost all advertising), education fraud (textbook industry, "student loans now, regret later" private colleges and universities - which since many student loans are federally guaranteed is also a form of privatization fraud), or "intellectual property" racketeering.


I think that a lot of this is because things that seem quite pointless on an individual level are actually very important at scale. When I started at Google my job was pushing pixels around on the search results page, and when I left my job was leading a team pushing the pixels around (in the middle I got to do actual cool stuff). On an individual level, this is pointless and boring. On a corporate level, some of those pixels were worth literally $1B+. I would love to have a billion dollar business, but I can't get there from scratch by pushing pixels around on a page.


Graeber's biggest point is that even when he talks to people and asks them if their job is valuable, and they should know better than anyone else, they often say 'no, not really.'

Yeah I know that's his point and that's why I describe his point of view as being naive.

It may be true that for some percentage of workers at any given time their role is experimental and may turn out to be unimportant or a failed attempt at change, and it may be the case that it take a while for these sorts of inefficiencies to be rooted out or resolved because the cost and time associated with doing so make it a low priority but that is something that people who are in those positions would not necessarily be aware of.

This is the same as when people say things like "I don't know why we are so precious with our kids these days. I never wore a bike helmet and I survived!".

Without the proper perspective people frequently form wildly inaccurate opinions.

I don't think he thinks "anyone not involved in hammering steel" is unnecessary, that's a bit of a strawman.

If all I have to do in order to make it a straw man is replace the word iPhone with horseshoe he's doing a pretty good job of constructing his own straw man.

The argument goes that big corporations have a tendency to hire more people than necessary because middle managers are rewarded based on how many people work under them. Or that people looking to keep their jobs, try to make themselves look more busy and valuable than they really are.

That story is completely ridiculous.

People waste a lot of time at offices doing things that aren't work. But our culture demands we be at work 8 hours a day, even if the work only requires 4 hours.

Then businesses have an opportunity to compete for talent by offering better working conditions. If they can do so profitably then they win.

Economists in the 1930's thought the world would be so rich by today, that people would only need to work a few hours a week. And the economy has grown vastly since then, and we are much richer. But I don't think most people have the option of working that little, even if they want to.

This is more a result of the spoils of productivity increases not being distributed evenly.

By having massive unemployment and under employment then expecting the remaining workers to work harder for less pay on average while more and more income is paid towards servicing private debt the finance, banking and real estate sectors have made out like bandits.

But that's a different subject entirely from automation.

I don't think so. It has a lot to do with automation because that is where the productivity increases have come from, but rather than the federal government guaranteeing full employment and setting a floor price for labour and a minimum set of conditions below which we consider employment to be exploitation, which would not only maintain demand in the economy but also reduce the inequality with which the spoils of those productivity increases are distributed, they have left people to fend for themselves and the economy has languished as a result.

Those jobs can really be valuable and still be vulnerable to automation. AI has improved a lot in the last 5 years. I don't think it's implausible at all to imagine most jobs done now being partially or totally automated.

Maybe in 1,000 years but I disagree that automation will have the type of impact people (particularly in the UBI advocacy camp) claim it will.


Same thing happens in software development at the library level:

"Why not use [Example OSS Library]?"

"Nah, I can do it in 15 minutes."

"You know, they have [X] contributors, lots of great documentation, and 6 months of work into it."

"Yeah, I know [but whatever.]"


I get the point that you are making, but if something really only takes 15 minutes to do, I wouldn't necessarily use an open source solution just because one exists.

Depending on the situation, the library may have a ton of extra bloat that your application will never use. Also, recklessly deferring to open source libraries can potentially snowball into something like the left-pad fiasco[1].

[1]: http://blog.npmjs.org/post/141577284765/kik-left-pad-and-npm


Yeah, and I get your point too. Eight lines of code? Yeah, you don't need a library. But usually when someone says "15 minutes" they mean 1 - 2 days. So things of that complexity is when it's obviously you should choose a good OSS lib.


It depends. In many cases you would be right, in many other cases you wouldn't.

It depends on many factors:

- what is the problem? (some problems are really hard, some problems are trivial and just seem hard)

- is developer experienced enough to build own solution?

- are existing solutions good or badly written?(I often refrained from using existing solution when I looked on bloated code in it).

- is solution suitable to project? ("too much bundle size", "doesn't support X", "not too configurable", "too slow", "too hard to use").

etc.


Completely unrelated to the article, but I'm curious what people's thoughts are on full-width lines of text characteristic of unstyled HTML.

I've seen some[1] bemoan its impacts on readability and others[2] complain about websites which artificially restrict themselves to (what can be) a narrow slice of the viewport.

[1]: http://bettermotherfuckingwebsite.com

[2]: https://plus.google.com/+LinusTorvalds/posts/2Ndtw3ZBRX2


On desktop I reflexively zoomed in to %125 and didn't think about it until I read your comment.

It also has a reasonable default font size my phone, so it's easy to read quickly. This is pretty rare since browsers gave up on text reflow.


I begin to think that nowadays websites are just designed for "zooming in" in mind (very small fonts, too many space etc.).

So maybe we shouldn't complain. These websites are just meant to be zoomed.

Although some web developers are cruel and block zoom with meta tag (I never understood why some developers did that, maybe for trolling people with poor sight?).


Pretty simple, we won't bother reading, I just read the comments instead of trying to read that awful formatting.

>7 fucking declarations.

Would have been nice


>>There’s also a wide body of research that’s found that decreasing latency has a roughly linear effect on revenue over a pretty wide range of latencies and businesses.

Anyone have pointers to such research?


There is no published research I'm aware of. However, you can find an article about every major tech company talking about how reducing their latency increased whatever their core metric was (not necessarily linearly though, sometimes it was and even steeper curve). I know for sure this is true at eBay, Netflix, and reddit because I've seen the numbers personally, and I have friends at Google, Amazon and Facebook who have said the same.


Came to the comments looking for these links, too.

Results of my research are below; everything I found is anecdotal, consistent with the other comments. The most robust is the presentation at Velocity in 2009. [4, 5]

My notes:

Small ecommerce vendor, 2012. One second delay in page-load causes 7% loss in customer conversions. [1]

Marissa Meyer, 2006 Web 2.0 conference. Extra 1/2 second in page load time dropped traffic by 20%. [2]

Greg Linden, 2006 (same link as above) "We had similar experience at Amazon.com. We tried delaying page in increments of 100ms and found very small delays result in substantial and costly drops in revenue. [2]

Greg Linden, Make Data Useful, 2006 - Every 100ms delay costs 1% of sales. [3]

Velocity, 2009. The User and Business Impact of Server Delays, Additional Bytes, and HTTP Chunking in Web Search Presentation. Multiple observations. My net: Delays under 500m measurably impact customer satisfaction metrics, magnitude increases commensurately with delay. (Microsoft, Amazon, Google) [4, 5]

1 - https://info.ensighten.com/rs/ensighten/images/just-one-seco...

2 - http://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20....

3 - http://www.gduchamp.com/media/StanfordDataMining.2006-11-28....

4 - http://radar.oreilly.com/2009/06/bing-and-google-agree-slow-...

5 - http://conferences.oreilly.com/velocity/velocity2009/public/...


This was an error by the author of the blog post. There isn't a wide body of research confirming this. There aren't academic papers. There is merely anecdotal evidence/marketing material from industry participants.


Otherwise, indeed, it's a basic evaluation error, as described as long ago as in the The Mythical Man-Month; it just seems that nowadays, the multiplier is not 9, but even bigger.

https://archive.org/details/mythicalmanmonth00fred Figure 1.1, page 5

To go from a program to a programming system (interfaces, system integration), you need to multiply the effort by 3.

To go from a program to a programming product (generalization, testing, documentation, maintainance), you need to multiply the effort by 3.

So to get a programming system product, you need to multiply the effort from the program step, by 9.

Nowadays, we need to compose indeed multipliers for distribution, reliability, availability, localization, targetting multiple UI platforms, etc.

And the original multipliers are also probably too small, when you see the effort required just to integrate testing and CI on some projects.


I have worked with one-man companies that did this and often they are shoddy products. Healthcare just seems to suck at software.

Full-stack developers are already a very fast evolution from the front/back end developers, which were an evolution from a guy doing just HTML/CSS in the 90s. Now you have to include sales, marketing, negotiation, project management, and business management skills on top of your full-stack development skills. The result is that something is going to suffer unless you are a genius, the product is super-simple, or you have 20-30 years to have learned all of this.

There's a much more interesting question in the form of: "Why's that company so big? A tight-knit team could do that in a weekend".


The technology of a big company is often a necessary condition, but it's seldom (if ever) a sufficient one.

We technically-minded people are often pretty terrible at accurately picturing reality as it exists outside of our respective spheres. Consider:

"Why is Armani a billion dollar company? I could make a beautiful suit in a weekend," said the talented seamster/seamstress.


> Recently, I was curious why an org that’s notorious for producing unreliable services produces so many unreliable services. When I asked around about why, I found that that upper management were afraid of sending out any sort of positive message about reliability because they were afraid that people would use that as an excuse to slip schedules. Upper management changed their message to include reliability about a year ago, but if you talk to individual contributors, they still believe that the message is that features are the #1 priority and slowing down on features to make things more reliable is bad for your career (and based on who’s getting promoted the individual contributors appear to be right). Maybe in another year, the org will have really gotten the message through to the people who hand out promotions, and in another couple of years, enough software will have been written with reliability in mind that they’ll actually have reliable services. Maybe.

This sounds like it could be Apple?


Besides underestimating the scope of the problem, one logical fallacy that programmers fall for is believing that a single developer should be able to juggle lots of tasks easily. If a small team can create a SaaS application, why do large companies need so many engineers for so many specific things?

I think, at a certain scale, it becomes necessary for organisations to ensure that there are certain people keeping specific things running. Let's say, a company doesn't have any specific person assigned for server administration and then one day, is confronted with a DDoS attack. Who is supposed to handle it? Whose responsibility is it? You have few people who know the basics of operating servers but no one who can swiftly respond to it. If no clear role is established, then the company is bound to face more attacks and possibly, lose more customers.


Looks like my primary search engine DuckDuckGo tackled this with a relatively small team.

https://en.wikipedia.org/wiki/DuckDuckGo

https://duckduckgo.com/


At an Alexa rank of 676, that's pretty impressive. I think the key to DDG's success with a small set of engineers is their small scope.

If they ever wanted to expand their product scope, for example, index their own searches instead of aggregation, they would start running into the obstacles discussed in the article. The Lucene comparison in the article doesn't apply to DDG (yet).


I was on the other side of this with a product that I was one of the main people on back in the late '90s - I looked at what we were selling and how much we were selling it for, and thought "Why are people buying this? Anyone with a decent IT department should be able to do this in-house."

But really when you sat down and looked at it, what we were selling was a supported and tested product that could be configured and active in less time, with excellent ROI, and at significantly less than the cost of dedicating an in-house developer to do it. What we were selling wasn't cutting-edge technology, it was time and cost savings for a department that was almost certainly already short of both money and staff time.


I think a lot of people are willing to "leave cash on the table" if it means a smaller, more effective team and leaner product.


Certainly not if you are VC driven.


I'm amused the central topic of this post is that google indexes 50M primes in random files in github, which they seem to not. 961748729 is one of the primes just before the 50M'th

Your search - site:github.com 961748729 - did not match any documents.


The article raises some good points but there is still a dichotomy here to me. Compare a successful startup with 2 founders with an established company with 2000 employees in the same business. The scale of the difference in employees vs the difference in the product is often shocking. BigCo product might be better, but that much better?

Obviously the quality of the product won't scale linearly with the number of engineers, but still, startups punch above their weight. Maybe it just highlights how much harder things get when the company reaches a certain size.


Those 1998 extra staff don't have to make the product 1000x or even 2x better (however you might quantify that). They just have to make it sufficiently better to maintain or gain market share.


Weird example, where it's assumed that a single machine can only hold 50GB of document data, which results in the assumption that you'd need 200k machines to store 1T documents...


Apps aren't just a pile of features, they're also ecosystems.

In the early years of Twitter, it would have been relatively easy to clone Twitter, but you won't have had the ecosystem.

Poster example: Google+


As a counter point, on every product I've worked on, on every time we've wanted to make a 'big leap' we've need to get a small expert team to go make it happen.


Dod that small expert team polish the product and maintained it for the next 2 years?

Something about 80%/20% rule...


I call this the difference between building a product and a business.

It's relatively easy to build a product and even to ship it.

The real art in building a business to support it.


> we want to maintain an artisanal search index of 1B documents. Then our cost comes down to $12M/yr.

Yeah, no. It doesn't cost that much to maintain an index of that size. Not even remotely close. My roommate coded a search engine back in 2001-02 as part of a class project/hobby and it easily had an index that large, probably larger till he shut it down. The key lies in not crawling it all in a single day. And your index doesn't populate overnight, it takes months of slow crawling. You can maintain an index of practically unlimited size for chump change (well, compared to the millions OP was throwing around). Do people think Excite, Hotbot, Lycos and Yahoo! in the early years had $12 million per year to spend on their indexing? Hell no. Every single one of them would have went bankrupt in a week. Those companies weren't even worth that much back in the mid to late 90s. Your biggest cost is going to be user loads on your servers (CPU/Ram/Bandwidth), not maintaining an index.


I doubt most who say this sort of thing mean it literally (although no doubt some naive sorts do, but they usually are self-identifying through the other outrageous things they say, and thus can be ignored). But one counterpoint to keep in mind is that the alternative view is every company is running as efficiently as possible. I doubt anyone really always believes that, either. For example, when a story about Twitter was posted on HN recently, it was widely discussed that they probably don't need anywhere near as many engineers as they actually have, and could perhaps reduce costs to the point where they are at or near profitability just by reducing staff. There might be some truth to that. Point being, it's not that simple, and there are no absolutes.


Building an app and building a business aren't the same.

Go out and sell the app to people and find out how easy it is.



From the article:

> There’s also a wide body of research that’s found that decreasing latency has a roughly linear effect on revenue over a pretty wide range of latencies and businesses.

I'd be interested in seeing that citation list, particularly for non-google/amazon businesses.


I feel if you have something better to offer, say a better way to look at information, not just a better page rank algo, but a different way to look at information that is scattered across, one that enables a better insight into existing web pages and less overhead of constant 'searching' by typing a query, something that automatically helps to "see" relevant information with less effort, then it can be an alternative to what a Google search does today and can position itself as a competitor in the long run even if it does not exist today.


If your problem is small, viable solutions to that problem are vast. If your problem is large (scale), there's usually only one or close to only one way to solve it. So choosing the right path is important. Since you can't predict product market fit (you cannot predict value), you have to figure out how to iterate and test as close to continuously as possible. This last thing is what most people just can't do. Programmers included. So you can see there's a technology side and a culture side and they have to be well-aligned.


Even if you're just building an open source app to give away, sure, you can get 99% of it done in a weekend. The last 1% will then consume 99% of your time (if you want anyone to use that app).



The question I was wondering about is: When does an organization realize and say, we have hired enough engineers to support our current systems?


Jeff Atwood made a fantastic writeup of a very similar sentiment about StackOverflow / the StackExchange network: https://blog.codinghorror.com/code-its-trivial/

Until reading that article, I often espoused similar views on applications. It was a really eye-opening read.


Fog Creek is something like 50 employees right? That seems like a reasonably sized team for their product, but Twitter/Uber are in a different ballpark.


I agree and disagree, I've been with companies and seen how scope can expand as you grow, building internal apps, monitoring, reporting, all kinds of stuff. On the flip side i know a guy that managed a team of 5, the 6 of them total managed the snippet editor in Visual Studio. Six people to manage a snippet editor? Really?


Reminds me of this great article about the 'this shit is easy' syndrome: http://steve-yegge.blogspot.com.ar/2009/04/have-you-ever-leg...


Hmm somehow it reminded me of brook's law CEO:

"Give me 9 women and I'll give you a baby in 1 month"


Features and scale are huge things.

Maybe you could make, say, a Twitter clone, in a weekend. But you couldn't make one that can handle anything near Twitter's scale nor that is as feature-rich as the actual thing, and besides, nobody will be using it.


To this I draw a 2x2 matrix on a whitboard. In each of the 4 quadrants I write:

"Product and Engineering"

"Ops"

"Sales and Marketing"

"Legal"

Enter tasks and goals for each quadrant. Iterate.

This is the minimum effort required to sustain any app. Developers are mostly only aware of the first quadrant.


One of my favourite tech jokes used to be "Google? I could build that out of wget, grep and baling wire!". The fact that "I could build that out of Lucene and money!" is not intended as a joke gives me pause.


Anyone find the 528 MB C++ file that is referred to in the Twitter screenshot near the end of the article? (With only 57 lines of code but, oh, an array with the first 50M primes hardcoded.) What's that about?


I'd guess it is probably for Project Euler.


that "probably" makes me wonder why you would say that? is there something about 50 million primes in there?


I don't think you really need that many primes for Project Euler, but I can certainly imagine someone tired of including a call to her prime sieve in tons of problems, and of having to waiting for it to run, and thinking, "let's just make a library that has all the primes I could possibly need in a constant array".


"I could do that in a weekend"

You could but you didn't so I can't use it...


Sometimes it takes a lot of time to reach something that is relatively trivial once you know how to do it.

I can reproduce E=mc² in a matter of minutes. Doesn't mean I could have ever come up with it in the first place.


What exactly is the point here? At scale, it's not the tech that got the product to the finish line.

The endurance part of the company has nothing to do with tech or even scaling or optimizing that tech. Not even close.


In France, the law (Art. R. 4228-10 du Code du travail) imposes one WC and one urinal per 20 male workers, and two WC per 20 female workers. So basically, it expects that up to 1 worker per each 10 workers will be busy with non-job related stuff.

Another datapoint, I heard that an accountant counted that the actual number was 1 for 50.

In any case, when you're alone or a small startup, this doesn't seem much, but when you have 100,000 employees, this means that between 2,000 and 10,000 equivalent employees are all the time in the WC (and being paid for that, basically).

Of course, there are a lot of other overheard that become very significant when you have a big corporation, but that still exist (and that we easily discount) for small startup and individuals.


Ever thought that maybe that 1:20 or 1:10 isn't about handling steady load but about handling pre-/post-lunch peak loads without half hour lines?

(And I've seen the half hour lines, in an office that had two WCs for 100 people. THEN you end up with people spending 10% of their day doing something other than work.)


In the US, most companies aren't hiring janitors, they outsource it to a facilities management firm or it's handled by building management.

I'm also not sure how that's relevant.


Do you stop thinking on the loo? I don't. For knowledge workers, that time isn't completely lost...


That $12M/yr figure for indexing 1B documents seems way off to me.


Yes and if you look at the footnote where the arithmetic is laid out, it seems quite confused. It conflates querying with indexing and also flatly assumes that each machine will index 5M pages, which seems low to me.


Too high or too low?


Too high. See http://news.appbase.io/scaling-elasticsearch-writes/ for example:

> In doing these tests, we indexed over 1 billion documents under just two hours. The peak merge rate we saw was 129 MB/s, yes our ElasticSearch cluster on a 24 node setup is gobbling data at this speed.

> Most mind boggling of all: At our peak sustained ingestion rate, we would have added over 13 billion documents in a single day and this would have cost us a mere USD 244.

Their test documents are much smaller (100 bytes) than a typical web page but if you multiply that figure by 50 to get to 5kb documents (probably a realistic average web page size minus all markup and assets), you get 12200$ for 13 billion documents or 1220$ for 1.3 billion documents (assuming that scales linearly). I'm probably way off in my estimate myself but I'm convinced that a few thousand dollars is much closer than the proposed 12M$.


Glad to see some sanity for once and people that realize building 'trivial' apps like Twitter and Uber are not feasible 'in a weekend'. Just making sure the app is secure takes months, and security becomes more complex as features are added. Pretty much every feature you add to the app can explode the complexity exponentially.

e.g: a sorted timeline if strings is easy, but add the simple feature of 'retweeting' and being able to 'unretweet' for example and things quickly become more complex.


Those numbers are whacked. We use ElasticSearch with >10B docs ingested and the cost is like a few thousand dollars a month for hosting.


Yeah right, except that you can't. It takes so much time to nurture something and build it in a way that satisfies the user.


I really enjoy how this guy's blog refers to research papers. It adds to the level of interest.


If you want to build a "boat" you start with a canoe, you don't build a ferry.


googles and ubers of the world are not just their end-user facing apps + the backend required to support them.

these are large businesses with end user-facing "IT" side of the house being just the tip of the iceberg.


The question is why do someone needs the first 50Mi primes in a C++ file


See making crc 4x faster in redis by keeping precommputed values in a lookup table: https://matt.sh/redis-crcspeed

Maybe something like that ?


Because the alternative: generating the primes at compilation time with a template functional expression would be too slow, and quite possibly would explode the compiler's memory limits. (No Turing Machine here), and doing it at run-time or deliverying it as a "resource file" is obviously out of the question.


No idea, but it would be a fairly reasonable way to test a prime-finding algorithm - or to replace a prime-finding algorithm with something less complicated.


crypto?


The code is the easy bit.


Generally it is Marketing..

And also "Hofstadter's law"


I personally have never seen this complaint leveled against the mighty Goog, nor really any company that's cash positive. When I see people giggle at bloat, it's more often than not twitter and still more often than not, it's not intrinsically because big company = waste, but because company that should be small is made to be big either by VCs or Wall Street, and then can't turn a profit anymore.

No, maybe if twitter was still a tiny company you couldn't use it in Arabic but maybe they also wouldn't constantly be on the edge of going out of business.

Edit: I think this is also a regular criticism of Twitter because Twitter has spent the last several years trying to Facebook itself and strayed far from the original idea. You can argue the pros and cons all day since they're pretty much infinite on both ends, but Twitter as a company has spent vast amounts of money on a lot of features that none of it's original userbase really wanted (and drove a lot of people off in the process).

Me, I only really ever used it as a time-waster so I'm not awfully attached but I listen to a ton of podcasts and I can't count the number of hosts who disliked more or less every new thing Twitter did.


Things like Dropbox are striking. I guess it was mostly written originally by Drew Houston over a year or so and was cool.

Then 2011, 50 million users 70 employees. Seems fair enough.

Now about 1500 employees. It worked fine and scaled fine with 70 employees so you wonder if hiring the other 1430 was good business. I guess if VCs are throwing money at you you may as well spend it on something. Though that's probably what Yahoo was thinking when it went to 11,700 employees and that didn't play out so well.


I bet its mostly sales. I had no idea about how much sales ppl you need at certain scale.


Sales persons are necessary when continuing growth requires efforts that do not scale as effectively. Marketing and advertising campaigns work sufficiently well for b2c with 1-2 people but you hit a wall if you want to selling to, for example, the Pentagon because the way those with the deepest pockets buy is nothing like how most efficient individuals and organizations buy. Groupon's massive needs for salespeople is an example of how even lower margin b2b even needs sales due to how scaling out b2b is so much less cost-effective. But for investors, landing a huge contract is not as much about the revenue as much as an indicator of stability in terms of both revenue and market signal (no Fortune 100 will buy some random fly-by-night vendor's stuff without code escrow even if it's the best thing ever because their sourcing overhead is so poor and big companies are immensely risk-averse by design).


Atlassian's 'b2b without salespeople' approach has worked well for them.


They definitely have sales roles in more recent years and have a sales ops team [1][2][3]. The Federal channel manager description even acknowledges that there's unique challenges in selling to the government that necessitates someone that knows the different contracting vehicles. What they've done with little turn to sales culture is pretty admirable but at the same time only possible with organic, engineer-to-engineer driven growth Github and Atlassian suites.

Salesforce grew somewhat similarly via rogue accounts on expense accounts, but to actually clinch the biggest growth there was no way to avoid salespeople to get over $500MM in revenue probably.

[1] https://www.smartrecruiters.com/Atlassian/99086565 [2] https://www.smartrecruiters.com/Atlassian/95395260 [3] https://www.smartrecruiters.com/Atlassian/93068238


Yea, looking at the employee page for Stack Overflow shows a vast majority of the staff being in sales.


Well VCs want to know what their money does. So expect a large part of controlling.


I think those people miss the point that not all engineers are working on the core products.

For example, as one grows they need to have a dedicated HR teams. These teams will demand for a HR software to make their life easier. If the software is an on premise solution then you require engineers to manage that software. Most HR systems require a RDBMS backend, so there is an additional need for a DBA. As this adds to the company, there is a need for a hosted Identity Management solution, which again requires dedicated engineers...so and so forth.

While there are many reasons for a company to be loosing money and most of them are related to building a sustainable business, from an engineering point of view it has a lot to do with hiring. Companies with lot of cash want to hire talented engineers and there is nothing wrong with that when getting the company off the ground. But as they grow the talented pool they can hire from gets smaller. So either they start overpaying for a great talent or offering above market rates for a mediocre talent. Its mostly the latter people who offer negligible intrinsic value while getting boat loads of money.


I'm definitely not claiming to know what's good/bad for a company I've never worked for, I'm saying the perception from outside is that companies hit these sizes, hire massive amounts of people to do something, and then the products almost universally become worse.

Maybe it's just big-company-itis.


Right, I feel like the author has a good angle with which to tackle some of this criticism, but choosing Google as the example company is really unpersuasive for me. I'd agree that Twitter would be a better example (or Uber, 2,000 engineers - doubled in size over the last year)


Uber need their 2000 engineers to write their 1000 microservices eh?

<snark/>


Joking aside I enjoyed that video (https://www.youtube.com/watch?v=kb-m2fasdDY - for those who didn't see it), the presenter was pretty honest and way more open than I expected. I'm just so curious how those services break down - if it's genuinely on the "lstrip as a service" level or whatever, whether that's "1,000 services" figure was actually a relatively simple API with 10 versions at various levels of deprecation, whether he meant it was 1,000 different service endpoints (250 services each with POST, GET, PUT, DELETE) or what. I want to know more basically :)


I am usually not someone who falls into the "what do they need so many people for, it should be simple"-trap but I was still surprised to learn that Lyft has 400 engineers and Twilio has around 300 too. I am note sure they would be that big if they had to finance everything out of their profits, but i guess it's a good way to maximize progress, on the other hand they also had a lot of challenges with growing dev teams.


What features has Twitter shipped? Moments??? Pfft.

On the contrary I'd argue they've hardly done anything new, and that is why they are struggling. Think of all the things Facebook has done and abandoned in the same time.

But I like Twitter - I just wished they fixed some pretty obvious issues. I mean... Tweetstorms are like a request for a feature right there, and Twitter's response is... silence.


> Businesses that actually care about turning a profit will spend a lot of time (hence, a lot of engineers) working on optimizing systems

Hm - every time I suggest optimization (of anything), I'm shot down with "premature optimization is the root of all evil!"


I'm sorry for the OT, but I feel I should post this: http://bettermotherfuckingwebsite.com/

This is unreadable as is.


Au contraire, it's a lot more readable than your link, because it respects my window size.


It's readable for some subset of people, but on a 32" monitor it's worthless - I tried to read it, but just abandoned it in frustration.


Resize your browser window?


Yeah, but asking readers to resize windows, resize the font, and tweak the CSS to get decent line heights is a bit of cheek when it's trivially easy to implement at least minimally pleasant-to-read typography and layout.

It's a bit shortsighted of a writer to completely ignore the realities of readability. The current layout may conform to some vague idea of purity or minimalism, but that puts form over function in a big way.


Resizing a browser window is hardly as involved as tweaking CSS or browser settings.


Every variation of this comment/mindset drives me nuts. It's unbelievably thoughtless.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: