

Ask HN: Why does Google need so many employees?  - gsivil


======
nostrademons
That's what I thought before I joined. Once I got into the swing of things and
started doing actual work, I was like "Goddamnit. We need _more_ employees.
There's too much work to do!"

The simple, non-confidential answer is that running a global service used by
billions of people entails facing a lot of problems, on a daily basis, that
most developers will never face. And that makes everything move slower.

Have you ever internationalized an app? It's not just a matter of translating
every string in the UI, although that certainly is a pain in the ass. You also
need to handle right-to-left text and mirrored UIs for Arabic/Hebrew/Farsi,
and English text mixed in with that mirrored text (this is common in Israel)
which requires a directionality reset. Did you know that you're not supposed
to bold Chinese characters? The typographic convention is to make them red for
emphasis - but _only_ Chinese characters, English text embedded in Chinese
characters (eg, "President Obama" in a Chinese news article) should still be
bolded as necessary, so you can't just use a global CSS rule. You know that
several European countries format a million bucks as $1.000.000,00 instead of
$1,000,000.00, right? And dates go DD/MM/YYYY instead of MM/DD/YYYY, and your
date parsing routines have to understand that as well as your date display
routines. Ever dealt with Russian plurals? In English, you know something is
plural based on whether it's one or many; in Russian, it's based on whether
the last digit of the number ends in 1, 2-4, or 5-0, and sometimes depends
upon its position in the sentence. Can you do case-insensitive comparisons in
multiple languages?

And every single UI engineer needs to be familiar with these pitfalls. A lot
are hidden behind frameworks and utility functions, but then every single UI
engineer needs to learn the frameworks and utility functions, and understand
the purpose behind the libraries so they know when they're appropriate to use.

Do you worry about efficiency when you program? Most programmers these days
grab something like Python or Ruby whenever they need to do something, but
doing a complex parsing routine in Python can easily run 100x slower than
doing it in C++, and at Google scale that means 100x more machines. There are
a bunch of open-source libraries for the type of internationalization problems
above - but the ICU libraries can easily pull in 50M or so of data files into
RAM. If you're running on 20K machines (number pulled out of a hat), that's a
_terabyte_ of additional RAM consumed because you wanted to use an open-source
library. Do you care about end-user latency? Most webdevs these days take for
granted that they'll have JQuery or Prototype available to deal with cross-
browser differences. One of the first things I did when I joined Google was
try to convince them to allow JQuery for development, and as part of that I
ran an experiment to figure out what the actual user impact would be. It
turned out it would've _doubled_ end-user latency of the time, and so we had
to keep writing everything from scratch based on browser primitives.

Then there're all the issues surrounding running a massively distributed
system. Machines fail, packets get dropped, and you have to handle those
failures gracefully. It's impossible to upgrade a distributed system all at
once without downtime, so every time you make a change that crosses a
component boundary, you need to make the new component backwards-compatible
with the old one and run both versions of the code in prod until the whole
fleet has been upgraded. Many familiar algorithms that work on one machine
don't parallelize effectively, so you're limited to what you can compute. Even
if they do, the standard libraries for them usually assume a single flat
address space, and so you have to really understand the algorithm and how to
convert it to a distributed network of computers, and then reimplement it
yourself.

If this sounds exciting, we're still hiring. :-) If it doesn't, well, if your
startup gets big you'll have to face these problems anyway. Hopefully you can
hire someone who finds them exciting.

~~~
jsnell
That's all true, but I fear it might be painting a slightly unrealistically
positive picture :-) There are also other kinds of reasons for why everything
moves slower than one would hope, and they aren't quite as interesting.

Some system your project depends got deprecated again, and you need to decide
whether to spend the time to migrate it to the replacement now, or hope it
continues working until the replacement has been deprecated too. The
development configuration files for other teams systems that you need to
inherit from to bring up a full test cluster for your own system changed
again, causing mysterious errors. Hopefully early on, rather than in the
output of a data pipeline that takes 20 hours to run.

Or the system your working on is somehow relevant to the outside world.
There's a legal/political/PR problem, and actual engineering effort is needed
to implement the minimal hack to get rid of the problem as soon as possible.
Or your project has inexplicably and apparently without anyone's knowledge
been marked as falling under Sarbannes-Oxley, and now some change needs to be
explained in great detail to a SOX auditor. Which would of course be much more
important than any work you were actually hoping to get done :-)

The code base is so massive and there are still so many unexpected
dependencies that even with distributed and cached compilation, just the
processes responsible for the distribution might be enough to drive your
machine to its knees. Of course there's a process in place to try to keep the
dependencies from growing out of control, but that's then going to make your
life much more difficult if you're working on the inconvenient parts of code
nearer the center of the dependency graph, and thus more strictly regulated.

Also, there have historically been a lot of misguided projects that in
practice failed but somehow didn't die. Usually they will get killed
eventually, but they do tend to grow larger and live longer than they should.
Maybe that's changed lately.

This is, of course, an unrealistically negative view. But at least in my
experience these kinds of issues were more significant drags on productivity
than the fun kind of issues you listed. Though that could be just because the
boring kind of drag is much more noticeable. Google's a great place to work at
in any case.

~~~
nostrademons
Yeah, I've run into all of those at times. There are some problems that are
just inherent to working at a big company.

Most of the really frustrating ones are fairly avoidable if you (or your tech
lead, if you're not leading the project) are reasonably adept at playing the
big-company game. My usual reaction to deprecation of critical infrastructure
is to complain loudly and persistently, and I can usually get a stay of
execution for 6 months to a year, enough for someone else to be the early
adopter and work out the kinks of the replacement, then move my own project
over to it once it's stable. I've saved many hours for my team by continuing
to use deprecated software until the replacement has been deprecated, and then
switching straight to the replacement's replacement once it's stable.

The massive codebase is reasonably easy to navigate once you become adept with
CodeSearch. Build performance problems are persistent, but I solve that by
getting a new machine every time the opportunity presents itself and using
them headless for builds & demos. I also open them up to my team so that other
frontend engineers I work with can use them...I've got a mini server farm
under my desk.

The legal/political/PR stuff can't really be helped, but I find that
exhilarating in its own way, since you get to see how those systems work. I've
learned to hate the Chinese government since working at Google, though.

BTW, all of these are potentially problems at a startup using open-source
software and doing interesting things. It's just that there:

1.) Instead of worrying about your OS libraries being deprecated, you pull the
version you want into your own source tree and never touch them, at least for
several years. Any attempt to upgrade to a new version is met with intense
pain, and so you push the problem off and hope that either your company will
go bankrupt or you'll get bought by some big company and the programmers there
will rewrite your software entirely.

2.) Instead of complying with the legal problems, you ignore them. If somebody
pays attention to you, you're sued out of existence and all look for new jobs.
If they don't, you go about your merry way and hope nobody looks too closely
at you.

3.) Instead of your dependencies growing out of control....aww hell, every
company I've worked at that's more than 6 months old has had dependencies that
were out of control.

4.) Instead of being on a misguided project that in practice failed but didn't
die, you're on a misguided project that will fail and die spectacularly. And
then you get to do it again for the next startup.

Startups can be a lot of fun too, but understand, you're screwed regardless of
what you do. :-) Engineering is hard. Let's go shopping!

------
phylosopher
I'm a former Googler who managed the deployment of their content distribution
network. Outside of the codebase challenges that nostrademons articulated very
well, Google has to deal with various big company concerns including shipping
and logistics, human resources, legal issues in the many countries it operates
in, hardware development, datacenter management, etc. To do this they have
many talented people across disciplines. I was constantly looking for people
to join our team while still maintaining the high level of ability and innate
Googliness that Google looks for in an employee. In many measurements Google
is still understaffed. Check out Royal Pingdoms profit per employee.
[http://royal.pingdom.com/2011/05/17/apple-staff-profit-
per-h...](http://royal.pingdom.com/2011/05/17/apple-staff-profit-per-head/).
The difference between 2008 to 2010 is incredible and factors in growth in
employees. Google still needs to hire... not for world domination but to
maintain and improve the services that we love.

------
byoung2
<http://www.google.com/intl/en/about/products/index.html>

Each one of those 50-odd services could be a full-fledged company on its own
with a few hundred or even a few thousand employees (in fact, many of them
were). I would even bet that as separate companies, they would add up to a lot
more than the estimated 20,000 Google employees.

~~~
cgallello
After reading what you wrote about each one being its own company, then seeing
that Google Toolbar still exists, I started thinking:

"We're delivering the next generation in browser toolbar software"

Oh dear god.

~~~
byoung2
That is actually a really good example. Look at the competition, Conduit
([http://techcrunch.com/2008/01/16/toolbar-company-conduit-
rai...](http://techcrunch.com/2008/01/16/toolbar-company-conduit-
raises-8m-more/)), a company that literally has a few hundred employees
(150-499 according to Glassdoor). I figure that Google has about the same
number as the competition does for each service, so the Chrome team would
roughly match Mozilla (~300 employees), the Docs team would match Zoho (~1500
employees), Maps would match MapQuest (~150 employees), etc.

~~~
skrish
Not to nitpick, but Zoho also has portfolio of several other products
including Accounting, CRM, Helpdesk, Network Manager among others. And their
total employee strength as you said is ~1500.

------
joelmaat
They waste a lot of engineering resources, but almost everyone somehow works
long hours. They could be _even_ more efficient.

~~~
vicngtor
And you are quoting this from?

~~~
joelmaat
I had a short and painful stint there, buddy (its been over a year, though).

Hopefully, given all the press, things are a bit better now.

------
snowleopard1010
The answer is simple: World Domination.

