
Ask HN: What Have You Learned at Google as a Software Engineer? - aalhour
Hello Googlers (former and current ones),<p>What have you learned at Google, as a Software Engineer, that you think was interesting and helped you up your game as an engineer? Anything you can share with people outside of Google that can help them improve as engineers?
======
Zach_the_Lizard
Not Google, but another Big Tech company.

Visibility is very important to getting a promotion at a large company.
Selling your work is important.

To move up, you must be playing the "choose a good project or team" game for
at least 6 months before you try to get promoted. Preferably for a year or
more to hit the right checkboxes for multiple cycles.

If you fail to do so, you can do absolutely amazing work but rigid processes
and evaluation criteria will conspire to defeat you in a promotion committee
setting.

At least, that's true in my company. From my ex-Google peers it seems to be
true there as well.

Being in a smaller office means you get fewer of the best projects available
to you. Reorgs sometimes steal them. Cancelling projects makes the last half a
waste of time from a promotion standpoint.

As for what constitutes a good project. It will:

* Let you lead it

* Have peers at your level + one or two

* Work with multiple other teams

* Ideally work with multiple teams outside of your group, e.g. you're in, say, a chat app team and get to solve issues for some other app

* Good product manager and designers with a well thought out product; it's not fair, but if the project is successful business-wise people will often incorrectly attribute that to your own skills

* Have large engineering scope. Committees get confused and sometimes think simple designs to solve complex problems are bad. You want to solve problems that are complex even in terms of the solution (or can be made to sound complex) to check more boxes.

* Allows you to solve problems for other teams

I haven't optimized for promotion and thats personally hurt me in the quest to
get to L6. I very, very close but didn't quite make the leap.

Don't ignore the flaws in this committee process. Exploit them.

Don't be me.

~~~
skizm
What does "promoted" look like for an software engineer at large tech
companies? In my company you're either an individual contributor or tech lead.
There aren't really any other rungs on the ladder without going to manager.

~~~
bt848
At Google there’s an IC ladder from level 3 (new grads) to 11 (Jeff and
Sanjay). Basically it’s a pay scale. Half of engineers are below 5, and 90%
are below 6.

Edit: pretty sure I was off by one originally. But you get the idea: it’s a
triangular organization with a lot of people at the base.

~~~
zuhayeer
For pay ranges check out some charts we’ve compiled at
[http://levels.fyi/charts.html](http://levels.fyi/charts.html)

~~~
generatorguy
I decided to focus on power engineering instead of software in the mid 2000s
since I thought a lot of software jobs would be offshored. Big mistake in
terms of compensation!!

------
jorawebdev
Other than having “Google” on my resume there is nothing special or applicable
outside of Google. Most tools are internal, isolated and the choices are
restrictive. Management is shitty - micro-management is in full bloom, display
lack of management knowledge, skills and there’s plenty of abuse of power.
They don’t show their appreciation to what we do. All developers are very
competitive. My entire time of over a year in 2 different teams is spent in
isolation and self learning without much help or directions. I’m currently
actively interviewing outside.

~~~
wolco
How can you have micro-management and isolation?

~~~
D-Coder
They ignore you for weeks, then complain about every line of code and solution
you've produced and tell you how you should have done it. Repeat every month
or two until crazy.

~~~
wolco
Complaining at the end is a different problem that gets solved during the
requirements phase. They are not micro-managing your day but the end results
against expectations.

------
dannypgh
I spent 9 years at Google, just left at the end of July.

The biggest thing I grew to appreciate was that for iterating large scale
production systems, rollout plans are as important as anything else. A very
large change may be cost or risk prohibitive to release at once, but with
thought you can subdivide the release into easier to rollout and verify
subcomponents, you can usually perform the same work as a series of lower risk
well understood rollouts. That's critical enough where it's worth planning for
this from the design phases: much as how you may write software differently to
allow for good unit tests, you may want to develop systems differently to
allow for good release strategies.

Google is a large company, and I think there are software engineers learning
very different things from what I focused on. I worked on large scale machine
learning in ads; someone working on chrome or Android likely learned very
different things.

~~~
ithkuil
(Hi Danny!)

Yeah Google has a lot to teach about building and maintaining reliable
systems.

I noticed that a lot of that, including the name SRE, gets cargo-culted by
other companies without really understanding what it is.

I also guess that there might be people within Google that don't end up
working in projects that invest in reliability (for whatever reason) and so
might have had different experience.

------
cmrdporcupine
8 years; what I've learned -- working in absolutely huge code bases, good C++
style, good code review culture, intellectual humility.

What I haven't learned: how to thrive in a huge organization with people more
bureaucratically ambitious/motivated than you. How to turn a blind eye to
broken waterfall method dysfunction. Starting to get to me. Probably looking
for the door soon; p.s. entertaining offers for remote (Canada) work.

~~~
lazyant
> entertaining offers for remote (Canada) work.

want to add an email or something to your HN profile :-)

~~~
cmrdporcupine
Is it not visible? It's there. Don't see an option to make it public.

~~~
detaro
put it in the about field. The email field is just for HN systems and staff to
e-mail you.

------
jeremysalwen
Everybody comments about how there are all these amazing tools that help your
productivity (which is true), but they don't talk about how the opposing force
of organizational complexity ultimately results in "just normal" levels of
productivity.

They don't cancel each other out exactly though, so some things are _way_
easier (spinning up a huge computation), while some things are _way_ harder
(making a small change to shared components).

~~~
tedmiston
I thought the monorepo setup made cross-project dependency changes easy?

~~~
skybrian
Well, it depends. If you have an API with only a few direct usages, you might
do it in a single commit. Changing API's that have many usages is difficult to
do atomically even if you have a monorepo, due to the difficulty of getting
approvals and the risk you'll have to roll it back.

Often it's better to introduce a new function and migrate usages to it across
multiple commits.

Either way, it's on you to fix everyone else's code, which is a different
mindset from releasing a library and letting downstream users do the migration
themselves when they feel like it.

~~~
mav3rick
In Chrome, more often than not it happens in one big commit.

~~~
skybrian
That's a single project (though a large one). I'm thinking of lower layers
where there are many downstream projects.

------
apollo_
I've been at Google for 1.5 years and before that startups. I've learned a lot
about working at a large company (getting adjacent teams on board with my
ideas, writing compelling design docs, etc). I haven't observed any changes in
the quality of my code or ideas.

------
mindfulplay
* Always be learning and doing. Never be stagnant. Even if you are in a very boring team you can still learn and supplement your skills / knowledge on the side. This may or may not help immediately but definitely helps long term.

* Really practice open/active listening and critical thinking: there are very smart people and it's really good to assume good intent when they talk. Critical thinking and question is more around 'where is the disconnect? How do I try and actively move towards and see their point of view'.

* Questioning authority and being able to say no.

* Don't use code reviews as a debate forum but instead as a learning forum. Granted time is of essence, so a timely resolution may be necessary but learning and actively listening through reviews helps your skills as a coder.

* Cut through heirarchy and organizational politics to achieve the larger goals for the team, the project and ultimately for Google.

------
bt848
The most important thing I learned there was that you hire and promote people
not so management can tell them what to do, but so they can tell management
what to do. I didn’t even realize it at the time but the first post-Google job
I had there was some clueless manager trying to set the roadmap and I was like
“LOL what is this guy’s problem?” That when I started noticing the reasons
that Google succeeds.

------
awinter-py
A friend who spent a short time there said "you know that tool that you want
to build before you start working on your real job? At G, it already exists"

This is less true now because a lot of their tooling has been exported either
directly or by rumors leading to it getting rebuilt outside. I'm talking
containerization / orchestration and protobufs specifically.

~~~
riku_iki
> At G, it already exists

quality of the tool has also significant impact on the end result. Looking at
the number of unsuccessful external projects, I bet they also have enough mess
inside.

~~~
mav3rick
Right search , maps , YouTube all run smoothly because of a "mess". Google
code is some of the cleanest most organized I've seen.

------
octocode
Not Google but similar. I learned that my motivation is proportional to the
quality and amount of free food I receive. This includes willingness to
participate in pointless meetings.

Note: this is not something I'm particularly proud of or anything. Just an
observation.

~~~
wyclif
The food isn't truly free.

~~~
octocode
I mean, nothing is truly free, but I'm not the one paying for it in this
situation, so for me it is free.

------
pembrook
1) All big companies inevitably devolve into Dilbert, no matter how cool they
seem from the outside

2) Humans are easily fooled by signaling and vanity metrics, exploit this
quirk

3) Depends on the team, but in general working at a big company is 10X easier
and more relaxing than working at a startup. You’ll probably never go back to
working at a small company after realizing this.

4) Buddying up to the right people will dramatically alter your experience in
a big company—meritocracy can never exist when human emotions are involved

5) the difficulty of the interview process is just plain bizarre given how
simple the daily jobs of 95% of the engineers are

6) you will have the nagging feeling a high school kid could do most of the
(non engineering) jobs in the company with a few months of on the job training

7) due to 5 and 6, unless you are a sociopath, you will feel guilty on a daily
basis for building generational wealth while an immigrant contractor
struggling to get by dishes out your free food

8) you will learn that at most big companies, the innovation is in the past

9) on the bright side, google does have really cool internal tools and perks

10) after leaving, people will grossly overestimate your intelligence and
skill level due to the name google being on your resume (see #2)

------
brianolson
Big organizational problems can't be fixed by writing better code. Meritocracy
sounds nice but you have to learn to play office politics. New products and
new features should have a business case (yes, even at Google) so that
eventually they can stand and be valued.

------
mav3rick
My quality of code has been immensely helped by code reviews by senior people.
My team also has a good culture of getting feedback on design. I've learnt a
lot on designing with security on mind - sandboxes, SELinux etc. I had no clue
about them before I joined.

Design docs and comments by great engineers like Jeff Dean are openly
accessible. Treasure cove of knowledge that I try to dip into everyday.

------
amelius
Perhaps nice to mention here: "Talks at Google"

[https://talksat.withgoogle.com/](https://talksat.withgoogle.com/)

Not restricted to just software engineering, though.

~~~
forgot-my-pw
I think Talks at Google is mostly not about Google?

For the engineering side:
[https://www.youtube.com/user/GoogleDevelopers](https://www.youtube.com/user/GoogleDevelopers)

------
aappleby
Even really smart people sometimes write really dumb code. Including you. :)

------
rictic
At Google for 9 years, startup before that. Roughly half my time at Google I
worked on individual products, and the other half on open source
infrastructure (common libraries, compilers, linters, IDE plugins, etc).

The biggest thing I've learned is how software changes over time. Previously I
thought of a good program as like a crystal. Write it correctly the first
time, and it doesn't need to change much, if at all. Sure you might want to
add features or fix bugs, but otherwise you write it once and it's good
forever, right?

Most programs are closer to organisms than crystals. They're adapted to
specific niche, and when the world changes around them they need to either
adapt or die. This is the biggest gap I notice between senior devs and strong
junior devs. Junior devs have mostly built things, released them or turned
them in for credit, and then moved on. It's rare to find a junior dev that has
maintained an app for the long haul, and it accounts for so much of the advice
that I found strange when I was junior:

* code is read many more times than it is written

* be very mindful of your dependencies, if push comes to shove you could end up owning them

* commenting why is often more important than commenting how. it's good to comment about both, but it's much more important to comment why. the source code explains how, but it has nothing about why

* it's ok to love your code, but delete it the first chance you get. it will save you and the people around you a lot of time

Do you think I'm overselling things? Maybe that's a problem for people who
write code in $cursed_language on $bad_platform but your tools favor writing
True and Long Lasting Solutions That Stand the Test of Time!

Let me give a handful of real world examples:

* A major security vulnerability is found in an API that previously seemed safe. Refactor your code not to use it.

* Turning on a certain compiler optimization breaks a pattern common in your code. After some experimentation, you discover an ugly fix that you then need to apply to everywhere in the code that uses that pattern.

* A compiler/browser/hardware update adds support for a feature that your code has been using a bunch of ugly workarounds to cope without. Do you use the new feature? Do you still support running without the feature? Is it a breaking change to users downstream of your code? How much does it improve things, how hard will it be for your users to update, and how many users do you have?

* You need to migrate to the new version of one of your dependencies. Maybe this is just a version bump, maybe you end up needing to review every single line of your app.

* Your app for phones and desktops is working great! We want to run it on TVs as well! This means a brand new (and super weird) matrix of features that are and aren't supported, from hardware to OS to UI.

In short, the only constant is change. Write your code so that you can go away
for six months and forget just about everything about it and still be able to
maintain it. Lots of tests, document why decisions were made, and it's like
Bruce Lee said, be like water.

~~~
EnderMB
I like your organism explanation. This is something I've learned over the past
decade, albeit not at any company at the level of Google.

The way I've often describe software to clients is likening it to Formula 1
cars. With a normal car, you turn the key and drive, and it should definitely
work for anyone. With a Formula 1 car, the tyres need to be warm, the driver
needs to be experienced, and there are so many reasons why a working car can
stall that careful monitoring, maintenance, and revision needs to be done
long-term by an experienced team.

Sure, the software on your computer might run like the former, but in the
background that huge pit stop team has been plugging away for years, and will
continue to plug away with optimisations and fixes.

------
zuck9
You should replace Google with "big tech company" or FAANG to get more
responses.

~~~
mav3rick
FAANG don't share the same engineering culture.

~~~
mlthoughts2018
Yes they do. In all these places except Amazon, the engineering culture is
essentially identical. I’ve worked at 2 FAANG companies and a few startups
that had a big influx of ex-Google engineers. In fact it’s a whole process
that startups go through to copy Google: monorepo + misguided in-house tooling
for it, homebrewed canary deployment strategies, naming things “mon” like
“altermon” or “datamon”, etc.

It’s a very miserable monoculture.

~~~
yibg
I assume the two FAANG you are referring to are the F and G. The rest of the
letters aren’t like that. At least no monorepo.

~~~
saagarjha
Or "mons".

------
justicezyx
Google showed me the fine engineering at large scale. And the key is the
people as they are behind everything.

That’s why I changed my view on who was the key person for google success. I
used to believe Larry and Sergei, then urs, now I put on Laszlo Bock.

~~~
samueljackson
they are ok at it, far from great. netflix is a good example of engineering at
scale.

~~~
justicezyx
Well, I do not meant to be complacent, but I am pretty sure that anyone think
Google are not the best at its scale, probably have only a superficial
understanding of the Google engineering machine's reach and influence.

------
samueljackson
decades of working for big companies on projects budgeted at billions.
Sometimes as part of large teams, other times a three man operation!!! seen it
all. Big companies taught me to be a politician and that skills alone were not
enough. You'd need to make a strong relationship with everyone and I mean
everyone. Insane working hours, a few projects had expectation of 24/7 and had
I refused, they'd politely replace me with another without notice. Lots of
talented people to do the work under that condition. Had enough of it, not
like they paid me millions and when I complained, they'd say "make your own
company and do it as you like it to be". Every big company I've been to had
that sort of culture. As for small companies, there isn't much to the future.
Lots of work to be done for free, always welcome to work for free. Finally
here I am, looking for a remote job of sorts that'd allow me to travel without
having to be online all day. At least I can enjoy programming and travel.

------
StephenAmar
Code search is a game changer.

~~~
kyrra
If you want to hint of what code search looks like a Google check out code
search for chromium.

[https://cs.chromium.org/](https://cs.chromium.org/)

What is internal is more featureful. And it covers the entire code base.

------
admils
Contract Python / JS engineer, working on the largest GAE app (internal, but
2400% RoI so well regarded). \- GAE was missing basic functionality that we
(as their biggest customer) couldn't get them to work on. The engineer in
charge of GAE had other tasks and didn't care for this role. We ended up doing
this work ourselves and I'm relatively sure this is now inside the App Engine
SDK.

\- Delivery was incredibly badly managed, I have to shut one one eye when I
sneeze due to sinus damage caused by working when ill on a project with global
visibility and no cover.

\- 'Decisions based on data' isn't a thing: if you want an outcome, you
research to that outcome or don't depending on how you feel.

As others have said having Google on your resume is great.

------
nailer
Contract Python / JS engineer, working on the largest GAE app (internal, but
2400% RoI so well regarded).

\- GAE was missing basic functionality that we (as their biggest customer)
couldn't get them to work on. The engineer in charge of GAE had other tasks
and didn't care for this role. We ended up doing this work ourselves and I'm
relatively sure this is now inside the App Engine SDK.

\- Delivery was incredibly badly managed, I have to shut one one eye when I
sneeze due to sinus damage caused by working when ill on a project with global
visibility and no cover.

\- 'Decisions based on data' isn't a thing: if you want an outcome, you
research to that outcome or don't depending on how you feel.

As others have said having Google on your resume is great.

------
Xlythe
I've been at Google for almost 4 years, mostly working at startups before
that. First and foremost, there's nothing special about Google (compared to
any other tech company) and it's best not to evangelize any one company.
Learning comes from experience, which means trying things (even if you end up
making mistakes) and figuring out the shortcomings of a particular solution.
I've seen a lot of bad (and a lot of good) code at Google.

What I learned, design-wise, came from a coworker. They planned everything out
in advance and would draw out components on a whiteboard and define what
they'd do. A few weeks would pass before they'd write any code. In contrast, I
would start with writing skeleton classes and intuit the breakpoints for
building a new class. We ended up with similar code structure at the end of
the day, since we were both designing the architecture (even if our methods
were different). But in code reviews, I would focus on if the new code made
sense within the scope of the change; are there any clear bugs, code
duplication, is there a more simple approach, etc. My coworker always went
back to the whiteboard and drew up the components again, making sure the new
code fit within the originally defined scope or if a larger change would make
more sense. Because my skeleton classes were quickly replaced with
implementations, I lost my original frame of reference and only focused on
what was in front of me. I could still tell when something was obviously wrong
(stop sending me CLs with global state to cross talk between components!), but
there was creep in the CLs I approved and the code would slowly get more
complicated until I realized it needed to be refactored.

There's a tradeoff (like in anything). You can't design your own solution for
every code review, as it takes too much time and removes autonomy from your
peers. But remembering to step back and look at the big picture was something
I needed.

There were other things I learned, like structuring code in ways that it's
hard to write bugs. For example, you could write a method like...

    
    
      void foo(Callback callback) {
        if (error1) {
          callback.onError();
          return;
        }
        if (error2) {
          callback.onError();
          return;
        }
        if (error3) {
          callback.onError();
          return;
        }
    
        doWork();
        callback.onSuccess();
      }
    

but it's easy to forget to call 'callback.onError()' in a future CL. A small
refactor, like...

    
    
      void foo(Callback callback) {
        if (bar()) {
          callback.onSuccess();
        } else {
          callback.onError();
        }
      }
    
      boolean bar() {
        if (error1) {
          return false;
        }
        if (error2) {
          return false;
        }
        if (error3) {
          return false;
        }
    
        doWork();
        return true;
      }
    

has the compiler help you catch mistakes. Unit tests obviously help too, but
doing both reduces the number of bugs that slip through. Similar tricks
include annotating methods as @Nullable so you don't forget to nullcheck,
annotating which thread is calling a method (eg. @FooManagerThread, @UiThread,
etc), and doing all the if checks as close to the top of a method as possible
so that you only do work if you're in a good state.

Oh, and here's one last tip that I only realized needs to be reiterated
because most of my coworkers forget it. Validate incoming data! Every API
needs a wrapper around the entry point that...

    
    
      * Verifies the caller is allowed to call that method
      * Verifies the method can be called at this point in time (eg. hackerNewsApi.postComment(threadId, msg) only works if the threadId is valid)
      * Verifies that the arguments make sense (eg. 'msg' is not empty/null/above the max comment size).
    

And this is needed at every layer (application, server, etc). Trust no one,
even if the only caller is supposed to be yourself.

Glossary: CL = changelist ~= pull request

~~~
crimsonalucard
Choosing the right language also helps reduce errors. There are languages with
no nulls and no runtime errors.

~~~
whateveracct
Not sanctioned for general use at google tho :D

------
opportune
All big tech companies are approximately the same, since people job hop a lot
and these places have tens of thousands of engineers, it’s basically the same
people at all of them anyway. I am not sure why people have these starry eyes
views of them

~~~
mav3rick
No it's not. Microsoft is way different from Google.

~~~
opportune
Having worked at both I can say it’s not that different. But I had a great
team at Microsoft

~~~
mav3rick
I had great teams at both but the bar was consistent across Google unlike
Microsoft. And 9/10 engineers will accept offers at Google over Microsoft
given both.

~~~
opportune
Oh I totally agree, I guess for me when I think of “work” I don’t really think
too much about other people’s engineering ability. It makes a difference but
less of one than most people think.

------
crimsonalucard
I think I'm a good engineer, but I don't think I have the intelligence to get
into google. Most people can't. There's definitely some sort of difference
between an engineer who can get in google and one who can't as displayed by
the interview.

Most companies that interview people outside of FAANG tend to have easier
interview questions in my experience. Some even forego algorithms all
together.

For googlers who worked at companies outside of FAANG or startups. Is there a
noticeable difference when working these people? Will you learn more from a
google level engineer then you will from someone who can't pass the google
interview? What are your thoughts about it?

If you think that FAANG engineers are genuinely better, say it. There's no
need to be politically correct here, honesty is appreciated.

~~~
opportune
I’m not sure why many people seem to think this, there are like tens of
thousands of engineers at google, and there are other good companies out there
that also have tons of engineers, you don’t need to be that special to get a
job at google. As long as you can get your first interview all you need to do
is be able to solve data structures and algorithms problems and talk about
software design. Let’s not put google on a pedestal

I will say, those problems do filter out a lot of bad people. But it only
ensures a minimum level of competence. It doesn’t mean everybody is insanely
smart.

~~~
crimsonalucard
The interviews at google are hard enough such that only a fraction of software
engineers can make it through.

Thus by statistics and logic, the majority of the worlds engineers cannot get
past the google interview. This is not something I am thinking, this is a
statistical reality.

Literally, in order for those engineers to work anywhere else the algorithm
questions must be significantly easier in other companies outside of the elite
FAANG. This is inline with logic and my observations.

My question is... for googlers who have spent time outside of google at a more
scrapy startup. Does the interview filter actually create a better class of
engineers? Do you learn much more working at google then you would at a
company who's engineers typically can't get into google? What's the experience
like if there is any?

Don't get me wrong. I'm not worshipping googlers or asking for tips. A real
dichotomy exists in the interview stage for the few engineers who pass and the
majority who don't... I'm just curious if this dichotomy becomes obvious
outside of the interview... like on the job? Do you notice a difference when
working with a google engineer vs. someone who probably can't ever get into
google. It's a complicated question with a complicated answer. I'm not really
looking for some humble answer telling me that google engineers aren't that
much different from other engineers when the interview essentially disproves
this statement by filtering out everyone except for a certain type of person.

I want an honest and candid answer.

~~~
opportune
Ok sure I’ll give you an honest and candid answer. It’s basically analogous to
top schools. Yes the average engineer at Google is going to be better than the
average engineer at some random company because google generally has a higher
hiring bar. Similarly your average student at Harvard is going to be a better
student than your average student at Random State University.

As a result you will on average learn more than at most other companies. Of
course there are trading firms, quant hedge funds, and a few other big tech
companies with even higher hiring bars where you will learn more. And some of
the things you learn, anywhere, while very useful to getting stuff done at
where you work, will not be readily transferable elsewhere.

But large companies as I’ve said before are basically all the same. The people
you interact with are a vanishingly small subset of the entire company so it
doesn’t even really make sense to speak in terms of what it’s like at the
entire company.

The interviewing filter mostly does one thing which is that it biases the
engineers at google to be generally good at solving data structures and
algorithms. This, by the way, is a skill that can be taught — and even though
it might roughly approximate engineering ability it doesn’t map directly.

~~~
crimsonalucard
Thanks for being candid, is this inline with your experience? Have you worked
at a smaller company and google as well? Your logic is really sound but I'm
also asking this question for the purposes of gathering anecdotal experiences
as well.

Your answer is great, btw, I just noted that you used the word "average" over
a personal example so I wanted to get an idea of where you're coming from.

~~~
opportune
Yeah I’ve worked at G, other big companies and smaller places. My own
anecdotes roughly fit my “average” explanations.

