
Ask HN: What is the development environment at Google like? - ralmeida
I take a lot of interest in the setup of development environments and how they affect productivity, and I hear a lot about the development environment at Google specifically (mainly on Quora and &#x27;word on the street&#x27;).<p>In particular, it seems that for most work, the actual machine being worked on is just a &#x27;terminal&#x27; for a more powerful machine accessed via SSH where the actual work (testing, builds, etc, is done).<p>I remember having read somewhere that developers are even not allowed to check out the repository in their local machines.<p>Is all this mostly true? Does it apply for all types of development (backend, frontend, mobile, tooling, etc)? I can see this working for backend development, but for mobile (or even maybe frontend), it sounds like a big hassle.<p>I would be interested in specific accounts of day-to-day work as well. For example, if you do use a remote machine for doing some type of work, does this mean that debugging is done accessing the IP of the remote machine directly from the local machine? Or all testing is done directly in the remote? Is this remote machine a remote desktop, or just a terminal? How is the connection between the machines made (apart from the SSH terminal the developer is operating)? Directly over the Internet? SSH tunnels? VPN?<p>I would also be interested in accounts from the other big players (Facebook, Amazon, Microsoft, Netflix, etc).
======
wsloth514
DISCLAIMER. I don't work there. But I know professors and met people that
did/still do. It does depend on the position.

First, their infrastructure and resources are mind-boggling impressive.
Second, it is more like any other job you would normally do. You have
deadlines, get it done. Somethings are more fun to work on than others. Yes
they have pros(including perks)/cons like any other job. Third, you loose your
life to the company. They normally are at the office all day and Saturdays. I
know people who love this and others that hate it. Essentially, all your
coworkers become your friends because you are always with them.

Yes, I am told they have their own Ubuntu flavor they use to work on. Servers
are constantly building software. Every company has a VPN, this is normal
business practice.

------
bminor13
I used to work for Google - voluntarily left earlier this year. I'm not sure
how much I am allowed to disclose about internal tools and processes, so I'll
refer you to this article: [https://cacm.acm.org/magazines/2016/7/204032-why-
google-stor...](https://cacm.acm.org/magazines/2016/7/204032-why-google-
stores-billions-of-lines-of-code-in-a-single-repository/fulltext) and call out
some parts that you might care about.

The workstation is not connected to the codebase via SSH on a server, but
rather the version control clients are remote and mapped into the local
filesystem via FUSE. So the workstation is less of a dumb terminal; programs
that access the code can run locally, even though the code itself is not
stored locally. Therefore, version control clients can be accessed with
various web-based tools as well.

Not all developers used a workflow that includes Piper/CitC, but most do. From
my limited view, the Piper/CitC workflows were the most painless.

Debugging - I only ever used a debugger for Python, and in those cases the
application I was running was local. Otherwise I never used a debugger but
rather logging (printf debugging, essentially). I'm of the opinion that if you
have to open a debugger to understand the code, you've already lost. Others
might have better stories.

Remote machine - as mentioned above, there is no remote machine. Things are
done on your workstation. The build commmands can build things locally (slow)
or farm them out to a cluster of build machines in datacenters (fast). See
[https://bazel.build/faq.html#does-bazel-require-a-build-
clus...](https://bazel.build/faq.html#does-bazel-require-a-build-cluster)

~~~
ralmeida
That's very interesting, thanks for your perspective. I was familiar with
Piper (from having skimmed the very article you linked), but didn't go much in
depth about how it impacts dev environment.

Debugging, I tend to agree with you to some extent - I'm even 'on the record'
with friends on saying that debugging should 'come from the mind' before
mindlessly stepping through a debugger. Being able to do it, however, may be a
valuable tool to help you 'refresh your mental model' of the code, or even to
build the mental model in the first place when you are first learning a new
system.

Building (Bazel) - Nice! Typically, when doing development on a compiled
system, would you retrieve the build back from Bazel/Blaze to 'play around'
with it locally? Or is most development test-driven? If the former, how many
builds did you do on a typical day? If you're able/willing to share, of
course.

------
fuball63
I have a friend that works there. He said everything he does runs in App
Engine, and when they go for a feature update, they just push it off to one of
several staging environments running App Engine, which is maintained by a team
of devops engineers. From his perspective, it's just push a button and go.

He didn't mention anything about a remote machine.

Disclaimer: this was all information I gleaned from casual conversation, so
take its accuracy with a grain of salt.

------
spcelzrd
The answer to this question depends quite a lot on what is being developed.
It's not like an engineer at Google could have their own local copy of
google.com.

Most of my career I've been in mobile development, so the code basis are
typically smaller. I've always had the full code on my local development
machine. But you have to connect to web services and back end databases, so
there's usually development, test, and production environments.

------
zerr
Interesting question indeed. E.g. myths such as everyone using either vim or
emacs... I wonder how common are "sane" IDEs, e.g. someone who uses C++ using
Visual C++ IDE, Windows instead of mac/Linux, etc...

------
Clownshoesms
I guess I'd ask, why do you care? Are you looking to optimize your own
workflow, or are just curious about Google's?

I don't think we need to be holding an advertising company up as our best
achievement. At least I hope not.

~~~
ralmeida
Fair question. I want mainly to optimize my own workflow, maybe try some new
'ways' do develop software - editing remote vs local, building remote vs
local, testing remote vs local, manual "printf" debugging vs debugger, etc.

I'll tell you why I chose to focus on Google specifically to start with.

First, it has a generally "analytic approach" to running their business, which
they use for many things, from actual engineering, to dev ops, to "people ops"
\- you might have seen an article which describes how they basically tackled
the problem of "how team dynamics" work with a large-scale internal social
experiment with a heavily quantitative methodology.

Second, being very large scale, they have plenty of incentive to try to
optimize developer experience, since even a small efficiency gain can make a
non-trivial impact.

Third, they share quite a bit of their 'opinionated approaches' about things
both in the form of research, articles and open source software (see
MapReduce, Piper, Kubernetes, etc), but not everything, which seems to spark
the imagination of people and generate some 'folklore' (as I mentioned in the
question, I've heard/seen quotes from 'no one can check out code locally',
'everyone uses vim', 'nobody uses vim', etc).

This is obviously only my impression, which may be even way off.

Finally, I am sure that other similarly sized companies may have very
interesting insights about this kind of thing. I chose to not make the
question too generic also to not dilute the interest of the very people
qualified to offer some insight.

------
bsvalley
BS. It's like anywhere else, you code on your machine and devops takes care of
the rest.

