
How Google Handles IT for Its Workers - sytelus
http://online.wsj.com/articles/how-does-google-handle-it-for-its-workers-ask-cio-ben-fried-1413751227
======
enneff
When I started at Google I was immediately impressed by the way IT is managed.

Need a new mouse or keyboard? Go and grab one from a nearby tech stop. No need
to ask, just swipe your badge and take it. Need something more esoteric? File
a ticket and they'll make it happen. I've never had an equipment request
denied at Google. The worst that's happened is I've had to wait a few weeks
for a particular item to come in stock.

~~~
opendais
Why is this impressive in 2014? This seems normal to me and I never worked for
a company like Google.

~~~
existencebox
As an employee at another one of your generic bigCOs who at one time was a
googler, it is very difficult to get e.g. another monitor in a reasonable
size, or new hardware on short notice or without manager buy-in. You typically
have to "steal one" from another employee who is moving/leaving. The process
at G was a breath of fresh air.

~~~
johnward
Yep as soon as someone announces they are leaving the vultures come in. We are
actually not even "approved" to have an external monitor based on my title.

------
nanoscopic
To chime in with contrasting experiences at other companies:

Where I used to work RHEL was used on all of the servers. I was part of a team
developing some new software using Mongrel2, MongoDB, ZeroMQ, Perl, and
Imagemagick ( among a whole collection of other esoteric small pieces ).

The plan for the project was approved and development underway. We encountered
problems getting everything to work properly with modern versions of the
various parts in the distribution of RHEL provided.

The company was mirroring RHEL, but only the core. They were not mirroring the
updates, nor were they mirroring the latest released version of RHEL. Outside
access was forbidden on the servers, so we could not fix the problem ourselves
without going around network policy.

We requested virtual machines; specifically to use Virtualbox on our desktops
to do development until the policy people could figure out how to give us a
server with the software needed. We were denied and told we had to requisition
a server to run VMs, that VMs on desktop were forbidden.

A $13k server was purchased for this purpose, and root access was denied to
it. We couldn't configure it as needed, and IT refused to help. It was
basically useless. The main department head approved going ahead with VMs
regardless of the network admin and head of software development saying we
couldn't.

In the end the project flopped; ultimately due to inability for any of IT to
get along or provide what was needed.

~~~
eitally
We are currently facing a situation where our data center guys only allow us
to use RHEL in production but since we standardized on PostgreSQL (moving off
MSSQL) we've been devving on 9.2/9.3. RHEL's officially supported version of
PostgreSQL is still 8.3/8.4. The devs are up in arms about being "forced" to
use a major version behind just because the DC team won't let them install and
self-support the latest stable version. Ridiculous.

~~~
nanoscopic
RHEL, CentOS, and Fedora are all basically the same thing.

The difference is which package versions are approved for use.

For the most part, you can add the yum configuration for CentOS or Fedora to
RHEL machines, and install newer packages without breaking it.

Note that some packages will end up pulling in a ton of other dependent
packages. By adding a Fedora repo to a RHEL machine, the following things will
happen: 1\. You will likely lose official support from RHEL. Anything that
goes wrong is your fault. 2\. Your OS install will start becoming Fedora; as
you install more and more stuff. 3\. Your setup could break entirely because
RHEL packages are not tested to be compatible with Fedora ones. The core
configuration is different in places, and generally you should not start
updating the core components as this will break things.

My recommendation is to setup the CentOS/Fedora yum repos on your RHEL box.
Run yum install for new PostgreSQL and see what yum wants to install. Eyeball
the results intelligently and take some guesses as to whether it will break
things. Cross your fingers and approve it, then disable the CentOS/Fedora
repos you added. Carry on with your life and don't tell IT what you did.

If you can't connect your RHEL machine to outside network, repeat the process
somewhere else to figure out what RPMs are needed. Download them elsewhere,
and stick them on the RHEL machine and install manually.

As an additional note; I have setup a Fedora repo on a RHEL machine and
changed the flag in the system to change the distribution ( there is a core
package everything depends on... ). We then updated all, and the resulting
system still functioned. Very evil, but it didn't explode. Also; it took quite
a while to install all the packages. It essentially is converting the whole
system from RHEL to Fedora in place...

~~~
shanemhansen
This isn't a good idea. Why? Because it takes IT from the position of being a
roadblock, which is something you can push to get fixed and puts you in the
wrong instead.

IT has valid reasons using RHEL and older packages, you presumably have valid
reasons for using newer packages. If newer packages are needed work with IT to
get those newer packages. This also means that they are back in the position
of being a roadblock if critical projects really can't go forward due to their
policies.

Sometimes the way to fix what is broken is to make your department stop being
a crutch for failed IT policies.

Also, randomly changing system packages in production is a dick move. That
sort of shit is (rightfully) going to result in getting your access revoked at
the minimum.

~~~
nanoscopic
Multiple hour long meetings with all of the internal authorities were
involved. They all refused to allow new packages to be installed.
Additionally, they were not even providing the proper update packages from
RHEL due to system misconfiguration. EPEL was additionally forbidden.
Regardless, the powers that be essentially said "You are not allowed to use
the software that your project specification states that it needs."

It was either use other components or circumvent IT. The head of the
department got fed up with the red tape and told us to circumvent. All proper
discussion was done before that, and as a result nothing I did was a "dick
move". The system was additionally not in production yet, so no changes in
production were being done.

Shame on you for making too many assumptions.

------
madengr
Asshole IT staff at my employer have banned Linux workstations from the
network, even though we need it for running Cadence and other intensive
simulation tools. The solution is for them to run a Linux server into which
one will VNC to do graphic intensive IC layout.

We all get a sanitized windows 7 shitbox, because a IC deign engineers needs
are the same as a secretary (not that we have secretaries anymore).

~~~
drzaiusapelord
> The solution is for them to run a Linux server into which one will VNC to do
> graphic intensive IC layout.

Why VNC? NX or just plain x11 window forwarding.

Maybe they want all their IC data on a server that gets backed up, etc as
opposed to littered on a bunch of random desktops that might or might not be
backed up.

~~~
madengr
I forgot to add, it's actually worse, in that it's Cadence running on Linux
under a VM on some Windows server, which itself is probably a VM. Remote
Desktop is allowed to the host running the VM, but not into the VM itself.

We had a centralized Linux server and tape system backing up all the Linux
workstations, but they fired the guy who ran that server due to stack ranking;
i.e. firing 10-15% of you staff yearly, even though he was a really sharp guy.

IT rules the roost around here, even though we are an engineering company that
produces physical items; not software. It's supposed to be the other way
around; IT is supposed to provide a service.

------
patio11
I know 4 years is ages in Internet time, but didn't it make international news
when they changed the policy from "Use whatever platform you want" to "Use
whatever platform you want as long as it isn't the one that corporate IT is
basically built to manage."

[http://www.ft.com/cms/s/2/d2f3f04e-6ccf-11df-91c8-00144feab4...](http://www.ft.com/cms/s/2/d2f3f04e-6ccf-11df-91c8-00144feab49a.html#axzz3GrMQOfkU)

~~~
melling
I was hoping that Google will be able to push Linux into the mainstream. One
of the big problems with Linux is getting commercial software. For example,
getting Photoshop and Illustrator working natively on Linux would be a big
win.

~~~
Igglyboo
Google actually maintains their own fork of Ubuntu for internal use.

[http://en.wikipedia.org/wiki/Goobuntu](http://en.wikipedia.org/wiki/Goobuntu)

------
pibefision
I'm starting to see a trend in Google PR.

I've read the latest book from Eric Schmidt and Jonathan Rosenberg, where the
main argument is "we are the best because we have the smartest people on the
planet working for us". That's the main point to explain many of the things
that they do.

This article seems on the same line.

~~~
awakeasleep
I'm an IT lead at a much smaller company, and I run the same kind of setup.
We're not the smartest people on the planet either.

But I do hold every hire to the standard of "will this person increase our
average capability?" And I think smartness or motivation does have something
to do with that. (We also have system administrator level people as our first
level support)

However, most companies don't put that level of effort into their IT org. This
article is worth reading because it says that your company can benefit by
holding the front-line support to a high standard, and paying to attract
talented people to that position. That is a radical proposition to most
companies. The only thing that could shake up the industry more is if they
said they required IT leadership to have technical ability and insight.

~~~
dethstar
>it says that your company can benefit by holding the front-line support to a
high standard, and paying to attract talented people to that position.

Doesn't that just apply to any position?

------
spindritf
Most businesses don't hire googlers and they will have a bad time if they
implement Google's policies. It's not "cultural." There's quite a difference
between someone who had been programming since they were 15 spending nights
reinstalling linux and someone who "can use technology" where that means MSO
and facebook.

How many people here serve as tech support for their families? And how many of
your relatives are otherwise successful professionals?

~~~
gedrap
Exactly. It works at Google because their engineers know their shit.
Meanwhile, at your average corporation... That might not be the case, and you
might spend loads of time trying to figure something out because your
colleague uses some semi-random tool 'because hey it's cool!'.

~~~
VLM
"because your colleague uses some semi-random tool"

For extra fun combine that with not invented here syndrome and retired in
place. "Git? whats git? never heard of it. Besides don't you know RCS is the
industry standard for version control?" (which was actually pretty much true
in 1985) And no this wasn't in '06 this was last year. I'm glad I don't work
in that particular team!

And add some inter group politics for extra extra fun.

~~~
RollAHardSix
Not to hijack the thread but here's a fun story, the place I'm working at now
actually considered 'version control', to be making a copy of the file before
you started making changes. There was no version control. This is a software
company that has been in business for ~20 years. I guess that's just part of
what happens when you land these big contracts, and hire the least common
denominator.

~~~
jpindar
I think many HN readers would be amazed at how often this is still true.

------
johngalt
In advance of the inevitable and one sided 'IT is crap' comments. Consider
that your company may have different goals and expectations of IT. If a google
engineer screws up or can't get his BYOD kit to work, it's not on google IT to
explain themselves. In <bigco IT> the expectation is different. Where anything
that is permitted to run must work provably and IT must explain any
deficiences.

------
bank_to_google
It is ironic this chap waxes lyrical about how great it is to let the staff
choose their IT equipment when he came from a bank previously, where they make
even the Unix SAs use the hideous corporate Windows desktops to try and do
their job.

Yeah, unless you are Google, IT equipment sucks donkey balls in the big corps.
Especially the banks. Still stuck on IE7 at our one. Sigh.

~~~
vince_refiti
I suffer this fate every day. It is made worse by being locked down so hard I
can't even use PowerShell. All I have is Notepad++ for development, searching,
mass text manipulation...

~~~
bhaak
How come you are not looking for a new job then?

It's one thing having to use Windows for development. I've done it, you can
work around lots of issues with Cygwin and other separate tools that enhances
the development unfriendliness of Windows. If you are developping in Java,
it's even less of an issue as Java itself and most of its IDEs are cross-
platform.

But being so locked down that the only thing to do development with is
Notepad++ (you can't even run vim or emacs?), that's actually so far beyond
that even a complete non-tech person should be able to see that this is
harmful to your productivity.

~~~
vince_refiti
I started less than a month ago, and it pays really well. I will stick it out
and hope to convince management to let me have admin rights to my machine.

~~~
jsymolon
You don't need admin rights.

What you do need is a reasonable set of tools to be installed and a quick
(under an hour) way to get items installed or updated.

What about a VM ?

Approach it from the tools use, do people use pads and pens or do they use
Word?

~~~
glynch
The bigger the company the more difficult this is. Where I worked our
workstations were locked down and required a package request to install
software. In addition, they would not bother packaging a new tool if it served
the same purpose of other tools; i.e. text editors. Doesn't matter if you
prefer Vim/emacs/Sublime Text 2/Notepad++, we already have a license for a
text editor you've never heard of before.

Depending on the company's hardware VM performance could be a real issue.
You'd be surprised how many developer jobs are done on old Windows XP boxes.
Big companies typically do upgrades in multiyear cycles.

------
wldcordeiro
The IT at the company I work for is odd. I work for a smaller subsidiary that
manages most things for itself but we're still required to have a parent
company approved workstation alongside our main development computers.

So I have this dinky little HP with a 15" monitor just to do the HR computer
"training" stuff related to our industry and then I have a full size
workstation with dual 24" monitors that I develop on. They're connected to
different networks, my workstation is on a Comcast business line and then the
other is on an internal network that has the strictest security policies on
the planet.

It's like having a weird cross between IT as it should be (as described in
this article) and IT hell you expect in large corporations.

------
davidw
> Now, what we do here at Google is really different.

> We can’t afford to have technology support where there are cookbooks and
> rules and every possible change is documented in advance

There's something ironic in that...

------
philip1209
They're able to be flexible with IT because no Google code lives on the
laptops - they are basically SSH clients with a Yubikey in the USB port and an
encrypted SSH key. All of the data that needs securing never makes it down to
the laptop.

------
lmedinas
"Letting Employees Choose Their Own Technology Is Beneficial, Says CIO Ben
Fried" -> This!

This means that people are more productive when they have the hardware and
software they are used to.

~~~
yitchelle
But this would mean that the majority of folks in the company are like minded.
Imagine, if I choose to use a Mac (because I am hipster), and my colleague on
my right choose to use a ChromeBook (because he is a cutting edge geek) , and
my colleague on my left choose to use a WinXP (because he doesn't want to
change.)

Interoperability would surely take a hit, and imagine the IT support required.

/sorry for the sarcastic stereotyping...

~~~
melling
No, I can't imagine the problems. I've had Mac, Linux, and Windows machines on
my home network at same time. Could you elaborate on the problems?

~~~
Brian-Puccio
So your department needs a document management system that will make sure you
are in compliance with all of the regulatory requirements that affect your
industry as well as interface accordingly with vendors and other departments
in your organization. The software has plugins with Word 2013 on Windows,
Acrobat 11 on Windows and Outlook 2013 on Windows, but now people are allowed
to pick whatever platform they want.

What do you tell the Mac user? What do you tell the Linux user? Boot a VM
whenever you want to work on a document? What do you tell the iPad user?

In many large organizations, simple file systems won't work because they don't
scale and you can't tell a legal department "oh, you need to
share/version/distribute documents? just use this program called git, it's
easy, let me show you and once you've got the hang of it, you call the
appellate judge and tell him that he just needs to do a git pull to get your
brief".

When every software package that's deployed in millions of installations
around the world that's only available in the Windows/Office/IE environment is
completely cross-platform, then yes, you'll have completely interoperability
between Windows, Mac and Linux machines, as well as iOS and Android devices.
Until, the "I've had Mac, Linux and Windows machines on a home network" line
has no bearing whatsoever on intolerability of software packages an enterprise
environment that needs to interface to hundreds of external organizations
daily.

~~~
wmf
Use whatever you want has the obvious caveat that you have to get work done
and you have to interop with other people.

------
Tepix
I hope my boss reads this.

~~~
mcherm
Print it out. Leave it on her desk.

Better yet, schedule a "brown bag lunch discussion" around the article, and
invite your boss as well as several of your colleagues.

Your corporate culture is something you must work within, but it is also
something you can change -- or at least affect.

