

Building a Devops team - sciurus
http://agilesysadmin.net/building-a-devops-team

======
unfocused
I just created this account to comment on this. I am a generalist. I started
out as a hardware and software designer, but then focused on software for many
years. Later moving from Development to Operations where I can write code and
deal with sys admin stuff at the same. Writing monitoring tools, installation
packages, automation for release and builds, as well reading all the
developers code that gets checked into the Source Code repository.
Troubleshooting servers, permissions, network issues, software bugs in
websites, client/server apps, SQL... whatever it took.

I got my previous first job about 8 years and I remember the person hiring me
saying that they wanted “specialists”. I firmly replied I am a generalist. I
didn’t hear back for a month or 2 but then got the call that I was hired. For
the first few years, the generalist mentality was actually how things ran.
Everybody was expected to dive into code, databases, server, etc. Suddenly,
they said that silos were forming and in order to improve the organization, a
re-org was needed and you had your various teams of architects, dev, support &
ops, DBAs, Q&A, project management, etc..you get the idea.

A year after this process the whole team spirit was crushed. The whole “not
our problem, it’s that team’s problem” starting gaining ground. People that
were once team mates were now (using the someone eles’s words) throwing
problems over the wall back and forth. People also threatened management that
if they ever got put on team ‘X’, they’re quitting. Yup, they made specialists
out of the wrong people. If only they had just left the generalist structure
be. For me, understanding something from the beginning to the end is VERY
important. How can someone write programs if they don’t understand how
security is handled on the web server, or how SQL statements affect
performance, or how deployment is done? I think having an assembly like
structure (specialists) can work, but it’s not as responsive as a generalist
structure. Products will get shipped with less silly mistakes if you know what
the destination looks like. Release and deployment is so misunderstood by many
developers. People need to get their hands dirty.

I have since left that organization and I’m now back to a generalist position
where I can get my hands dirty. It’s tough because you’re starting from
scratch again and trying to hire once again. Hiring is hard. Typically, new
organizations need generalists more than established ones. That is not to say
that established organizations should have less of a use for them. I tried to
change the shift from specialists to generalists at my previous work, but you
start hurting your career if you’re constantly grinding it out with upper
management. You have to know when to let go.

I really like this term Devops. Because you Developers do need to be able to
understand Operations, and vice versa.

~~~
devicenull
I often wonder how developers are actually able to develop without being able
to throw together the entire stack they use without following a detailed
tutorial.

~~~
unfocused
@r00fus: You hit it on the nail. I was actually involved in creating all of
the initial reports for upper management for statistical purposes (metrics).
It's because management had no clue what anyone was doing because the team
grew too quick too fast. I was fortunate at looking at the numbers of 30+
developers. Everybody had to enter their time. Even if you spent 10 minutes
reading email, you enter it. I am not exaggerating. All the numbers were
bullshit. Everybody made sure to have 7.5 hours. All the numbers were padded.
The metrics didn't actually reflect the truth, just how people padded their
numbers.

@devicenull: I have done that as well. You basically program using the "leap
of faith" methodology. You're never 100% sure that what you wrote won't break
something. You rely on others to validate your work. Assuming the "others"
know what to do.

------
illumin8
I really like his interview process. Having you actually install WordPress and
troubleshoot a broken MySQL installation, while having full access to Google
and other resources, is much closer to real sysadmin work than the type of
interviewing that Google and the rest of the tech elite seem to like.

I am so sick of getting asked basic administrivia that you never need to know
in your daily job, or can easily be googled within 3 seconds. Our brains have
selective memory - we remember what is important to us.

If I have to sit through one more interview being asked about esoteric junk
like the structure of an inode on disk or how to code an algorithm that you
never use outside of a CompSci course in University I'm going to hang up on
the interviewer.

------
thirdstation
I've always thought this type of person was called a 'generalist'. I like
devops a bit better.

The problem I've found being a devops-type person is that hiring managers have
a hard time figuring out what you do and why you're valuable (author of post
excluded).

I've received comments that my resume is too unfocused -- they'd ask if I
wanted to be a sys admin or a developer, to which I'd answer "both!"

------
viraptor
Could the timing be worse? "Brian Henerey heads up Operations Engineering in
the Online Technology Group at Sony Computer Entertainment Europe." talking
about devops? It's almost as fun as Sony's scheduled tweets, inviting to play
online while their network was down. Whatever he wrote in this article, I'd be
thinking about their security problems and how they relate to what he's
promoting...

------
jamesli
Great articles, including those from the links.

I have troubles in finding such a position, even though I focus on start-ups
only.

------
andreadallera
Just wondering: how much do they pay? I'm asking this because 1) people with
the skill set they're looking for are extremely rare 2) tend to be self-
employed or startuppers 3) might leave any moment if the conditions they're
looking for are not met.

So what's the kind of money you're offering to retain these guys?

~~~
glenngillen
Disclaimer: I've previously worked with the owner of the linked blog, but not
in the environment being discussed in this post.

I can't give you the specifics for the employer (Sony) mentioned in the blog
post, but I can offer answers based as my own experience. I went back to
freelancing/self-employed about 5 years ago following 7 years of being in a
full-time job. At the time it was primarily motivated by the rates I'd seen
advertised for roles similar to what I was already doing, with grand plans of
building a team of other contractors that I'd farm work out to and skim a %
off the top and ultimately build a successful dev shop of my own. I quickly
realised the economics and scale required to make that happen, so instead
focussed on just making sure I had enough work to keep myself busy. What I've
realised in that time is:

* Demand definitely exceeds supply at the moment. I've constantly had more paying work than I have time available to do it.

* We (developers) get paid stupidly good for what we do. Not movie star/sports person good, but still pretty amazing given there is very little risk to our health and much of it can be self taught.

* Primarily chasing the money is almost always destined to end in disaster.

I spent the better part of 3 years working with Stephen (the author of the
linked blog) for a client. Almost everyone there was self-employed, and most
had been there for longer than I had. I was definitely getting paid below
market rates, but I stayed there for that long because:

* They were honest and open about everything in the business, I never felt like I was being screwed over or that my contribution was undervalued.

* The working environment was flexible, I had Monday's off (which as a contractor is actually another 20% hit on your weekly rate), and my performance was measured by delivering rather than face-time and adherence to a schedule.

* It was just a generally great place to be.

I've since worked for clients where the environment has been just the
opposite. The worst places have always been the highest paying, they've always
been cashflow rich and figure money can solve all their problems. If I've had
any complaints about the way things were done and said I was leaving, they'd
throw more money at me. I'd justify accepting it at the time, and then start
to feel dirty about the fact you're just whoring yourself out. I can't be the
only one to have conclusion, the fact I've worked with so many talented people
happily earning less than their potential I think confirms that.

Because lets be honest, I could be earning 50% of what "market rates" are
these days and still be a hell of a long way from the poverty line.

I guess what I'm trying to say is that while great developers might be rare, I
think great developers that are primarily motivated by their daily rate are
even rarer. If you (as an employer) are aware that all your employees, whether
they are contractors or not, might leave at any moment if you're not offering
them an enjoyable place to work... well then I think you're half way there.
Then build it, and they will come :)

~~~
getsat
So, do you actually have any numbers to answer the grandparent's question? I'm
guessing $80,000 USD + benefits (or equivalent) at a minimum.

~~~
glenngillen
No, and I obviously didn't make my point clearly enough. It was to stop
thinking that you have to offer a minimum of $x to attract people. I tolerated
a 49% drop in effective pay rate to work somewhere I loved.

Offer a really great place to work, at a minimum.

~~~
andreadallera
It's your life and you make the calls but why couldn't you just work half the
time then? You'd get the same money and 12 or so days off every month. You can
do plenty of interesting work during that spare time.

~~~
glenngillen
Because I've seen plenty of people try that (myself included) and it's rarely
worked as planned. If you've got an interesting idea that you _genuinely_ I
think being tied up doing something else for 8 hours a day is just an excuse.
You can either fit it in to your down time, or quit completely and be truly
motivated to get it done.

Instead I opted to have a long weekend every week, and spend my days doing
something I enjoyed for people I liked for an amount of money that let me live
a still very comfortable lifestyle in what is reportedly one of the most
expensive cities in the world. I've still got a tonne of time to do
interesting work, and a family that is happy I'm not coming home complaining
about "the office".

~~~
unfocused
>a family that is happy I'm not coming home complaining about "the office"

That was why I quit my last job. I was emotionally invested. Couldn't help it.
When you come home and 90% of the time you're complaining to your family about
work - it's time to leave.

My current job is a 10 minute walk. 9 to 5. And I'm paid well. I come home
quickly and have plenty of time to spend on whatever I want. And right now,
it's my family.

