Hacker News new | comments | show | ask | jobs | submit login
Building a Devops team (agilesysadmin.net)
50 points by sciurus on June 20, 2011 | hide | past | web | favorite | 18 comments

I really like his interview process. Having you actually install WordPress and troubleshoot a broken MySQL installation, while having full access to Google and other resources, is much closer to real sysadmin work than the type of interviewing that Google and the rest of the tech elite seem to like.

I am so sick of getting asked basic administrivia that you never need to know in your daily job, or can easily be googled within 3 seconds. Our brains have selective memory - we remember what is important to us.

If I have to sit through one more interview being asked about esoteric junk like the structure of an inode on disk or how to code an algorithm that you never use outside of a CompSci course in University I'm going to hang up on the interviewer.

I've always thought this type of person was called a 'generalist'. I like devops a bit better.

The problem I've found being a devops-type person is that hiring managers have a hard time figuring out what you do and why you're valuable (author of post excluded).

I've received comments that my resume is too unfocused -- they'd ask if I wanted to be a sys admin or a developer, to which I'd answer "both!"

I just created this account to comment on this. I am a generalist. I started out as a hardware and software designer, but then focused on software for many years. Later moving from Development to Operations where I can write code and deal with sys admin stuff at the same. Writing monitoring tools, installation packages, automation for release and builds, as well reading all the developers code that gets checked into the Source Code repository. Troubleshooting servers, permissions, network issues, software bugs in websites, client/server apps, SQL... whatever it took.

I got my previous first job about 8 years and I remember the person hiring me saying that they wanted “specialists”. I firmly replied I am a generalist. I didn’t hear back for a month or 2 but then got the call that I was hired. For the first few years, the generalist mentality was actually how things ran. Everybody was expected to dive into code, databases, server, etc. Suddenly, they said that silos were forming and in order to improve the organization, a re-org was needed and you had your various teams of architects, dev, support & ops, DBAs, Q&A, project management, etc..you get the idea.

A year after this process the whole team spirit was crushed. The whole “not our problem, it’s that team’s problem” starting gaining ground. People that were once team mates were now (using the someone eles’s words) throwing problems over the wall back and forth. People also threatened management that if they ever got put on team ‘X’, they’re quitting. Yup, they made specialists out of the wrong people. If only they had just left the generalist structure be. For me, understanding something from the beginning to the end is VERY important. How can someone write programs if they don’t understand how security is handled on the web server, or how SQL statements affect performance, or how deployment is done? I think having an assembly like structure (specialists) can work, but it’s not as responsive as a generalist structure. Products will get shipped with less silly mistakes if you know what the destination looks like. Release and deployment is so misunderstood by many developers. People need to get their hands dirty.

I have since left that organization and I’m now back to a generalist position where I can get my hands dirty. It’s tough because you’re starting from scratch again and trying to hire once again. Hiring is hard. Typically, new organizations need generalists more than established ones. That is not to say that established organizations should have less of a use for them. I tried to change the shift from specialists to generalists at my previous work, but you start hurting your career if you’re constantly grinding it out with upper management. You have to know when to let go.

I really like this term Devops. Because you Developers do need to be able to understand Operations, and vice versa.

I think silo formation is pretty much "how business operates" when it's not forced to be super efficient. It benefits certain types of middle management (who are interested in fiefdom building), and allows for "metrics" so that "workforce improvement" can be sold upwards to incompetent or distracted upper management.

Generalists are also hard to hire and manage (it takes another generalist to detect bullshit... and there's always the tendency for the manager to hire folks less talented than him/her self).

I say this from years as a specialist (software) now doing more generalist work... the only thing I don't like is that I get less stuff done on any given angle, but I do get to look at (and own) the whole picture now.

I often wonder how developers are actually able to develop without being able to throw together the entire stack they use without following a detailed tutorial.

@r00fus: You hit it on the nail. I was actually involved in creating all of the initial reports for upper management for statistical purposes (metrics). It's because management had no clue what anyone was doing because the team grew too quick too fast. I was fortunate at looking at the numbers of 30+ developers. Everybody had to enter their time. Even if you spent 10 minutes reading email, you enter it. I am not exaggerating. All the numbers were bullshit. Everybody made sure to have 7.5 hours. All the numbers were padded. The metrics didn't actually reflect the truth, just how people padded their numbers.

@devicenull: I have done that as well. You basically program using the "leap of faith" methodology. You're never 100% sure that what you wrote won't break something. You rely on others to validate your work. Assuming the "others" know what to do.

Could the timing be worse? "Brian Henerey heads up Operations Engineering in the Online Technology Group at Sony Computer Entertainment Europe." talking about devops? It's almost as fun as Sony's scheduled tweets, inviting to play online while their network was down. Whatever he wrote in this article, I'd be thinking about their security problems and how they relate to what he's promoting...

Great articles, including those from the links.

I have troubles in finding such a position, even though I focus on start-ups only.

Just wondering: how much do they pay? I'm asking this because 1) people with the skill set they're looking for are extremely rare 2) tend to be self-employed or startuppers 3) might leave any moment if the conditions they're looking for are not met.

So what's the kind of money you're offering to retain these guys?

Disclaimer: I've previously worked with the owner of the linked blog, but not in the environment being discussed in this post.

I can't give you the specifics for the employer (Sony) mentioned in the blog post, but I can offer answers based as my own experience. I went back to freelancing/self-employed about 5 years ago following 7 years of being in a full-time job. At the time it was primarily motivated by the rates I'd seen advertised for roles similar to what I was already doing, with grand plans of building a team of other contractors that I'd farm work out to and skim a % off the top and ultimately build a successful dev shop of my own. I quickly realised the economics and scale required to make that happen, so instead focussed on just making sure I had enough work to keep myself busy. What I've realised in that time is:

* Demand definitely exceeds supply at the moment. I've constantly had more paying work than I have time available to do it.

* We (developers) get paid stupidly good for what we do. Not movie star/sports person good, but still pretty amazing given there is very little risk to our health and much of it can be self taught.

* Primarily chasing the money is almost always destined to end in disaster.

I spent the better part of 3 years working with Stephen (the author of the linked blog) for a client. Almost everyone there was self-employed, and most had been there for longer than I had. I was definitely getting paid below market rates, but I stayed there for that long because:

* They were honest and open about everything in the business, I never felt like I was being screwed over or that my contribution was undervalued.

* The working environment was flexible, I had Monday's off (which as a contractor is actually another 20% hit on your weekly rate), and my performance was measured by delivering rather than face-time and adherence to a schedule.

* It was just a generally great place to be.

I've since worked for clients where the environment has been just the opposite. The worst places have always been the highest paying, they've always been cashflow rich and figure money can solve all their problems. If I've had any complaints about the way things were done and said I was leaving, they'd throw more money at me. I'd justify accepting it at the time, and then start to feel dirty about the fact you're just whoring yourself out. I can't be the only one to have conclusion, the fact I've worked with so many talented people happily earning less than their potential I think confirms that.

Because lets be honest, I could be earning 50% of what "market rates" are these days and still be a hell of a long way from the poverty line.

I guess what I'm trying to say is that while great developers might be rare, I think great developers that are primarily motivated by their daily rate are even rarer. If you (as an employer) are aware that all your employees, whether they are contractors or not, might leave at any moment if you're not offering them an enjoyable place to work... well then I think you're half way there. Then build it, and they will come :)

So, do you actually have any numbers to answer the grandparent's question? I'm guessing $80,000 USD + benefits (or equivalent) at a minimum.

No, and I obviously didn't make my point clearly enough. It was to stop thinking that you have to offer a minimum of $x to attract people. I tolerated a 49% drop in effective pay rate to work somewhere I loved.

Offer a really great place to work, at a minimum.

It's your life and you make the calls but why couldn't you just work half the time then? You'd get the same money and 12 or so days off every month. You can do plenty of interesting work during that spare time.

Because I've seen plenty of people try that (myself included) and it's rarely worked as planned. If you've got an interesting idea that you genuinely I think being tied up doing something else for 8 hours a day is just an excuse. You can either fit it in to your down time, or quit completely and be truly motivated to get it done.

Instead I opted to have a long weekend every week, and spend my days doing something I enjoyed for people I liked for an amount of money that let me live a still very comfortable lifestyle in what is reportedly one of the most expensive cities in the world. I've still got a tonne of time to do interesting work, and a family that is happy I'm not coming home complaining about "the office".

>a family that is happy I'm not coming home complaining about "the office"

That was why I quit my last job. I was emotionally invested. Couldn't help it. When you come home and 90% of the time you're complaining to your family about work - it's time to leave.

My current job is a 10 minute walk. 9 to 5. And I'm paid well. I come home quickly and have plenty of time to spend on whatever I want. And right now, it's my family.

(I really don't want to come off as bragging, please bear with me.)

I'm a programmer and I've got my reasons to believe I'm pretty good at my job. I love my job. But it's a job, and the amount of money they pay me to do that is directly proportional to the likelihood I'll accept their offer. For me, working in a management-infested big firm at 200€/hr beats working for a small, agile startup at 100€/hr hands down. I'd go for the big firm even if the startup offered 150€/hr.

Why? I can always work on my projects, and I do, on my spare time. If I work at twice the rate, I can take half my month off and take home the same amount of money I'd get for working at the startup.

Also, I really don't think we programmers get paid well for what we do. Think about our job: it needs a lifetime of experience and you always have to keep updated. When they say that the best programmers are the one with passions, think about it - there's no other way to be even a decent programmer if you're not very passionate about it, because it needs so much time and practice it is just not possible to be a programmer the same way you can be an accountant (just to say, nothing personal against accountants).

My idea customer is one who pays me a ton of money to work with shitty tools on problems he can't even define himself, and then blames it all on me when things go fubar (as long as he pays me). I really don't care - I don't want to be appreciated, I don't want to be cuddled - I'm a professional and I'll give 100% when on shift, I don't need a pat on the back. I will have my satisfaction during the huge amount of spare time that I'll be able to take for myself because you're paying me lots more than a "friendly" customer does.

Really? I know at least in Australia and the UK (only places I've filed tax returns so only places I can speak with any degree of confidence) that becoming a professional accountant requires:

- 3-4 years (at least) of university - industry accreditation if you want to work for yourself - a minimum number of hours every year spent maintaining that accreditation - trawling through government budget and tax amendments every year to know the impact for all or your clients - knowing the legal intricacies of each country/state/county you operate in to ensure you're doing the best job for you clients, and keeping abreast of any changes to those.

There's no way I'd trade being a programmer for having to stomach that burden. And most accountants I know are on a lower rate than most devs I know, unless they're working for a big 4 consultancy.

My ex, and accountant and, shortly later, a CPA, was never asked in an interview what sort of outside accounting projects she worked on. Companies didn't ask for a 'passionate' accountant, they asked for someone with who could reliably get the work done on time. The companies she worked for sent her to training and testing on their dime and time.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact