

DevOps Engineering: Who does your automation? - emcooke

Here a brief intro for those that aren't familiar with DevOps:
http://www.jedi.be/blog/2010/02/12/what-is-this-devops-thing-anyway/<p>Before being introduced to the concept of DevOps, we at Twilio struggled with what to call the role. It requires a different kind of thinking.  To us, DevOps is about automation to improve productivity, reduce risk, and achieve scale.  DevOps engineers should always be striving to deprecate themselves.  It has been more then 2 year since we founded Twilio and we've never had a sysadmin or an ops person.  This has forced us to write better software automation from day one.  However, having someone that can provide leadership on DevOps issues (and fundamentally understands distributed systems/CAP/etc) to the whole team is extremely valuable.<p>Configuration management tools like Chef and Puppet are the beginning but a more complete view of infrastructure and service automation (e.g., http://redeyemon.sourceforge.net/) seems inevitable in the long term.<p>We are looking to hire a Lead DevOps engineer.  Here is the detailed job description if you know someone that might be interested:
http://twilio.jobscore.com/jobs/twilio/devops-engineer/bSHjYWq2Sr37U6eJe4aGWH
======
mmt
Sometimes I wonder if DevOps isn't really just those who would truly prefer to
be developers but don't mind sysadmin work and, for whatever reason, started
out doing the latter.

I, too, cringe a little when I hear the term, perhaps because I've always
viewed myself as a "pure" sysadmin. That is, this is the kind of work which
continues to bring me joy after 2 decades. Application development, on the
other hand, despite an ability to code, does not.

CM tools (current and their predecessors like cfengine) as they tend to be
implemented strike me as a way of avoiding administering the whole _system_ ,
by focusing on a single component (servers) and reducing them to a lowest
common denominator.

It also fits the pattern of trying to solve problems entirely in a custom
software solution, a trait I associate with developers, not sysadmins. This is
the source of my inference of DevOps' true leanings.

The reality is that the system consists of considerably more than fungible
servers[1]. There's the underlying hardware, the peripheral hardware like
disks, database software[2], network hardware and configuration, and service
providers.

These are all traditional sysadmin areas, with _plenty_ of room for
professional growth and overall benefit. So much so, that I have difficulty
imagining why any SA (not a developer in SA's clothing) would wish to do more
dev at the expense of increaaed mastery of the infrastructure.

By way of example, something I wrote on LinkedIn, in response to why I'm not a
fan of puppet:

    
    
      I'm uncomfortable with any system that attempts to maintain state in place
      on a running system, as well as possibly override the native packaging ethos. 
      That the client takes its own local file inventory on a fresh system, rather 
      than trusting the one the package manager has, is wasteful at best. What's 
      worse, to me, is that it creates yet another system to administer. I haven't yet 
      found anything similarly suitable for the Ubuntu/Debian world, but I very 
      much liked Cobbler a couple years ago, as it was primarily a wrapper for 
      what could otherwise be standard, standalone services (kickstart, package 
      repo). Even without such a wrapper, yum/rpm or apt/dpkg can do nearly all 
      the work"
    

[1] Virtualization and "cloud" notwithstanding, as anyone who's experienced an
outage with a hosting provider or an i/o perofrmance issue with a cloud server
can attest.

[2] Which can be like an encapsulated OS itself

------
chuhnk
Yea I think like some the term DevOps makes me cringe as well since people use
it as a job title when it should be more thought of like agile development,
just another term to describe a process of working.

I use chef because I needed a way to consistently deploy software in exactly
the same way every time to multiple servers. A repeatable deployment process
that scales. Why did I pick chef over puppet or cfengine? Because I also
wanted to learn ruby and the entire thing is in ruby, win. Plus we have a ruby
on rails team in the office, if ever I needed a hand, they were there or if
they wanted to write recipes to deploy software thats not an issue either.

DevOps is just a new buzz word. My title is System Administrator. I have a
depth of knowledge in the infrastructure of all our technologies and how they
are interconnected. I know how to deploy, maintain, scale and do it all over
again. My knowledge in programming in second to that but is necessary to
automate repetitive processes. DevOps I guess is being classed as a way to
bring the IT team together, making sys admins and developers aware of each
others "agendas" when it comes to running apps on servers. I as a sys admin
have to understand what each of our applications does throughout its life
cycle, from the browser right down to the database. Because if something goes
wrong, I'll be the first to figure out where that happened and relay this back
to the developer. By understanding core functionality of the code I can make
it clear to the developer, "look you are executing a query on the database
thats scanning millions of rows because its not using an index, its occurring
in this controller in your code". We are seeing this term DevOps be classified
now as more and more startups are emerging, because its essentially alot
easier to execute in a startup environment where everyone is talking to each
other to roll things out, everyone is trying to solve problems together. In
large companies you'll see a massive separation and segregation along with
processes to keep people in their place and slow results.

I hope that all makes sense. DevOps as a process is not new, the
classification of it as this term is.

------
fragmede
Linky - [http://www.jedi.be/blog/2010/02/12/what-is-this-devops-
thing...](http://www.jedi.be/blog/2010/02/12/what-is-this-devops-thing-
anyway/)

Call it what you will, 'DevOps' is just what you'd expect a _good_ sysadmin to
do in a sufficiently large org. That there is on going friction between
developers and sysadmin ("it works on my box" is a valid excuse?) is perhaps a
larger sign of the culture and people involved.

~~~
WestCoastJustin
Whenever I hear that term I cringe a little. I think this was a word invented
by developers (please correct me if I'm wrong) to bring sysadmins into the
fold. It is definitely good to have an understanding of the application layer
but developers also need to understand the systems running their applications.

I work in a large organization (5000+ employees) as a unix/linux sysadmin. You
have to automate deployment, monitoring, configuration management because
there just isn't time to do anything else. We have a group (about 4) out of
30+ that have an overall view of what you might call devops. The rest
specilize in development or sysadmins. We meet once a week and review all
progress on the dev/ops issues.

I like the term OpSec (<http://en.wikipedia.org/wiki/Operations_security>)
someone who looks after Systems/Security ;) Since a good system admin should
also know a great deal about security. Maybe Dev/Op/Sec.. haha, but really ;)

p.s. I found that job posting to be full of PR fluff. It would be nice to see
job posting actually post what their looking for in a distilled form.

~~~
emcooke
Thanks for the pointer to OpSec term. DevOpsSec... I sense a tshirt in the
making?

~~~
WestCoastJustin
I first heard about OpSec from Roland Dobbins with Arbor. Really smart fellow
who has many suggestions on network design / defence.

I just think people need to specialize in their field (+ a general knowledge
of how the overall systems works).

Almost how a carpenter, plumber, and electrician each specialize but probably
have a general idea how everything works. Do you really want a plumber
building the foundation of your house? Just seems a little silly to me. Each
domain is so large that I would rather have deep expert knowledge in a area
than a general handyman who might cut corners or waste time with solutions he
doesn't know about.

------
gaius
40 years ago IBM called these guys "systems programmers". That this is a new
concept to some people is symptomatic of a wider problem in the industry:
everything needs to be shiny and new so every 10 years we reinvent the wheel
with new jargon.

~~~
blueben
DevOps is not just "writing automation tools". If that's all you're getting
out of this, you're missing 90% of what's going on.

~~~
starkfist
What else is it, then?

------
nordgren
There is a bit of the same movement in CCP Games where I work, it's not called
DevOps here, but there are groups forming in both the EVE Online software team
and the Core technology group with an engineering focus on QA, which in
practice translates pretty well to the DevOps style of thinking about the
needs for this type of development and quality work. Here's one of the
advertised positions for example:

<http://www.ccpgames.com/en/jobs/job-details.aspx?jobid=101>

