
Not only coders are hard to recruit - alexgotoi
https://blog.sysadminsarena.com/not-only-coders-are-hard-to-recruit-54c74ba9f742#.wo8aiem2i
======
Peroni
The premise that technical people with in-demand skills are hard to recruit
has very little to do with the screening process. The screening process is not
a difficult problem to solve.

The real difficulty in recruiting in this space is _finding and engaging_
these people, not testing them.

If this is a Hacker Rank for sysadmins, then great. I'm sure you'll have
plenty of users but we're still a long way away from solving the bigger
problem.

~~~
zzzcpan
> The real difficulty in recruiting in this space is finding and engaging
> these people, not testing them.

Tons of people in this space hang out on various places online, where you can
simply advertise or do some PR and many do, is it really that difficult?

~~~
Peroni
When was the last time you engaged with a company with a view to working with
them because you saw an ad they happened to post in the right fora?

~~~
dijit
I've hired 3 people from an IRC community I run, they all turned out to be
exceptional (sysadmins and developers).

I've never been hired from IRC though.

~~~
Peroni
Don't get me wrong, it's entirely possible if you do it right. I've hired
quite a lot of folk as a result of interacting with them on HN alone. My point
is that it's a significantly more difficult problem than screening those
people.

------
otoburb
The Cisco Certified Internetwork Engineer (CCIE) certification used to be
_the_ gold standard amongst network engineers 10-20 years ago.

Becoming a CCIE required a passing a notoriously difficult hands-on 8 hour in-
person lab examination[1]. SysAdminsArena reminds me of a remote version of
the CCIE Lab Exam for sysadmins.

Fun fact: the CCIE program was established in 1992 and was originally going to
be called "Cisco Top Gun"[2], but they had to choose a more serious name.

[1]
[https://learningnetwork.cisco.com/community/certifications/c...](https://learningnetwork.cisco.com/community/certifications/ccie_routing_switching/lab_exam_v5)

[2] [http://www.bradreese.com/blog/how-the-cisco-ccie-program-
was...](http://www.bradreese.com/blog/how-the-cisco-ccie-program-was-born.htm)

~~~
alexgotoi
20 years later -> SysAdmins Top Gun :))

------
pmoriarty
There's been a lot of criticism of hackerrank.[1][2]

Are you doing anything to make sure your service gets more respect?

[1] -
[https://news.ycombinator.com/item?id=12825953](https://news.ycombinator.com/item?id=12825953)

[2] -
[https://news.ycombinator.com/item?id=12667174&source=techsto...](https://news.ycombinator.com/item?id=12667174&source=techstories.org)

~~~
alexgotoi
Glad you asked that. As i saw in the treads mentioned by you, the common
problem is that the tests are more about theory, and less about practice.
Which is true. Sometimes, in interviews, you are asked about algorithms that
you studied in college and never used before. It's very difficult to score
high in this tests.

We propose to eliminate this gap by having real job, day to day, scenarios
like: install apache, mysql, php, clone site, install database etc.

These are recurring task in the daily activities of a sysadmin.

What do you say?

~~~
bovermyer
Those are pretty baseline things that get automated away pretty quickly. More
useful to me (as both a practicing devops type and a manager in that capacity)
would be tests gauging ability to write bash scripts to do various things, or
simulated problems that need to be diagnosed/fixed using command line tools
(e.g., strace, netstat, etc.).

~~~
alexgotoi
Indeed, those are pretty basic tasks. We will have 3 levels of difficulty and
the examples mentioned are in the easiest category. In a early phase of a
recruitment process, an experienced sysadmin would solve in less than 3
minutes and this is helping screening process when you have a huge number of
applicants. But for latest stages of recruitment process, we will have more
advanced scenarios for the tests.

Definitely your suggestions are great and we will discuss having tests in this
area.

Cheers!

~~~
marcosdumay
> an experienced sysadmin would solve in less than 3 minutes

Why do you think so? And, I'm hopping you don't rely on timing on this level.

An experienced sysadmin would have little experience with basic tasks, since
he solves them around once or twice per job. And you never know the
environments they tried it before, that may be completely different from what
you are presenting them right now.

I'd say "installing X" is a test for encyclopedic knowledge, and a very
unreliable test for expertize. That's the kind of test that gave bad names to
IT certification at the early 2000's.

~~~
alexgotoi
What scenarios would you suggest to have?

~~~
marcosdumay
No idea, but the certificates that focused on those things became enduring
jokes.

------
davidgerard
So this is to have a test VM they have to fix? Excellent! I heartily approve
of this method for finding a sysadmin.

This is a thread where I talk about how we did this:
[https://news.ycombinator.com/item?id=8332104](https://news.ycombinator.com/item?id=8332104)

We used to use it a lot - a trivial Java webapp running in a slightly
misconfigured Tomcat with a slightly misconfigured Apache in front on a
slightly misconfigured Unix box - though it's fallen by the wayside because we
never quite get around to the faff of setting up a slightly broken test system
for them. So having that as a service would be pretty good.

If this is what you're doing: I suggest not just keystrokes, but that they
keep a log of what they're doing and thinking. Because we never expect them to
fix _everything_ wrong with the box, but we do want to know how they approach
it.

~~~
alexgotoi
Hi David, thanks for your thoughts. Very useful the thread provided. And YES,
this is what we are doing.

Keystrokes is just one of the metrics we are about to collect and analyze. We
are also thinking about number of commands wrote, how many were correct, how
many wrong, also a log of the entire test. In future phases, we may take into
consideration pair working or capturing a movie of the test.

What other ideas of relevant metrics do you have?

~~~
lisivka
So, if I will follow best practices and will prepare backup, then restore
backup locally (to check that backup is OK), then will play with backup until
problem will be solved (to not harm target server), then prepare runbook, and
then will just execute it at target server, I will pass your test? Just two
commands.

~~~
johnny_snq
Yes. you should pass the test. This is one of the scenarios we are thinking.
you get the backup location do the restore. done you passed.

~~~
lisivka
I'm talking about copy-pasting of solution.

~~~
johnny_snq
Ok. Maybe I misunderstood you. If you are thinking of having the tests leak
and people just finding solutions on Google I'm just thinking of dynamically
generating the problem. How about 3 characters missing from the config file
for apache at random. You have to understand what is required and then try to
fix it. Does this solve your case?

~~~
lisivka
I had similar idea to your and found no solution to problem of copy-pasting.
Good problem sets are hard to create but their solutions are easy to copy.
Moreover, this is the good practice to try to solve solution locally, then
copy-paste solution to target server.

I hope, you will find a solution for that.

------
tristor
Speaking of sysadmins, a note for yours. You are currently serving
blog.sysadminsarena.com over HTTPS with an HTTP to HTTPS redirect, but are not
doing the same for your website. In fact, you don't have an HTTPS endpoint for
your website at all. Going to your website by modifying the URL from the blog
entry (because clicking your logo takes you to the front page of the blog, not
the website) results in a `ERR_CONNECTION_REFUSED`.

Just an FYI :)

~~~
johnny_snq
I'll fix it tonight after work with LetsEncrypt most probably. The blog is
hosted on medium.com so they provided the ssl termination. But there should be
no excuse for not having https on a website in 2016.

~~~
xiphias
Now this looks like a real interview task, not installing apache (which I as a
non-sysadmon could do easily as well)

------
devonkim
Interesting problem you're addressing but I've been working on this problem
from a completely different set of metrics. Fundamentally, I really don't care
how a problem is fixed as long as it doesn't have bad repercussions. If a web
server is "slow" users don't care why - giving freedom to admins to approach
higher level problems while still measuring them is something I'm confident
that can be done while I take a number of issues with assessing anything based
upon commands entered even in realtime (I'm mostly concerned from a security /
bad or illegal practices in prod perspective, to be frank).

The approach I'm working on is easier to do technically but requires some more
costs.

With that said, glad to see I'm not the only one that's been frustrated with
hiring devops / sysadmin / SRE roles. Google's even commented how hard hiring
for SREs is for them and I think we have a huge gap between people stuck in
the 90s approaching systems and those that modernized.

~~~
alexgotoi
We are also glad that there are other people that are working on this issue.
Good luck!

------
freddref
Isn't everyone hard to recruit? The difference in perception of coders might
be that the evidence for a bad recruit is obvious in the code they produce.
There could be many bad recruits in all other areas but are less obvious.

I generally like the idea of clearer quantification of a persons output but it
should be done very delicately. As in, if you are going to measure everything
someone produces, maybe that data should only be available to that person and
it's up to them how they share it.

The uncertainty principle likely applies to people too. Hard to measure people
without influencing them. Trick is to do it in a positive way. Managers
measuring people for the purpose of firing them would be perceived, and have a
negative effect on morale, than measuring people to identify opportunities for
education, training or skills development.

~~~
alexgotoi
Great idea to offer the data to the one that is producing it and he/she
decides what to share. Thanks

------
alexgotoi
Feel free to ask anything or provide feedback. Thanks

~~~
globalgobble
I am not sure you can do this I believe sysadmin/sre/devops person wears too
many hats. What are you going to test ?

~~~
johnny_snq
We are going to give real life scenarios based on the various roles you are
testing with.(basicly a ssh login to the test machine) And we plan to asses
the candidates way of solving the problems based on what commands and what
process it uses to identify and solve the problem.

~~~
globalgobble
Please allow me my honesty but that's a stupid way of thinking about
candidates and the roles. Or I do not get it. So you are basically going to
judge the candidate based on what criteria ? If they use vi instead of cat to
append to a file ? So decide what is the best way of solving the problem ? Or
you give +1pt for using tail instead of opening the whole file in VI ? Are you
aware of the fact that there are many types of systems and products that
sysadmins use ? They do not behave the same. It makes no sense to use "top" on
Solaris even the number for "uptime" reported by Solaris are different. What
about Windows sysadmins ? What about stuff like Veritas ? What about docker ?

~~~
languagewars
When I used to work in OS support my actions were optimized for the
assumptions of a default minimal install and being able to describe actions to
a person of questionable knowledge over the phone and interpret what they
would most likely try to read back at me. The certifications were along those
lines too.. While the exact nature of the certifications was ridiculous and
would be even more ridiculous in terms of judging a general user's skills,
dropping an administrator on minimal installs with broken ttys, etc makes a
lot of sense. If you can't recover a system when vi is not an option then you
have a problem, if you spend a lot of time recovering a good term when you
could have used tail, then you are also not ideal.

The only suggestion I would make is that a good tool is present and working
then I don't expect a penalty. I expect tools to not be present accordance
with standard minimalist setups and to pay extreme penalties if I need to call
a package manager or move to a less minimalist install to complete the task.

~~~
globalgobble
I strongly disagree. We should not test edge cases and specific scenarios. We
are trying to test the candidate and his abilities in daily tasks. If I say I
use vi instead of "commands" I will probably solve most of the "text" editing
tasks faster than an admin that only knows how to get around using commands.

I can come up with tons of scenarios when where you do not really know the
internals you can get stuck easily. Like if I do chmod -x on the chmod binary.
Does it prove anything if the candidate is not able to solve such task ?

~~~
CmdrSprinkles
While I mostly agree with you, I can speak from experience with how we handled
the first point:

We were interviewing a new admin for the sysadmin/maintenance team and her
answers to "how would you solve this problem?" were "Open that file and check
that these variables are set and, if not, set them". One of the more jerkish
people there went down the rabbit hole of "How are you going to edit that
file? What if you don't have vim? Okay, what if you don't have emacs? Nano is
gone too" leading to the response that made us hire her: "Well, in that case
the system is probably completely hosed and we need to stop configuring things
and focus on recovery. But I think you are looking for me to tell you a
sequence of commands that involve cat. If you give me five minutes I can check
stack overflow and get back to you with those"

Which I personally think should be the response for stuff like that.
Understand there is an alternative and a way to resolve the corner cases and
know how to get that info if needed.

~~~
globalgobble
Brilliant answer :-). I totally agree with you. But this is the problem..you
cannot implement such a thing to a tool like sysadminarena.

Actually I believe being a sysops/devops or whatever you want to call it is
one of the "most" challenging jobs in the IT industry. You get zillions of
technologies you have to administer and maintain but it's impossible to master
all of them. What makes a good admin is the ability to learn,adapt,ask for a
help and knowing where to look for an answer. It's easy to look up the
parameters if you know the fundamentals.

Interviewing for me as a devops person is one of the most frustrating
experiences. You are often asked questions that are specific to the
environment of a company you are interviewing with. It's unbelievable how many
times I've been asked about parameters of tar, du etc. It's totally stupid.
And imagine interviewing with a company that uses only a specific subset of
technologies and they heavily emphasize on that.

~~~
CmdrSprinkles
I would argue that basically any tech job (and probably other fields) is about
knowing how to solve a problem (I really do wonder what the medical doctor
equivalent of stack overflow is).

And just targeting a limited subset is kind of reasonable. Because, odds are,
you will be. If you are running infrastructure for a web server you probably
will have decided on a few solutions and have time to experiment with new ones
on a new platform. Scientific computing? Same deal. It is the same logic by
which coders should learn a range of languages and tools but will probably use
a very small subset at any given job.

And I think that the semi-automated tools COULD detect stuff like that. One
point for each solution presented and maybe something akin to those god awful
rubrics for the more complex ones. So going with the tar approach (after
twenty years, I know xzvf and czvf and can use a test file to figure out the
order):

1 point for a correct answer 0.5 points for typing "man tar" or "<google> tar"
And then a subset of the remaining 0.5 for each letter and the order of
operations. So something like tar xvf foo.tar.gz ./foo would get you most of
the credit.

The big problem with stuff like this is that it should ONLY be used for an
initial filter, but it will inevitably be used as "Only the top N people
matter".

------
hopster
When will be the beta version available to test?

~~~
alexgotoi
We proposed to have it available on mid December. Subscribe on
www.sysadminsarena.com and we will keep you up to date.

------
kesor
When you have a very low supply of high quality people, and at the same time a
huge demand for those same people, it is a major problem. You claim to solve
this problem, by making it even harder? How does that help?

~~~
alexgotoi
Kesor, we do not claim to solve this problem. We propose a product to help
assessing easier the technical skills, not to improve the sourcing. In the
recruitment process flow, our product comes after there is a short list of
candidates (screened based on their resume) and want to perform a more deep
screening.

We know there is the gap mentioned by you. Hopefully, in the future, we will
be able to provide help in this area too.

