
Linux Sysadmin/DevOps Interview Questions - metmac
https://github.com/chassing/linux-sysadmin-interview-questions
======
eulji
Generally good questions.

However I would not ask a candidate or more specifically I would not show him
the code for fork bomb. I would ask him what fork bomb is or demonstrate it on
some demo VM and ask him how would he prevent himself..

There's too many technologies already. I would not ask specific questions
about the staff we use but instead a very generic ones.

Like really it's almost impossible to be a specialist these days and you can
pretend your company is full of rockstars and works on the next big thing but
in fact you are just bunch of posers.

Nobody really knows everything to the detail. I really hated when my colleague
asked new people really deep stuff about Puppet and then we have switched to
Ansible and half of the team was clueless and had to learn along the way.
That's what good engineers do. They adapt. They learn the concepts and focus
on delivering a good service. They do not memorize man pages.

When it comes to scripting / programming do not print scripts on paper and
make the candidate to describe you what is it for or find a bug / mistake just
by reviewing the paper. It's insulting. Especially if you do not follow any
conventions or your code is not idiomatic.

Also I would never ask a question "What does the following command / parameter
do". These are not good questions. Ofc they are asked often during interviews
but are just plain stupid.

How would you display last / first 10 lines of a file ? How would you remove
an X string from a line / row of text ?

and so on...

------
quasse
Honestly I think this is a good list of questions to test out a search engine.

As far as interviewing a DevOps guy, maybe? I consider myself a decent DevOps
person out of necessity, but questions like "How to add a new system user
without login permissions?" would kind of annoy me, because that's the exact
kind of thing I can't remember because I automated it years ago at my current
job. I probably add a new user twice a week... from a Fabric script. I
couldn't tell you if it's useradd or adduser because my workflow would be to
google the exact wording of the question if I actually needed to do it
manually for once.

~~~
txutxu
I don't remember the flag, but this could be my answer:

Most linux distributions, come with /bin/false as a valid and registered
shell, for login to the system (but it's a program that returns false and does
not open a shell).

I know that useradd and adduser allow a flag for change this at creation time
(probably --shell /bin/false).

It could be more safe to be explicit since the creation time, than using chsh
(changeshell) to avoid race conditions in security, having a user created with
a shell until chsh is run.

You see, I didn't remember the flag, but I did try. That's the kind of answer
that maybe some interviews are waiting for.

~~~
nailer
/sbin/nologin actually does some cool logging stuff that /bin/false doesn't,
which is why the distros generally use it.

~~~
txutxu
Cool stuff as in display a message that the account cannot login, which leads
to brute-force service, process or other stuff system user names.

Failed logins will be logged either way.

I think still could go with the output of `type -P false`. Unless the project
requests that feature of messaging to the (forbidden) user, its forbidden
logins.

It could work even on embedded devices having only 'false' available.

------
SFJulie
Oh my.

Nothing about knowing if the sysadmin knows how to find information (man,
docs, info, apropos, perldoc) offline and online (advanced bash scripting
guide), which shell to use to be the most portable, and where to go to be up
to date regarding vulnerabilities.

------
derFunk
That's a good pool of questions to choose from. How long do you think would
not take an applicant to fill all of the questions?

My questionnaire for the same purpose (hiring) has 30 similar questions, yet
including some that query soft skills.

E.g.: "you got the following email from a customer, please write down your
answer to her inquiry".

Or: "the CFO, the CEO and the CTO are approaching you with the following
different requests, how do you react to them and in which order would you
process them".

------
jwingy
There's some decent questions here, but also too many "how to" questions that
most people won't bother memorizing since they take 2 seconds to google.

------
KaiserPro
The key thing I found in interviews (from the interviewer side) is that
getting someone to sit down and write the test is not very useful.

Its far better to get them to talk you through it. It allows you to asses
their knowledge (which can be taught) but more importantly it allows you to
see _how_ they work and _if_ they can explain their shortcomings.

I want someone to say "I know that x does y, but what happens when z comes
along I'm not sure" I can then provide a nugget of more information and see if
they are able to grab onto it.

It shows you their ability to communicate, and will bring out more of their
latent personality. You're going to spend more time with your co-worker than
your significant other, so you need to have some idea that you won't hate
them.

Its far more important to have someone who knows when they are out of their
depth and handles it gracefully than someone who seems to have more knowledge
but hacks about wildly when they fall off the deep end.

------
znep
This is a horrible interview question for all but a tiny handful of jobs:

    
    
      Is 300.168.0.123 a valid IPv4 address?
    

...but it is a great late night fun trivia question about "how can adding one
character to that address make it into an address accepted by many tools".

~~~
tyingq

                                                                                                                        Stick a zero in front?  0300 octal == 192
    

Edit: Fixed so it's not such an obvious spoiler.

------
hacoo
"Unpack test.tar.gz without man pages or google"

Ha, guess I'm not the only one :)

~~~
nailer
Remember to 'eXtract Ze Files', twirling your moustache while you say that.

~~~
Rapzid
You can actually drop the Z these days in a lot of cases; tar will detect the
compression algorithm.

~~~
nailer
I occasionally log into older GNU, or non-OSX BSD systems where tar can't
detect the compression automatically. [1]

[1] this isn't true, I just enjoy saying 'extract Ze Files' in a silly accent

------
nailer
> Write a get_prim method in python/perl/bash/pseudo.

What does get_prim do?

~~~
minikomi
Returns prim

------
kinai
Isn't the biggest skill of a sysadmin to solve problems that he has not seen
before? Learning all those answers to the questions wouldn't help you much in
a real life scenario.

Instead of asking "what is/how do you do this?" it is better to give example
scenarios and listen to what someone would do, to see how they approach the
problem.

~~~
Karlozkiller
Being a hobby linux sysadmin myself I'd say you're right. The hard and often
interesting part is to figure out why everything suddenly stopped working.
sometimes I see the same problem several times and try to figure out how to
stop it from happening again, most of the time though I see completely new
problems. But it always helps to know your Linux system and how to hunt down
error messages and log files. Buy then I'm no professional.

------
djent
How can I use this as a self-placement test? I'm in college still, and I have
answers to many of the general, simple Linux, and devops questions. What is
expected to be known for varying levels of employment?
(Intern/junior/normal/senior)

~~~
banashark
I was a sysadmin, a devops engineer, and a build/release engineer. Now a
developer.

These questions are all decent. There's nothing here that would be out of the
question to ask in a junior interview.

Usually for junior interviews, they want to see if you understand the
fundamentals and not advanced details that a senior might need to know.

One example of this is the question "What is swap and what is it used for?"

For an intern, I'd ask if they knew what it was for. For a junior, I'd add
questions about any ways to control swap. For a midlevel/normal, I'd add
scenario questions about configuring swap and the performance effects it can
have. For a senior, I'd add troubleshooting questions.

To be honest though, I've only ever been asked the base question in
interviews. I'm guessing that my answer sounded confident enough that the
interviewer determined that it would be more worthwhile to explore areas I was
not so strong in to see how I problem solve/deal with foreign scenarios.

Most decent interviews understand that people have different levels of
understanding for different topics which might fall under the umbrella of
"DevOps". A candidate could be a docker wizard, knowing all of the best
practices of versioning, CI, and deployment, but could not know much about
VLANs because "they never had to use it".

Keep in mind that most companies use a certain tech stack and are usually
looking for people familiar with it. Interviewers always seem to be impressed
when I told them I "checked it out over the weekend" and was able to answer
questions decently about their stack. Skimming the documentation, setting up a
vm to play around with, and finding the most popular blog posts about current
best practices for a technology can go a long way towards closing the gap
between your current knowledge and the ideal candidate's knowledge.

Hope that helps.

------
dmourati
Here are a couple I use:

\- How many hosts are there in a /23 network?

\- Estimate ping time from your workstation to your local gateway?
www.google.com? Server on the east coast from the west coast? Server in Europe
from the west coast?

\- What law(s) of physics control(s) the round trip times?

~~~
tyingq
"How many hosts are there in a /23 network?"

Having interviewed a lot of sysadmins, I see this trip people up, mostly
because the slash/bits network notation isn't taught/documented well. The
"memorize a table" approach seems to be common.

Most of them, with a bit of prompting, get that an ipv4 is 32 bits, and that a
/23 means (32-23==9) 9 bits of host addresses. Most also can figure out what
111111111 is. Some remember to account for the network and broadcast address
not being available for hosts.

------
bsaul
The "simple questions" really reads to me like a proof of the superiority of
the gui over terminal. With "how to delete a directory and its content"
winning the award.

It could almost read like a wishlist for linux improvments.

~~~
Someone
Change it to "how to delete the contents of a directory, but keep the
directory itself" and it becomes less simple.

But even this can lead to an interesting discussion. From a CLI, there's quite
a few things to consider. Do you need sudo? If not, you shouldn't use it, IMO.
Is the directory name dangerous? If so, would _rm -rf --_ help? Does rm
support -- as an argument at all? I wouldn't know. And would it always prevent
problems, even with truly weird directory names (with spaces, asterisks,
slashes, $HOME, unbalanced quotes)? Of course you shell should tab-complete
names and correctly escape them, but should you check your shell (brand and
version) first if the name is weird, and know what versions have a known
problem?

------
wyaeld
Does anyone know where to find a similar list for Microsoft Sysadmins?

~~~
autotune
"How do you enable DHCP or $feature on Windows Server 2008?"

"Click the "Add Role Wizard" and then click next until you're done."

~~~
lstamour
Q: "Ah but how would you enable it in Windows Server 2012/2016 Core editions?"
/ A: "PowerShell and Google, not necessarily in that order"

(Of course, Bash and Google would be the equivalent in *nix-land, so, maybe
not that great an answer...)

Taking it more seriously, well, I'm not a sysadmin, I'm more of a developer
(usually on Mac) with a passing interest in Windows servers. But here's my
list: at least one question could relate to knowing any advanced Windows terms
they might come across in a sysadmin context, like VHDs, different types of
installers/drivers, differences between client, server, core, nano editions of
Windows. Bonus points if they know of Git, Nuget or Chocolatey? Oh, and
they're good in my books if they know about Microsoft Virtual Academy and
Channel 9, especially the early Scoble days ;-) I do like the direction
Microsoft's been going lately...

~~~
dsp1234
_Q: "Ah but how would you enable it in Windows Server 2012/2016 Core
editions?"_

A: Most likely, add it to the Server Manager in the RSAT machine, then use the
add roles wizard and click next till done.

------
ex_amazon_se
Ex Amazon syseng here. Please be advised that the questions are really basic,
even the "hard" ones and won't get you very far in an interview in various
companies.

------
lazyant
Thanks for the effort.

These type of questions are OK for a filter phone interview or entry-level
Linux job imho as in they rely on "what is" questions (which can be memorized
or replied without much experience) rather than "how would you" questions (the
"demo time" at the end, that's the good part). You want to ask scenario and
troubleshooting questions rather than trivia for jobs other than basic ones.
(You can certainly ask more advanced non-scenario -still relatable or
practical- questions about things like system calls, interrupts, CPU
scheduling etc, "system internals" stuff that I don't see at all or barely in
these questions. A Linux questionnaire without mentioning strace or perf hurts
my heart.)

I can basically nit-pick, find a fault or find a better way of asking in
pretty much every question.

The Expert section can be dropped (only one question).

Mysql section can be dropped, maybe replace with basic SQL section.

DevOps section can be dropped, we can see here how definition of DevOps is
still fluid, just add Linux/Networking/General questions to their respective
sections (also "GIT", really?).

One reference link at the bottom doesn't work.

\----

Examples of issues with questions and how to improve them:

\- What commands do you know that can be used to check DNS records?: instead
ask how to troubleshoot DNS issues, like how to check if a name server has an
entry. How does local DNS work? (as in what's the order in which files/servers
are checked to resolve a name, where is that stored) how does DNS in general
works? (as in you ask for a record to a name server and it doesn't have it,
now what). How much TTL would you set in a record and why. Why does DNS run on
UDP and TCP?

\- "tac", (really? and no "grep" "sort", "uniq" or "sum"?), rather ask how to
print a field from a command like 'df' or 'free' or how to add column values
from a text file (basically if they can slice horizontally with grep and
vertically with awk / cut and then count occurrences or sum numbers.

\- What does & disown after a command do?: rather ask when you would use
'disown', or better, inversely how to de-attach a running command from
terminal

\- What is Virtual Memory? : not sure what you want them to answer here, maybe
rather ask about a server with most of its total memory used, what does Linux
cache (there's buffer and inode/dir caching besides files but only two memory
caching areas), how to find that, when does the OS write stuff in RAM to disk
(what are dirty pages), how do you change that frequency, when is better to do
it more/less frequently or what's the tradeoff etc

\- What is swap and what is it used for?: this is more advanced but you can
ask if a server can have no swap, is it possible for a server with plenty of
free RAM to page (write to swap)? can you reduce/increment tendency to swap?
why would you do that, how much swap partition would you set up (OK these are
a bit of tricky questions, more to gauge experience)

\- How to search for the string "my konfi is the best" in files of a directory
recursively? : I actually like this one

\- How to connect to a remote server or what is SSH? : you can split in "list
all the possible ways you can think of so you can communicate (or send
message/file) between two servers on a network" and a question about how SSH
works from Linux point of view (not protocol), sshd settings (idea, not
memorized params), how to set up passwordless (key-based) SSH, hardening tips,
how to troubleshoot if you can't ssh to a server (no firewall, you can
telnet/whatever), what does it mean when you get a message "key has changed
etc", what's a private and public key...

\- What is the sticky bit? : this depends on OS, I guess explaining different
behaviours is good.

\- What does the immutable bit do to a file?: this is not part of linux VFS
but of particular filesystem implementations, btw no questions about what's a
filesystem, what's a journaled one, difference between ext4 and xfs for ex.
Also what about LVM?

\- What is the difference between hardlinks and symlinks?: while a good
question I'd rather ask when to use one or the other, also just to confuse,
ask how to "link/map" a directory in a partition that is running out of space
to a directory in a new partition with plenty of space, so that it works
transparently.

\- What is MAJOR and MINOR numbers of special files?: I guess device files,
rather or besides ask difference between block and character devices, about
pseudo dirs in file system: /dev /proc /sys , how to find hardware or process
info (everything is pretty much in /proc or /sys , also dmesg )

\- Describe a scenario when deleting a file, but 'df' not showing the space
being freed. : this is good but it can be better built into a scenario:
there's an "no space left on device" error, how to find biggest files/dirs.
You find it's a huge web server access.log file, now you delete it and disk
space doesn't seem to shrink, why (the df vs du), what can you do?, can you
close the open file?

\- Instead of "how does ps/top work?" ask something specific or scenario
based, like how do you find if there's a disk I/O problem by looking at 'top'
(there's at least two possible indicators). Zero questions about I/O, or for
the matter, for server utilization metrics, which ones are they, how to
measure each, how to find if they reached saturation, what happens in that
case for each, how to look for errors in each. If a server is using 80% CPU is
that good or bad?

OK I got tired here.

If you have read until this dear reader, would you be interested in my trying
to create a Docker container or Amazon image with a Linux box and have these
types of scenarios set up?

~~~
effie
> _Describe a scenario when deleting a file, but 'df' not showing the space
> being freed._

I came across this, seems to occur when disk is almost full. Is there any
probable reason why this happens?

~~~
lazyant
if the deleted file is open 'df' doesn't show cleared space, 'du' does

------
metmac
Any additional questions anyone can think of?

~~~
dozzie
One ultimate question: "describe a project of yours, current or past, personal
or or job-related, that you are most proud of". You don't need to ask a
professional anything else if he's anywhere near worth his salt; he'll
volunteer all the information you could judge him by.

Having that said, most of the questions from "general questions" part come
either from assuming that the candidate is dumb and possibly doesn't
understand how computers work or from some lousy layman psychology.

Many of the questions further down are incorrectly assigned. "What is an A
record?" is not a Linux-related question, and git is hardly a "devops tool".

Very poor list, in my opinion. In an interview you need to assess whether the
candidate can do the job. Since the only heuristics available to the
interviewer is how the candidate did in the past, the interviewer needs to
allow candidate to show what he has done, how, and why.

And just for fun:

> A careless sysadmin executes the following command: chmod 444 /bin/chmod \-
> what do you do to fix this?

My favourite way of fixing it is by running /bin/chmod. It blows minds of most
smart-asses who ask this question.

~~~
Ysx
> > A careless sysadmin executes the following command: chmod 444 /bin/chmod -
> what do you do to fix this?

> My favourite way of fixing it is by running /bin/chmod. It blows minds of
> most smart-asses who ask this question.

\----

Not sure what I'm missing here?

    
    
        $ docker run -it ubuntu
        # chmod 444 /bin/chmod
        # /bin/chmod
        bash: /bin/chmod: Permission denied

~~~
dozzie
ELF interpreter/loader.

    
    
      objdump -s -j .interp /bin/chmod
      /lib64/ld-linux-x86-64.so.2 /bin/chmod

