
Ask HN: Must have (Linux) sysadmin skills? - zabana
Hey Everyone !<p>What would you list as the most essential skills one must be able to demonstrate to be hireable ? (Coming from a developer). What are the most common things a sysadmin does daily ?<p>Cheers
======
nelsonmarcos
* understand the basic directory structure (/, /boot/, /var, /usr, /opt, /etc) * file management (commands: tail, head, cat, awk, sed) * process management (commands: ps, top, kill) * package management (yum, dpkg) * one configuration manager (puppet, chef, ansible) * basics of tcp/ip _network management (ifconfig /ip, netstata/ss, ping, trace) _ addiontal commands if you want to be more than a junior (strace, lsof, iostat, vmstat)

I'd like to point out that some sysadmins are focused on linux internals while
others focus on application in production. So, of course, the list may vary
according to the position you're looking to be hired.

~~~
caw
Good list, I'd also say permissions. The number of problems I fix by
understanding users/groups and assigning the appropriate permission (not
`chmod 777 *`) is huge. It shows up in a variety of "Cannot access file" type
messages or "File does not exist" application errors and people sometimes
freak out because the file is there.

~~~
himlion
To be fair 'chmod 777 *' fixes those issues as well, just at a big cost to
security.

~~~
Jugurtha
I don't know if "fix" is the right word. Chopping your right arm off "fixes"
the itch, although you're literally giving your right arm.

------
citrin_ru
There is a very big overlap in what good developer and sysadmin should know
about Linux. There are a lot of tasks, which in one teams performed by
developer and in other — by sysadmin. Random topics which IMHO more specific
for sysadmin work:

1\. Troubleshooting and performance analysis. There is excellent site about
performance:
[http://www.brendangregg.com/linuxperf.html](http://www.brendangregg.com/linuxperf.html)
I don't know a good resource on troubleshooting, but some tools are the same.

2\. Problem can be anywhere including network so the next topic is networks.
There is excellent book TCP/IP Illustrated — old, but still relevant.

3\. Sysadmin often spends a lot of time in a shell so it is good to know it
very well (and common shell commands too).

------
citrin_ru
> What are the most common things a sysadmin does daily

\- reacting to monitoring alerts and critical messages in logs. First you need
to understand what given alert/message mean, and then goes troubleshooting if
root cause is not obvious \- improving monitoring system
settings/thresholds/metrics (if there no separate teem for this) \- deployment
(but it often performed by developers) \- upgrading OS and 3rd party software
(if 3rd party software not deployed alongside own code) \- performance tuning
\- learning software used in production (both: 3rd and written in the company)
\- writing numerous config files and ensuring that all settings adequate to
given environment/system \- if system is not entirely in cloud - replacing
broken hardware and parts (HDD), usually using "remote hands" in datacenter.
\- managing ACLs / credentials for different systems, e.t.c.

------
vram22
Well this one is not really a skill, but quite a useful guideline or
precaution to follow - and consistently:

Make your actions reversible (as far as possible). Had learned it in the
field, later also saw it mentioned in a sysadmin book.

I can't count the number of times I've seen colleagues just blindly charge
ahead and do, for example, non-trivial surgery (a.k.a. edits) on important
system config files (in a haphazard, lets-see-if-this-works manner, without so
much as making a backup copy of the file), which sometimes resulted in
screwing up the system further, sometimes irrevocably (because, you guessed
it, no backups - or no current ones, anyway). And yes, on production systems.
Got a few stories about that, may mention them some other time.

------
jlgaddis
The skills listed on Red Hat's RHCSA [0] page might be a good start.

[0]:
[https://www.redhat.com/en/services/certification/rhcsa](https://www.redhat.com/en/services/certification/rhcsa)

------
codegeek
I love linux even though I am not an expert devops or sysadmin. I get around
working with servers for my product and thats about it. I stackoverflow shell
scripts whenever needed. Here is my 'must know' list for linux:

\- ssh: you need to know what that means and how to use it

\- Difference between ftp and sftp

\- cd, ls, pwd (directory stuff)

\- scp and rsync (move files from server to server)

\- find, cat, grep, sed, awk, head, tail (manipulations and discovery)

\- chown, chmod (permissions)

\- cp, mv, rm, rmdir, mkdir, touch

\- Don't do rm -rf unless you know what you are doing

\- Setup aliases on your shell. e.g. .profile file for bash etc.

\- top, ps (check processes)

\- sudo and su (switching users or running commands as root)

\- Know various OS differences like CentOS vs Ubunut.

\- Know where to find logs for web servers like Apache or nginx (e.g.
/var/log/nginx in Ubuntu).

\- Star/Stop services as needed

------
atmosx
I want to do a blog post series on this topic.

Systems administration is a huge topic with so many interconnected parts and
such vast variety of tools. Even breaking it down to segments can fall apart
easily as nearly all topics overlap from user management, to filesystems, etc.

My only advice is this: Get a copy of "UNIX and Linux System Administration
Handbook" and read the topics that you work on. It's a reference not something
that you can read from cover to cover, but it's I own the 4th edition and it's
depth is amazing. On 18 Aug there is the 5th edition coming out. Grab a copy.

My only issue is that I'd love to have *BSD included along Linux, Solaris and
AIX (which is hard to find these days...).

~~~
Tsiklon
Is that the Evi Nemeth et al. book? I didn't think they were doing another
edition after she was lost at sea? I'll happily buy an updated version.

~~~
atmosx
Yes! It's an amazing book for sysadmins. I bought my copy (4th edition) a few
months ago. I'm still reading various bits and pieces.

The new book I'm sure will have a new section on virtualisation, automation
(ansible, etc.), docker, docker orchestration tools, service discovery, etc.

I'll buy a copy on Christmas.

------
autotune
While not a fan of most certs, I would highly recommend the RHCSA to get an
understanding of common sysadmin tasks as it is a hands-on exam and fairly
relevant to the real world, and RHCE as well. In addition you should find some
random service in your language of choice on GitHub that requires a database
and try to implement in a cloud provider taking into account high
availability, deployment types (Blue/Green vs Rolling, for example),
durability, backups, security, logging, and monitoring by rolling out the
services yourself or through managed services at a given cloud provider using
a CM like Ansible and/or Terraform.

------
lumberjack
Not sure why nobody mentioned it yet but iptables.

