
Devops/Sysadmin Cheatsheet - wlll
http://rubytune.com/cheat
======
js2
Some random thoughts:

\- Instead of 'while true;' you can use the shorter 'while :;'. ':' is a null
command.

\- On OS X, dtruss is kinda, sorta like strace (it's a wrapper around dtrace,
and the dtruss name comes from Solaris).

\- For basic host to IP resolution, I prefer ping as it calls gethostbyname(),
like most other programs will do. host/dig are suitable for querying/testing
DNS independent of how the local box is configured as they call the resolver
library directly. For example, host bypasses nsswitch.conf.

(Separately, OS X's stub resolver has some neat tricks. e.g., using
/etc/resolver/<domain_name> to route DNS requests for a particular domain to a
specific name server.)

\- I had never heard of <http://michael.toren.net/code/tcptraceroute/> before.
I don't think I've ever encountered a situation where the issue was outbound
ICMP/UDP packets being blocked, but rather it's the return ICMP Time Exceeded
packets.

\- 'find . -size +100M' is shorter than 'find ./ -size +100000000c -print'.

\- ctime is NOT when a file was created, though this is a common
misconception. Unix does not record when a file is created. Rather, ctime is
the last time the file's status (i.e. inode information aka metadata) was
changed. This differs from mtime which records the last time the file's data
was changed. ctime is a superset of mtime. atime is the last time a file was
accessed, unless the filesystem is mounted with noatime (common for NFS file
ssystems). atime/mtime can be set to arbitrary values (permissions allowing)
via the utime() system call.

\- Modern find supports -delete instead of '-exec rm {} \;'. In any case, my
muscle memory still defaults to '-print0 | xargs -0 rm'.

\- 'dd if=/dev/zero of=file.txt count=1024 bs=102' seems like an odd way to do
that. I think that 'bs=1024 count=102' might be more efficient depending upon
buffering?

~~~
samuel
Regarding ctime, some Unix versions(or I should say filesystems) really store
the creation time(birthtime) of the inode. FreeBSD's find has the -Bmin switch
for searching for true creation time.

~~~
UnoriginalGuy
Can anyone link something about this? We have two contradictory opinions and
I'm not sure who to believe. The Wikipedia article on ctime isn't at all
helpful either:

<http://en.wikipedia.org/wiki/Ctime>

~~~
samuel
I really wasn't clear about what I wanted to say, sorry. birthtime is
different from ctime on those systems. Ctime is last change on the inode as
parent said.

I just wanted to note that some Unix systems really store the creation time.
See <http://www.daemon-systems.org/man/fstat.2.html> for an example.

------
falcolas
Great list. A few more (related or building on those there)

Quick mysql "why is my database under so much load suddenly" that doesn't rely
on the slow query log:

    
    
        pt-query-digest --processlist h=<host> --interval=0.01 --print > /tmp/queries.out
        pt-query-digest /tmp/queries.out
    

Tracking down what files exactly are filling up the disk. Go to /, and follow
the big numbers:

    
    
        du -smc *
    

Combine strace (for the file handle no), lsof (for the file handle to ip/port)
and netstat (to find the process on the remote host) to track RPC calls that
are hung.

For 'ps auxww', throw on an f to get a visual representation of child
processes.

And finally, ssh tunnels (which can be chained together) to get past bastion
servers easily:

    
    
        Host <alias>
            HostName <ip>
            ProxyCommand ssh -q <bastion host, can be another alias in .ssh/config> "nc -w 3600 %h %p"

~~~
AaronBBrown
I prefer using tcpdump instead of processlist when dealing with pt-query-
digest. You get everything instead of just what happens to show up during the
interval specified. This is, of course, assuming that you don't allow
localhost socket connections.

~~~
falcolas
Very true - extremely fast queries can slip between the processlist checks.
The problem I've encountered with tcpdump is that you can cause packet loss by
running it. It's not common, but it happens enough that I've become gun shy
about using it on production systems.

The pt-query-digest function will cause additional load on the DB, but it
won't interrupt communications.

~~~
X-Istence
What OS will drop packets while tcpdump is running?

~~~
falcolas
Linux. :)

If tcpdump can't keep up with the incoming traffic, the kernel will drop the
packets in its buffer (or rather overwrite them with new packets).

Throw in TCP's flow and congestion control protocols, and dropped packets can
have disastrous effects on your database.

Google has many references you might find useful on this subject.

------
adamgravitis
That's not devops - that's system administration. Not complaining; it's still
a well-presented list.

~~~
sudara
Thanks! You are completely right. Just basic sysadmin stuff. It's titled that
way simply a) I thought I'd see if I could hang with the term and b) the
"source" of these commands (including myself) are from developers who
specialize in rails ops work (so we do write app code, but specialize on the
server side)

~~~
anon987
Christ, just admit you did it so you could be associated with the term when
someone Googles your name.

~~~
sudara
YES, this is all part of my master plan to be THE KING OF DEVOPS!!11one.

------
reinhardt
Meta question: where do you actually learn this stuff? I am a developer with
an intermediate desktop level command line knowledge learnt mostly on the job
but would appreciate a crash course on system-level command line, diagnostics,
devops, etc.

~~~
eLobato
In my experience, you pretty much learn this on the fly. You don't know how to
do X, you google it, if you do X often enough, you'll eventually learn. After
a while you end up mastering most of the unix suite (ls, grep, awk, du, etc..)
so instead of googling stuff you write it.

Unlike fundamentals (think books like SICP, Introduction to Algorithms, K&R,
the Dragon book, et al), this is just a collection of useful commands that do
not bring barely any collateral learning aside from learning the 'UNIX' way.

If you really want to put up the effort to learn this, and you don't have any
projects or anything that requires this knowledge, I've noticed lots of tech
offices have a copy of UNIX in a nutshell always have a copy around. I've
checked it out myself and it's pretty useful (I already 'know'), not sure how
good is it for learning this stuff from the ground up.

~~~
sudara
This is a great reply. "On the fly" is definitely my experience too. Needing
to do something. Doing things slowly, inefficiently, with googling until you
find or get told the more efficient way of doing things.

Also, talking with/demanding explanations/debating with geekier than thou
friends has been really helpful with the bigger concepts.

~~~
exDM69
Learning by doing is probably the way most people learn their unixcraft. It
also helps if you have a friend or a team member who is a lot more experienced
than you, a go-to guy with infinite patience to your stupid questions.

I had the privilege of working with a badass russian unix hacker for some
time, he taught me how to do black magic with find, grep, sort and uniq.

------
njharman
[Edit oh gads I see now the commands are in text boxes and the full command is
not visible. It is "ps aux | head -1 && ps aux | sort -k 4 -nr | head" which
maybe correct is even more ridiculous.]

> ps aux | head -1 && ps aux | sort

Is a wasteful construct and doesn't even do what is claimed "List the top 10
memory hogs". Depending on what you consider "memory" something like this is
shorter and correct.

ps aux --sort=-resident|head -11

Other errors: They list same command "du -hs" for twice. I believe they meant
"df -h" for "overview of all disks". Although, it's correctly overview of
mounted file systems.

~~~
sudara
Thanks! I've updated that with your version, much MUCH more concise. (Also
fixed the df thing, definitely copy paste error!)

------
ghjm
Okay, I'll admit it. I don't actually know what devops is. I know what
developers do and what sysadmins do. Is devops just a buzzword for one person
who can do both? Or does it mean something other than that?

~~~
darkarmani
I will admit that I have a fuzzy understanding as well. My take on it is a
holistic view and integration between development and operations instead of
formal or informal walls between the two.

Wikipedia: <http://en.wikipedia.org/wiki/Devops>

In small companies it usually always exists by accident. I think the "hype" is
more around large companies that have huge barriers and sometimes friction
between development and IT or sustaining Operations.

------
cwh
I would warn anyone that read this to understand what each command does before
actually running anything on a machine.

Things like "sudo !!" are INCREDIBLY dangerous, and I would never put that on
a cheat sheet.

~~~
AaronBBrown
I use this one like a dozen times a day.

    
    
      user@host$ less /var/log/syslog
      Permission Denied
    

crap...

    
    
      $ sudo !!
      sudo less /var/log/syslog
    

Woo!

~~~
biturd
why would that be bad? You would get to read your file.

~~~
AaronBBrown
It's not bad. It's extremely useful. If someone is so negligent that they run
something potentially destructive and then immediately sudo !!, they probably
shouldn't be administering systems.

------
sudara
This is just a small curated collection from myself and old friends — mainly
rails devops. I'd be interested in hearing from other rails devs/ops what you
use on the command line on a daily basis, what cool thing you know about that
no one else does, etc! I'll be adding more as we go....

~~~
DougN7
It would be nice if the title mentioned it was rails focused.

~~~
wlll
I didn't mention Rails in the title because most of the commands aren't Rails
specific.

------
eggsby
For getting around linux I've found no better helpers than manpages and the
cheatsheets by Peteris Krumins[1]. I notice this isn't offered as a pdf or
image, is the idea that someone will come and visit the site when their server
is in trouble? That said, I didn't know about some stuff here like
scriptreplay -- thanks!

[1]: <http://www.catonmat.net/projects/cheat-sheets/>

~~~
nicholassmith
I'd second a quick pdf version of it as well, I'm sure more than a few of us
have a directory with cheatsheets in somewhere on their drive as a 'just in
case'.

~~~
sudara
Here's a work in progress, still needs some loooovvvee:
[https://www.dropbox.com/s/a0rx9c28euwokaz/cheatsheet-pdf-
ver...](https://www.dropbox.com/s/a0rx9c28euwokaz/cheatsheet-pdf-
version-1.pdf)

~~~
nicholassmith
Nice work!

------
AaronBBrown
Here's one I like for testing bandwidth between two machines:

    
    
      host1$ while : ; do nc -l 6666 > /dev/null; done
    
      host2$ pv /dev/zero | nc host1 6666 
      156MiB 0:00:17 [9.46MiB/s] [    <=>              ]

~~~
imbriaco
You should check out iperf as well.

~~~
AaronBBrown
Looks like a useful tool. Thanks!

------
fduran
A little gotcha: "iptables -L" doesn't list the NAT table rules; do "iptables
-L; iptables -t nat -L" instead.

~~~
js2
I also almost always add '-n' to iptables commands to disable reverse name
resolution, which can sometimes take a while. Similarly when calling netstat.

~~~
thyrsus
Me too. Even more important is that a host on multiple networks may have the
same PTR value for each address, and thus one can lose information in the
reverse lookup. "host w.x.y.z" ( or Solaris "nslookup -type=ptr z.y.x.w.in-
addr.arpa" ) is available if you don't recognize the IP address.

~~~
X-Istence
dig -x <ip>

Will also correctly do a PTR query for the reverse entry for that IP. Also
functions correctly with IPv6, so you don't need to remember to split the IPv6
address up into a lot of dots :P

------
jlkinsel
This is a UNIX cheatsheet. Nothing devops about it.

And I concur with the comments about "sudo !!" - !! in general I've totally
removed from my UNIX vocabulary. At least do something like !?string[?]

------
roozbeh18
Checks the speed of the network. if you are looking to good connectivity on
your VPS.

    
    
        wget cachefly.cachefly.net/100mb.test -O /dev/null
    

check the write speed of the disk. Mostly used to check what kind of a write
speed you get on the machine.

    
    
        dd if=/dev/zero of=iotest bs=64k count=16k conv=fdatasync && rm -rf iotest

------
X-Istence
Devops/Sysadmin Cheatsheet for Linux

Most of the commands listed rely on various GNU extensions to the various
utilities or are only applicable to Linux.

FreeBSD:

strace -> dtrace/ktrace/ lsof -> sockstat/fstat watch -> no idea

sudo -> Almost never installed by default on FreeBSD ps aux
--sort=-resident|head -11 -> \--sort is not valid ...

And the list goes on ...

~~~
vacri
If you're going down that path, add windows servers to your list and find more
things missing.

~~~
X-Istence
I don't know a lot of people running Ruby services on top of Windows ...

~~~
smoofles
I don’t know a lot of people running Ruby services on FreeBSD, either. I’m
pretty sure I know one or two who, at least, develop on Windows.

But, yeah, noone’s stopping you to make your FreeBSD version of the
cheatsheet, what with the open source spirit and all… :P

------
3amOpsGuy
There's a few typos in this, in case it helps:

    
    
        ps auxww -> ps auxww -H (H is hierarchy)
    

Faster than lsof, and only displays files - although most things in unix are
files! lsof -p -> ls -l /proc/$PID/fd

Run something forever watch command

Overview of all disks du should be df.

Find files over 100mb find . -size +100M

Low hanging fruit for size ls -al | sort -nk5

Files created (modified) within the past 7 days: find . -mtime -7

Find files older than 14 days: find _.gz -mtime +14 -type f This will break
when you have more archived files in the directory than the shell's glob char_
can support. Use: find . -mtime +14 -type f -name '*.gz' and it will run
quicker too.

TCP Sockets in use, "netstat -antp" will be faster and also lists the process
id.

EDIT: formatting

~~~
activepeanut
Instead of:

    
    
      ls -al | sort -nk5
    

I suggest:

    
    
      ls -larS

~~~
3amOpsGuy
I upvoted you the other day and it seems coming back today i have the upvote
option again? Weird.

Either way, have an upvote!

------
julian37
Small typo, "Overview of all disks" should be

    
    
      df -h
    

not

    
    
      du -sh

~~~
sudara
Thanks for noting. Last minute copy/paste fail :(

------
tszming
Be careful when you think you really understand the meaning of atime/mtime of
findutils.

E.g. man find

    
    
      -atime n
        File  was  last  accessed  n*24  hours ago.  When find figures out how many 
        24-hour periods ago the file was last accessed, any fractional part is 
        ignored, so to match -atime +1, a file has to have been accessed at least 
        two days ago.

------
patja
Thanks for this nice list.

Small nitpick/question: why put the commands in inputs? Do you edit them on
that page? Or is it just to format them?

I tried to do a copy/paste on the page contents to my personal offline
notebook but the most important bits, the commands, didn't paste.

~~~
sudara
Ack! I'm sorry! I started out with "pre" elements, but they were causing
problems with copying and pasting (it looked odd, and it added a dreadful
newline so the command would execute on paste!)

The plan is to have a pdf shortly; that will most definitely be copy and
pasteable.

------
thaumaturgy

        du -h --max-depth=1 -x
    

I use this frequently to keep an eye on disk used by user directories. It
returns the disk space used by each directory in the current directory.

~~~
falcolas
So does:

    
    
        du -sh *

~~~
thaumaturgy
Correct with one important caveat: because __* is a shell expansion, it will
not process any directories starting with a ".". e.g., imap maildir
directories.

~~~
falcolas
That's fair, but for the proposed use, checking the size of home directories,
.* directories are of little concern.

Honestly, when I have to get fine grained enough to dig into disk usage using
'du', I'm rarely concerned with hidden directories (and I typically append a
'c' to the command as well, so if I do have to pay attention to . directories,
I will notice the discrepancy between reported sizes).

~~~
thaumaturgy
We have lots of imap maildir directories; e.g., ".Junk", ".Trash", ".Sent",
".Inbox", for each mail account on the server. It's kind of a special case
though; I'll agree that in all other cases, your version is simpler and easier
to remember.

------
royblow
this is a computer operator cheatsheat. there are no systems administration
tasks being implemented here, just basic unix.

------
joshdotsmith
Just a UX suggestion, but I would highlight the text on the input focus for
easy copy/paste.

~~~
sudara
Good point. I considered doing this, but then it ends up being a "surprise" to
users when they quickly go to triple-click (select all) and the interaction is
different than expected.

What I find really useful is you can immediately type some characters to
filter, and then tab to the command you want. It's automatically selected in
that case.

I'll likely stick a "copy" button on hover shortly — had some initial problems
with z-indexes (it uses flash) but this solves the "one click to copy" issue
without affecting the default and expected interaction.

------
jthomp
Nice list

~~~
sudara
Thank you!

