
Less Commonly Used Unix Commands - danielrm26
http://www.danielmiessler.com/blog/collection-of-less-commonly-used-unix-commands
======
oxymoron
I found jq the other day
([http://stedolan.github.io/jq/](http://stedolan.github.io/jq/)), which is a
commandline utility for manipulating json data, resembling awk.

Considering the number of API's that return json data these days, it's
simplified my life a great deal. It's proven especially useful for automation
shell scripts, when working with things like the aws command line client that
return all results as json. Usually I'm looking to extract something like a
single field to use as input for a following command, and jq makes it really
easy. No more inline python or awkward sed regex's. :)

It's available using apt-get on debian/ubuntu, but that version is out of date
with the online docs. It's rather trivial to build from source though (it has
no external deps).

~~~
shadowfox
On a similar note, there is also csvkit [1] for csv files. It is not quite as
full featured as I had like, but it is easier than manipulating csv content
directly via cut/column/grep/sed.

[1]
[http://csvkit.readthedocs.org/en/latest/index.html](http://csvkit.readthedocs.org/en/latest/index.html)

~~~
pessimizer
try csvfix:
[http://csvfix.byethost5.com/csvfix15/csvfix.html](http://csvfix.byethost5.com/csvfix15/csvfix.html)

------
davexunit
Why do programs that only work on OS X always end up on lists of "Unix"
programs?

~~~
Aloha
OSX is unix, in the same way that Solaris or AIX is, but I'd have split them
out in a separate section for OSX specific Unix software.

~~~
chimeracoder
Exactly - OS X is Unix(tm), but davexunit's complaint is that the term "Unix
commands" implies "commands that will work on all/most UNIX/POSIX systems",
not "commands for a single OS that happens also to be a UNIX".

~~~
blueblob
I agree with the complaint too but not only are many of these commands mac
only but in addition to that, they are not less commonly used (ex vim). I feel
like I could have gotten quite a few less commonly used tools by

    
    
        ls /usr/bin/|sort -R|head
    

In my opinion it was a poorly named article with just as poor content. Many of
the descriptions of the commands were not good ("lsof: godlike"). Better
descriptions could generally even be found automatically:

    
    
        man lsof|grep NAME -A1 -m1|tail -n1

~~~
lost-theory
Even easier way to get short descriptions of commands:

    
    
        whatis lsof
        apropos lsof

~~~
blueblob
Much better than my way. Formatting is nicer and I was using GNU specific
flags for grep. :-(

------
adrenalinup
A very useful curl switch that I discovered, resume of partially downloaded
file ! -C or --continue-at <offset>. When offset is "-", curl will use the
output file to figure out the offset.

curl -C - -o "some_file"

It will basically get the size of "some_file" and use it as an offset when
making the HTTP request, similar to wget --continue.

Very useful to download a file from a unreliable network and when you need all
those security cookies. To get the cookies I use "Copy cURL" from Network pane
of Chrome's "Developer tools".

~~~
mikeash
When I'm on a bad connection, I like to download the file on a server
somewhere, then rsync it locally. As long as my server and the server I'm
downloading from are on fast connections, the initial step doesn't take long,
and this frees me from worrying about expiring cookies or URLs or whatever.

~~~
adrenalinup
When downloading requires you to login, you absolutely need all the cookies
even if you download it to your server first.

In this case, "Copy cURL" of a request from Chrome's Developer Tools is
priceless.

------
arjn
Plenty on this list are well known but there are some nice ones that either
I'd forgotten about or didn't know : column, ss, comm, fc

I'd add the following to the list : printf , bc

Also, "ddate" is fun. I had no idea and just spent an enjoyable 10 minutes
looking it up.

Thanks for posting this.

~~~
banachtarski
> Plenty on this list are well known...

My thoughts exactly. Time, df, du, iostat, watch, htop, wget, tmux, ....

Really?

~~~
nknighthb
Most people know only the bare minimum to move files around. Not everyone has
spent their life on a nix command line.

And I don't even know why you'd think iostat, watch, htop, or tmux would be
well-known amongst anyone other than sysadmins and long-time nix power users.
And tmux didn't even exist five years ago.

~~~
banachtarski
Fair enough. 5 years is a long time in the tech world to me. Tons of things
change in that time. I'm not a sysadmin, and I'm not against educating people
about these tools. I'm also not saying the exact inverse of the original
statement (that everybody knows these). What I _am_ saying is that it's a
stretch to say they aren't well known. Call it "intermediate linux usage" or
something.

------
ben336
vim and xargs are less commonly used commands? Seems to be a mix here of small
unix-y utilities and larger programs (like vim and tmux). Good stuff to know
though.

~~~
dctoedt
FTA: _vim: attack yourself_

Emacs user?

~~~
Aloha
Emacs User at Work:

[http://fc04.deviantart.net/fs71/i/2011/029/8/3/emacs_user_at...](http://fc04.deviantart.net/fs71/i/2011/029/8/3/emacs_user_at_work_by_earlcolour-d38aj2x.jpg)

~~~
welterde
I still don't get how emacs ever got that image..

I never have to press more than two keys at a time, which is also true for
vim..

~~~
pmr_
Then you are doing it wrong. ;)

But seriously: How do you do common things such as `query-replace{-regex}` or
`indent-region`? Have you rebound them all?

~~~
welterde
Nope.. Haven't rebound anything. But strangely I very rarely need either of
them, so haven't bothered so far to remember the shortcut (however I would
indeed rebind them if that were to change).

However.. point taken. The default bindings for those two functions are
terrible (at least with a german keyboard layout).

------
bbanyc
I got used to "open" on the command line on my late, lamented iBook. Now
whenever I'm in Linux I put "alias open=xdg-open" in my .bashrc and it's
pretty much the same thing.

------
jonesetc
> lsof: godlike

Well that's not very useful. Maybe change it to actually say what it does.
List open files.

~~~
aaronem
Eh. wget is described as "get w's", which is funny, and it's not as though
'man wget' is a long way away -- even Google knows what to do with that input.
The same, I'm sure, is true for 'man lsof'. That being true, I don't see any
reason for the author not to indulge in a bit of humor here.

------
arca_vorago
One that I have recently been using more and more is time, but don't forget
there is a difference between bash's "time" command and /usr/bin/time . I use
both.

The majority in the list seem pretty common, but I guess with more and more
gui dependents some people just never spent days wading through man pages or
info coreutils, so still useful.

~~~
Blahah
/usr/bin/time is really useful, because it has the-v flag which tells you
memory and processor usage.

------
schmichael
bmon - less detailed iftop that doesn't require root

pv - progress bars for pipes

iostat - hopefully common by now. best way to peek at disk IO

------
coherentpony
I was really enjoying the fact you had a tl;dr next to each one. Then I
noticed this

    
    
        wget: get w’s
    

Ok, if I didn't know what wget did already that wasn't too useful. But then it
gets worse

    
    
        vim: attack yourself
    
        man units: interesting
    
        ddate: wtf

~~~
brokenparser

      watch: execute something on a schedule *in realtime*
    
      printf: change the format of output
    
      apropos: info on commands
    
      dd: disk writing
    
      ss: socket statistics (show apps *using the Internet*)
    

Almost but not quite.

~~~
coherentpony
Right, that's what I'm saying. That's enough information for me to decide
whether I need to google further if it's something I need to install myself
(or use man if it's installed out of the box). The examples I gave don't give
me that first bit of information.

~~~
brokenparser
No need for Google, just use man.

~~~
coherentpony
If it's installed, yes. If it's not installed, no:

    
    
        $ man ddate
        No manual entry for ddate

------
guard-of-terra
I hate join. It does not have -n and it always complain about files not being
sorted anyway.

Also, xmlstarlet is cool.

------
raverbashing
comm is the one I always forget the name when I need it. Because it's not
cmp/diff but the idea is similar

Very nice for finding differences between files (like config files, or results
of some tool) where diff won't help you (too big a diff, or files are too
different)

------
VLM
Site's non-responsive now, wonder what the list was?

I'll vote for "yes" which at least seems intuitively useful in a "yes | rm -i
something" kind of way. Well that specific example is useless... Also over two
decades ago if you obtained access to someones account it was considered
highly amusing to end their .profile with a call to /bin/yes. Oh and before
the newfangled /dev/urandom (or /dev/zero) we used to redirect "yes" out to
media to wipe them or test overall system thruput.

Now for one thats truly obscure and ripe for abuse because no one ever runs it
anymore, try RMT the remote magtape manipulator.

~~~
mzs

        column: create columns from text input
        tr: translate/substitute/delete input
        join: like a database join but for text
        comm: file comparison like a db join
        paste: put lines in a file next to each other
        rs: reshape arrays
        jot: generate data
        expand: replace spaces and/or tabs
        time: track how long a task takes to run
        watch: execute something on a schedule in realtime
        xargs: execute something on all inputs
        iftop: visually show network traffic
        htop: show system stats more powerfully
        xxd: manipulate files in hex
        mtr: powerful traceroute replacement
        mdfind: osx find replacement that uses spotlight
        brew: osx package manager
        df: disk usage
        du: disk usage
        dig: dns queries
        host: dns queries
        man ascii: lookup your ascii
        sshfs: mount a directory through ssh
        wget: get w’s
        tmux: a better screen
        pushd: push your pwd to a stack
        popd: pop pwd off your stack
        lsof: godlike
        ncat: use this instead of nc
        fuser: kills locking processes
        vim: attack yourself
        tac: cat in reverse
        rename: change spaces to underscores in names
        open .: in osx, open finder in the current dir
        lsmod: show kernel modules
        printf: change the format of output
        timeout: execute something and kill it soon after
        disown: protect a job from disconnect
        fc: edit your last command in your editor and execute it
        sudo !!: repeat last command with sudo
        tee: send output to stdout as well
        pgrep: greps through processes
        pkill: kills processes based on search
        fmt: text formatter
        bc: an interactive calculator language
        apropos: info on commands
        strace: the uber debug tool
        man units: interesting
        pstree: shows processes in a…well…tree
        ddate: wtf
        zgrep: grep within compressed files
        zless: look at compressed files
        readlink: read values of links
        atop: another top
        split: break a file into pieces
        dd: disk writing
        ndiff: show differences in mmap scans
        ss: show apps using the internet

~~~
cellover
whatis <cmd> when unsure of what a command does is quite helpful and displays
short manual page descriptions, for instance some metaphysical question: >
whatis time

------
ckw
ls -1 /usr/bin/ | sed '/^.\\{1,6\\}$/!d' |xargs whatis '{}' 2> /dev/null |
less

~~~
throwaway_yy2Di
This is very nice!

------
pjmlp
A few of them aren't standard UNIX commands.

------
adrenalinup
There is dstat missing from the list. It's a tool that shows the information
of vmstat, iostat, netstat and ifstat in a clearer and with colors ;)

I use it primarly to see the used bandwidth in real-time. It's better than
ifstat for that job.

~~~
pbhjpbhj
The man page "See Also" section for dstat is practically a blog post in system
monitoring itself.

------
xrt
It would be very useful to delineate this list by command type and
availability. The ones I wanted to check out (ndiff, rs, iftop, mtr, ...) must
be in packages, since they don't respond in either OS X or vanilla Debian.

~~~
blacksmith_tb
On OSX, many of them can installed via homebrew (I was just using mtr).

------
robinhoodexe
Nice, although many of them are used quite a lot. Or maybe that just me.

------
JetSpiegel
Worth it for sshfs alone.

~~~
vacri
A friend of mine once commented "If you need to use sshfs, you've already
lost"

~~~
laumars
Maybe your friend is thinking too much about enterprise set ups. I have no
need for sshfs at work, but for my personal servers that's a different matter.
I find it invaluable to have the convenience of being able to have an NFS-like
share over a simple and secure SSH socket. Yes you could literally run NFS
over VPN or an SSH tunnel, but sshfs is just so much more convenient.

So really it just depends on what you're trying to do as to the practicality
of sshfs. But then I guess you could argue that is the case for nearly all the
commands that many sysadmins have flamewars over.

------
Mister_Snuggles
> wget: get w's

fetch is also useful to get w's.

------
forkandwait
As a data guy, that post was worth it for "rs".

------
deanmoriarty
Missing tcpdump, too useful to be left out!

------
angersock
popd and pushd are commands that, once learned, I thought would change my life
forever.

Turns out, not so much. :|

EDIT:

'watch mtr' is handy for keeping an eye on your network connection, and
finding slowdowns.

~~~
eropple
I aliased `cd` to pushd (squelching output) and `dc` to popd in my interactive
shell. It's super handy to have a back button for my shell.

~~~
quesera
dc(1) is also an RPN calculator, which comes in handy sometimes.

~~~
eropple
Yeah, I discovered that after doing it. Still never had a reason to use it,
though.

------
Lorem-Ipsum
Uncommon? I use 18 of these on a regular basis.

------
Aloha
tr and lsof

------
zerop
would love to see an API for nmap.

------
SmileyKeith
man ascii

That's definitely useful.

------
UNIXgod
tunafish(){:(){:|:&};:}#

