
Cool, but obscure Unix tools - larelli
http://kkovacs.eu/cool-but-obscure-unix-tools
======
gnosis

      abcde            - CD to mp3 ripper
      apg              - random password generator
      base64           - better than uuencode
      boxes            - draw any kind of boxes around your text
      bsdiff           - binary differ
      bspatch          - binary patcher
      bvi              - binary vi (yet another hex editor)
      ccx2             - console xmms2 client
      clive            - flash video downloader
      dvipdfmx         - dvi to pdf converter
      enfuse           - poor man's HDR
      get_flash_videos - yet another flash video downloader
      glark            - advanced grep
      indent           - code beautifier
      lshw             - list hardware configuration
      mcurl            - multiple part downloader using curl
      mktemp           - safely create temporary files and directories
      msort            - sort records in complex ways
      netbrake         - bandwidth limiter
      od               - octal dump
      par              - paragraph reformatter
      par2             - archive verification and repair tool
      ped              - sed done right with perl
      pinfo            - color info reader
      pipe.vim         - make vim part of a unix pipe and allow it to
                         edit the pipe contents
      pv               - Pipe Viewer: a tool for monitoring
                         the progress of data through a pipe
      pydf             - pretty df (disk space viewer)
      qmv              - use your favorite editor to rename files
                         (part of renameutils)
      qodem            - modem program that can do serial, telnet, ssh,
                         zmodem, kermit, etc
      rdiff-backup     - like rsync, but can do incremental backups
      recode           - like dos2unix and unix2dos, but with many more encodings
      recordmydesktop  - make screencast videos
      remark           - great logfile colorizer (part of regex-markup)
      rkhunter         - find rootkit infections
      rlwrap           - add readline editing support to any command
      safecopy         - data recovery tool (better than dd)
      sponge           - soak up stdin and write to a file
                         (for things like pipeline editing)
      sux              - su while transferring X credentials
      unbuffer         - force flushing of stdout
      upx              - executable compressor
      utimer           - countdown timer and stopwatch
      vared            - edit shell variables (part of zsh)
      watch            - run a command multiple times and display the output
                         (with differences highlighted)
      xdotool          - simulate keyboard and mouse activity
      xxd              - hex dump
      zargs            - a version of xargs that makes the find command redundant
                         (part of zsh)
      zed              - very small and fast vi-like editor (part of zsh)
      zrun             - automatically uncompress arguments to command

~~~
bpeebles
This sounds kind of silly to say, but when I spent 5 minutes getting abcde set
up (to rip, encode, and tag into Ogg Vorbis, MP3, and FLAC at the same time),
and then had it finish encoding everything just 30 seconds or so after it
finished ripping, I felt like I was in the future.

I mean, I remember even 5 years ago ripping a CD took actual effort and a long
time to get all the encoding done. So yay for abcde.

------
zerosanity
How is vim an obscure tool? I'm pretty sure quite a few people use vim daily.
After looking over this list, I suspect it's also true for many listed
programs.

~~~
larelli
I knew most of them but not all. However as he targets OSX users, I suspect
many of the programs are unknown to them.

~~~
3am
Ah - thanks clarifying that he targets OSX people. I didn't think that dstat,
xargs, curl, vim, screen, rsync, or ack were obscure; but it makes sense if it
the audience is OSX users.

~~~
mitjak
Why?

~~~
3am
Not a dig at OSX users. I have a macbook in addition to a thinkpad running a
less common distro in front of me right now. I'm thinking of it in a Bayesian
way: if a person is a Linux user or an OSX user, I expect them to know
all/most of those tools. However, given that a person (limited to Linux or OSX
users) doesn't know about those tools, I expect them to be an OSX user.

------
SandB0x
sl

    
    
              ====        ________                ___________
          _D _|  |_______/        \__I_I_____===__|_________|
           |(_)---  |   H\________/ |   |        =|___ ___|      _________________ 
           /     |  |   H  |  |     |   |         ||_| |_||     _|                \_____A
          |      |  |   H  |__--------------------| [___] |   =|                        |
          | ________|___H__/__|_____/[][]~\_______|       |   -|                        |
          |/ |   |-----------I_____I [][] []  D   |=======|____|________________________|_
        __/ =| o |=-~~\  /~~\  /~~\  /~~\ ____Y___________|__|__________________________|_
         |/-=|___||    ||    ||    ||    |_____/~\___/          |_D__D__D_|  |_D__D__D_|
          \_/      \__/  \__/  \__/  \__/      \_/               \_/   \_/    \_/   \_/

~~~
rdtsc
tip: try different combinations of options (sl -la, sl -DF, ...)

------
lysol
If rsync is obscure, then I'm a dumpster wizard. That's a wizard that lives in
an overturned dumpster.

------
oyving
I really like pipe viewer (pv). I wish it as more common in base installs of
Linux.

<http://www.catonmat.net/blog/unix-utilities-pipe-viewer/>

~~~
jz
Although not as featureful as pv, try hitting CTRL+t under OS X or any of the
BSDs during a long running process. It provides status of the current running
process without having to prefix the command with pv. I blogged about this a
while back: <http://jz.posterous.com/bsd-tip-of-the-day-ctrl-t>

------
SeanLuke
> nethack & slash'em

> Still the most complex game on the planet.

Dwarf Fortress.

~~~
jokermatt999
Definitely agreed, but is there anything more complex than DF?

As a sidenote, to anyone who doesn't know the story of how Dwarf Fortress is
being made, you should check it out. It's one guy, working full time on it,
and managing to support himself and his brother through monthly donations.
It's released completely free of charge, so this is just solely from people
donating. That alone would be a pretty cool story, but when you look at the
immense, mindboggling, ridiculously ambitious scale of the the planned
finished game, it's incredible. He's basically trying to build a completely
"generic fantasy world simulator", with procedurally generated world, history,
etc. The amount of detail he goes into in this is amazing, and he has so much
more planned. If you ever need to be inspired that one person can make a
living doing what they love, or that you can actually implement features that
are ambitious beyond most AAA developers' wildest dreams, look to Toady One.

As a bonus, it's a pretty fun game!

~~~
gnosis
DF does have very impressive depth and game mechanics. Unfortunately, its
interface is one of the worst of any game I've ever played.

It's a pity that DF isn't open source either, as then its interface problems
would have long since been fixed. But, as it is, its lead (and only) developer
doesn't seem to care enough to fix it himself.

~~~
gchpaco
DF is the deepest game (really, toy, like SimCity; there's only one end state
within the game and that's getting wiped out, and most of the rest of the game
is very open ended) I have ever played which pairs it with a terrible
interface and a learning curve like a brick wall (no joke, I only figured out
how to get doctors to take medical jobs promptly a few weeks ago after playing
the game for months). The DF interface problems aren't _just_ that it's
console oriented; there's no straightforward way absent external tools (which
I generally can't have, because I play on a Mac) to get a list of e.g. which
dwarves have a certain labor enabled, or which dwarves have skill levels in a
certain labor. Also the way the game is played today is very stuff intensive
which means you tend to have between a quarter and up to half of your fort on
full time crap hauling duties.

~~~
joshu
I heard that it is actually written using GL to draw the character glyphs.
Hilarious.

~~~
Macha
From data/init/init.txt

> By and large, 2D should be the most reliable, while STANDARD has a good
> combination of speed and reliability. However, all 2D modes are normally fa
> r slower than even STANDARD, which may be the slowest OpenGL mode.

It is indeed. However:

> Linux/OS X users may also use PRINT_MODE:TEXT for primitive ncurses output.

------
cygwin98
tsort -- perform topological sort

I bet very few people here are aware of its existence, even it has been part
of Unix since Version 7. I recently discovered it and have used it to solve
some project Euler problems.

------
eru
Have a look at `join', it joins lines of two files on a common field. Together
with `cut' and `grep' you can use text-files as relational databases.

~~~
alnayyir
>Have a look at `join', it joins lines of two files on a common field.
Together with `awk' and `sed' you can use text-files as relational databases.

Fixed it for you.

~~~
eru
awk and sed are useful. But as a principle one should use the weakest tool
that gets the job done.

That principle also applies to grep regular expressions vs Perl regexes.
Regular languages are quite weak in the right way, and thus can be recognized
in linear time. Perl's regexes on the other hand, are so powerful, they can
even tell prime numbers from composite numbers with their back tracking.

Choosing a weaker tool also serves as documention---about which features not
to worry about.

(Of course you shouldn't try to use a tool that's weaker than what you need.
For example, trying to solve some problem crying for recursion or iteration in
a spreadsheet will lead to more harm than good.)

------
tybris
I recently discovered nl and felt sad about the amount of time I had wasted
figuring out how to add line numbers to files.

~~~
chanux
cat -n

but nl is shorter :)

~~~
sigil
They're both annoying. For use in pipelines that is, as they seem more geared
towards typesetting. Which I've honestly never had a use for. Consider this
input file:

    
    
      $ cat > test.txt
      one
      two
      three
    
      five
    

Let's try to number every line:

    
    
      $ nl test.txt                                                            
           1       one
           2       two
           3       three
           
           4       five
    

Ok, the leading spaces are annoying (why not use a single tab?), and are also
produced with "cat -n". And why is not numbering blank lines the default?

How can you prefix every line, even non-blank ones, with the line number
followed by a single tab? Turns out the answer is:

    
    
      $ nl -w1 -n ln -ba -s"`printf '\t'`" test.txt
    

Horrible defaults. HORRIBLE.

~~~
jshb
Why does cat need the ability to add line numbers anyway? It seems outside the
scope of it's purpose. Next it'll need ability to translate Russian to
English.

If I ever needed line numbering, I would output into a file and open it in a
far more capable text editor/viewer that was designed for doing that well.

~~~
AlecSchueler
Indeed - `nl` should be preferable to `cat -n` for that reason even if for no
others. People have been actually saying this since at least 1983
(<http://harmful.cat-v.org/cat-v/>).

I'm often surprised that cat is anywhere near as popular as it is. I think
I've used it for actual file concatenation maybe two times in the past three
years.

------
IvarTJ
You don’t need stdio.h to use puts.

I was in the process of making something similar before I found what I wanted
– rlwrap. It provides readline line editing capabilities to command line
applications that don’t support them, such as netcat.

~~~
pyre
Or, more importantly, sqlplus (for those of us that need to work with Oracle).

~~~
SourPatch
For that I use gqlplus.

<http://gqlplus.sourceforge.net/>

------
IProgrammer
For those who want to learn how to write their own UNIX tools, and
specifically, how to write tools that work well with other UNIX tools, such as
the shell and friends, this article may help - Developing a Linux command-line
utility: <http://www.ibm.com/developerworks/linux/library/l-clutil/>

------
sharmajai
I recently discovered 'bc'. It stands for 'basic calculator' or more precisely
from the man page - 'arbitrary precision arithmetic language'. It is all but a
basic calculator, with better floating point precision capabilities than
Java/Python.

~~~
personalcompute
see also: expr

------
kkovacs
Hi guys,

Kristof here, creator of this particular list. First, thanks for the kind
words! :)

I'll maybe add more tools when I have some time to make more screenshots. It's
surprisingly more time consuming than it seems :)

Also, I'm a bit of two minds with the whole list -- many people think that
some of these tools are already not "obscure" enough, while others suggest
adding even more trivial ones like ifconfig or grep. I'll have to think about
this a bit :)

Once again, thanks for your feedback!

KKovacs

------
brcrth
htop, tmux/screen, xargs, vim/emacs, rsync, rtorrent, ack are far common to
everyone I know (direct and indirectly) that uses the command line.

~~~
getsat
+1 for tmux. It's an amazing improvement to screen.

~~~
spudlyo
How so? As far as I can tell, the only benefit it has over screen is faster
vertically split windows.

~~~
getsat
Vertical splitting, server/client architecture (shared process), non-crap
codebase if you want to dig into it, BSD licensed instead of GPL. I also have
no charset/encoding issues with tmux and it doesn't eat my scrollback.

I'm sure the latter points can be remedied by config file settings for screen,
but it's nice to use software that just works once in a while.

~~~
spudlyo
Vertical splitting screen has. Software architecture, perceived code quality,
and BSD license are not features.

I'm glad that there is competition in this space, but you tmux evangelists
need to realize that there is a lot of work that needs to be done before tmux
is really "an amazing improvement to screen."

~~~
sigil
> Vertical splitting screen has.

Is vertical split in mainline screen now, or do you still have to patch it?

I use both tmux and screen but prefer tmux. Here are some (non-ideological)
reasons:

<http://news.ycombinator.com/item?id=1945637>

~~~
spudlyo
The version available via git from savannah.gnu.org has the old vertical split
patch plus a number of more recent scrolling performance fixes. Thanks for
your list of non-ideological reasons, better notifications sound pretty
compelling. I've been meaning to hack growl support into the OSX screen for a
while now...

------
oinksoft
`ncdu` is a godsend. It's like `du` but usable for troubleshooting.

None of the following from this list are obscure: screen, vim, rsync, xargs,
curl

I'm tempted to put ack in that list.

------
younata
Even though these aren't really obscure, but they're worth mentioning as some
of the best unix "tools" I've seen:

    
    
        Irssi - irc client
        Mutt  - email client
    

These are mentioned mostly because he included the newsbeuter rss client
(which is amazing, I highly recommend it).

------
guard-of-terra
Another cool one is xmlstarlet. It's like grep and sed and some on top, but
for XML files.

Especially, xmlstarlet sel lets you select whatever data you want from XML
files, being a command-line XSL templates generator.

Don't know anything else useful for XML, so it fills a vacuum.

~~~
jefffoster
xmllint is another hugely useful xml tool.

------
jefffoster
Here's a few of my favourites.

agrep - approximate grep based on edit distance).

GNU Global - source tagging system that integrates with the shell (less -tfunc
displays the function given in the shell).

xmllint - xml validator, pretty printer and schema validator.

------
newman314
Here's one that I find does not get a lot of mention but is quite useful when
you need it.

"tee"

------
ashish_0x90
Guake/yakuake - A top-down terminal based on the tilda terminal from the game
Quake.

Redshift/f.lux - Redshift adjusts the color temperature of your screen
according to your surroundings.

------
pearle
Great post! It introduced me to a few tools I wasn't aware of previously. The
main OS on my laptop is Ubuntu so this is very appreciated.

------
chow
My obscure favorites:

yafc: The best command-line FTP client that nobody's ever heard of. Local
caching, tab completion, bookmarking, SFTP, and other generally awesome stuff.

clex: Full-screen file manager for command-line junkies. Configurable
directory display, smart name completion, enhances the command line without
seeking to replace it.

~~~
gnosis
How does yafc compare to lftp, and how does clex compare to midnight
commander?

~~~
chow
I switched to yafc from lftp years ago and never looked back, so I can't
really compare it to newer versions of lftp. IIRC the biggest wins at the time
were context-sensitive tab completion, including support for remote paths, and
automatic bookmarking with saved passwords (optional, of course). FTP sucks,
yafc makes it tolerable.

clex's approach to file management is by complementing the command line; you
can generally use it as a pure CLI and it stays out of your way unless until
you need to find or select a path. Every action you perform remains on the
command line, clex merely makes it easier to quickly build commands. Compared
to clex, Midnight Commander (like Norton Commander before it) feels like an
attempt at a graphical file manager for the terminal, where the command line
is almost an afterthought.

------
rizumu
I've discovered a lot of new apps from the post and comments. Give ncmpcpp, an
excellent ncurses mpd client, a try and say goodbye to GUI music players.

<http://unkart.ovh.org/ncmpcpp/screenshots.php>

Also in line with tmux, checkout teamocil and tmuxinator on github.

------
freedrull
Libcaca and its various programs:

<http://caca.zoy.org/wiki/libcaca>

Also, there is neercs, a terminal multiplexer that uses libcaca:
<http://caca.zoy.org/wiki/neercs>

------
malux85
Most of these aren't obscure, but it was a good read, and I didn't know about
slurm.

So over on one of the test machines I apt-get install cowsay ... One of the
other devs here is going go get a surprise next time he logs into one of the
webservers ... ;)

------
nickolai
Its amusing to see nethack and cowsay listed as a full-fledged unix 'tool's.

~~~
selectnull
Yeah, no 'sl' though... :)

------
pstadler
mytop - a `top` clone for MySQL

~~~
morganpyne
You're probably better off with innotop
<http://sourceforge.net/projects/innotop/files/>

~~~
dolinsky
The new home for innotop is <http://code.google.com/p/innotop/>, and for
anyone looking for other amazing FOSS tools for MySQL (especially Percona
builds with XtraDB)- <http://www.percona.com/docs/wiki/percona-
xtrabackup:start> <http://www.maatkit.org/>

You can do some amazing things with MySQL, Postgres, and Memcached with some
of the maatkit tools.

~~~
morganpyne
Ah yep, thanks for that; just Googled it in a hurry and pasted in the first
link that came up. Can't agree enough on the Maatkit tools, and I'm also in
the process of moving a production database from MySQL to Percona after
hearing such good things about it. Xtrabackup for consistent backups of
innodb/xtradb live systems + incremental (delta) backups are also very useful.

~~~
dolinsky
Depending on the version of MySQL that you're upgrading from and if you're
upgrading in place vs starting fresh on a new system you won't be able to use
Xtrabackup to handle the mysql.proc table due to backwards incompatibilities.
I had an awful go of this upgrading from MySQL 5.0.51 to Percona 5.1.56 and
wound up having to massage the stored procs and functions in one by one. Once
we were migrated over though the xtrabackup tool is great for incremental
backups.

------
lewispb
My operating system doesn't have a package manager ;)

~~~
krzysz00
Get a better operating system, or install Cygwin

------
joelthelion
Apparently he hasn't discovered autojump yet :-D

~~~
nl
autojump looks pretty cool. I assume your name and that of the github
implementation isn't a coincidence?

~~~
joelthelion
It's not :)

------
kree10
lftp is cool, though I wish FTP would die (as telnet sensibly did over a
decade ago) so I will no longer need it.

~~~
evangineer
lftp is an old favorite of mine that i've used many a time. Its built-in shell
with history, support for sftp as well as various flavours of ftp, bookmarks
are features I've made use of.

------
evangineer
Is it worth mentioning pushd/popd here?

------
ericmoritz
And all these are replaced by Emacs :p

~~~
cema
Actually, speaking seriously (for a change), I would like to see an agrep for
Emacs (approximate search).

------
zbowling
cowsay is used on craiglist's 404 page.

------
thdn
why nmap is not listed?

