abcde - CD to mp3 ripper
apg - random password generator
base64 - better than uuencode
boxes - draw any kind of boxes around your text
bsdiff - binary differ
bspatch - binary patcher
bvi - binary vi (yet another hex editor)
ccx2 - console xmms2 client
clive - flash video downloader
dvipdfmx - dvi to pdf converter
enfuse - poor man's HDR
get_flash_videos - yet another flash video downloader
glark - advanced grep
indent - code beautifier
lshw - list hardware configuration
mcurl - multiple part downloader using curl
mktemp - safely create temporary files and directories
msort - sort records in complex ways
netbrake - bandwidth limiter
od - octal dump
par - paragraph reformatter
par2 - archive verification and repair tool
ped - sed done right with perl
pinfo - color info reader
pipe.vim - make vim part of a unix pipe and allow it to
edit the pipe contents
pv - Pipe Viewer: a tool for monitoring
the progress of data through a pipe
pydf - pretty df (disk space viewer)
qmv - use your favorite editor to rename files
(part of renameutils)
qodem - modem program that can do serial, telnet, ssh,
zmodem, kermit, etc
rdiff-backup - like rsync, but can do incremental backups
recode - like dos2unix and unix2dos, but with many more encodings
recordmydesktop - make screencast videos
remark - great logfile colorizer (part of regex-markup)
rkhunter - find rootkit infections
rlwrap - add readline editing support to any command
safecopy - data recovery tool (better than dd)
sponge - soak up stdin and write to a file
(for things like pipeline editing)
sux - su while transferring X credentials
unbuffer - force flushing of stdout
upx - executable compressor
utimer - countdown timer and stopwatch
vared - edit shell variables (part of zsh)
watch - run a command multiple times and display the output
(with differences highlighted)
xdotool - simulate keyboard and mouse activity
xxd - hex dump
zargs - a version of xargs that makes the find command redundant
(part of zsh)
zed - very small and fast vi-like editor (part of zsh)
zrun - automatically uncompress arguments to command
This sounds kind of silly to say, but when I spent 5 minutes getting abcde set up (to rip, encode, and tag into Ogg Vorbis, MP3, and FLAC at the same time), and then had it finish encoding everything just 30 seconds or so after it finished ripping, I felt like I was in the future.
I mean, I remember even 5 years ago ripping a CD took actual effort and a long time to get all the encoding done. So yay for abcde.
This is a very nice list. I didn't know most of these tools before, and they look very useful; I'm going to try and learn some of them. I found the link in the topic a bit disappointing. It mostly contained tools for system diagnostics rather than work flow optimisation.
How is vim an obscure tool? I'm pretty sure quite a few people use vim daily. After looking over this list, I suspect it's also true for many listed programs.
Ah - thanks clarifying that he targets OSX people. I didn't think that dstat, xargs, curl, vim, screen, rsync, or ack were obscure; but it makes sense if it the audience is OSX users.
I don't think OSX is specifically that important - I've met plenty of "new" linux users who have no idea htese tools exist either - linux has a gui too, you know.
Converseley, there are plenty of long-term unix experts out there who also find that a mac makes a nice workstation that still satisifies their need for unix tools plus other stuff.
Not a dig at OSX users. I have a macbook in addition to a thinkpad running a less common distro in front of me right now. I'm thinking of it in a Bayesian way: if a person is a Linux user or an OSX user, I expect them to know all/most of those tools. However, given that a person (limited to Linux or OSX users) doesn't know about those tools, I expect them to be an OSX user.
There really is no "target" audience of this post. :) It just started out as a list of "recommended" tools, then people started suggesting stuff -- some of which I knew, some of which I don't, some of which run on every unix, some only on Linux.
And it was easiest to take the screenshots from my OS X laptop. For some tools, like powertop, I had SSH-d into various Linux or FreeBSD boxes.
Some of these are uncommon enough that they are not ported to every package manager system, too. :(
Edit: looking at the site again, at least one of those screenshots is from OS X. htop lists /sbin/launchd and several processes from /Library/Frameworks.
Although not as featureful as pv, try hitting CTRL+t under OS X or any of the BSDs during a long running process. It provides status of the current running process without having to prefix the command with pv. I blogged about this a while back: http://jz.posterous.com/bsd-tip-of-the-day-ctrl-t
Definitely agreed, but is there anything more complex than DF?
As a sidenote, to anyone who doesn't know the story of how Dwarf Fortress is being made, you should check it out. It's one guy, working full time on it, and managing to support himself and his brother through monthly donations. It's released completely free of charge, so this is just solely from people donating. That alone would be a pretty cool story, but when you look at the immense, mindboggling, ridiculously ambitious scale of the the planned finished game, it's incredible. He's basically trying to build a completely "generic fantasy world simulator", with procedurally generated world, history, etc. The amount of detail he goes into in this is amazing, and he has so much more planned. If you ever need to be inspired that one person can make a living doing what they love, or that you can actually implement features that are ambitious beyond most AAA developers' wildest dreams, look to Toady One.
DF does have very impressive depth and game mechanics. Unfortunately, its interface is one of the worst of any game I've ever played.
It's a pity that DF isn't open source either, as then its interface problems would have long since been fixed. But, as it is, its lead (and only) developer doesn't seem to care enough to fix it himself.
DF is the deepest game (really, toy, like SimCity; there's only one end state within the game and that's getting wiped out, and most of the rest of the game is very open ended) I have ever played which pairs it with a terrible interface and a learning curve like a brick wall (no joke, I only figured out how to get doctors to take medical jobs promptly a few weeks ago after playing the game for months). The DF interface problems aren't just that it's console oriented; there's no straightforward way absent external tools (which I generally can't have, because I play on a Mac) to get a list of e.g. which dwarves have a certain labor enabled, or which dwarves have skill levels in a certain labor. Also the way the game is played today is very stuff intensive which means you tend to have between a quarter and up to half of your fort on full time crap hauling duties.
> By and large, 2D should be the most reliable, while STANDARD has a good combination of speed and reliability. However, all 2D modes are normally fa r slower than even STANDARD, which may be the slowest OpenGL mode.
It is indeed. However:
> Linux/OS X users may also use PRINT_MODE:TEXT for primitive ncurses output.
I think he's said it's on his to-do list, but he wants to add more 'cool' stuff first. Also you can get tools like dwarf therapist which take some of the pain out of managing the UI.
Just an example of the "amount of detail": the game tracks things like beard length, nausea level, ratio of indoor time to outdoor time, noise level, and it simulates the entire history of the planet before you start.
It's just too bad he can't come up with a UI that would appeal to any but the most dedicated players.
There's a player trying to get a sample of saguaro wood so they can measure some of its material properties empirically and add those to the game. And that's just one small part of some very large, recent threads researching real-world stuff to improve the game's model.
Someone actually went through about a couple of hundred random types of stone and wood to find out more information on them and input proper data for them, rather than using the defaults for wood and stone.
That thread is somewhere on the Suggestions forum.
Dwarf Fortress was already suggested by somebody, and I very much like it, too (haven't had the time to look deeply enough, though), but it's not a terminal game. Yes it uses characters; but one actually needs OpenGL for the game.
I bet very few people here are aware of its existence, even it has been part of Unix since Version 7. I recently discovered it and have used it to solve some project Euler problems.
awk and sed are useful. But as a principle one should use the weakest tool that gets the job done.
That principle also applies to grep regular expressions vs Perl regexes. Regular languages are quite weak in the right way, and thus can be recognized in linear time. Perl's regexes on the other hand, are so powerful, they can even tell prime numbers from composite numbers with their back tracking.
Choosing a weaker tool also serves as documention---about which features not to worry about.
(Of course you shouldn't try to use a tool that's weaker than what you need. For example, trying to solve some problem crying for recursion or iteration in a spreadsheet will lead to more harm than good.)
They're both annoying. For use in pipelines that is, as they seem more geared towards typesetting. Which I've honestly never had a use for. Consider this input file:
$ cat > test.txt
one
two
three
five
Let's try to number every line:
$ nl test.txt
1 one
2 two
3 three
4 five
Ok, the leading spaces are annoying (why not use a single tab?), and are also produced with "cat -n". And why is not numbering blank lines the default?
How can you prefix every line, even non-blank ones, with the line number followed by a single tab? Turns out the answer is:
Why does cat need the ability to add line numbers anyway? It seems outside the scope of it's purpose. Next it'll need ability to translate Russian to English.
If I ever needed line numbering, I would output into a file and open it in a far more capable text editor/viewer that was designed for doing that well.
Indeed - `nl` should be preferable to `cat -n` for that reason even if for no others. People have been actually saying this since at least 1983 (http://harmful.cat-v.org/cat-v/).
I'm often surprised that cat is anywhere near as popular as it is. I think I've used it for actual file concatenation maybe two times in the past three years.
> That's because numbers look better when they are right justified.
They don't look better to a computer though. And that's what I don't get. Why are the defaults geared towards typesetting (human consumption) while cut/sort/join friendly output is so painful to produce?
This is a very common scenario when processing log files: line numbering, then sorting on another field followed by join/grep/head to grab a subset of interest, then recovering the original line numbers for the subset of interest.
Honestly half the time I end up doing this, just because it's the shortest invocation that comes to mind:
Yeah, I usually use awk or perl for this (depends on my mood). Just find it annoying that the unix program whose stated purpose is to number lines is basically dead to me.
I was in the process of making something similar before I found what I wanted – rlwrap. It provides readline line editing capabilities to command line applications that don’t support them, such as netcat.
For those who want to learn how to write their own UNIX tools, and specifically, how to write tools that work well with other UNIX tools, such as the shell and friends, this article may help -
Developing a Linux command-line utility:
http://www.ibm.com/developerworks/linux/library/l-clutil/
I recently discovered 'bc'. It stands for 'basic calculator' or more precisely from the man page - 'arbitrary precision arithmetic language'. It is all but a basic calculator, with better floating point precision capabilities than Java/Python.
Kristof here, creator of this particular list. First, thanks for the kind words! :)
I'll maybe add more tools when I have some time to make more screenshots. It's surprisingly more time consuming than it seems :)
Also, I'm a bit of two minds with the whole list -- many people think that some of these tools are already not "obscure" enough, while others suggest adding even more trivial ones like ifconfig or grep. I'll have to think about this a bit :)
Yep, lots of them are known to some subset of people. However, if people find one or two really useful ones that they didn't know about previously, the list has done its job.
For me, the major winner was dstat. Did not know about that tool - I've been struggling with the output of iostat. I'll also be experimenting with tmux as a screen replacement/alternative.
Vertical splitting, server/client architecture (shared process), non-crap codebase if you want to dig into it, BSD licensed instead of GPL. I also have no charset/encoding issues with tmux and it doesn't eat my scrollback.
I'm sure the latter points can be remedied by config file settings for screen, but it's nice to use software that just works once in a while.
Vertical splitting screen has. Software architecture, perceived code quality, and BSD license are not features.
I'm glad that there is competition in this space, but you tmux evangelists need to realize that there is a lot of work that needs to be done before tmux is really "an amazing improvement to screen."
The version available via git from savannah.gnu.org has the old vertical split patch plus a number of more recent scrolling performance fixes. Thanks for your list of non-ideological reasons, better notifications sound pretty compelling. I've been meaning to hack growl support into the OSX screen for a while now...
It's not in the version of screen that ships with OSX. The macports version of screen supposedly supports vertical splits with the +vertical_split variant, but that didn't work for me last time I tried. And that's how I discovered tmux.
It's not any one thing, but it's a lot of small things. When I last looked a couple of years ago, vertical split wasn't in screen mainline and so it was non-starter for me. Check out the list of tmux commands: it's a very powerful program that's continuously being enhanced.
Dunno if screen has this, but in tmux IIRC, you can have two different tmux servers running in totally different processes, and transfer a window from one server to another. That's pretty cool, imho. The graphical equivalent would be running two VNC servers, and moving an xterm window from one to the other.
Another cool thing: pipe the entire contents of a window in realtime to an arbitrary program.
I always install it by downloading standalone version from http://betterthangrep.com/ There is a command on the homepage you can copy paste to shell and execute.
Ack is simply too often used and I don't want to type ack-grep all the time.
<nit>
$ man alias
No manual entry for alias
$ man bash
ALIASES
Aliases allow a string to be substituted for a word when it is used as the first word of a simple command.
</nit>
alias is not a standalone, it's part of the shell (in bash at least).
Alias (though not 'man alias' :) ) could work. But I prefer my way (on server) because I have one thing less to worry about, namely putting 'alias ack=ack-grep' to .bash_aliases.
Yeah, it's quick and dirty I guess...
yafc: The best command-line FTP client that nobody's ever heard of. Local caching, tab completion, bookmarking, SFTP, and other generally awesome stuff.
clex: Full-screen file manager for command-line junkies. Configurable directory display, smart name completion, enhances the command line without seeking to replace it.
I switched to yafc from lftp years ago and never looked back, so I can't really compare it to newer versions of lftp. IIRC the biggest wins at the time were context-sensitive tab completion, including support for remote paths, and automatic bookmarking with saved passwords (optional, of course). FTP sucks, yafc makes it tolerable.
clex's approach to file management is by complementing the command line; you can generally use it as a pure CLI and it stays out of your way unless until you need to find or select a path. Every action you perform remains on the command line, clex merely makes it easier to quickly build commands. Compared to clex, Midnight Commander (like Norton Commander before it) feels like an attempt at a graphical file manager for the terminal, where the command line is almost an afterthought.
I've discovered a lot of new apps from the post and comments. Give ncmpcpp, an excellent ncurses mpd client, a try and say goodbye to GUI music players.
Most of these aren't obscure, but it was a good read, and I didn't know about slurm.
So over on one of the test machines I apt-get install cowsay ... One of the other devs here is going go get a surprise next time he logs into one of the webservers ... ;)
Ah yep, thanks for that; just Googled it in a hurry and pasted in the first link that came up. Can't agree enough on the Maatkit tools, and I'm also in the process of moving a production database from MySQL to Percona after hearing such good things about it. Xtrabackup for consistent backups of innodb/xtradb live systems + incremental (delta) backups are also very useful.
Depending on the version of MySQL that you're upgrading from and if you're upgrading in place vs starting fresh on a new system you won't be able to use Xtrabackup to handle the mysql.proc table due to backwards incompatibilities. I had an awful go of this upgrading from MySQL 5.0.51 to Percona 5.1.56 and wound up having to massage the stored procs and functions in one by one. Once we were migrated over though the xtrabackup tool is great for incremental backups.
lftp is an old favorite of mine that i've used many a time. Its built-in shell with history, support for sftp as well as various flavours of ftp, bookmarks are features I've made use of.
