

This must already exist - spydez
http://lbrandy.com/blog/2009/10/this-must-already-exist/

======
silentbicycle
While it's not a perfect solution, the Unix command _apropos_ (a program that
searches the summaries of the man pages for utilities) can help immensely:

    
    
       $ apropos pipe
       IO::Pipe (3p) - supply object methods for pipes
       perlipc (1) - Perl interprocess communication (signals, fifos, pipes, safe subprocesses, sockets, and semaphores)
       pipe (2) - create descriptor pair for interprocess communication
       ***tee (1) - pipe fitting***
       funzip (1) - filter for extracting from a ZIP archive in a pipe
       pv (1) - monitor the progress of data through a pipe
    

I find it helpful to run the following once in a while:

    
    
       $ for f in $(ls $BINPATH); do whatis $f >> funs; done
    

(where BINPATH is /bin, /usr/bin, /usr/local/bin, etc.), skim through the list
of utilities, and look at the man pages for anything unfamiliar.

As with Emacs, the tool you want probably already exists, and you may already
have it installed, you just need to search with the right keyword. (Learning
the terminology takes some time, though.) Any system with that many features
is difficult to completely remember, but remembering the details can _also_
get pushed onto the system.

~~~
jberryman
Relevant to this discussion is hoogle:

<http://haskell.org/hoogle/>

It beats the pants off of google/apropos/anything else simply because the type
signatures for functions in a purely-functional, statically-typed language
practically tell you what the function does!

Hoogle is a search engine for haskell's standard libraries, and lets you
search by name, or type signature. It even helps you discover abstractions
that you never knew about before:

<http://coder.bsimmons.name/blog/2009/04/why-i-love-hoogle/>

~~~
silentbicycle
Apropos and the like could also be a lot more perceptive if they only worked
for one language, but it's a _major_ trade-off. On this computer, apropos is a
uniform interface to search all the BSD userland utilities, the C, Perl, TCL,
and Erlang standard libraries, system design docs (e.g. hier(7)), etc.

Come to think of it, apropos could also search Haskell functions by type
signature -- Haddock would just need to generate stub man pages for them. (I
don't use Haskell anymore, but someone might find it useful.)

Incidentally, the pipe-able Unix utilities are so useful because they all have
the _same_ type (line-buffered text stream -> line-buffered text stream),
which is a particularly easy one to use across languages.

------
vilya
How about putting the commands into a Makefile and running it with make -j 4?
Easy!

For a more heavyweight alternative, there are lots of solutions around for
running batches of commands across multiple machines. For most/all of these,
multiple commands on a single machine are just one of the simpler cases they
can handle. Systems that spring to mind are GridEngine
(<http://gridengine.sunsource.net>), Qube (<http://www.pipelinefx.com>) and
Condor (<http://www.cs.wisc.edu/condor>). There are plenty of others too
(including a lot of home-brewed brewed systems knocking around inside VFX
houses).

------
RiderOfGiraffes
With regards parallel jobs, what's wrong with sticking a "&" at the end of the
line? Everything he listed is pretty trivial.

It comes down to the idea that the command line is a "non-discoverable"
interface. If you don't know what you want, there's pretty much no way to find
it.

Any suggestions how to fix that? Google solves a lot of them, but not all.
Having said that, I've had fair success recently with just typing entire
questions into it as if it were a programmer I'm talking to ...

~~~
dreish
I learned probably around 80% of the shell commands I use through apropos.

For example, if I'd been searching for the "pipe thingy" lbrandy's blog entry
starts out searching for, I might have typed "apropos pipe", and lo and
behold, there it is at the bottom of the list:

    
    
       ...
       pipe(2)                  - create descriptor pair for interprocess communication
       pipe(8)                  - Postfix delivery to external command
       rotatelogs(8)            - Piped logging program to rotate Apache logs
       tee(1)                   - pipe fitting
    

"pipe fitting" isn't the most useful description, but given how few other
things there are, it's not too many man pages to read, especially after
culling out the things I already knew about back when I didn't yet know about
tee(1).

~~~
ars
On my machine (debian lenny) it's listed as:

    
    
      tee (2) - duplicating pipe content
    

tee (1) does not show up in the search for pipe, but it's description is:

    
    
      tee (1) - read from standard input and write to standard output and files

------
joshu
phrase your processes as make rules. then use gmake -j - this will let you
restart properly in the case of failures, as well.

------
kyleburton
I've had to run those kinds of jobs before and wanted exactly that kind of
tool, not being able to find it at the time, I wrote one:

    
    
      http://asymmetrical-view.com/personal/code/perl/parallel-jobs.readme
    
      http://asymmetrical-view.com/personal/code/perl/parallel-jobs
    

Example 'command file': # specify several downloads to be run in parallel wget
<http://some.host/software.tar.gz> wget <http://some.host/database.mdb> wget
<http://some.host/movie-trailer.mpeg> wget <http://some.host/linux-
distrubtion.iso>

Run as:

    
    
      parllel-jobs --cmdfile=file.cmd --maxjobs=4

~~~
tfh
It turns out you didn't have to write your own program. Sometimes I know
there's somewhere a unix tool that does what I'm looking for, but writing it
can be pretty fun.

------
clawrencewenham
Everybody's jumping in with suggestions for a better command line. What about
ways to know what you need to know when you need to know it?

Maybe we could figure out a thingy that watches what you're doing and finds
ways to do it better.

~~~
silentbicycle
Doing that well is difficult, and (worse still) when it breaks down, it's
typically very irritating. There is a sort of uncanny valley of automated
helpfulness, where sincere attempts to assist common interactions just end up
feeling like Clippy to end users.

------
jimbokun
Wow, xargs can parallelize commands? That's amazing, and could be extremely
useful.

~~~
rbranson
xargs + ssh = poor man's hadoop.

------
guelo
This is why they invented perl.

    
    
      #!/bin/perl
      use Parallel::ForkManager;
      $pm = new Parallel::ForkManager(4); 
      while ($filename = <*.jpg>) {
        $filename =~ s/.jpg//;
        $pm->start and next;
        system("convert $filename.jpg -resize 320x240 $filename-sm.jpg");
        $pm->finish;
      }

~~~
sync
So much more readable too.

------
jermy
A sysadmin friend of mine swears by xapply - although it is largely a BSD-only
application (and tricky to build on linux, to say the least) it appears to
have been the inspiration for xargs's -P option.

------
skorgu
See also the Related Work for shmux: <http://web.taranis.org/shmux/>

------
timf
awk? cut.

~~~
codyrobbins
Every time I go to use awk for something I always think, 'I feel like I would
be able to do this with cut so much more easily, but I also know that I always
feel like that, but am never able to get cut to actually do what I want.'
Despite feeling that way, I usually optimistically go spend 30 minutes trying
to get cut to do something simple like give me a specific column from the
output of _ps aux_. But it's just 30 minutes of banging my head against the
keyboard, and it always reinforces my original strategy, which is to use awk
for everything and never, ever use cut.

~~~
silentbicycle
If your issue with cut is what I susect, you need to flatten the whitespace
first with sed.

At that point, I'd just use awk, though. It has better defaults for field
delimiters.

~~~
jsrn
> [...] you need to flatten the whitespace first with sed

by "flatten the whitespace" do you mean turning many spaces into one space? If
so, I have always used tr with the -s (squeeze) option for that:

    
    
        $ echo "one       two     three" | tr -s " " | cut -d " " -f 3
        three

~~~
silentbicycle
Didn't know about that option, I always did it with sed. Thanks! :)

