
Bash redirections cheat sheet - johndcook
http://www.catonmat.net/blog/bash-redirections-cheat-sheet/
======
dllthomas
One thing worth noting is that "redirection of streams" is a poor metaphor.

If I have two streams,

    
    
      1 ---------- 1'
    
      2 ---------- 2'
    

and I redirect the 1>2

    
    
      1 ----
            \
      2 ---------- 2'
    

it seems to work okay.

But if you have three,

    
    
      1 ---------- 1'
    
      2 ---------- 2'
    
      3 ---------- 3'
    

and try "redirecting 1 to 2 then 2 to 3", the natural reading would be

    
    
      1 -
         \
      2 -----
             \
      3 ---------- 3'
    

But that's not what you get, because you're not "redirecting streams" - you're
copying file descriptors. This is partially fixed by informing people to do
the operations backwards, but there are corner cases where that breaks down,
and "copying file descriptors in the order they appear on the line" never does
because that's genuinely what's going on.

    
    
      1 -> 1'
      2 -> 2'
      3 -> 3'
    

foo 1>2 2>3

This is "run foo copying fd 2 to 1 and 3 to 2"

which gives

    
    
      1 -> 2'
      2 -> 3'
      3 -> 3'

~~~
wethesheeple
Is there a typo: s/genuinely/not geniuinely

I guess what you're explaining is the process that happens as the kernel
reserves fd's? And when you start your program there isn't necessarily any
guarantee that a particular fd will be available, except for 0,1,2, right?

~~~
dllthomas
Copying file descriptors is genuinely what's going on, so understanding it
that way always works.

Edited for clarity:

The way file descriptors work is that the kernel maintains, for each process,
a table mapping from numerical file descriptors to structures describing the
open file/socket/device associated with that descriptor.

When the shell is setting up to spawn a new process (or subshell, for that
matter), it processes the redirections in order, left to right. For
redirections of the form "A>&B" this is a call to dup2(B, A), which has the
kernel copy the entry in the table at B over the entry at A.

Incidentally, you will note the direction of the symbol doesn't matter when
cloning file descriptors: 5>&7 and 5<&7 mean the same thing - you're just
copying a file descriptor. It does matter when opening a file, as the file
will be opened for reading or writing before before being dup2'ed over the
requested descriptor. Having said that, of course use the correct symbol -
it's good documentation.

~~~
wethesheeple
Thanks for the this. In my opinion, there's too little written about passing
fd's (use of dup2). It seems like a technique that is more useful than just as
part of shell redirection. Safer than popen()?

~~~
dllthomas
I'm pretty sure popen is just a wrapper around this stuff. It's basically,

pipe: get two fds connected by a pipe

fork: create a child process

dup2: move the read fd of the pipe to 0 (stdin) in the new process

exec: run the program in the child process

There's some cleanup but that's the gist - popen isn't a syscall (or
reasonably close to one), so must necessarily rely on other stuff to get its
work done where it actually interfaces with the kernel.

~~~
wethesheeple
That makes perfect sense. It's the fork+exec that makes it dangerous (e.g. in
the CGI context). This is something I've been wondering about and you have
just provided a jolt of clarity. Many thanks, again.

~~~
dllthomas
Happy to help :)

------
gwillen
I think I previously had all the pieces required to do this, but only just
figured out that it worked:

    
    
      Glenn-Willens-MacBook-Pro:~ gwillen$ exec 3<> /dev/tcp/<website>/80
      Glenn-Willens-MacBook-Pro:~ gwillen$ echo "GET / HTTP/1.0" >&3
      Glenn-Willens-MacBook-Pro:~ gwillen$ echo "" >&3
      Glenn-Willens-MacBook-Pro:~ gwillen$ cat <& 3
      HTTP/1.1 200 OK
      Date: Mon, 10 Sep 2012 21:43:33 GMT
      Server: Apache
      Last-Modified: Sun, 06 Sep 2009 18:44:42 GMT
      ETag: "30525-0-472ed20537a80"
      Accept-Ranges: bytes
      Content-Length: 0
      Connection: close
      Content-Type: text/html
    
      Glenn-Willens-MacBook-Pro:~ gwillen$

------
dllthomas
One thing missing is <<\- heredocs, which strip (some) leading whitespace. It
lets you keep up your indentation in a shell script without passing a bunch of
whitespace to the commands.

------
shabble
I've found

    
    
        cmd <(another_cmd)
    

to be particularly useful in the past for pushing generated data into
something that doesn't accept - as an alias for STDIN. I'm not sure if it's a
different way of expressing one of the listed items though; iirc it creates a
temporary fifo

~~~
daragh
That FIFO temporary file is the file descriptor of the subprocess' stdout.
You're accessing it through a pipe.

It's most useful when you want to diff the result of multiple commands, e.g.:
diffing the contents of two directories:

    
    
      $ diff <(find /first/path | sort) <(find /second/path | sort)

~~~
akldfgj
Aha!

In zsh "=(cmd)" does that, but "=" doesn't work in bash. "<(cmd)" works
everywhere... why did zsh trick me with "="?

~~~
ralph
zsh had the idea first IIRC.

------
bajsejohannes
I've always done

    
    
       (cmd 2>&1) > file
    

to redirect stderr and stdout to a file. I always thought the parenthesis were
necessary, but according to this I can just do

    
    
       cmd > file 2>&1
    

It seems very unintuitive, though. Does anyone know the rationale behind
ordering of redirects?

(That said, I'm going to use 'cmd &> file' that I just learned from now on)

~~~
rcsorensen
cmd is going to output to STDOUT and STDERR, which are file descriptors 1 and
2. In a basic shell command execution, those route out to the console.

Reading from left to right:

    
    
      cmd
    

Run command cmd. File descriptors: {1: /dev/console, 2:/dev/console}

    
    
      > file
    

Open a file descriptor for "file" and copy it into our first file descriptor
(stdout). File descriptors: {1: fopen('file'), 2: /dev/console }

    
    
      2>&1
    

Copy the file descriptor stored in 1 (stdout) into 2 (stderr). File
descriptors: {1: fopen('file'), 2: fopen('file')}

------
dredmorbius
The network pseudo filehandles (/dev/tcp/<host>/<port> and
/dev/udp/<host>/<port>) are explicitly disabled in certain distros (notably
Debian). Ubuntu appears to support it.

Though not limited to just shell redirections, one of the true masters of bash
(and zsh) is Larry Peek whose written/given multiple books, articles, and
presentations on the topic.

~~~
dfc
Do you know why debian did that? Any links to debian-devel threads?

~~~
dredmorbius
<http://lists.debian.org/debian-user/2006/07/msg00204.html>

The feature is confusing (it uses virtual files visibly only to bash, and not
extant on the filesystem) and overlaps existing functionality (netcat).

Also:

<http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=146464>

<http://lists.debian.org/debian-user/2003/04/msg01591.html>

<http://lists.debian.org/debian-devel/2006/11/msg00794.htmlu>

<http://lists.debian.org/debian-user/2006/07/msg00204.html>

------
luriel
I can never remember the (ba)sh redirection syntax, and I used it for decades.

Rc on the other hand is much more clean: <http://rc.cat-v.org>

------
AceJohnny2
the "cmd <<EOL ..\n..\n EOL" part has been mystical to me for ages, especially
as I didn't know what to google to find out. I recently learned it was called
a "Here Document". <http://en.wikipedia.org/wiki/Here_document>

For extra credit, try using a "Here Document" as input for 'echo' and 'cat'.
Explain the results.

~~~
akldfgj
`man bash` , search for "<<" see the "Here Strings" section.

You can also search <https://www.google.com/search?q=bash> and follow links to
"Documentation"

    
    
        xargs echo <<EOL  
        > line 1
        > line 2  
        > EOL
        line 1 line 2
    
        $ cat <<EOL
        > line 1
        > line 2 
        > EOL
        line 1
        line 2
    

line-terminators get eaten by xargs

------
darkstalker
The PNG version is rendered with subpixel antialiasing, which is kinda
annoying to see. Besides that, awesome job.

~~~
jdpage
Is there a particular reason you'd go for the PNG version over the PDF
version? It's a more suitable format for these "cheat sheet" type pages, since
it'll use the platform text rendering, and also keep the vector information
(meaning it'll print better if you want to hang it up next to your desk).

------
zwilliamson
Where is EOF? I use that a lot in my Bash scripts. I think it should be on
this cheat sheet.

------
dllthomas
I like the mention of PIPESTATUS - $PIPESTATUS[*] is at the start of my PS1

------
michaelhoffman
Thanks! Much more useful than the other recent alphabetic cheat sheets.

------
trusko
Great! Thanks for sharing, very helpful.

------
methoddk
Very useful! Thank you for this.

