
Unix: When pipes get names - conductor
http://www.itworld.com/operating-systems/375359/unix-when-pipes-get-names
======
Erwin
Don't forget the ordinary UNIX domain socket: you can think of them like TCP
connections bound to 127.0.0.1 but not assigned a port number but a path name.
This can be useful if you have e.g. 3 instances of the same program that needs
to do some client/server work: rather than figuring out a port number
assignment it can have a UNIX socket. Linux has also an extension to this
mechanism, where the UNIX socket does not create a corresponding file.

In addition to doing cool things like passing open files over a UNIX socket,
or credentials (your remote server can verify the UID of whoever opened the
file in the other end!) the SOCK_DGRAM UNIX sockets are reliable and accept
packets beyond normal UDP limits. This is useful for load balancing (you don't
have to proxy your connection but can pass the new accepted socket directly to
whowever is going to send data to it).

So if you want to make a simple server yet requesting/replying good size
packets you can bind it to AF_LOCAL/SOCK_DGRAM, then the client calls bind()
and sendto() while the server calls bind, then recvfrom() (which fills in the
address of the caller) and sendto() back.

~~~
beagle3
Can you give a link to good example code and documentation about this?

~~~
denysonique
[http://nodejs.org/api/net.html#net_net](http://nodejs.org/api/net.html#net_net)
provides one example, instead of binding to a host/port it listens on a UNIX
socket.

------
saljam
I feel I should mention Plan 9's pipeline branching. Not named pipes, but
something which I guess served a significant subset of named pipes' use cases.
The Plan 9 shell, rc, had a feature where when a subprocess was invoked with
the syntax:

    
    
      <{ls}

that string would get replaced by a filename, which when opened would be
hooked up to the stdout (or stdin for >) of the invoked command.

Used as an argument, this allowed you to essentially do none-linear piping in
a mostly transparent way. I say mostly because I don't think you could seek.

Every once in a while I find myself wishing there was something like this in
bash. Maybe there is...

~~~
RamiK
You shouldn't bring up Plan9 when discussing Linux.

It's akin to bringing up Mozart's writing symphonies at the age of 8 to the
proud mother of a somewhat disabled child that has finally mastered the potty
at the tender age of 12.

Seriously, you can literally list every single new IPC introduced in *nix in
the last 3 decades and counter it with a better design and execution under
Plan9. It's simply the result of a proper design as opposed to an
afterthought. No real point in bringing it up unless you can constructively
use the information to fix Linux.

~~~
zobzu
yea your comment certainly is constructive.

~~~
RamiK
It's not. That's the point. You can't get constructive about this comparison.
I suppose the only exception is to go the Wayland path and push IPC outside
the kernel into the user land. At least there you can reason redundancy vs.
redundant... But I honestly can't think of a single instance when that led to
actual work getting done ;)

------
yason
Speaking of pipes, what would be a good way to branch pipes for some work and
then join them back again in order? Running:

    
    
      ls | pee "./cruncher" "./muncher"
    

will produce output from both ./cruncher and ./muncher in a mixed order. But
often it would be more useful to synchronize the output so that all of
./cruncher output comes first and then ./muncher output.

You could redirect ./cruncher and ./muncher output to individual named pipes
and then pipe a 'cat cruncherpipe muncherpipe' to follow that:

    
    
      mkfifo cout mout
      ls | pee "./cruncher >cout" "./muncher >mout" | cat cout mout
    

But that requires the manual precreation of the named pipes which isn't very
clean. Using <(./cruncher) on the command line would automatically create an
output pipe but doesn't allow redirecting the pipeline to each of the
subcommands.

Any brilliant ideas?

~~~
anonymous
You need to buffer the input in any case, why not just

    
    
        LS=$(ls)
        ./cruncher <<END
        $LS
        END
        ./muncher <<END
        $LS
        END

------
RexRollman
This is why I visit here regularly. Even though I have been tinkering with
Unix/Linux/BSD for years, I was unaware of named pipes. I might never have a
need for it but it is cool to know.

Conductor, thanks for posting the link.

------
telephonetemp
Are there any particularly clever uses for named pipes in shell scripts? I
don't think I've ever seen any.

~~~
gizmo686
I don't think named pipes work particuarly well in shell scripts. A program
writing to a pipe will not terminate until the output is consumed. However,
consuming the output will not happen until the program finishes and the script
arrives that the program that reads from the pipe. Similarly, a program
reading from the pipe will not terminate until the program writing to it does.
If you start reading before you start writing, then you simply wait. Anonymous
pipes solves this problem by launching all of the programs at the same time.
You could work around this by telling the shell to run the command in the
background (append & in bash).

Having said that, I have had occasion to use named pipes in a shell script.
Basically, I had two programs and I wanted to determine which lines were in
one, but not both, of their outputs. Using named pipes, I was able to do
something like:

    
    
      mkfifo foo
      ./prog1 > foo &
      ./prog2 > foo &
      cat foo | sort | uniq -u
    

Effectively, this provides a way to combine the output of multiple commands. I
took it as a matter of faith that doing this won't interleave lines. I assume
that as long as both programs flush only at line breaks then there is not any
problem.

~~~
derleth
Does:

    
    
        sort < foo | uniq -u
    

not work with named pipes?

~~~
dllthomas
It works fine.

The advantage to

    
    
        cat foo | sort | uniq -u
    

is that it is (marginally) easier to drop in another filter in front of sort.

~~~
knz42
This argument is less strong in the face of:

    
    
      <foo  sort | uniq -u
    

(redirections can be at the beginning too.)

Also: sort -u instead of sort|uniq.

~~~
dllthomas
It's still a reason to disprefer the form that was given. I don't see any
reason to prefer the cat version over yours, except habits, though. That said,
there's only weak reason to avoid the cat.

------
binocarlos
Great article! I didn't know how to make named pipes just anonymous ones.

I was recently messing around with Arduinos and Raspberry PI's - totally
clueless about baud rates and Serial in general but had the bug so kept going,
burned a few chips then a few more and finally did something that didn't smell
of smoke : )

Then it went ding - the | in Linux was passing data one byte at a time like a
Serial connection to an Arduino was. Somehow I've never looked at computers
the same way since. Time to go play with some named pipes!

~~~
ars
A named pipe will not change the buffer size. The way pipes work (both
anonymous and named) is they send data in groups that are exactly as large as
they came in.

So probably something is using the write function with just one character at a
time. The standard wrapper over write (i.e. the stdio functions like puts and
printf) can be set to buffer the data first then write it out.

------
dbbolton
Is there something you can do with named pipes that you couldn't also do with
a regular text file?

~~~
hamburglar
A regular file has two obvious disadvantages: first, you write all the data to
disk and read it back again, which costs you time and disk space. In a pipe,
the data is streamed, with the OS doing buffering for you. Second, if you have
one process writing to a plain file and another process reading from it, you
need some way of signaling when there's data ready to be read and when the
reader should wait, or when the stream has ended. A pipe provides this for
you: in blocking IO mode, the reader just issues a read() and blocks until the
writer writes something.

~~~
antocv
What if the file is in /tmp or shared memory then it wont cost that much time
and no disk space/access?

The blocking/signalling is still missing though.

~~~
hamburglar
I don't understand _why_ you'd do it, but sure, if you're determined to avoid
pipes even in situations where they'd be really well suited to the problem, a
scheme using files on a RAM-based filesystem and roll-your-own
signaling/blocking/buffering could conceivably be made approximately
equivalent, performance-wise, to a pipe.

Out of curiosity, is there some good reason you'd do this instead of just
using a mechanism that's specifically made to solve this problem?

------
agumonkey
I wonder how much of go (et al.) channels abstractions could be done this way.

~~~
nine_k
In Go you can control the length of the channel queue, and chan create a zero-
buffering channel which blocks the writer until a reader is available.
Unfortunately mkfifo does not seem to allow this at all.

------
luikore
Is there a stack-like mkfi __l __o ?

