
Ask HN: How are Unix tools composable? - iamwil
The second rule of UNIX is to make applications that can work with each other. All the stuff I read have to do with using text streams. My question is, even if it&#x27;s a text stream, don&#x27;t programs need to parse the text stream to get some semblance of a data structure they can work with? So if it can be any arbitrary text stream, how do they ensure composibility?
======
informatimago
Yes. Read
[http://web.mit.edu/~simsong/www/ugh.pdf](http://web.mit.edu/~simsong/www/ugh.pdf)
Then read:
[http://www.catb.org/esr/writings/taoup/html/](http://www.catb.org/esr/writings/taoup/html/)

You are responsible, as programmer or script writer, to ensure that the data
you give to some tool is compatible!

But notice also that the pipe is not the only IPC mechanism available on unix;
there are a lot of other ways unix processes can communicate, from shared
memory to external files, thru pipes, sockets, semaphores, signals and others.
Each of these IPC mechanisms can be used to compose tools, as long as they are
designed to use those mechanisms. In all cases, you need to document the
specifications of the interfaces and data format.

------
PaulHoule
Most of them assume records separated with a record separator (typically \n)
and fields separated with a field separator (often a comma, tabs, space.)

That might not be a universal data model, but it is close enough for rock and
roll.

If you need more complex data structures than that, take a look at jq.

------
dozzie
> So if it can be any arbitrary text stream, how do they ensure composibility?

You got the composability thing wrong. The tools are composable because you
feed them on STDIN or in command line arguments. You still need to do plenty
of work to convert one format to another, and that's what awk, grep, sed, and
other tools are for. The only two common (and just _common_ , not _universal_
) things are to split fields on whitespaces and to split records/entries on
newline.

