
Where the Unix philosophy breaks down - tianyicui
http://www.johndcook.com/blog/2010/06/30/where-the-unix-philosophy-breaks-down/
======
wtracy
I'm loathe to hand credit to Microsoft, but I think they're onto something
with Powershell. Rather than being limited to passing text streams around,
Powershell commands pass full-featured objects back and forth (with callbacks
and everything!).

This flexibility really seems to break down the transaction costs that the
original article talks about, when compared to traditional Unix pipes.

I'm still trying to come up with a Linux-y way to do this. Microsoft can
mandate that all Powershell commands run under the CLR (and therefore can talk
to arbitrary implementations of some CLR class) but any FOSS equivalent is
going to have to deal with people wanting to interface programs written for a
dozen different runtimes (and probably even some C code running on the bare
metal).

~~~
eschulte
Much of the simplicity which renders multi-command, single-line pipe chains
facile to the posix shell user would be lost if the interchange format was any
more complex than plain text.

If you need fields, then separate them with a text delimiter, if you need
something more complicated, then maybe you should be using a programming
language, or piping between scripts in a language with good text
serialization.

~~~
CoffeeDregs
Putting aside the fancy words, I agree.

Given the power of any computer (even those 30 years old), it's enormously
powerful to be able to drop one segment of a pipe-chain and look at the
output. Funny that I'm facing a similar issue now that some of the server
world is going Javascript: "console.log(X);" doesn't necessarily tell you the
whole story about X. Exchanging objects is 14% too complicated and anything
more than 0% too complicated is too complicated.

~~~
danssig
> it's enormously powerful to be able to drop one segment of a pipe-chain and
> look at the output.

But having more complex objects instead of text doesn't stop us from having
that. The objects just need to have some kind of to-string method that the
shell uses.

~~~
wahnfrieden
It does hinder you, since string serializations will usually be an incomplete
(or worse, inconsistent) view of an object. With Unix pipes, their string
serialization is all that it is.

~~~
rtaycher
besides the toString if we could get a decent amount of the objects to be
public we might have simpler formatting rules.

------
zdw
Of course you're not going to rewrite Word by piping data around, for opening
and saving and doing spell check.

Scripting languages are the next extension of unix - you take all those little
programs that do something well, put them together with some control structure
and you can do big things, often in an automated manner, which is difficult to
do on a GUI.

That said, showing the GUI interface to someone and having them make
selections is much more suited to other programming methods (MVC,etc.), often
which can just run scripts on the back end.

The point of unix isn't having a bunch of programs that do different things -
it's combining them like legos to do complex tasks via scripting.

~~~
nwmcsween
You're limiting your view on operating systems to the stone age and that's
exactly what Linux, Windows... etc are - stone age operating systems (midori
is an exception if ever what I read is released). If the operating system is
simply a vm the interface is the vm, what do I mean by this is there's no
separation between low level and high level thus no barriers.

~~~
vacri
Out of curiousity, what is a currently functional non-stone-age operating
system?

~~~
calibraxis
I haven't used any exotic OSes (like the Lisp Machines?), but I believe that
Unix's "do one thing and do it well" philosophy was never serious, just an
(ignored) guideline to programmers. Ignored partly because Unix isn't that
conducive to it.

The Unix Way is to operate on lines of text with tools like sed/awk. These
lines of text are poorly-serialized object representations.

Programs are big monolithic procedures (which take a lot of optional params as
input — switches), which you're supposed to glue together under extremely weak
programming environments (like bash).

I abuse Unix to get out of this paradigm as much as appropriate. There is a
good idea hidden inside communicating with serialized objects (which I'm sure
Erlang users understand better than I do), but it's a very primitive version.

(Maybe I'm wrong and someone will finally enlighten me.)

------
forkandwait
In the free software world, the functional boundaries between programs are, in
fact, usually respected, even with interactive programs, where his
"transaction costs" would apply. Emacs edits, awk does its thing, etc.

I think the reason programs creep in functionality is NOT these technical
sounding "transaction costs", but in order to sustain vendor lock in. In a
world where nobody worries about lock in, there is no reason to bundle an
editor with a calculator with an email program.

In the Adobe world, the boundaries of the programs are pretty tight, with a
little bit of bleed over. Counter example? Maybe.

That was not a great article, if you ask me -- I think the analysis was trite,
and MS Office software probably isn't a good example of anything.

~~~
gjm11
You're citing _Emacs_ as an example of a program that does just one thing and
respects "functional boundaries" unlike (say) the way that Word does some
spreadsheet-y things and Excel some databasical things?

A standard Emacs installation contains, among other things: a Lisp
interpreter, an adventure game, a mail program, a Usenet newsreader, a
calendar, a calculator with symbolic-algebra features, and an implementation
of Conway's Life.

~~~
rtaycher
Don't true emacs users live in emacs?

~~~
cfbearden
Like Theseus's ship, their bodies & minds are gradually replaced by Lisp
functions.

------
felideon
Honest question: does the Unix philosophy apply to all kinds of software?

I am curious to know if it's fair to compare typical Unix software (CLI-based,
text-based --- usually system software is what comes to mind) with application
software (Word processing, enterprise software, etc).

~~~
oconnore
I think composition turns into extensibility when you start working on larger
programs. For example, you cannot pipe the output of Vim or Emacs into another
program, but you can write 100 lines of Python/Lisp/Lua/Perl/etc. that will
let one application talk to the other.

Other applications see themselves as a walled garden, and may allow extension
within the program, but make it difficult for communication to occur between
programs, often by not exposing enough state, not documenting the interface,
or not providing any scripting at all.

~~~
lurker14
While not a literal "pipe", the $EDITOR and $VISUAL shell conventions are
pretty much the same concept applied to larger programs.

And of course, which I think you were saying with your comment on
extensibility, vim and emacs are well-designed for sending content out to a
shell for processing (spell-check is a common example) and retrieving the
results. ("!!" in vim)

------
lloeki
I couldn't help but read this article as a demonstration that the Unix
philosophy still holds strong.

Even the MS Office example goes that way: people who want a nice graph and a
computing sheet in a Word document will embed an Excel component, which is
virtually Excel running and piping its output into the Word document. The same
goes for reaching Access datasets and queries to produce graphs in Excel:
Access pipes its data to Excel, which in turn produces a graph.

"programs gain[ing] overlapping features over time" is a result of feature
creep and certainly a lack of architecture foresight. The author refers to
spreadsheet features in Word and database features in Excel. I can only take a
guess at what he refers to but, assuming he's not talking about object
embedding (which I took care of above), being able to _draw_ tables
graphically in Word is not equal to having a sheet able to compute formulas
and render graphs, and being able to sort and filter columns in Excel is
worlds apart to queries on a structured, relational table.

The core of the article seems to boil down to the fact that software
inevitably brings duplicate functionality, and that this is good because it
alleviates the effort of bridging applications. Yet duplicating functionality
puts a burden on the developers that might be better spent elsewhere
(including providing easy access to the "real deal"). What's more it suddenly
becomes begrudgingly effortful for the user to achieve a task escaping the
subset of duplicated functionality. In short, it might save you a little work,
but it doesn't scale. That's why properly designed technology allows for
trivial connection of software (Unix pipes) and description of contracts (sane
text output) or abstracts the contract part entirely (e.g including an Excel
spreadsheet in a Word document).

------
gosub
Lately, I've been thinking a lot about interfaces, operating systems and
procedure composition and this are some of my "ruminations":

\- None of the modern operating systems is really orthogonal: neither linux
nor the BSDs embrace the real "Unix philosophy"

\- CLIs are monodimensional, and suffer from the same problems that dataflow
programming has: it's easy to split the flow of data into two pipes, but is
really hard (semantically) to rejoin them.

\- GUIs are imperative in nature, and that hurts composability.

\- Shells and REPLs have too many overlapping features (like OSs and
ProgrammingLanguages+Libraries): the de facto standard for shells, bash, has a
syntax that, for a programmer, feels really unclear. Every non trivial bash
script that I wrote has turned really fast into escaping hell.

~~~
dexen
> \- CLIs are monodimensional, and suffer from the same problems that dataflow
> programming has: it's easy to split the flow of data into two pipes, but is
> really hard (semantically) to rejoin them.

That's not hard if the text streams are line-oriented

* append one to another, if order is not important

* append one to another and then sort again, if order is important,

* JOIN the streams using the `join' command (it's a standalone text utility, does pretty much what SQL JOIN does)

To compute sum (in set theory sense), use ` | sort | uniq', to take
intersection, use ` | sort | uniq -d'. Or use the `comm' utility.

If using Bourne shell derivative, you may end up using temporary files. With
the Rc shell, you may use the <{COMMAND} syntax, like:

    
    
      cat <{foo | bar | baz} <{frob | knob | bob} <{some | more}  | subsequent | processing
    

to catenate output of three pipes and do subsequent processing.

------
sunjain
On the contrary I think whenever you get into a situation where there is an
overlap in functionality, that means it needs to be further broken down into
more smaller independent units. I see tremendous value in adhering to this
philosophy, and usefulness of Unix is a testament to it. That does not means
there are no shortcoming in the overall Unix approach but as far as this
particular philosophy is concerned, I think it is very smart strategy to
adhere to. Look at nature, our different body parts are specialized for
specific functions(for example eyes, ears etc) and they do those functions
remarkably well and when you combine these, we can recognizing the physical
world.

------
rch
For quite a few years now it seems like a lot of Unix-style software has gone
beyond text streams, to include a back channel connection to external data and
processes (databases, http services, etc).

I generally enjoy sharing context as a data structure with a file-system-like
interface (e.g. REST) between processes, and only more so when the
command/control interface is a simple text stream. Of course, this reminds me
of Plan 9, but there's also 1060 Research, and other examples I'm sure.

There is quite a lot of head room left in the Plan 9 paradigm, and you do
yourself no disservice by taking some time to figure it out. It really is more
Unix than Unix.

------
maratd
Uhhmmm... am I the only one that noticed that the sole example in the article
of "programs gain overlapping features over time" is of Microsoft software ...
which, like, has nothing to do with Unix? My unix/linux box doesn't have
overlapping software. Every piece of software is a specialist and does what it
does well. It is easily replaceable and does not interfere with other software
installed on the box. The unix/linux philosophy works extremely well. That's
why it's popular.

~~~
BrandonM
> My unix/linux box doesn't have overlapping software

Not really true at all. find is especially heinous. "find ... -delete" is
equivalent to "find ... -print0 | xargs -0 rm --"

~~~
lloeki
To me that does not count as a duplicate, but as a usability and scalability
feature.

* "find ... -delete" probably just calls unlink, just as rm does, which is a libc feature. No subshell spawned thus lean and fast.

* "find ... -exec 'rm {}'" allows to run a command without xargs, but does not allows you to filter and transform the stream. Propably will spawn a subshell for each call.

* "find ... -print0 | sed 'bar' | grep -v foo | frobznicate qux | xargs -0 rm --" is where the power lies. A few more subshells created.

This, to me, shows that 'find' actually does what it should do, and allows you
to choose the best way you can do it WRT the task at hand.

~~~
rtaycher
and massive gui programs are?

------
RyanMcGreal
> For example, think of the Microsoft Office suite.

Beneath the covers, don't the Office applications share a lot of common code?

~~~
DaveMebs
Yes, but I would not think about it as code sharing. MSI (the installation
technology used by Office and most Windows programs) supports the notion of
"Shared Components," which allows applications to share common resources. A
component is a discrete installation unit, and in the case of large products
such as the Office suite, many components make up any one product (Word,
Excel, etc.). Rather than sharing actual source code or bundling the same DLLs
with each Office product, components are shared between the products. You can
envision that rather than each product containing the same code used to
generate tables, they all use the same piece of code when generating tables.
There are a variety of advantages, of which perfect interop between pieces of
software is by far the greatest.

More info on shared components:
[http://blogs.msdn.com/b/heaths/archive/2009/12/21/about-
shar...](http://blogs.msdn.com/b/heaths/archive/2009/12/21/about-shared-
components.aspx)

------
troyk
Employee transaction costs are not lower than contractors. Employers typically
pay vacation, sick pay, workers comp, health care, 1/2 social security plus
other payroll taxes, equipment, office space, and incur certain liabilities
for the employee.

~~~
tbrownaw
All those things still have to be paid for by _someone_ when you hire
contractors. Which is probably why I always hear about contractors getting
seemingly-absurd high hourly rates, if it _didn't_ all come out about even,
either we'd all be contractors or nobody would choose to be a contractor.

The _transaction costs_ here are things like all the negotiation that has to
go in to getting anything done with contractors every time you want something
done.

------
edderly
It can break down, but it's a damn good place start, especially if you don't
have a good philosophy to replace it.

Does it need stating how useful and powerful text based output can be?

~~~
oskarkv
50-minute talk on simplicity (simple meaning not compound) as a place to start
(and a little bit on Clojure): <http://clojure.blip.tv/file/4824610/>

------
vlisivka
> Piping the output of a simple shell command to another shell command is
> easy. But as tasks become more complex, more and more work goes into
> preparing the output of one program to be the input of the next program.
> Users want to be able to do more in each program to avoid having to switch
> to another program to get their work done.

I see no logic here. Can anybody explain that?

------
vlisivka
We can handle objects in CSV/TSV format with commands sort, cut, join, paste,
awk, perl, python, etc.

We can handle XML objects with commands xml_grep, xmlpath, xmlpatterns, perl,
python, js (JavaScript has builtin support for XML/XPath), etc.

JSON format in new, but we can handle it with js (native format), perl,
python. CouchDB already uses JSON over pipe to communicate with tools.

