
Unix doesn’t follow the Unix philosophy - prajjwal
http://www.johndcook.com/blog/2012/05/25/unix-doesnt-follow-the-unix-philosophy/
======
rjknight
I agree with some of OP's points, but I think he could explain the mechanisms
at work better. OP quotes the example of the 35+ command-line switches
supported by _ls_ , some of which are clear violations of the 'Unix
philosophy'. However, he moves on from this to suggest that users actively
_dislike_ using multiple tools to complete a task, and prefer a single all-
purpose tool even if it results in that tool becoming very complex. So users
would rather pass another flag to ls than learn how to achieve the same effect
by piping the output from ls to _grep_ , or _sed_ , or what-have-you.

I don't think this is the right way to look at it. Passing in additional flags
is the preferred option not because users can't comprehend multiple tools, but
because there's no discovery mechanism for finding out that using ls and
_grep_ together can achieve a particular outcome. The user's primary
reference, the man page, provides no guidance here. Googling might help, as
might asking a colleague, but by this point the cost of finding the answer
begins to exceed the benefit.

What Unix misses right now is a system to help users figure out how to combine
operations. Yes, it's obvious once you know how, but for beginners it's really
not obvious at all. The point is that there's nothing inherently difficult
about doing things the Unix way, it's mostly a problem of documentation and
discoverability. There's nothing that can compete with the single man page
listing all of the options for a single tool in one place.

This problem exists at higher levels too. Often web API documentation fails
because it only tells you how to use the API in question, when a tiny bit of
extra documentation on how to mash that API up with other popular APIs would
be much more beneficial. If you want people to adopt your tool, you really
need to tell them how to make it work with their other tools.

I guess this is another example of the same principle behind the success of
PHP - being 'right' is nowhere near as important as having documentation that
tells people how to solve relevant problems.

~~~
lubutu
I think I agree with you, that it's (for those who actually do like Unix) a
discoverability problem, but then I think that can be helped by simply
reducing the number of command line tools. If you have a small set of tools
you know extremely well, the pipelines necessary become immediately obvious
whenever you encounter a task... I think part of the problem is that many
modern Unix commands are non-orthogonal. It would be interesting to look at
ways to improve the discoverability of manpages, though, or at least have some
kind of well-structured "if you want to X, you could use Y" index.

~~~
rjknight
Our main problem is that computers have never been good at making suggestions
to users. "It looks like you're trying to..." Clippyism was a disaster, but
nobody ever came up with a better alternative. The trend on tablet computers
seems to be toward minimalism, solving the problem by reducing the set of
possible interactions to something easily comprehended by a user, but with
very little scope for creative combination of tools.

------
dexen
<http://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs> follows the Unix
philosophy. A lot of legacy has been shed. I can count 13 options to _ls_ , 11
options to _sed_ and just 5 to _sed_.

The standard Plan 9 shell, Rc, is described in mere ~500 lines of manpage,
while Bash takes whooping ~5400 lines.

Oh, and there is no `dll hell' in P9 :-)

~~~
telemachos
> A lot of legacy has been shed.

An example I remember from the UNIX to Plan 9 command translation page[1]:

    
    
        UNIX                       Plan 9
    
        head                       sed 10q
    

[1]: [http://www.plan9.bell-
labs.com/wiki/plan9/unix_to_plan_9_com...](http://www.plan9.bell-
labs.com/wiki/plan9/unix_to_plan_9_command_translation/index.html)

~~~
DeepDuh
Looking at this list I can't really tell why this is supposed to be better.
Shedding complexity ok - but getting rid of many shorthands that are there for
a reason? It reminds me of how Java wanted to replace printf with a
ridiculously complex string formatting system.

~~~
sp332
The point is that "head" on your system is not the same as "sed 10q". There
are two different codebases. By default head shows the first 10 lines but you
can change it with flags etc. Well why bother when you already have sed to do
that?

    
    
      head -n 5         sed 5q
    

For most cases, head is not a shortcut. 10 is just a special case.

~~~
DeepDuh
I agree with that change. I don't with many others.

------
udp
_> Why does ls have dozens of tangled options? Because users, even Unix users,
are not overly fond of the first two points of the Unix philosophy. _

No, ls(1) is flexible because the philosophy is to do one thing and do it
_well_.

ls(1) lists information about files. That's what `man ls` says it will do, and
whichever flags you pass, that's what it's going to do. Providing a lot of
control over _how_ exactly it does that is (imho) doing it _well_.

~~~
telemachos
> No, ls has dozens of tangled options because the philosophy is to "do one
> thing and do it _well_ ".

Do you mean that in order to truly do well its one thing (list information
about files), that ls _had to_ incorporate things like these?

    
    
          -o      List in long format, but omit the group id.
    
          -T      When used with the -l (lowercase letter ``ell'') option, display complete
                  time information for the file, including month, day, hour, minute, second,
                  and year.
    
          -x      The same as -C, except that the multi-column output is produced with entries
                  sorted across, rather than down, the columns.
    

Some degree of feature creep is probably inevitable, but a lot of the common
utilities now seem to lean towards "offer a flag for everything" rather than
"do a small number of permutations on one thing; create pipelines for the
rest". (Note: These three options are all from OSX's BSD-derived ls. GNU's
version is almost certainly more complex still.)

~~~
masklinn
> that ls had to incorporate things like these?

Or this bunch:

    
    
         -p      Write a slash (`/') after each filename if that file is a directory.
    
         -r      Reverse the order of the sort to get reverse lexicographical order or the oldest entries first (or largest files last, if combined with sort by size
    
         -S      Sort files by size
    

* -p's information is redundant with the _first_ character of the long format

* -S (and -r) are redundant with `sort` (ls -lSr -> ls -l | sort -k5; ls -lS -> ls -l | sort -rk5)

~~~
lmm
-p is immensely useful. Of course it's redundant in long format; it's for using in short format.

-r is also very useful (I use -tr all the time, which I can't immediately see how I'd replace with sort), and while I could replace ls - _l_ S with piping into sort, you seem to be assuming ls will _always_ be outputting one file per line, which is far from its most useful output mode when running it in a terminal.

~~~
masklinn
> you seem to be assuming ls will always be outputting one file per line

No, I am assuming a "do one thing" `ls` would be used in long format (or as
inode numbers) whenever it need to be worked with, and would be massaged into
form by further commands. For `t`, `r`, `S` and a bunch of others you'd ls,
sort and then cut/awk/whatever to format into a "short form".

Not that it matters a bit.

Also, I didn't claim these weren't useful, I claimed they were redundant.
Which they are.

------
adamtj
The idea that Unix is or should be a collection of small tools is only a
corollary to the real point, which is that Unix is not a machine, but a
language. A machine can only do what it has buttons for. A language can
express ideas that have yet to be thought. They are greater than the sums of
their parts. As with all languages, words get expanded and used in ways they
weren't intended. This is good. English is messier than Esperanto, and better
for it. Unix is a language where merely speaking an idea makes it reality. We
call this magic.

------
romaniv
_It’s a lot easier to maintain an orthogonal design when your software isn’t
being used._

Mentally, I'm translating this to "it's okay to have messy design if your
software is used". This is probably the most common excuse for writing bad
software.

Some things in Unix really _should_ follow Unix philosophy more closely.

~~~
raldi
I'd rephrase it more like, "Clean design is a means to pleasing users; it's
not an end in itself."

When is it okay to get messy? When it's a net win for the user experience.

~~~
romaniv
That's exactly what I'm talking about. "Pleasing users" is often used as an
excuse for horrible internal design. People cite examples like ls and proceed
to conclude that the Unix philosophy is a concept of abstract philosophy not
applicable to real life, when they _should_ conclude that there are simply
some issues with ls design.

Nearly _every time_ I hear "but it's okay to be messy, because it's real life"
or "I must do this bad thing to please the users" the person saying it
advocated something that is an insignificant short-term gain for a tiny
_subset_ of users at the expense of very significant long-term losses for all
the users.

Internal design usually does not affect end users directly, but it has
tremendous indirect effects in terms of performance and reliability, which
people do care about.

LS is a good example of this. Having some obscure flag might have solved a
particular problem in the past, but now everyone has to deal with extra
documentation, complexity and bugs.

~~~
raldi
That may be the case sometimes, but not always. Here are some examples of what
I think were good uses of messiness:

ls output can be sorted in all sorts of ways with just a few characters of
typing, even though it would be cleaner to make people use sort.

grep has a flag for counting lines that match, even though wc is supposed to
be used for counting lines.

rm can remove directories with the right flags, even though rmdir is supposed
to be the tool for removing directories.

I think those were all net wins for the user, both in the short term and the
long term.

------
dmethvin
> It’s a lot easier to maintain an orthogonal design when your software isn’t
> being used. Software that gets used becomes less orthogonal and develops
> diagonal shortcuts.

Yep, and that's not necessarily a bad thing. Human language is exactly the
same way. Irregular verbs, contractions, and slang are all diagonal shortcuts.

There may be a few unneeded adornments to some Unix utilities, I agree. But
remember that the `ls -l` that you might have typed in 1980 on a minicomputer
running BSD still has the same output as the `ls -l` from a PC running Linux
in 2012. That's just awesome backcompat right there.

~~~
andrewem
I really liked the phrase "diagonal shortcuts" from the OP. It brought to mind
a physical metaphor for what's meant by software having only orthogonal
functions: on a college campus they usually pave orthogonal sidewalks, and
inevitably you find diagonal paths where students have cut through the grass
and trampled it down to bare dirt. It's ugly in some sense, but it's what
everyone wants.

~~~
dmethvin
Yeah, a literal case of voting with your feet.

------
lsd5you
In my opinion this is a superficial view of the problem which misses the
fundamental point. The Unix shell tries to be 2 things and suffers as a
result. It is a) an interactive shell, optimised for quick navigation and b) a
programming api.

The programming api should be as simple as possible, but no simpler,
everything should be well structured and in long form and with typed
arguments. e.g. 'ls' would be 'listDirectory' and it would take arguments as
named parameters. It would also be directly bindable into programming
languages (rather than messing around with strings to 'execute'.

The interactive shell would be a program built on top of the programming api
(an api which does everything, not posix), and would contain all the
abbreviations and implicit type conversions. This would probably create more
choice/fragmentation/innovation than currently exists it would be more
practical to deprecate commands and arguments.

------
zhiel
The basic flaw in the Unix philosophy is that the rules 1. and 2. are
incorrect. The rules should be as follows:

1\. Write libraries that do one thing and do it well. 2\. Write libraries to
work together.

The third rule is correct, because that is what programs essentially are:
black boxes to modify input streams to output streams. I think this correction
should be followed in the future.

------
kamaal
Unix is basically a REPL for the C programming language. Its more than a
operating system, its actually a development environment. And most of the Unix
utilities that you see today are sum total of people trying to solve problems
they faced repeateadly. There fore what you see is tiny utilities with a lot
options each designed to solve both a operating system and a programming
problem. That is precisely what helps people to write almost real looking
programs by just sewing commands using the pipe.

I love the all the three points the author mentions, because together they
form a a very powerful platform to work with and solve problems.

>>Why does ls have dozens of tangled options? Because users, even Unix users,
are not overly fond of the first two points of the Unix philosophy. They don’t
want to chain little programs together. They’d rather do more with the tool at
hand than put it down to pick up new tools. They do appreciate the ideal of
single-purpose tools that work well together, but not too strictly.

I don't agree with this. I can tell you that I just love Unix the way it is
now. People feel they are messy because,as I said they always keep looking at
them as _commands_ and not as small programs that can be used together to
solve problems. Most people don't realize this, because they don't know how to
use it. It takes a lot of time to mature on the Unix platform. You have to
learn what the commands do, then you need to go out and learn how to use them
in combinations. Then you need to start learning the Unix way of representing
data and transforming data to solve problems.

Like always most people don't realize the fact that data is a actually 'Text'.
And not learning how to work with Text/Strings/Data generally creates all
sorts of other problems. Although projects which are RDBMS heavy don't
encounter them usually. But a lot of projects that are not DB heavy really
pass through frustrating patches of time trying to figure out easy ways of
representing, transforming and working with Data(Text). The most common format
which is heavily abused are XML's. Followed by JSON these days. But in my
experience if only they had learned how to use a little grep, cat, head, tail,
sed, awk and learned to represent their data properly they would save
themselves writing several hundreds to thousand lines of code. This problem is
especially prevalent in the Java world. They are plagued with writing endless
programs that actually do a subset of what sed or awk do. Its sad, but most
people express shock and surprise while shown how their 500 line Java program
is actually a one line sed command. Or how a simple awk command actually does
what they spent a whole day writing. Not understanding Unix forces you just re
implement Unix utilities badly!

Unix utilities were precisely invented to solve these kind of problems. In
fact if you find yourself solving a problem a particular way, then ideally you
should use a tool that can be programmed that particular way to solve the
problem.

A tool like sed is implicitly designed to open files, iterate line by line,
provide you with matching, manipulating, counting, printing abilities. It also
contains variables to hold data, and do all other kinds of manipulations. If
you were to keep writing programs in permutations of logic to perform each of
these facilities specifically as needed(Which is pretty common, if your data
is in files), you might as well just go ahead and learn sed. Because that will
ensure you not just save on writing code, but will also save you time.

Now this is just sed, Awk can go further. It can help you do a lot more stuff.
It gives you conditionals, more sharpened regexes, better ways of handling
field separated data, variables, functions and much more. Combined with stuff
like wc, less, cat, tr, xargs, grep, column etc you really get a full fledged
programming language(Without even realizing that you have one :)) in which you
can hack real fast without worrying about bugs.

The thing is 'Text' is actually data. And knowing how to play with Text and
learning how to manipulate it well helps you solve problems elegantly, quickly
and in a far more easier way than writing a specific program for each task.

As a final leap, you mature as a Unix programmer when you learn Perl. Its the
King of scripting languages. You master Perl, you begin to truly appreciate
the beauty of what it means to be a backend Unix programmer. Its a world whose
beauty and power is known to those who program in it. It completes the total
Unix equation and suddenly you recieve your enlightenement and realize why
despite Unix's messiness it still is the best thing in the world.

Worse is Indeed, Better!

~~~
bandy
Perl is a shining example of OP's complaint - it is the kitchen sink. Sure, it
allows you to do a ton of things, most all of which duplicate standard UNIX™
commands (those you'd find in PWB or v7), wrapped in a syntax that makes APL
programmers recoil in horror.

The argument the OP is making is that Unix has fallen into doing things The
DEC Way, with gigantic monolithic programs that not only perform their obvious
function, but have been turned into one of those twenty-pound Swiss Army
Knives that look pretty but no-one actually uses, because they're too heavy to
lug around. Does ls really need to emphasize its output with color? Better
yet, in a default scheme that's often near-unreadable in an xterm that's
running with its default colors? (I thank CentOS4/RHEL4 for this stunning
example) No, it doesn't. Looking at ls output in a rainbow of colors is just
as opaque to someone who hasn't read the man page (and grokked the system
config file(!) for ls) as are the old single-character flags (with the
exception of '/') that the F option gave back in '81 on BSD. Unix won over
VMS, LISPMs, TOPS-20, CMS, etc., and all of those people have migrated to our
platform, resulting in the world we have today.

~~~
kamaal
Isn't that the crux of why worse is better.

Turns out there is a huge mismatch between real world needs and text books
demands of elegance. Tools that seem to be massively monolithic are extremely
helpful in solving problems and do them far too well to do them away.

Every time I hear why some programming language is better than Perl, I always
see some Utopian dreams being displayed as the only thing that every matters
to a programmer and nothing else. In reality there are many competing demands
a programmer has. And providing them all generally leads to 'Swiss Army Knife'
kind of designs. Because you are trying to provide many things in a
package(That is the demand and the requirement at the first place). That sort
of design might not be text book elegant, but it is extremely helpful in the
real world.

There fore the 'beauty' and 'utility' design goals don't often match. This is
the unfortunate problem every programmer to face sooner or later. A programmer
has to abide by time, budget and some metric of quality every time.

Alas many of those demands don't lead to goals which may require an elegant
design.

In early days of Unix, Lisp programmers used to call the _problems_ Perl
programmers solve, itself as stupid. Telling the user why his needs are
invalid isn't a nice way to sell things.

------
conformal
this is a troll with very little substance.

summarized as "ls does not follow the unix philosophy", it points out that an
example of a tool that doesn't follow the philosophy as close as it could or
should. there are exceptions to every rule, what a substantive exposition.

ORANGE ALERT, send out the bubble!

------
blt
Linkbait article with hardly any meat on its bones.

------
papsosouid
Unix followed the unix philosophy just fine. Looking at current "unix-like"
software and the huge amount of bloat and cruft isn't a reasonable way to
criticize unix. Unix has been obsolete for decades, and the systems that took
over (BSDs, linux, GNU) didn't follow the unix philosophy very much (or at all
in the case of GNU).

~~~
unimpressive
> (or at all in the case of GNU).

What did you expect? RMS was a MIT Hacker. Which means that his background was
using Lisp, ITS, and Lisp machines.

~~~
papsosouid
I didn't say I expected GNU to follow the unix philosophy. I pointed out that
it doesn't, and that saying "unix doesn't follow unix philosophy" while
pointing at GNU tools for proof is absurd.

