
Unix Is Not an Acceptable Unix (2015) - l1n
http://mkremins.github.io/blog/unix-not-acceptable-unix/
======
quanticle

        Unix permits programs to communicate with one another, and with the user, 
        exclusively through character streams. You can’t write a function that 
        returns a list of files because the shell doesn’t know what a “list” is, 
        doesn’t know what “files” are, and couldn’t tell you the difference between 
        a “function” and a “program” if its life depended on it. Programs don’t 
        “take arguments” and “return values”, they read characters from stdin and 
        print characters to stdout!
    

And this is why I consider PowerShell a more Unix-y shell than bash.
Powershell commands don't just operate on character streams. They operate on
CLR _objects_. For example, the PowerShell equivalent of grep, Select-String,
doesn't return a string, or even a list of strings. It returns a list of match
objects, which contain the regexp used, the matching string, and the file that
matched. PowerShell often gets dinged for its verbosity, but honestly, I find
it much easier to compose PowerShell commands because I know exactly what each
command does, and the command-line flags are mostly intuitive. Meanwhile, Unix
commands are shorter, yes, but it's a lot more difficult for me to remember,
e.g. that if I want to run a command across a number of files, I have to pass
the command to find.

~~~
milesrout
That's completely orthogonal. GNU bash has nothing to do with what arguments
you can pass to GNU ls. You could use bash to run notepad.exe just as you
could use Powershell to run GNU ls.

If that's not the case, if Powershell actually does integrate most of what are
on Unix-like systems separate commands, then that's terrible.

>Meanwhile, Unix commands are shorter, yes, but it's a lot more difficult for
me to remember, e.g. that if I want to run a command across a number of files,
I have to pass the command to find.

You don't have to do anything of the sort. If you want to run things across a
number of files, you need a list of those files, and you need to know what you
want to run across them. That list of files can come from anywhere. Does it
come from a text file? A script? Or find? Doesn't matter.

If you do get them from find, you run things on them in precisely the same way
you run things on any other list of files: a for loop.

    
    
        for file in $(find ...); do
            rm $file
        done
    

Yeah you can also do

    
    
        find ... -exec rm {} \;
    

or

    
    
        find ... -print0 | xargs -0 rm
    

or any other number of other ways of doing it. But you don't have to. I'd
probably use xargs to delete those files, honestly.

~~~
BlackFingolfin
Of course one any of those filenames contains a space, your first two examples
break down, and in the worst case result in the wrong files being deleted. Of
course one can work around that, but in reality, this quickly gets difficult,
and often is forgotten. The print0 works for some cases, but not in general
(e.g. when one needs to do more complex processing)

~~~
milesrout
The second one doesn't fail, and the first one only fails because I wrote `rm
$file` instead of `rm "$file"`, which I have done every single other time I've
written the same thing this century.

~~~
uzoodoo
Writing in first one won't save you since "in" splits by whitespace, not
newline. If you really have to do that, use "find ... | while read file; do"
instead.

------
coffeemug
Deciding whether a program does "one thing well" is a matter of granularity.
`ls` with its 38 flags is absolutely tiny compared to, say, finder, which
displays multimedia previews, allows you to connect to remote servers, and
probably does a bazillion other things.

Yes, `ls` isn't _perfectly_ clean in a mathematical sense, but it's still tiny
compared to modern software, and passing a couple of flags for common tasks
now and then is far more convenient than writing `ls | map ... | sort ...`.

Unix is not a mathematically perfect Unix, but it is most definitely an
acceptable one.

~~~
vacri
The article hinges far, far too strongly on the "do one thing" part of the
definition, and largely ignores the "do it well". The optional flags to 'ls'
are there to help "do it well".

If we're going to use the overzealous puritism of the article's definition,
why not go further? My keyboard driver isn't unix-ey, because I can use it to
input a 'k'. Or a 'K'. Or a lot of other characters. The argument in the
article would mean that for a keyboard driver to be unix-ey, it should only
output a single character. Nevermind the fact that I have 104 options on the
HID at my fingertips, if my driver allowed that, it wouldn't be "doing one
thing"!

~~~
fenomas
> My keyboard driver isn't unix-ey, because I can use it to input a 'k'. Or a
> 'K'. Or a lot of other characters.

You're missing the article's forest for its trees. The author is claiming that
"filter or sort the items in a list" and "get a list of files in a folder" are
two different "things". That's not a frivolous, hair-splitting thing to say -
the difference between generating a list and re-ordering it should be apparent
to anyone who writes code.

After all, when you write a function that returns an array, do you typically
have it take in a boolean argument for whether or not to reverse the array's
contents?

~~~
flukus
> After all, when you write a function that returns an array, do you typically
> have it take in a boolean argument for whether or not to reverse the array's
> contents?

And the granularity of unix is much more course than a single function. Have
you never written a program or application that returns the array and sorts
it?

~~~
Someone
Yes, but I preferably compose it of two functions: one that gets the data and
one that sorts it.

Only rarely that isn't the right thing to do. For example: if the data comes
from SQL, I tend to do the sorting in SQL, it I see that as a problem with
SQL: it composes badly.

~~~
flukus
Is there anything stopping you from doing that with current unix tools? You
can do that, I'll continue using the more pragmatic option.

~~~
Someone
There isn't, but if you see the base system as the library to build your tools
on, a library function to get all files in a directory and, optionally, sort
them by name, doesn't seem like a good design.

------
charles-salvia
What's funny is that some modern scripting languages like Python, with things
like the "os" module, are _almost_ passable as a general purpose shell
scripting environment. I actually tend to write Python scripts instead of bash
shell scripts for one-off file manipulation scripting, mostly because even
after a decade-plus long career, I've never become much of a wizard at bash
shell staples like sed or awk.

But it's just that a general purpose programming language like Python doesn't
treat executing subprocesses in enough of a first-class manner, in the way the
bash shell does, to make it feel like a seamless shell-scripting experience.
You still have to go through the motions of "import os", etc. just to get
started, and you can't directly execute other programs, instead having to use
"system" or the more cumbersome subprocess module. But still, it's....
tantalizingly close to something like a "better" shell that includes support
for lists and other data structures instead of just "everything is a string".

~~~
theonemind
The near-miss of Python is why I write perl scripts. It supports the
'backtick' notation, and pretty much kills the problem domain of awk, sed, and
bash.

~~~
tannhaeuser
I can see where you come from, but try to run some obscure nontrivial Perl
package of the 90s and see how far it goes. I had the displeasure to exhume
legacy Perl programs three times in the last two decades. It always ended with
a total rewrite in another language. In contrast, awk is such a nice and
minimalist language with multiple implementations available.

------
TheOtherHobbes
"Unix considered harmful"

I think "a demonstrably inhumane environment" may be my favourite description
to date.

There are many terrible assumptions in the design of the shell. Not least:

Use short commands (because they're quicker to type in theory, even though
you'll waste time huge amounts of time in practice looking up commands and
switches, because even the average god-like greybeard can't remember more than
a few tens, and beginners have no chance)

Use terse switches (ditto)

Text is universal and computers are text processors (not true back then, even
less true now)

Make no attempt at standardisation (switches are effectively random between
different commands)

Quick hacks are better than a smart and structured environment (this was
always bullshit and has caused endless grief, not just in Unix, but in
everything touched by Unix)

Software forks are good (not in the shell, they're not)

And now computing is buried under the dead weight of accumulated cruft and
poor theory and practice - probably forever.

But that isn't even the real problem. The real problem is that research into
humane OS design seems to have stalled. CS academics could band together to
design something smarter, better, and friendlier than the current shell. But
there seems to be zero interest in starting a project that would generate huge
productivity and reliability gains and also make low-level computing more
accessible to ordinary users.

I find that baffling, and actually rather disturbing.

~~~
adrianN
There are shells other than SH and derivatives. They're not terribly popular.

~~~
oblio
That's the IE/Safari/Android/vi problem: defaults.

------
MikeTaylor
It seems that the lessons of Rob Pike's 1983 talk "cat -v Considered Harmful"
have still not been absorbed 34 years later. Here is the abstract of the talk,
together with the link to the paper (co-authored with Brian Kernighan) that
evolved from it: [http://gaul.org/files/cat_-
v_considered_harmful.html](http://gaul.org/files/cat_-
v_considered_harmful.html)

~~~
pjmlp
I is kind of interesting that FOSS community argues for the _spirit of UNIX_ ,
when its creators all agree that UNIX as OS model is long overdue.

~~~
flukus
Where are the better options? There are plenty of warts, as you'd expect from
something that old, but dealing with those warts (or beauty spots) is still
the best option I know of.

~~~
mfukar
The better options are struggling for support while the rest of us pour our
money on the problematic UNIX, hoping we can 'deal' with its warts. In one
word, inertia.

------
Animats
There was a time (PWB Research Unix) when one typed:

    
    
        ls | sort | mc
    

to get an alphabetical list in multiple columns. That didn't last.

This reads like someone discovered functional programming and now wants
everything to be functional.

~~~
oneeyedpigeon
Isn't that kinda the point? Why would that be a bad thing? Seemingly, my
version of ls doesn't provide a 'sort alphabetically' flag, but if it did, the
equivalent of

    
    
        ls | sort | mc
    

would be something like

    
    
        ls -C --sort=name
    

and, as discussed, the ls program has to be much, much more complicated.

~~~
Animats
UNIX pipeline data is mostly raw strings. To do anything non-trivial, you have
to know the format of the previous pipeline item and have a parser for it. It
would be amusing to have a system where all commands output JSON, so that
pipelined programs can process the previous input. Then you could have

    
    
        ls | sort --key=modification_time | mc
    

For display, you'd have a convention that if there's a "text" field in the
JSON, that's what's displayed in the console window. Otherwise you see the raw
JSON. So "ls" would output an array of JSON records with all the usual fields,
but if you just wrote

    
    
        ls
    

the user would see a list of file names.

You could have "format" and "select" programs:

    
    
        ls | select --exclude="owner=root" | sort --key=modification_time | format  
            --fields=name,modification_time,size 
    

This would be quite powerful and would allow the creation of complex,
functional pipelines. The end result is basically equivalent to SQL, but it's
functional!

~~~
sbierwagen
This was the point of Powershell, where programs pass around objects, rather
than plain text:
[https://en.wikipedia.org/wiki/PowerShell#Pipeline](https://en.wikipedia.org/wiki/PowerShell#Pipeline)

~~~
Animats
Cute. So, while Unix is not an acceptable Unix, maybe Windows now is.

------
thomastjeffery
Unix is a poor implementation of the Unix philosophy. Just like QWERTY is an
awful keyboard layout, etc. This is yet another case where we are steeped in
tradition. It would be fantastic if we could just instantly overcome these
shortcomings, but it's difficult enough to make an OS compatible with the ones
we already have. With manufacturers like Nvidia and Broadcom keeping such a
tight control over their drivers, it's difficult enough to even make an OS!

It would be fantastic to have a more cohesive environment, especially in (and
including) the shell, but most people are happy with the status quo.

Even worse than *nix tools, there is really no environment outside the realm
of text-based UI that follows the Unix philosophy. A GUI that does just one
thing, let alone well? Ha. AFAIK, there is no existing way to create GUI
software that can cohesively inter-operate with the expressive power that
pipes and subshells give command-line tools.

~~~
alkonaut
With unix that's exactly the mistake - what stops us from changing how some of
the standard tools work in unix? They are used in existing scripts!

Making the same set of tools be interactive (so we need them to be concise,
simple) and used in scripting (so we need them to be constant over time) has
led us here. They now form a programming API instead of a set of tools that
each do one thing well.

------
MichaelMoser123
I think the unix design philosophy is great in theory, but in practice you get
a lot of parts that have to be combined, this combination then gets tricky, so
you will want a utility that just covers all the edges.

Lets see, I often use ls -tr

* the good thing about this is that it is very fast to type

* the bad part is that you have to memorize the incantation.

You could probably do it by a pipe (ls | sort <now it gets complicated here>)

* good: the good part is that every program does its job

* bad: also two programs have a larger footprint than one

* good: you can have it as an alias to make it short again

* bad: aliases can't be used in bash scripts, unless you explicitly source the file that defines the alias

* good/bad: you just shifted complexity from ls to sort

....

~~~
nailer
> ls | sort <now it gets complicated here>

In Powershell, from memory:

    
    
      ls | sort -property time
    

[https://msdn.microsoft.com/en-
us/powershell/reference/5.1/mi...](https://msdn.microsoft.com/en-
us/powershell/reference/5.1/microsoft.powershell.utility/sort-object)

~~~
flukus
"man ls" in bash showed me what I need, doing the same in powershell gave me
some nonsense about Get-ChildItem.

~~~
ksk
What you read might not work on another UNIX. BSD has its own semantics making
writing cross platform UNIX scripts (depending on how far you stray from
POSIX) a pain. Though I admit, its not fair to compare it to a single vendor
system like NT. :)

~~~
flukus
Wouldn't the man pages for bsd be updated to reflect any changes? I'm not sure
if PowerShell​ will fare much better when it comes to program inconsistencies.

------
f2f
Back in 1983 the authors of UNIX were already complaining about the
proliferation of incompatibilities. As far as "true unix" is concerned nothing
this paper refers to is considered "true unix".

    
    
        It seems that UNIX has become the victim of cancerous growth at the hands of
        organizations such as UCB. 4.2BSD is an order of magnitude larger than Version
        5, but, Pike claims, not ten times better.
    

[http://harmful.cat-v.org/cat-v/](http://harmful.cat-v.org/cat-v/)

------
glangdale
What always gets me about this class of articles is that they are a classic
case of bikeshedding. Is anyone _really_ having that much trouble with ls?

Yes, things like ls could be better - but is switching between "ls --plus -a
--pile --of --options" and "ls | one | damn | filter | after | another" a
particularly core concern?

I tend to agree with this bit... "Rather than re-evaluating the Unix command
line with an eye towards improving its usability under the greatly relaxed
technological constraints of modern hardware, we’ve written terminal emulators
that faithfully reproduce the constraints of the mid-1970s",

... but feel that the author decided that it was a lot more fun and easy to
complain about ls instead of properly exploring this idea.

~~~
ygra
Unless Unix dies a swift death for some reason or another, going forward there
will be many more people who have to _learn_ this than there are that _have
learned_ it already. Surely, once you're using those things on a daily basis
it doesn't matter how inconvenient, illogical, or badly designed things are,
you just know how to do them right. Is that a good reason to scoff at anyone
who points out that things _are_ badly-designed? Backwards-compatibility seems
a far better argument in favor of keeping things like they are than »it
doesn't bother me much in practice«.

------
bigger_cheese
I think in some ways using flags to modify the behavior of a command is an
elegant solution. The alternative is to rewrite a new version of "ls" for each
sort of functionality you'd like.

Without flags you'd need "ls1", "ls2" etc. like you do with system calls ala.
dup()->dup2()->dup3(), pipe() -> pipe2() etc.

Linux has gone further than some *nixs and started using flags to extend
system calls as well see:
[https://lwn.net/Articles/585415/](https://lwn.net/Articles/585415/)

~~~
JasonSage
The alternative is not to write multiple `ls` commands. The alternative is to
let `ls` list directory contents and to leave filtering, sorting, or
customizing the output display to a separate utility which specializes in
those.

~~~
Rotten194

        ls | filter-hidden | sort | add-directory-slashes | mark-symlinks | mark-executables | colorize | display-in-grid
    

This seems really inconvenient, but who cares because it's ideologically
sound!

~~~
JasonSage
You could make it as convenient as existing flags are, only you wouldn't have
to learn a different set of flags for every cli. No need for sarcasm.

~~~
flukus
How would you make it as convenient as existing flags?

~~~
JasonSage
Not sure if you're being serious or not, but if I had to do it today, instead
of:

`ls | filter-hidden | sort | add-directory-slashes | mark-symlinks | mark-
executables | colorize | display-in-grid`

I'd start with this:

`ls | nodot | sort | slash | dec | c | grid`

I imagine there's an even better way, I just haven't spent very much time
thinking about it.

There's an argument that this could be even more convenient than existing
flags—`slash` could work on output from other commands, like `find`—only I
don't have to pull up a man page for `find` since I'm not as familiar with it
as I am with `ls`. That's the idea behind this kind of composability.

If this is actually too much typing or you're not happy with the defaults, a
few bash aliases would go a long way towards making this shorter. For
instance, you could alias `ls` such that it colorizes, sorts, and hides
dotfiles by default.

~~~
pjc50
How many of those filter systems would be useful for things that aren't ls?

Also, these days the cost of doing an extra fs lookup + stat call for each
file in each filter is pretty small, but if you want to run that pipeline in a
directory of >100k files it might be noticeable.

~~~
JdeBP
The cost of the extra lookups goes away entirely if one then extends the
design in the ways already discussed elsewhere on this very page. Instead of
the pipeline passing along a stream of bytes, which every program then parses
into a stream of filenames, the pipeline passes along a stream of _objects_.

Then "sort" becomes a utility that sorts a stream of objects by a chosen set
of fields; "slash" becomes a utility that inspects the "file type" and "file
name" field of an object and makes an appropriate "display name" field; and
"ls" becomes a utility that outputs a stream of objects that have "file type"
and "file name" fields, straight out of the directory entry. "ls" is the only
part of the pipeline that looks at the disc.

------
upofadown
>...it was developed by and for a small group of highly specialized experts in
an environment where every keystroke, every character displayed on the screen,
came at a real and meaningful cost.

Have keyboards gotten better/faster? How does the existence of a 80 character
line length change the best way to specify a directory listing?

This seems to be based on the logical fallacy that since the old thing is bad
that a better new thing must be possible.

------
nsb1
The arguments of ls are simply an optimization for things which are done so
frequently that simply typing them over and over becomes a huge waste of time.
In the early days, you can imagine people first typing everything out
manually, then creating macros like 'ls | awk {stuff} | sort -k2 |..." and
then just getting fed up with your .xxxrc file and modifying the ls source to
trim it down to a couple of arguments.

Taken to extremes, this article seems to suggest that all command line flags
for every executable ought to be replaced with separate executables and pipes.
Just imagine how many files would be in /usr/bin, and how many of them you
would have to remember just to do useful work every day. We'd have to increase
inode limits just to hold all the manpages. An OS like this would not be one
I'd like to use.

------
tempodox
Lisp machines once had all it took to make better interfaces but were wiped
out by the worse-is-better logic behind Unix. Now the weight of all the stuff
we built on top of that makes it harder to change. And almost everyone still
subscribes to worse-is-better, albeit under a different name.

~~~
progman
McCLIM is an attempt to revive the awesome GUI of the Lisp Machines. There is
no need to invent special data interchange formats like XML or native blobs
since Lisp uses simple lists as AST.

[https://common-lisp.net/project/mcclim/excite.html](https://common-
lisp.net/project/mcclim/excite.html)

------
huherto
I do a lot of things from the command line. But as soon as I have a few lines
and I need to write a script I just switch to Perl(which I know better than
Python). It just seems easier. I still call unix commands but Perl seems
better to glue them together.

------
progman
What's the problem with all those options? IMHO ls is not bloated at all. It
simply follows the Unix philosophy: Do just _one_ thing, and do it "right"
which means: do it as well as possible. ls actually does that. I never had any
desire for another ls because ls already gives me almost all possible
functionality. I am glad that I have all those options so that I don't have to
write my own tool to get a special function if I really need that.

If you don't need all the options, ignore them. If ls is too complicated for
you then do just as the author recommends: "write a simpler alternative to
ls", and name it lf ("list files") or the like.

------
bjourne
> Unix permits programs to communicate with one another, and with the user,
> exclusively through character streams.

It communicates using byte streams. For the original authors of Unix where
ascii ruled supreme, it didn't make a difference but now it does. For example,
see this stack question: [http://unix.stackexchange.com/questions/167814/get-
first-x-c...](http://unix.stackexchange.com/questions/167814/get-first-x-
characters-from-the-cat-command)

------
faragon
Unix is OK. That article is bullshit, in my opinion.

