
Unix is not an acceptable Unix - lfpa2
http://mkremins.github.io/blog/unix-not-acceptable-unix/
======
useerup
Interesting examples, especially in light of the recent debates about Unix
shells and PowerShell.

In PowerShell, the `ls` command _is_ a simple command that produces output
without flags to control output format, ordering or filtering. Like with
higher order functional programming, those tasks are delegated to cmdlets like
Format-List, Format-Table, Select-Object, Where-Object.

Some PowerShell examples:

To list the files and directories (children) of a directory:

    
    
        ls
    

To get just the names or the full path:

    
    
        ls | select Name
        ls | select FullName
    

To discover _which_ properties are available from `ls`:

    
    
        ls | gm
    

To select only files larger than 200MB:

    
    
        ls | ? Length -gt 200MB
    

(ls is alias for Get-ChildItem, gm is alias for Get-Member, ? is alias for
Where-Object)

It is somewhat ironic that PowerShell is more true to the principle of "do one
thing and do it well" than the Unix shells.

~~~
crdoconnor
My one wish for UNIX coreutils would be that they all had a switch to output
JSON.

ls --json

ps aux --json

If they did that, all that powershell stuff would become trivial.

Wouldn't even be hard.

~~~
oinksoft
Ten years ago this would be --xml. Who knows what it would be ten years from
now. Text streams are timeless.

~~~
alkonaut
Text streams are just an ad hoc structured format. Even an out of date format
is better than one you have to invent or decode on a per case basis.

The whole xml vs json debate feels pretty pointless, they are for all
practical purposes equal (in size, complexity etc). Sure xml has some weird
design decisions around namespaces and so on, but if you use it as a simple
hierarchical markup it's pretty much equivalent to json with different
brackets? and often easier to parse for humans because of closing tags instead
of }}} and of course, comments.

The xml-hate I think isn't really xml-hate it's the reaction against the xml-y
things from 10-15 years ago: the IBM/Struts/Blah we all had to wade through.
These days it feels frameworks pick json over xml even when it's clearly an
inferior choice (such as for config files). Json is an object/message markup,
not a good human readable config format.

~~~
crdoconnor
XML hugely complex and difficult to parse despite not being any more useful
than JSON as a means of representing or transmitting data.

YAML is a far superior format to XML for config files, and is basically JSON
in a readable format.

There's literally zero reasons to choose to use XML where you could use
JSON/YAML instead.

~~~
vezzy-fnord
YAML may look quite simple on the outside, but the specification of it is
surprisingly complicated.

------
andmarios
As I see it ls does one job (list directory contents) and does it well (by
supporting 37 flags).

If ls was a human, we would call him an expert.

Since unix tools expect input as plain text, the flags that ls supports are
absolutely useful and I believe this is obvious to anyone who has written more
than 50 lines of bash. When your input doesn't support a standard complex
structure (e.g json with some standard set fields), claiming that “sort by
creation time” is a simple filter is funny.

First you would need a ls flag so that ls will output creation times in an
easy to process time format (seconds since epoch), then you would need to ask
sort to work on a certain (1 flag) numerical (1 more flag) field. Last you
would have to pass sort's output to a stream processor to get rid of unwanted
fields.

Current “list files by creation time”:

    
    
        ls --sort=time --time=ctime
    

Replace sort flag with filters:

    
    
        ls -l --time=ctime --time-style="+%s" | sort -n -k 6 -r | awk '{ $1=$2=$3=$4=$5=$6=""; print substr($0,7) }'

~~~
joepvd
Nitpick, sorry. `ctime` is not Creation Time, but Change Time. The traditional
*NIX file systems do not have a meta data field for creation time.

~~~
andmarios
Thank you; I naively assumed since we have modification time (mtime), ctime
would stand for creation time. :)

------
pjmlp
Yes, there is this strange idea in new generations that GNU/Linux is UNIX,
without having tried DG/UX, Tru64, Solaris, HP-UX, Aix, Irix and many others.

Just drop a GNU/Linux user into a default HP-UX installation and watch them
getting around the system. Hint, those nice GNU flags and utilities aren't
there.

As for what UNIX turned out to be, I think Rob Pike as one of its creators is
a good quote:

<quote>

I didn't use Unix at all, really, from about 1990 until 2002, when I joined
Google. (I worked entirely on Plan 9, which I still believe does a pretty good
job of solving those fundamental problems.) I was surprised when I came back
to Unix how many of even the little things that were annoying in 1990 continue
to annoy today. In 1975, when the argument vector had to live in a 512-byte-
block, the 6th Edition system would often complain, 'arg list too long'. But
today, when machines have gigabytes of memory, I still see that silly message
far too often. The argument list is now limited somewhere north of 100K on the
Linux machines I use at work, but come on people, dynamic memory allocation is
a done deal!

I started keeping a list of these annoyances but it got too long and
depressing so I just learned to live with them again. We really are using a
1970s era operating system well past its sell-by date. We get a lot done, and
we have fun, but let's face it, the fundamental design of Unix is older than
many of the readers of Slashdot, while lots of different, great ideas about
computing and networks have been developed in the last 30 years. Using Unix is
the computing equivalent of listening only to music by David Cassidy.

</quote>

Taken from [http://interviews.slashdot.org/story/04/10/18/1153211/rob-
pi...](http://interviews.slashdot.org/story/04/10/18/1153211/rob-pike-
responds)

~~~
FooBarWidget
> Yes, there is this strange idea in new generations that GNU/Linux is UNIX,
> without having tried DG/UX, Tru64, Solaris, HP-UX, Aix, Irix and many
> others.

I think that it's not so strange, seeing that you have to fork over lots of
$$$ to get your hands on hardware that would run Solaris/HP-UX/AIX/etc, plus
licensing fees. You can't easily rent a cloud server for an hour to play with
it (with some exceptions, like SmartOS on Joyent). The high barrier to
tinkering reduced proprietary Unices to expensive niche environments that are
willing to pay lots of cash.

~~~
justincormack
IBM offers free access to AIX machines[1]. Joyent is the easiest way to try
Solaris.

[1]
[http://www-304.ibm.com/partnerworld/wps/servlet/ContentHandl...](http://www-304.ibm.com/partnerworld/wps/servlet/ContentHandler/stg_com_sys_power-
development-platform)

------
icebraining
I found this a weird post. The shell does know what files are (how else could
you do "echo * "?) and programs do take arguments and have return values.

In any case, I think the author is kind of tilting at windmills; as far as I
know, that Unix and Linux don't really follow the Unix philosophy is
reasonably accepted today, even if it wasn't back in '83, when Rob Pike and
Brian Kernighan made the presentation called _" UNIX Style, or cat -v
Considered Harmful"_, which pointed out exactly those problems, which were
already in development:

    
    
      Bell Laboratories
      
      Murray Hill, NJ (dec!ucb)wav!research!rob
      
      It seems that UNIX has become the victim of cancerous growth at the hands of
      organizations such as UCB. 4.2BSD is an order of magnitude larger than Version
      5, but, Pike claims, not ten times better.
      
      The talk reviews reasons for UNIX's popularity and shows, using UCB cat as a
      primary example, how UNIX has grown fat. cat isn't for printing files with line
      numbers, it isn't for compressing multiple blank lines, it's not for looking at
      non-printing ASCII characters, it's for concatenating files.
      
      We are reminded that ls isn't the place for code to break a single column into
      multiple ones, and that mailnews shouldn't have its own more processing or joke
      encryption code.
      
      Rob carried the standard well for the "spirit of UNIX," and you can look
      forward to a deeper look at the philosophy of UNIX in his forthcoming book.
    

Yet, Rob Pike himself uses Linux nowadays - the problem today isn't so much
acceptance, but backward compatibility.

------
soyiuz
I think the author misses the point. As far as I see it, Unix tools are
optimized for one thing: the human user. If `ls` is bloated, it is only
bloated from the system point of view. As a user, I only see a small command
that is quick and consistent (to a reasonable extent) with other tools. I
don't need to know about 39 flags--that complexity is revealed with time, as I
need to accomplish more complex tasks. For now, I only use and "see" two or
three. And if more is needed, the program is small enough to be reasonably
well documented. `man ls` perhaps with `grep` is all most people need.

This brings me to the second point. "Streams of text." Just like `ls`, streams
of text are an optimal format/convention for humans. Many other things are
better at being more compact or more efficient etc. But as formats and
conventions proliferate, streams of plain text remain: readable and universal.
Humans will ALWAYS be able to work with text. It is something that all humans
kind of agreed upon--which cannot be said for any other formats or standards,
which can offer various technical benefits at the expense of longevity,
universality, and readability.

These two features (`ls` being relatively light and text streamy) leads to the
"bootstrapping" effects sought after by the first generation of Unix
developers. Learning about pipes and filters in one part of the system is
applicable to all others. These tools scale with your level of expertise. They
grow with you because despite the small quirks, there's a remarkable
consistency of interface: text! Consequently `ls` (along with many other core
tools) is implemented in the same way across a staggering variety of
platforms. It has survived decades of alternatives touted as "better,"
"faster," "more usable," etc. etc. etc. That is the remarkable achievement of
*nix / GNU etc. approach to creating human-centric software. As we architect
ever more complex systems, we would do well to understand why and how Unix has
endured, warts and all.

~~~
iopq
No, I'd rather be working with FILES, not paths to files.

This is how you get `rm -rf $STEAMROOT/*` problems

------
adambatkin
This article gets it all wrong: The ls command does one thing well (LiSting
files) and it isn't optimized for terminal size, it's optimized for PROGRAMMER
productivity. The ls command gives you a default view that is exactly what you
want most of the time and is easily copy/pastable for other uses (it's a
single column) or more importantly, piped to some other program for further
processing. And then there are easy-access options for common tasks (extra
details, hidden files, sorting by attributes, etc...)

Powershell largely does exactly what the author is talking about. And yet it's
completely unusable for interactive use (though it's great for writing longer-
lived automations).

------
mangecoeur
I think there's a bit of a disconnect between how people talk about/defend the
unix shell and the reason it's actually still there. People often defend the
simplicity or philosophy - despite, as the article points out very well, much
of it being clearly poorly designed.

But really the reason most of it exists unchanged is simply path-dependancy -
we used it then so we use it now. It works well enough that there's not enough
incentive to drive a large change. There are dozens of different shell-like
environments and scripting languages, but they never really took over out of
plain intertia. That, and the conservative attitudes of people defending their
Unix traditions (related in no small part to how much effort they put into
learning the thing in the first place). For new people arriving on the scene,
its still less effort to learn things, even when they get truely arcane, than
to use something which relatively few other people use or even roll your own
alternative...

------
ised
I have never thought that UNIX approach is the "best" in a universal sense.

But when compared to how large, slow, complicated and opaque the
"alternatives" are, UNIX is the clear choice for me.

I can modify and recompile UNIX to meet my own ends. That is all but
impossible if I chose an alternative such as Windows.

For example, if I do not want ls to have 30+ options, I can trim it down to
just a few options and recompile.

There are other utilities besides ls for viewing file information, e.g., BSD
stat(1) or mtree -cp. The later displays mode information in octal which is
something ls, despite its 30+ options, does not do.

Or I can write my own simple utility. I am given the full UNIX source code.
Where is the source code for Windows?

Personally I keep my filenames short and never use spaces, so I sometimes use
the shell's echo builtin and tr(1) to get a quick list of files.

    
    
       echo * |tr '\040' '\012'
    

If there were non-UNIX alteratives that were small, simple and transparent,
perhaps I might not be using UNIX.

Because I have become very comfortable with UNIX, any alternatives that others
suggest have to be comparable with UNIX on size and simplicity before I will
take them seriously.

Currently, I use a kernel source tree that compresses to under 40MB; I can
compile kernels with about 200MB of RAM and fully loaded kernels are about
17MB. Userland utilities are usually around 5MB as I prefer to put them in the
kernel's embedded filesystem. I do not like to rely on disks. My "IDE" is the
same system I am compiling. There is no GUI overhead, everything can be done
in textmode. The importance of the preceding two sentences cannot be
understated.

I am always willing to consider non-UNIX alternatives that can offer the same
or better flexibility, size constraints and simplicity.

But after decades of being open to alternatives, I am still not aware of any.

~~~
throwawaykf05
Why do you need the source code to write a utility when all you need is an
API? I've written software for Linux and Solaris but never had to look at the
source code, so I imagine it's the same for Windows.

~~~
ised
"... you need the source code to write a utility..."

Where in the comment is this statement?

Personally the primary reasons I would want the source code for the kernel and
utilities would be 1. to assess its quality and, assuming the quality meets my
standards, 2. to modify it to meet my own ends. In my case, the less I have to
write things from scratch the better.

Let me know if you still have questions.

~~~
throwawaykf05
_> Where in the comment is this statement? _

_> Or I can write my own simple utility. I am given the full UNIX source code.
Where is the source code for Windows?_

I agree with your reasons, it is useful to have the source code. Just saying
tat in the many, many years I've been writing software that's deployed on
Linux, I've never had to look at it.

------
mwcampbell
The essay "Free Your Technical Aesthetic from the 1970s" [1] by James Hague is
appropriate here. See also The UNIX-HATERS Handbook; much of it is outdated,
but I think some of it is still relevant, particularly the part on the
shortcomings of pipelines.

I think it's unfortunate that even if a good open-source PowerShell clone is
developed (or PowerShell itself is open-sourced), it will probably never gain
widespread acceptance outside of Microsoft shops. Tribalism so often trumps
actual merit. But if someone did a PowerShell-like thing on top of Node.js, I
could see that taking off, particularly if it were integrated with the Atom
editor and its package manager.

[1]: [http://prog21.dadgum.com/74.html](http://prog21.dadgum.com/74.html)

~~~
bishop_mandible
> But if someone did a PowerShell-like thing on top of Node.js, I could see
> that taking off

I think it should also use React.

------
vanilla
I can't follow the comparison.

Linux is just a Kernel, most distros use the coreutils package which is
developed/maintained by GNU [1].

So any comparison is not between Linux and Unix, rather the GNU part in
GNU/Linux

[1]:
[https://www.gnu.org/software/coreutils/coreutils.html](https://www.gnu.org/software/coreutils/coreutils.html)

~~~
justizin
Right, and OS X is based on FreeBSD.

------
sigsergv
Linux is not Unix, that's just fine, I'm happy it is not. I've tried “re-
evaluated” shell (Powershell), no, thanks, it's not intended for using by
humans. Currently shell is still simple and powerful. Powershell is powerful
and NOT simple.

~~~
UK-AL
What does intended for humans even mean? At least powershell has command names
that describe what they do.

~~~
dijit
Powershell is readable but barely writeable.

Bash is the opposite, it's very easy to do complex things if you have some
basic knowledge.

for instance a one liner nested loop to replace a string in a bunch of text
files which are a few directories deep. Easy to write but it takes a while to
read.

------
chris_wot
Someone needs to port PowerShell to Unix :-)

I'm confused about the response to his tweet though - ls has a lot of
switches, but I can't really see any that don't do something with listing
files.

And then the response is "Unfortunately, the lineage is _not_ Unix -> Linux.
It is Unix -> Plan 9. Linux doesn't follow Unix philosophy :-("... Except
Unix's ls eventually had 19 options, and the ls used by Linux is a port Gnu's
coreutils.

Nevertheless, I tend to agree. ls does a lot of formatting that could be moved
to ancillary utilities. I guess what I like about PowerShell is the way it
allows for redirecting into objects - that really is an innovation that Unix
shells don't have (at least as far as I'm aware...).

------
guard-of-terra
"A system composed of a zillion tiny modules is itself a pile of mud"

Having so many args to `ls' might be an overkill, but I certainly don't want
to combine four programs to get full coloured sorted listing of current
directory.

~~~
wz1000
Many shells already come with a few default aliases that call ls with
different arguments. Your "four programs" can be combined in a simple alias.

~~~
nly
And what would you call an alias that gave you a coloured, sorted listing of
the current directory?

~~~
mseepgood
lsc

------
ggchappell
> You can’t write a function that returns a list of files because the shell
> doesn’t know what a “list” is, doesn’t know what “files” are, and couldn’t
> tell you the difference between a “function” and a “program” if its life
> depended on it.

Could it be that what this article is really groping towards is Perl? I mean
Perl not as it has become, but as it was -- Perl 4 perhaps: basically the
capabilities of the shell, augmented with a couple of data structures. So
there are strings, lists, hashes, and functions, and that's pretty much it.

------
golergka
The `ls` example is especially interesting: it made me think about perfomance
of pipelines. Imagine a folder with 10^6 files. Listing them would take some
time; if you only need 10^3 of files selected by the simplest filter (LIMIT
1000, for example), piping the whole list, in whatever format, to the filter
program would be very far from optimal.

What would be a perfect solution is to use iterators instead of lists (the
best example that I know of is Linq, but it seems that this approach is common
enough).

However, now we don't even exchange data between different programs; now we
couple them, running at the same time, by a complicated interface! It seems
that if we take one more logical step in this direction, we'll get generics.

Don't you start to feel that this kind of ideal shell, piping different
functions together in the most correct way possible already exists, and it's
already installed on your computer? I'm talking about interpreted languages
like Python, Perl, Javascript, Ruby or whatever else you fancy.

Of course, I don't seriously think that we should abandon shells for language
interpreters. It's just that balance between complexity and universality is a
very delicate thing, and convention, however bad, is good just because it's a
standard.

In the meanwhile, I'm quite happy with the Fish shell.

~~~
Camel2
> It seems that if we take one more logical step in this direction, we'll get
> generics.

I agree, Unix would have been so much better with generics, ADTs, higher-order
functions, monads, higher-kinded types, dependent types and structural pattern
matching.

As I always say: "Those who do not understand generics were condemned to
invent Unix, poorly."

~~~
Mr_T_
If the Unix guys were smart they would have used an Idris REPL as default
shell.

~~~
bishop_mandible
Why not Nimlang?

~~~
Mr_T_
Because Nim is not dependently typed. You can't get sh*t done without
dependent types.

------
dbbolton
Seems like the article, and several comments here, equivocate Linux with GNU.
While they are most commonly used in conjunction with one another, I still
don't believe that "this GNU program does too many things; therefore Linux
doesn't adhere to the Unix philosophy" is a valid argument.

------
Myk267
I think it's easy to misinterpret any of the "Unix philosophies". It's Ye Olde
"MIT vs New Jersey" again, and Unix is relentlessly straight out of NJ.
Simplicity in the "NJ" sense isn't what you'd expect if you're of the "MIT"
mindset.

~~~
jstimpfle
That's a good point. Makes me think again of "slow devices" and interrupts
leading to EINTR or partial reads/writes. In my eyes a beautifully pragmatic
solution to a real world problem, and yet at some point SA_RESTART was
devised.

------
mtrn
> Granted, we’ve developed graphical user interfaces that keep “ordinary
> users” away from the command line, but we still expect “serious developers”
> to drop down into a demonstrably inhumane environment to get anything
> meaningful done.

Maybe I am too familiar with it, but I have yet to encounter a graphical
environment that matches the effectiveness of the terminal and command line
programs. I saw systems, that tried to replace shell one-liners with a hundred
thousand line GUI application, that no one cared to use.

------
ezy
"Many of the usability issues raised by Don Norman in his 1981 criticism of
Unix have gone largely unaddressed"

That's because Don's complaints were not about intrinsic issues, but
complaints from someone who had a frustrating time developing a mental model
of the components of a system and who tried to use the just-so story of
"cognitive design" to justify his inability. The non-point about prompting y/n
to delete is particularly misguided.

Similar to this article.

"ls" is called "ls", not because of slow 80-char terminals, but because
_humans_ are, on average, not fast typists. One wants to list the contents of
a directory _a lot_ , and "ls" is quicker than "catalog". Similarly, "ps" vs
"tasklist" and "cc" vs "compile-a-c-program" :-). Notably, in powershell, most
people have aliases that do this for powershell's necessarily verbose query
functions.

It's primarily a command shell for controlling the machine, not a programming
language. And the UI choices made over the last 30 years reflect that.

That fact that ls takes arguments is a UI improvement that keeps you from
having to write tiny programs all the time and pipe it's output to sort and
awk all day long. It is also why the "one job well" crew is only half right,
and why Powershell kind of feels awkward even though it seems to get more
things "correct" in that regard.

This is just yet another just-so story by someone who wants ideological purity
in UI, which is completely misguided when UI, by definition is oriented
towards a universe which is not ideologically pure.

------
nspattak
Do one thing and do one thing well doesn't equal do one simple thing.
Furthermore, ls can actually do One Thing Only if this is enough for you and
that is plain ls. What would be the better alternative? Have multiple ls, ls-
color, ls-long-color, etc?

The discussion about Unix philosophy, standards, etc is a different story but
I did not see that from the articles examples.

~~~
Dewie3
"Doing one not-simple thing" is sometimes "doing multiple simple things which
they couldn't decompose" in disguise.

------
solidsnack9000
UNIX emerged in an era of more structured systems, though. Many OSes of the
time did not have files in a naive sense -- they had record-based formats.
"Everything is a file" was a revolutionary idea, and made UNIX simple and
lightweight and portable.

It is interesting that the author doesn't highlight "everything is a file"
which I have always understood to be the fundamental UNIX philosophy. ("Do one
thing well" applies to so much more than UNIX.)

Our understanding of how to use types and how to structure data on disk and
for streaming has greatly matured; but that doesn't mean we can ever afford to
pay more to get nothing. Say for a moment that `ls` provided data in tab-
separated value format (a format which is, unlike CSV, line filterable).
Wouldn't that go a long way toward allowing for the HOFs the author mentions?
And it wouldn't involve a shell type system (or "Shell Object Model").

------
taylodl
I dunno, the shell combined with sed and awk is pretty powerful and so far has
taken care of everything I need to do

------
gumby
I think you have it backwards: there are two legitimate counterarguments.

One is that 'ls' is the common program for querying the directory data
structure (which in original Unix was just a kind of file). So it makes sense
for it to output the data in different kinds of ways.

The second is that its a user mode command. It's easier than having a lot of
little commands (ls for all files, basically ls -1; lsF for showing file
types, etc). In fact I'll often to 'ls' then realise I want more info so will
type 'ls -l' (say) which is really my brain saying "oh yea, just like what I
just asked for but more info".

Better to pick something more deranged like tar or cpio!

------
darkr
One of the reasons for ls having additional functionality, is that it is used
in restricted shells - such as those used on rssh (SCP,SFTP, rsync and the
like). There are no filtering utilities available in these shells:

    
    
        -rbash-3.2$ ls $PATH
        chmod  cp  groups  ls  mv  rm  scp

------
sudioStudio64
these arguments about unix philosophy, to me, seem to miss the point about
what Unix actually is...which, again...to me, is a bunch of tools to get
things done.

being able to get things done with Unix means that you just put up with the
weird bits and move on. there doesn't need to be an overriding philosophy.
there are places where there seems to be consistent concepts, but their
implementation isn't and doesn't really need to be consistent.

the way that new ideas get so intensely opposed needs to be really looked at
though. there's no need to be so insulting if something isn't a good idea...it
just won't get used.

~~~
iopq
Maybe for you, but for me it's just a bunch of tools I don't really know that
well with lots of arcane options. I just google how to do what I want because
it's not possible to memorize through light usage.

~~~
sudioStudio64
I agree. No one ever memorizes all of it...or even a common core of it. It's
weird like that.

------
knowledgesale
Incidentally, there is plenty of ls alternatives like lsp:
[https://github.com/dborzov/lsp](https://github.com/dborzov/lsp)

------
Garlef
In other words: The shell should be a lisp.

~~~
GlennS
So, [http://scsh.net/](http://scsh.net/)?

------
peter303
Ed is still one of favorite editors out of UNIX-like terseness.

------
markhahn
ls's plethora of flags doesn't stop "echo *" from working.

------
Dewie3
Meta: I think someone made a comment that forced the whole page to be wider
than it normally is? like when you post an image on a forum that forces the
whole page to widen to fit it horizontally.

~~~
avn2109
Yeah this is super annoying. I am Chrome/OSX and scrolling is awful :(

