
The Unix Philosophy - dorsatum
http://www.catb.org/esr/writings/taoup/html/ch01s06.html
======
fauigerzigerk
I keep wondering about what seems to be the most important component of Unix
philosophy: write many small programs that do one thing well and interface
using text streams!

Yes, modularity is important. However, in some cases, this philosophy has
resulted in the "tangled mess held together by duct tape" kind of systems
architecture that no one dares to touch for fear of breaking things.

I think Unix philosophy is struggling with a fundamental dilemma:

On one hand, creating systems from programs written by different people
requires stronger formal guarantees in order to make interfaces more reliable,
stronger guarantees than interfaces within one large program written by one
person or a small team would require.

On the other hand, creating systems from programms written by different people
requires more flexibile interfaces that can deal with versioning, backward and
forward compatibility, etc, something that is extremely difficult to do across
programming languages without heaping on massive complexity (CORBA, WS-
deathstar, ...)

I think HTTP has shown that it can be done. But HTTP is also quite heavy
weight. It doesn't exactly favor very small programs. Handling HTTP error
codes is not something you'd want to do on every other function call.

In any event, I think Unix philosophy is a good place to start but needs a
refresh in light of a couple of decades worth of experience.

~~~
jalfresi
Recently I've been exploring the idea that small programs should consist of
two parts; a "backend" that you can communicate with via a strict message
protocol like protobuf. It can handle multiple requests because it is a mini
server, the message format is locked to a "type" and you communicate with it
via RPC.

The second part is a "frontend", a command-line client that does all the unixy
stuff with text streams etc; but is just another RPC client to the backend.

This provides flexibility; for quick duct-tape situations you use the front
end and pipe anf ilter to your hearts content. Then when stuff needs to get
serious, you can bypass the command-line front end and connect directly over
RPC using a strict message format which has the ability to handle a couple
(doesn't have to many) of concurrent requests.

Of course this is early doors yet, but I think it might have legs in the
flexibility in the text streams/hard message format debate, for very little
additional work.

~~~
forgottenpass
You're sort of describing the design idea common in unix-y areas to build
libfoo that does all the work, then a foo frontend that makes tasks useable
from the command line. I don't know how widely used this model is, but it's
certainly out there.

The difference is that in what I've described, the ABI is the common interface
and an RPC server would be just another consumer of the library.

~~~
jalfresi
Yup, even better!

------
alfiedotwtf
People will hang shit on ESR, but The Art of Unix Programming is one of my all
time favourite books. If you haven't read it, even if you're not a *nix
developer, do yourself a favour and just skim the table of contents...
something may pique your interest and you may learn a thing or two.

It's also free online:

    
    
        http://www.catb.org/esr/writings/taoup/html/

~~~
jumblesale
It's a really great read and did a lot to inform my approach to writing all
kinds of software. It's a shame that I cringe to recommend in now due to the
politics of its author.

~~~
yarou
That's really bizarre. A person's political views have no bearing on the
validity or usefulness of what they write.

~~~
peteretep
Undoubtably a deep philosophical argument, but brighter minds than ours have
written at length about it both in science and jurisprudence. "Research
ethics" and "Exclusionary rule" would be good illustrating Wikipedia articles.

~~~
arethuza
Out of interest, what are these political views that are so terrible that they
should influence our views of his thoughts on operating system design?

Edit: I really don't have any idea what they are...

~~~
philh
Among other unpopular opinions, ESR denies AGW, believes that race and IQ are
correlated, and was strongly in favor of the Iraq war.

~~~
TeMPOraL
None of these seem even remotely relevant to engineering. Moreover, research,
discovery and innovation requires letting people having freedom to think. That
includes holding unpopular, controversial or politically incorrect opinions.

We are so worried about Evil Government dictating what we can and cannot think
that we haven't noticed the current organic trend to prosecute every other
person for thoughtcrimes. It's not the jackboot that keeps us on the ground,
it's social media, and the public outrage you get when you disagree with
whatever's the most popular opinion on a topic this week.

~~~
dredmorbius
ESR's views particularly on guns, libertarian economics and politics, and AGW,
all present pretty standard cases of assuming a frame and fitting all data to
that frame. Chopping, discarding, and/or fabricating data as necessary to do
so.

That actually _directly_ calls into question engineering validity, as _solid_
engineering is solidly based in reality and a realistic interpretation of
facts. Also the ability to discard frames which no longer fit.

My own work and research of the past several years puts a very high
significance on both frames (or more generally, models), and on the psychology
of interacting with those, with strong emphasis on denial in various forms.

ESR's political views call much of his work into question. I say that as
someone who was strongly influenced by much of what he said, and enjoyed a
fair bit of it. He's become a tremendous disappointment.

TAOUP has its merits. It's rather like recommending Ted Kaczynski's
_Manifesto_ a a social-technological critique. It's got some really solid
points (see what Bill Joy's had to say on it:
[http://archive.wired.com/wired/archive/8.04/joy.html](http://archive.wired.com/wired/archive/8.04/joy.html)).
But damned if the rest of the author's views and actions don't muddy the
waters a tad.

~~~
philh
> ESR's political views call much of his work into question.

Presumably this question, whatever it is, can be answered by looking at his
work. Do you think ESR's technical work and technical writings fail to stand
up to scrutiny?

~~~
dredmorbius
It definitely gives me pause. ESR clearly doesn't know when he's out of his
depth (classic Dunning-Krueger).

I'm not enough of a programmer to judge his programming texts, though I _am_
enough of a sysamdin to find his Unixy sysadminish stuff generally valid.

I've found CatB itself aging poorly and question a number of the assumptions
behind it, particularly as concerns anthropology. It seems shaky. Though I
think the general principles behind Free Software and the open source model
have their merits. Just, possibly, not quite those ESR describes.

------
zvrba
> The ‘Unix philosophy’ originated with Ken Thompson's early meditations on
> how to design a small but capable operating system with a clean service
> interface.

Except that a program-to-program interface based on formatting and parsing
text is anything but clean.

~~~
mbrock
What do you mean by "text"?

~~~
zvrba
Semi-rigid, informally specified output meant for easy inspection by humans.
The output of `ifconfig -a` for example. (That's the worst offender, most
other commands have a better-formatted output.)

As a general rule of thumb: if you have a command pipeline that would break
because programs in it handle locale differently, it's most definitely text-
based program.

I like the idea of Power shell: programs pipe formatted (binary) objects
between them, and there exist commands to print them out in a nicely formatted
way to the terminal. But it's objects. I can extract information through named
fields instead of through magic indexes and regexpes in awk/sed/perl
oneliners.

IOW, proper object (meta)model, not sucky text streams. Text is great for
long-term storage, but awful for data in flight. UNIX ignored this distinction
and decided to use text for everything.

~~~
mzs
env -i ifconfig -a | ... solves a lot (like locale or other special env vars),
but yeah...

------
McElroy
At my place of work, we have a client relationship to a company like ours in a
neighbouring country. They develop a software that is of much use to us so we
pay a license to them for it. They are nice people but god damnit, there is
one thing they just never got right. They only support one platform, one that
is certified UNIX, yet no matter how severe an error, their cli tools and
scripts exit with 0 no matter what. I'm the guy who writes some smaller tools
and scripts on our side integrating with their software so you probably
understand why I get a bit upset about this at times. Still, I enjoy my work
and as I said they are nice people and also they are a quite small team so I
don't want to burden them with these concerns when there are other things that
our company need from them more.

Anyway, I've been with my company for a few years and soon my contract expires
and I'm going to study the field our company is in and get a degree in that,
then I'm going to apply for a position doing our core business. I would still
like to be involved with the software my current position is touching on,
though, if possible. (Our company has 1000+ employees and several different
sub-sections, so even though I might get back into the company, it's not a
given that I'll be working with the group of people I am now even though I'd
like to.)

I also sometimes think that if possible, perhaps I'd like to work for that
other company in our neighbouring country for a few years and be on the dev
team of the software. After all, I have experience from the user side which
the dev team has not and the dev team has seen some of the tools I've made and
a couple of the guys seemed to think that some of that stuff was pretty
decent.

~~~
JoeAltmaier
This can be a very good fit. Smart companies hire their customers' best
engineers with the most familiarity with their product. The best, smartest
feedback comes with that package.

------
jokoon
This should be taught in any programming class.

> Rule of Diversity: Distrust all claims for “one true way”.

Although, does the python rule "There should be one-- and preferably only one
--obvious way to do it." contradict this one ?

> Rule 5. Data dominates. If you've chosen the right data structures and
> organized things well, the algorithms will almost always be self-evident.
> Data structures, not algorithms, are central to programming.

I wonder what rob pike has to say about OOP or java, I wish I could listen to
it.

Also it says that text is a good representation of data, but I think he meant
it as intermediary. I don't think xml or html are really good choices when you
see all the CPU cycles spent parsing those.

> Rule of Economy: Programmer time is expensive; conserve it in preference to
> machine time.

I prefer this rules versus the "no premature optimization" rule.

~~~
vmsisbetter
Rob Pike doesn't know Smalltalk, Self, or Common Lisp/CLOS--why should you
care what his opinions are concerning OO as a paradigm?

~~~
pjmlp
Funny how HNers have such a high regard for his opinion on a domain he is well
known for being sceptical and yet disregard his opinion, as UNIX co-author,
that UNIX is well past its due date and is time to move on.

~~~
kasabali
He's not a UNIX co-author at all, he joined Bell Labs in 1980.

~~~
pjmlp
Except he contributed to many of the userland tools and was part of the UNIX
working group, which does make him a co-author, even if he didn't wrote the
first kernel lines.

~~~
kasabali
I must admit you have a strange definition of "co-author".

~~~
pjmlp
I must admit you have a strange way to ignore Pike's contributions to UNIX
initial GUI architecture, his book together with Kernighan about the UNIX
programming environment and other contributions.

~~~
kasabali
Please don't twist my words. I'm just saying "co-author" have different
meanings for you and me.

edit: Really? Downvotes? Not cool.

~~~
pjmlp
I didn't downvote you.

As for co-author, the Cambridge dictionary says:

"To write a book, article, report, etc. together with another person or other
people:"

Which given Pike's participation on what defined the initial UNIX graphics
userland, makes him a co-author.

~~~
kasabali
That's where we disagree. You think Blit is one of things what makes UNIX UNIX
while I don't. We have different interpretations of "co-author". I don't see
any further benefit in discussing that.

------
sytse
When making a microservice application you have to choose between a text
(JSON) or binary (Thrift) interface. You cloud argue that the unix way is to
make it JSON until it becomes a performance problem.

~~~
ajtulloch
This is a false choice - you can serialize/deserialize Thrift objects to/from
JSON. I think the choice here is between having a clearly-defined /schema/ or
not.

~~~
sytse
Thanks for the insight.

------
coldtea
> _Make each program do one thing well. To do a new job, build afresh rather
> than complicate old programs by adding new features._

Which means that no program is more un-UNIX-y than Emacs...

~~~
agumonkey
Because Emacs is a system of its own, the Lisp Machine legacy leaked too much.
Inside of emacs many things are tiny components, after all it's a lisp, which
had a stronger unixy feel than so many things since. I mean lambda, compose,
map... hard to separate concerns more than that.

------
oldpond
Can't upvote this enough. My takeaway from this gem is that we need to keep
thinking about our craft even as we evolve our technologies, and we need to
have the courage to stand up for it. The hardest thing in the world to see is
your own point of view because you have to step outside of it to see it.

------
thinkmoore
The Unix Hater's Handbook ([https://en.wikipedia.org/wiki/The_Unix-
Haters_Handbook](https://en.wikipedia.org/wiki/The_Unix-Haters_Handbook)) has
a few sections on of some of the ills of the Unix philosophy.

~~~
marcosdumay
The UNIX Haters Handbook shows its age, it's criticizing a system that has
only a passing similarity with modern unixes.

But yes, it's funny.

~~~
nickpsecurity
Nonetheless, the specific examples show the UNIX system contradicted many of
the supposedly "UNIX" principles. The article seems a bit revisionist in a
pro-UNIX way in that sense. Unsurprising.

Saying that these are modern UNIX principles might be more fair. Yet, Kemp's
article inspires doubt in even that:

[http://queue.acm.org/detail.cfm?id=2349257](http://queue.acm.org/detail.cfm?id=2349257)

------
jkot
How debacles such as micro-kernels and XServer fits into Unix Philosophy?

~~~
grymoire1
Sun had the NeWS windowing system before XWindows. Be glad that did not become
the standard. Be very glad.

~~~
arethuza
I rather liked NeWS - certainly it was a lot more fun to develop for at that
time than X and had some wonderful UI toolkits (particularly HyperNeWS).

For example, being able to draw a shape in a a graphical editor and then paste
the shape directly to a window as the shape that window should take was rather
cool - and this was in the early 90s!

------
hitlin37
in case of mankind extinction, we should preserve the above text and pass it
on to the next race.

------
el33th4xx0r
I wonder how relevant is it nowadays. Many good things we can't have, like
systemd, if we have to follow strictly to the unix philosophy.

~~~
userbinator
For a lot of people, the set of "good things" does not include systemd.

~~~
vezzy-fnord
Not to mention you can certainly get feature parity with systemd while still
being Unix philosophy-robust. See the nosh service manager.

------
digi_owl
the rule of separation makes me wonder if the long term legacy of Wayland will
be as a hardware accelerated backend for X.

~~~
jkot
X as a protocol is long gone. Direct memory access, kernel drivers etc are not
really compatible with network protocol. Wayland is just streamlining and
formalizing current situation.

~~~
umanwizard
At my old job (which I left about ten days ago) I ran X servers and clients on
different machines daily. Typical example was running my IDE on my desktop,
connected to an X server running on my laptop.

~~~
forgottenpass
The X developers didn't break remoting, but that doesn't mean contemporary gui
toolkits still use the features of X that made X "network transparent." As I
understand it, most features of the protocol are largely ignored except for
the parts needed to pump bitmaps over a network.

Here's an LCA talk by a dude what works on X and Wayland:
[https://www.youtube.com/watch?v=RIctzAQOe44](https://www.youtube.com/watch?v=RIctzAQOe44)

~~~
digi_owl
The main reason, as best i can tell, is that every DE out employ OpenGL
compositing that effectively bypass X.

Meaning that to draw the desktop they effectively pain a single large window
inside X that is then filled with the output of the GPU.

Thing is that Wayland seems more comparable to svgalib than X. This in that
Wayland pretty much a lib/protocol for talking to the graphics hardware.
Something else, be it their reference implementation Weston, GTK, Qt, or some
other alternative, has to handle the handling of windows, desktops etc.

Right now you can use Wayland as a driver for Xorg.

