
The Beauty of Unix Pipelines - 0x4FFC8F
https://prithu.xyz/posts/unix-pipeline/
======
gorgoiler
Pipes are wonderful! In my opinion you can’t extol them by themselves. One has
to bask in a fuller set of features that are so much greater than the sum of
their parts, to feel the warmth of Unix:

(1) everything is _text_

(2) everything (ish) is a _file_

(3) including _pipes_ and _fds_

(4) every piece of software is accessible as a file, invoked at the _command
line_

(5) ...with local _arguments_

(6) ...and persistent globals in the _environment_

A lot of understanding comes once you know what execve does, though such
knowledge is of course not necessary. It just helps.

Unix is seriously uncool with young people at the moment. I intend to turn
that around and articles like this offer good material.

~~~
jcranmer
> (1) everything is text

And lists are space-separated. Unless you want them to be newline-separated,
or NUL-separated, which is controlled by an option that may or may not be
present for the command you're invoking, and is spelled completely differently
for each program. Or maybe you just quote spaces somehow, and good luck
figuring out who is responsible for inserting quotes and who is responsible
for removing them.

~~~
gorgoiler
To criticize sh semantics without acknowledging that C was _always there when
you needed something serious_ is a bit short sighted.

There are two uses of the Unix “api”:

[A] Long lived tools for other people to use.

[B] Short lived tools one throws together oneself.

The fact that most things work most of the time is why the shell works so well
for B, and why it is indeed a poor choice for the sort of stable tools
designed for others to use, in A.

The ubiquity of the C APIs of course solved [A] use cases in the past, when it
was unconscionable to operate a system without cc(1). It’s part of why they
get first class treatment in the Unix man pages, as old fashioned as that
seems nowadays.

~~~
jcranmer
There's a certain irony in responding to criticism of that which you're
extolling by saying not to use it.

And the only reason I might be pushed down that path is because the task I'm
working on happens to involve filenames with spaces in them (without those
spaces, the code would work fine!), because spaces are a _reasonable_ thing to
put in a filename unless you're on a Unix system.

~~~
enriquto
> because spaces are a reasonable thing to put in a filename unless you're on
> a Unix system.

Putting spaces on a filename is atrocious and should be disallowed by modern
filesystems. It is like if you could put spaces inside variable names in
Python. Ridiculous.

~~~
HeadsUpHigh
I'm not going back to CamelCase or underscores for my normal day to day file
naming. The problem with spaces only exists inside the IT world and it's
something they should find a way around.

~~~
enriquto
I agree. The file load/save dialogs of all GUI should work in such a way that
spaces typed by the users in a filename field are always transparently changed
to something different (for example, the unicode non-breaking space).

~~~
skissane
> (for example, the unicode non-breaking space)

That's potentially even worse in a file name than plain old space is. It looks
like a normal space, but isn't.

If you are going to replace spaces in file names with something else,
underscores are the best option I think.

------
cowmix
I use pipelines as much as the next guy but every time I see post praise how
awesome they are, I'm reminded of the Unix Hater's Handbook. Their take on
pipelines is pretty spot on too.

[http://web.mit.edu/~simsong/www/ugh.pdf](http://web.mit.edu/~simsong/www/ugh.pdf)

~~~
cuddlybacon
I mostly like what they wrote about pipes. I think the example of bloating
they talked about in ls at the start of the shell programming section is a
good example: if pipelines are so great, why have so many unix utilities felt
the need to bloat?

I think it a result of there being just a bit too much friction in building a
pipeline. A good portion tends to be massaging text formats. The standard unix
commands for doing that tend to have infamously bad readability.

Fish Shell seems to be making this better by making a string which has a
syntax that makes it clear what it is doing:
[http://fishshell.com/docs/current/cmds/string.html](http://fishshell.com/docs/current/cmds/string.html)
I use fish shell, and I can usually read and often write text manipulations
with the string command without needing to consult the docs.

Nushell seems to take a different approach: add structure to command output.
By doing that, it seems that a bunch of stuff that is super finicky in the
more traditional shells ends up being simple and easy commands with one clear
job in nushell. I have never tried it, but it does seem to be movement in the
correct direction.

~~~
code-faster
It's less that pipelines are friction, they're really not.

It's more that people like building features and people don't like saying no
to features.

The original unix guys had a rare culture that was happy to knock off
unnecessary features.

------
atombender
Pipes are a great idea, but are severely hampered by the many edge cases
around escaping, quoting, and, my pet peeve, error handling. By default, in
modern shells, this will actually succeed with no error:

    
    
      $ alias fail=exit 1
      $ find / | fail | wc -l; echo $?
      0
      0
    

You can turn on the "pipefail" option to remedy this:

    
    
      $ set -o pipefail
      $ find / | fail | wc -l; echo $?
      0
      1
    

Most scripts don't, because the option makes everything much stricter, and
requires more error handling.

Of course, a lot of scripts also forget to enable the similarly strict
"errexit" (-e) and "nounset" options (-u), which are also important in modern
scripting.

There's another error that hardly anyone bothers to handle correctly:

    
    
      x=$(find / | fail | wc -l)
    

This sets x to "" because the command failed. The only way to test if this
succeeded is to check $?, or use an if statement around it:

    
    
      if ! x=$(find / | fail | wc -l); then
        echo "Fail!" >&2
        exit 1
      fi
    

I don't think I've seen a script ever bother do this.

Of course, if you also want the error message from the command. If you want
that, you have to start using name pipes or temporary files, with the
attendant cleanup. Shell scripting is suddenly much more complicated, and the
resulting scripts become much less fun to write.

And that's why shell scripts are so brittle.

~~~
fomine3
set -e makes another pain for command that nonzero isn't mean failed (ex.
diff). It changes semantics for whole script.

~~~
atombender
If by pain you mean you have to handle errors, yes, that's what you have to
do. It's no different from checking the return code of functions in C.

~~~
fomine3
Yes. So I don't use set -e.

------
geophile
I love pipelines. I don't know the elaborate sublanguages of find, awk, and
others, to exploit them adequately. I also love Python, and would rather use
Python than those sublanguages.

I'm developing a shell based on these ideas:
[https://github.com/geophile/marcel](https://github.com/geophile/marcel).

~~~
jraph
Your project looks really cool.

I am pretty sure I've seen a Python-based interactive shell a few years ago
but I can't remember the name. Have you heard of it?

~~~
ihumanable
Perhaps you are thinking of xonsh [https://xon.sh/](https://xon.sh/)

Edit: x1798DE beat me to it :D

~~~
jraph
Xonsh, that's it!

Thanks to you and x1798DE, and to geophile for the attempt :-)

I could not find the right keywords on two different search engines to find
it. This never happens.

------
ketanmaheshwari
Unix pipelines are cool and I am all for it. In recent times however, I see
that sometimes they are taken too far without realizing that each stage in the
pipeline is a process and a debugging overhead in case something goes wrong.

A case in point is this pipeline that I came across in the wild:

TOKEN=$(kubectl describe secret -n kube-system $(kubectl get secrets -n kube-
system | grep default | cut -f1 -d ' ') | grep -E '^token' | cut -f2 -d':' |
tr -d '\t' | tr -d " ")

In this case, perhaps awk would have absorbed 3 to 4 stages.

~~~
salmo
I think people can go through a few stages of their shell-foo.

The first involves a lot of single commands and temporary files.

The second uses pipes, but only tacks on commands with no refactoring.

The third would recognize that all the grep and cut should just be awk, that
you can redirect the cumulative output of a control statement, that
subprocesses and coroutines are your friend. We should all aspire to this.

The fourth stage is rare. Some people start using weird file descriptors to
chuck data around pipes, extensive process substitution, etc. This is the Perl
of shell and should be avoided. The enlightened return back to terse readable
scripts, but with greater wisdom.

Also, I always hear of people hating awk and sed and preferring, say, python.
Understanding awk and especially sed will make your python better. There
really is a significant niche where the classic tools are a better fit than a
full programming language.

~~~
young_unixer
My most complex use of the shell is defining aliases in .bashrc and have never
felt the need to go further. Do you recommend learning all of that for someone
like me?

If so, what resource do you recommend?

~~~
salmo
I may be a bad person to ask. I like history a lot. A lot of this stuff was
originally picked up by secretaries and non-computer people working on manuals
for Bell Labs. If you understand it, it will save you time. It will also make
you use and understand Linux/Unix better, which will bleed into any other
programming you do (assuming you write other code). The skill also comes up a
lot in strange places like Dockerfiles, CI Pipelines, and a lot of that kind
of surrounding infrastructure for applications.

The video referenced in the article is actually pretty interesting, as is the
book "The Unix Programming Environment". Both will refer to outdated
technology like real terminals, but I think they help understand the intent of
the systems we still use today. Any of the Bell Labs books are great for
understanding Unix.

Also do a minimal install of Linux or FreeBSD and read all of the man pages
for commands in /bin, /usr/bin. Read the docs for the bash builtins.

The old Useless Use of Cat is great:
[http://porkmail.org/era/unix/award.html](http://porkmail.org/era/unix/award.html)

But writing real scripts is probably the only way to learn. Next time you need
to munge data, try the shell. When you read about a new command, like comm,
try to use it to do something.

Also force yourself into good practices:

\- revision control everything

\- provide -h output for every script you write (getopts is a good thing to be
in the habit of using)

\- use shellcheck to beat bad habits out of you

\- try to do things in parallel when you can

\- assume everything you make is going to run at a much larger scale than you
intend it to (more inputs, more servers, more everything)

\- after something works, see where you can reduce the number of
commands/processes you run

\- write things like a moron is going to use them who will accidentally put
spaces or hyphens or whatever in the worst place possible, not provide
required arguments, and copy/paste broken things out of word

\- don't let scripts leave trash around, even when aborted: learn trap &
signals

The two things you should probably understand the most, but on which I'm the
least help on are sed and awk. Think of sed as a way to have an automated text
editor. Awk is about processing structured data (columns and fields). I
learned these through trial and error and a lot of staring at examples.
Understanding regular expressions is key to both and is, in general, an
invaluable skill.

Oh and if you are a vi(m) user, remember that the shell is always there to
help you and is just a ! away. Things like `:1,17!sort -nr -k2` (sort lines
1-17, reverse numerical order on the second field) can save a ton of time. And
even putting some shell in the middle of a file and running !!sh is super-
handy to replace the current line that has some commands with the output of
those commands.

~~~
mturmon
You’ve left some good tips. sed, awk, and grep. getopts. Read the man pages.
Try to automate even simple series of commands.

Don’t be shy about running a whole pipeline of commands (in an automation
script) just to isolate one field within one line of one file and add one to
it, and store the result in a shell variable.

------
russellbeattie
I think there's an interesting inflection point between piping different
utilities together to get something done, and just whipping up a script to do
the same thing instead.

First I'll use the command line to, say, grab a file from a URL, parse, sort
and format it. If I find myself doing the same commands a lot, I'll make a .sh
file and pop the commands in there.

But then there's that next step, which is where Bash in particular falls down:
Branching and loops or any real logic. I've tried it enough times to know it's
not worth it. So at this point, I load up a text editor and write a NodeJS
script which does the same thing (Used to be Perl, or Python). If I need more
functionality than what's in the standard library, I'll make a folder and do
an npm init -y and npm install a few packages for what I need.

This is not as elegant as pipes, but I have more fine grained control over the
data, and the end result is a folder I can zip and send to someone else in
case they want to use the same script.

There is a way to make a NodeJS script listen to STDIO and act like another
Unix utility, but I never do that. Once I'm in a scripting environment, I
might as well just put it all in there so it's in one place.

~~~
mongol
I have never come across a NodeJS script instead of a shell script. Can you
suggest an example to look at that shows the advantages of this?

~~~
russellbeattie
Here's a JS script [1] I wrote a little while ago just for my own use that
queries the CDC for the latest virus numbers, then calcs the average, formats
the data and prints to the command line. You can pass in a number of days,
otherwise it pulls the last 14 days.

$ node query-cdc.js 7

It's nothing special, but I wouldn't want to try to do this with command line
utilities. (And yes, it was a bit uglier, but I cleaned it up of random crap I
had thrown in before posting it.)

1\.
[https://gist.github.com/russellbeattie/f9edf91115b43d6d7ca3c...](https://gist.github.com/russellbeattie/f9edf91115b43d6d7ca3cf0b4f9eb060)

~~~
mongol
Thanks, appreciated! I guess using shebang #!/usr/bin/node does not work as #
is not a valid comment?

~~~
russellbeattie
I think Node has an exception for that built in, so you can do that or
#!/usr/bin/env node and it'll work.

------
nojito
Debugging Linux pipelines is not a fun experience.

This is one clear area where Powershell with its object model has got it
right.

~~~
pwdisswordfish2
Please provide an example. Thank you.

~~~
fomine3
PowerShell supports breakpoint.

[https://docs.microsoft.com/en-
us/powershell/module/microsoft...](https://docs.microsoft.com/en-
us/powershell/module/microsoft.powershell.utility/set-
psbreakpoint?view=powershell-7)

~~~
pwdisswordfish2
Can you give a _working_ example?

This way you can watch a UNIX user "struggle/fail" as they "attempt" to debug
a script using set -x, set -e, $?, LINENO, utilties like ktrace, strace,
recordio, etc. and your argument will be taken seriously.

~~~
pjmlp
Some people also insist to keep debuging with printf(), whereas the rest of
the world is happily using InteliJ and friends.

~~~
pwdisswordfish2
I just use gdb and ald. I heard IntelliJ is a memory hog and requires a
powerful computer.

~~~
pjmlp
InteliJ isn't the only option, hence why I mentioned "and friend".

------
antipaul
A similar philosophy has made the "tidyverse" a much-loved extension of the
statistical language R.

Compare the following 2 equivalent snippets. Which one seems more
understandable?

    
    
        iris_data %>%
            names() %>%
            tolower() %>%
            gsub(".", "_", ., fixed=TRUE) %>%
            paste0("(", ., ")")
    
    

or:

    
    
        paste0("(", gsub(".", "_", tolower(names(iris_data)), fixed=TRUE), ")")

~~~
chubot
Yup, I'm working on Oil shell and I wrote an article about data frames,
comparing tidyverse, Pandas, and raw Python:

 _What Is a Data Frame? (In Python, R, and SQL)_

[http://www.oilshell.org/blog/2018/11/30.html](http://www.oilshell.org/blog/2018/11/30.html)

The idea is essentially to combine tidyverse and shell -- i.e. put structured
data in pipes. I use both shell and R for data cleaning.

My approach is different than say PowerShell because data still gets
serialized; it's just more strictly defined and easily parseable. It's more
like JSON than in-memory objects.

The left-to-right syntax is nicer and more composable IMO, and many functional
languages are growing this feature (Scala, maybe Haskell?). Although I think
Unix pipes serve a distinct use case.

------
theshadowmonkey
Pipes are like one of the best experiences you’ll have whatever you were
doing. I was debugging a remote server logging millions of Logs a day and was
aggregating a little on the server. Then all it required was wget, jq, sed and
awk. And I had a powerful log analyzer than splunk or any other similar
solution on a developer Mac. Which you think is awesome when you’re paying a
fortune to use Splunk. And for getting some insights quick, Unix pipes are a
godsend.

------
CalmStorm
Unix pipelines are indeed beautiful, especially when you consider its
similarity to Haskell's monadic I/O:
[http://okmij.org/ftp/Computation/monadic-
shell.html](http://okmij.org/ftp/Computation/monadic-shell.html)

Unix pipelines actually helped me make sense of Haskell's monad.

~~~
AnimalMuppet
Could you be more specific? I don't get it.

~~~
msla
It helps if you avoid the syntactic sugar of do-notation:

    
    
        main = getArgs >>= processData >>= displayData
    

main is in the IO monad. The >>= function takes a value wrapped in a monad and
a function which accepts a value and returns a value wrapped in the same type
of monad, and returns the same monad-wrapped value the function did. It can be
used as an infix operator because Haskell allows that if you specify
precedence, in which case its left side is the first argument (the value-in-a-
monad) and its right side is the second argument (the function).

The (imaginary) function getArgs takes no arguments and returns a list of
command-line arguments wrapped in an IO monad. The first >>= takes that list
and feeds it into the function on its right, processData, which accepts it and
returns some other value in an IO monad, which the third >>= accepts and feeds
into displayData, which accepts it and returns nothing (or, technically, IO
(), the unit (empty) value wrapped in an IO monad) which is main's return
value.

See? Everything is still function application, but the mental model becomes
feeding data through a pipeline of functions, each of which operates on the
data and passes it along.

~~~
AnimalMuppet
Thanks. That's helpful.

------
nsajko
Surprised there is no mention of Doug McIlroy:
[https://wiki.c2.com/?DougMcIlroy](https://wiki.c2.com/?DougMcIlroy)

------
lexpar
Interesting comment in the header of this site.

[https://github.com/prithugoswami/personal-
website/blob/maste...](https://github.com/prithugoswami/personal-
website/blob/master/layouts/partials/head.html)

~~~
nsajko
It's a meme:
[https://www.reddit.com/r/copypasta/comments/5we0ny/if_youre_...](https://www.reddit.com/r/copypasta/comments/5we0ny/if_youre_reading_this_youve_been_in_a_coma_for/)

~~~
giantrobot
It's from the naughty strings list. It's meant to be a "meat exploit" instead
of an exploit for the computer.

------
rzmnzm
Powershell is the ultimate expression of the Unix pipeline imo.

Passing objects through the pipeline and being able to access this data
without awk/sed incantations is a blessing for me.

I think anyone who appreciates shell pipelines and python can grok the
advantages of the approach taken by Powershell, in a large way it is directly
built upon existing an Unix heritage.

I'm not so good at explaining why, but for anyone curious please have a look
at the Monad manifesto by Jeffrey Snover

[https://devblogs.microsoft.com/powershell/monad-manifesto-
th...](https://devblogs.microsoft.com/powershell/monad-manifesto-the-origin-
of-windows-powershell/)

You may not agree with the implementation, but the ideas being it, I think,
are worth considering.

------
akavel
I think it will be _on topic_ if I let myself take this occasion to once again
plug in a short public service announcement of an open-source tool I built,
that helps interactively build Unix/Linux pipelines, dubbed "The Ultimate
Plumber":

[https://github.com/akavel/up/](https://github.com/akavel/up/)

I've also recently seen it being described in shorter words as a "sticky REPL
for shell". Hope you like it, and it makes your life easier!

~~~
edeion
I'm surprised this doesn't get up-voted. The tool automates very nicely my
shell command writing process: pipe one step at a time and check the
incremental results by using head to get a sample. Looks cool to me!

------
tekert
To my understanding, this is the same pattern where every "object" outputs the
same data type for other "objects" to consume. This pattern can have a text or
gui representation that is really powerful in its own nature if you think
about it, its why automation agents with their events consumption/emition are
so powerful, its why the web itself shifts towards this pattern (json as
comunication of data, code as object), The thing is, this will always be a
higher level of abstraction, i think that a gui of this pattern should exist
as a default method in most operating systems, its would solve a lot learning
problems like learning all the names and options of objects, i would be the
perfect default gui tool, actually, sites like zapier, or tools like huginn
already do this pattern, i always wondered why this pattern expands so slowly
being so useful.

------
benjaminoakes
I find gron to be much more Unix-y than jq. It "explodes" JSON into single
lines for use with grep, sed, etc and can recombine back into JSON as well.

[https://github.com/TomNomNom/gron](https://github.com/TomNomNom/gron)

~~~
majewsky
Since jq is sed for JSON, by the transitive property, you're saying that sed
is not Unix-y. ;)

Seriously though, I use both, and IMO they serve different purposes. gron is
incredibly useful for exploring unknown data formats, especially with any form
of

    
    
      something | gron | grep something
    

Once you've figured out how the data format in question works, a jq script is
usually more succinct and precise than a chain of
gron/{grep,awk,sed,...}/ungron.

So in practice, gron for prompts and jq for scripts.

------
parliament32
Related: Pipelines can be 235x faster than a Hadoop cluster
[https://adamdrake.com/command-line-tools-can-
be-235x-faster-...](https://adamdrake.com/command-line-tools-can-
be-235x-faster-than-your-hadoop-cluster.html)

------
sedatk
It's ironic that the article ends with Python code. You could have done
everything in Python in the first place and it would have probably been much
more readable.

~~~
enriquto
> You could have done everything in Python in the first place and it would
> have probably been much more readable.

Python scripts that call a few third party programs are notoriously
unreadable, full of subprocess call getouptut decode(utf8) bullshit. Python is
alright as a language, but it is a very bad team player: it only really works
when _everything_ is written in Python, if you want to use things written in
other languages it becomes icky really fast. Python projects that use parts
written in other languages inevitably gravitate to being 100% Python. Another
way to put it, is that Python is a cancer.

------
enriquto
If you eval this simple pipeline

    
    
         file -b `echo $PATH:|sed 's/:/\/* /g'`|cut -d\  -f-2|sort|uniq -c|sort -n
    

it prints a histogram of the types of all the programs on your path (e.g.,
whether they are shell, python, perl scripts or executable binaries). How can
you ever write such a cute thing in e.g., python or, god forbid, java?

~~~
junke
\- This fails on badly formed PATH, or malicious PATH

\- When PATH contains entries that no longer exist / are not mounted,
substitution with "sed" gives a wildcard that is not expanded, and eventually
makes "file" report an error, which is not filtered out

\- If PATH contains entries that have spaces, the expansion is incorrect

~~~
enriquto
More realistically, it also fails if you have directories on your path that
are symlinks to other directories in the path. In that case their programs are
doubly-counted.

Anyways, if your PATH is malicious then you have worse problems than this
silly script :)

~~~
junke
I agree, but "more realistically"? those cases were pretty realistic I think.
If you run the script on a box where you have users who can do whatever they
want, at least the script should fail on bad inputs.

    
    
        PATH=rm -rf :/bin

------
stevefan1999
I think it is terrible, as if everything is a text you can't precisely
describe some data structure, e.g. circular list.

------
tarkin2
I wish there were gui pipes. Pipes are wonderful but they’re neither
interactive nor continuous.

Pipes that loop, outputting a declarative gui text format, and listen for
events from the gui, would be marvellous.

I can’t think how to do that without sockets and a bash loop. And that seems
to create the kind of complexity that pipes manage to avoid.

------
praveen9920
I recently wrote a golang code for fetching rss feed and displaying gist based
on requirement.

Looking at this code, I am tempted to reimplement this using pipes but saner
mind took over and said "don't fix something that is not broken"

I probably would be still do it and get some benchmark numbers to compare
both.

------
staycoolboy
The first time I saw pipes in action I was coming from Apple and PC DOS land.
It blew my mind. The re-usability of so many /bin tools, and being able to
stuff my own into that pattern was amazing.

If you like pipes and multimedia, checkout gstreamer, it has taken the custom
pipeline example to real-time.

------
hi41
Thank you! I did not know you get to a "sql like group by" using uniq -c.
That's so cool! I think I used to pipe it to awk and count using an array and
then display but your method is far better than mine.

~~~
quicklime
I use "sort | uniq -c" all the time, but I find it annoying that it does it in
worse-than-linear time. It gets super slow every time I accidentally pass it a
large input.

At that point I usually fall back to "awk '{ xs[$0]++ } END { for (x in xs)
print xs[x], x }'", but it's quite long and awkward to type, so I don't do it
by default every time.

At some point I'll add an alias or a script to do this. One day.

edit: OK I finally did it; it's literally been on my to-do list for about 10
years now, so thanks :)

$ cat ~/bin/count

#!/usr/bin/awk -f

{ xs[$0]++ } END { for (x in xs) print xs[x], x }

------
Silamoth
That was a great article! Pipes can definitely be very powerful. I will say,
though, that I often find myself reading pages of documentation in order to
actually get anything with Unix and its many commands.

------
miclill
Great write up. One thing I would add is how pipes do buffering / apply
backpressure. To my understanding this is the "magic" that makes pipes fast
and failsafe(r).

------
not2b
I've been using pipes for decades to get my work done, but it was cool to
learn about jq as I have much less experience with JSON. It's a very cool
program. Thanks.

------
sigjuice
Minor nitpick: It is Unix "pipes" (not pipelines).

~~~
lsofzz
I think this reply needs more up votes! ;)

------
jakubnarebski
The first example in the article is what `git shortlog -n` does; no need for
the pipeline.

------
kazinator
> _If you append /export to the url you get the list in a .csv format._

Text filtering approach narrowly rescued by website feature.

Phew, that was close!

------
stormdennis
The video from 1982 is brilliant great explanations from an era before certain
knowledge was assumed as generally known

------
mehrdadn
What kills me about pipelines is when I pipe into xargs and then suddenly
can't kill things properly with Ctrl+C. Often I have to jump through hoops to
parse arguments into arrays and avoid xargs just for this. (This has to do
with stdin being redirected. I don't recall if there's anything particular
about xargs here, but that's where it usually comes up.)

------
0xsnowcrash
Aka The Ugliness of Powershell. Would be fun to see equivalents to these in
Powershell.

------
JadeNB
What does it mean to say that the video shows "Kernighan being a complete
chad"?

~~~
dodwer
It's an alt-right in-joke related to the incel subculture.

~~~
Aaronstotle
I wouldn't be so quick to say it's an alt-right joke. Plenty of my friends
will describe something/someone as a Chad and it has nothing to do with
incel/alt-right cultures. Are those kind of phrases thrown out in those
circles? Yes, but that's more of the general meme/internet lingo as opposed to
subscribing to an ideology.

~~~
junke
This is brogrammer speak (e.g.
[http://new_words.enacademic.com/662/brogrammer](http://new_words.enacademic.com/662/brogrammer)).

~~~
JadeNB
Thanks! In fact, your link suggests that it may have originated (in this
context) from Rob Spectre calling his alter ego 'Chad':
[https://www.businessinsider.com/the-ultimate-guide-to-
learni...](https://www.businessinsider.com/the-ultimate-guide-to-learning-
brogramming-the-hard-way-2011-9)
[https://twitter.com/ChadSpectre/](https://twitter.com/ChadSpectre/)

------
known
The capacity of pipe buffer is important for doing serious producer|consumer
work;

------
nickthemagicman
I have problems understanding what commands I can pipe. Some work some don't.

~~~
ketanmaheshwari
The commands that read their input from standard input could be used as
receiver in a pipeline. For instance: `A | B` is a short form of `A > tmp.txt;
B < tmp.txt; rm tmp.txt`

There are some commands that need literal arguments which is different than
standard input. For instance the echo command. `echo < list.txt` will not work
assuming you want to print the items inside the list. `echo itemA itemB itemC`
will work. This is where xargs comes into play -- it converts the standard
input stream to literal arguments among other things.

~~~
nickthemagicman
How do you know which ones receive input from standard input or how can you
find out efficiently?

~~~
ketanmaheshwari
The man page of the command helps in most cases. Often the description section
has info if the command reads from standard input, a file or both.

------
unnouinceput
Cygwin and you can do exactly this on Windows too. For me Cygwin is a bless.

------
hbarka
abinitio.com was borne from these principles.

------
Dilu8
Good article. Thanks for information

------
billfor
I think systemd can do all of that.

------
niko0221
Esta genial Laexplicacion. Se aprende mucho aqui Soy nuevo y este articulo es
bastante genial.

------
fogetti
I am sorry for playing the devil's advocate. I also think that pipes are
extremely useful and a very strong paradigm, and I use them daily in my work.
Also it is not an accident that it is fundamental and integral part of
powershell too.

But is this really HN top page worthy? I have seen this horse beaten to death
for decades now. These kind of articles have been around since the very
beginning of the internet.

Am I missing something newsworthy which makes this article different from the
hundreds of thousands of similar articles?

~~~
hu3
Perhaps it's useful to reinforce pipes' utility to those who are not aware.

As for being top most article, if I had to guess it's because people feel
related to the tool's power and enjoy sharing anecdotes of it.

------
adamnemecek
Unix pipes are the a 1970's construct, the same way bell bottom pants are.
It's a construct that doesn't take into account the problems and scale of
today's computing. Unicode? Hope your pipes process it fine. Video buffers?
High perf? Fuggetaboutit. Piping the output of ls to idk what? Nice, I'll put
it on the fridge.

~~~
JdeBP
An informed criticism would be based upon the knowledge that Doug McIlroy came
up with the idea in the 1960s. (-:

