
The growth of command-line options, 1979-Present - janvdberg
https://danluu.com/cli-complexity/
======
choward
> that's the beauty of command line options -- unlike with a GUI, adding these
> options doesn't clutter up the interface

GUIs provide discoverability so often times you don't need a manual. If you
don't know how to use a command line tool you must open the manual. So in that
way I would consider the man pages as part of the UI and those definitely get
cluttered.

~~~
socialdemocrat
The discoverability of a GUI deteriorated as you add more options.

~~~
K0SM0S
I've worked with music software GUIs, these are notoriously plagued by
feature-creep.

For a desktop paradigm, there is _always_ a way to do better with GUI as you
add more features; but you'll refactor / redesign a lot on the way and that
has a cost in familiarity, which directly impacts user productivity. It's a
trade off.

The dirty secret of discoverability is that only a tiny subset of features is
relevant at any time in the UX, and identifying that subset + triggering user
awareness about it is the "art" part. We learned a lot with mobile in that
regard because the harsh limit on real estate becomes incentive to identify
the _essentials_.

Discoverability doesn't have to suffer from complexity of the software, but
it's not just UI design, it's UX.

~~~
perl4ever
"The dirty secret of discoverability is that only a tiny subset of features is
relevant at any time in the UX"

For some reason, designers and some users don't like it, but a hierarchy
(folders, menus, etc) is something many people find useful and effective.

But it seems pretty common to declare people can only handle a few options
plus search.

~~~
oblio
Hierarchies suck. That's why early web "search engines" failed, the old
hierarchical portals, and Google won (once it added a good sorting algorithm
to the what the initial search engines returned).

The UI is one input and the "computer" figures out the rest.

Regular users don't want/understand hierarchies, they get easily confused.
They also don't want to organize things themselves, see the many failed
attempts at tagging files, websites, images, etc. manually.

~~~
perl4ever
Every time this is mentioned, people say "shut up shut up shut up, nobody
wants hierarchies". Obviously it's dogma. But if you have to keep saying it...

I am probably not a "regular user", but I got a job about a year ago defined
by and intended for _regular users_ , and one of the major parts of it was
managing emails in a departmental account, using (you guessed it) hierarchical
folders, and another was managing project documents in SharePoint, using (what
appear to be) hierarchical folders.

I'm aware that people will tell you not to do the latter in SharePoint, that
it's not really designed to be used like a filesystem, just use lists, etc.
But purely from an anthropological viewpoint, I see how people shove
everything on a shared drive, and then they try to move it to something like
SharePoint and recreate a folder structure.

So, you know, it's not about what _I_ want, but it's an objective fact people
do want hierarchies, even when other people don't want them to. Google is
pervasive, but it's not all of computing.

------
IHLayman
Isn't it possible that, like human languages that evolve over time for
multiple reasons
([https://en.wikipedia.org/wiki/Language_change](https://en.wikipedia.org/wiki/Language_change))
such as language contact, principle of least action, and pejoration, that
computer languages also experience similar shifts? All of this grumbling about
how command line options have exploded or why doesn't everyone use language X
or shell Y or OS Z that has existed for forever is similar in tone to me about
people who grumble about Oxford commas or starting sentences with conjunctions
or (heaven forbid) using emojis?

Of course some principles of speech and syntax should stay around (as they
probably exist for a good reason), but it is foolhardy to grip tightly onto
any language, human or computer, thinking it will always stay the same, when
it plainly won't.

~~~
dexen
_computer languages also experience similar shifts?_

Partly, but less so.

Human language vocabularies approximate Huffman Coding - the more frequently
used words (and phrases) tend to be proportionally shorter. This is a gradual
process for spoken languages, and somewhat quantized for written languages.[1]
It's possible because with human languages, "close" is "good enough"; one can
read english written with _ye olde_ orthography, if with a bit of extra
effort.

Less so in the lifespans of particular programming languages. Per Postel's law
we tend to be liberal with extending but conservative with breaking backward
compatibility. I think the case of command line tool parameters is
particularly conservative for various reasons; among them because the
'knowledge' of options is spread among multitude of separate project - many of
them obscure and bespoke - and coordination would be nigh impossible.

However there are some signs of similar trends; note how the "function" and
"integer" and similar keywords of older languages tend to turn into "fn" and
"int" in newer languages.

\--

[1] Not sure if there's a comparable process for grammars, but I guess it's a
fair bet...?

~~~
IHLayman
I agree, that's why I mentioned principle of least action. But you also forget
the power of idioms. We would connect to a file with a file handle using
something like open("file"), and then have to close it with close(handle).
This got tedious (and dangerous because people would forget to close files),
so people would write macros that would automatically open and close file
handles, and soon languages would have the "with" idiom that would handle the
context for you. And this is not the only case of an idiom causing a paradigm
shift in languages.

Pejoration happens too. When is the last time you used a GOTO in code? And
yet, 40 years ago, you couldn't write a BASIC program without one. And the
revulsion against NULL causing a shift to Maybes and optional types to the
point where the concept is reified in many languages now?

All I'm saying is that there are multiple mechanisms that are changing the
languages we write code with, and that it is changing far faster than people
realize.

~~~
close04
Systems grow in complexity and as a result you want to do more _with_ them and
_for_ them, all while maintaining decades of backwards compatibility. Which is
why all those basic tools presented in the article, that have existed over the
past 4 decades still do the same and more. They have to cater to scenarios
that didn't exist 4 decades ago but also to the ones that did.

~~~
hhas01
That’s an excuse. Of _course_ demands will increase over time—all the more
reason to build a system that scales well! Unix Philosophy doesn’t demand
much: just small simple _composable_ tools, each of which _does one thing
well_.

So why does `ls` have a recursion flag? Why does `cp` have a recursion flag?
Why does `chmod` have a recursion flag? Why does `chown` have a recursion
flag? Why isn’t recursion its own command, that takes as argument the command
to apply at each level?

Ditto untyped, untagged streams, which mean every tool implements its own
custom UI behavior when attached to a terminal, instead of outputting tagged
data and letting the consumer run it through system-standard data formatters.

This is not good—or even borderline competent—systems design. This is
bullshit. Because the people who slapped it all together are bums, and the
people who accepted it didn’t care enough to call them on it.

But noooo, we can’t change anything now; because “backwards compatibility”.

Protip: when you build a system that _can’t_ evolve, that grinds to a halt
under its own impossible weight, it’s time to confess “yeah, we totally
ballsed that one up” and start over again, lessons learnt. Because maintaining
that status quo is a tyranny: dragging everyone else down to the level of its
incompetence, and beating them to death with intransigence. No prizes for
guessing who that benefits: the bums who built it. And who pays for it:
everyone else.

It’s a scam. Perpetrated by and for those who cannot admit they are wrong.

Smash it down. Because whatever comes after it should be so much better, the
_only_ question will be “Why the frell did we wait so long?”

\--

“Those who make peaceful revolution impossible will make violent revolution
inevitable.”—John F Kennedy

~~~
close04
> So why does

Why did _ls_ even need 11 parameters to begin with? Why not exactly 1? Or
none? Why did _tar_ need 12? Why did _find_ need 14? The 28 tools in the
example (excluding non-existing ones) had an average of ~3 parameters per tool
in '79\. Should 2017 have brought us 200 extra individual tools to provide the
same additional functionality? You want another tool that does recursion for
_ls_ to save us from "incompetent bums" and "grinding to a halt"?

Everything around you grew in complexity to adapt to _your_ needs because
there's value in consolidation. It's why you are now typing from a device that
is orders of magnitude more complex and integrated than the one used to write
down the Unix philosophy.

You ask the question "why" but fail to answer the "why not". Why would a tool
that still does _exactly_ what it used to do in the past not be able to also
do more especially when it's still within its sphere of relevance? And not the
"out of principle" answer, not the handwaivy, tin foil hat answer, not the
offensive and insulting answer that you provided above (bums? incompetent?
bullshit? scam?).

Times change and philosophies adapt to remain relevant. "We must use time as a
tool, not as a couch" \- JFK.

~~~
pdimitar
> _You want another tool that does recursion for ls to save us from
> "incompetent bums" and "grinding to a halt"?_

Yes I absolutely do. Remember UNIX? "Many tools, each small, each doing one
thing well". Adding hundreds of flags to older tools does NOT achieve that.
Your point here?

> _Everything around you grew in complexity to adapt to your needs because
> there 's value in consolidation._

Please don't cross into abstract philosophy. It adds absolutely nothing to a
specific technical discussion.

> _It 's why you are now typing from a device that is orders of magnitude more
> complex and integrated than the one used to write down the Unix philosophy._

I use my computers as the complex messes and a series of train-wrecks that
they are, and now hear me out please, _because I literally have no other
choice_. Be it Win10, Linux, Mac, it's honestly almost irrelevant. One
example: to this day nobody can make a reliable system-wide sleep mode that is
not messed up by extra hardware (like my external HDDs), on neither of those 3
major OSes. In 2020.

I can't go back to my Apple IIc. I will lose my job if I do.

What's your point here? That such a complexity is inevitable? Is that it? But
who argues against that? Of course it's inevitable, we see it in every area of
life -- it's one of the aspects of entropy. What's being discussed is: "isn't
it time to take a firmer stance and start using newer tools that don't have
these problems and protest against the old ones by not using them"?

I personally answered "yes" and I am gradually doing more and more of it --
already replaced grep, ls, find/xargs, about 13 in total IIRC. You know what?
My productivity on the terminal skyrocketed. I started ditching languages with
too much magic and implied behaviour and too weak/random typing (PHP/JS come
to mind) and moved to others which, while not covering everything that I'd
like to have (like Elixir), are still letting me sleep that much better
knowing that `[] == 0` is actually false and I won't have to curse my way
through logs in the next morning.

> _You ask the question "why" but fail to answer the "why not"._

If you actually frequent HN and aren't just looking for conflict then you'd
know that literally every month we have a few rants by people who are now
burned out that decry the current state of affairs. But if you really want me
to answer then I can likely summarise it to:

"Because our tiny brains don't work well with so much complexity in the long
run. We can cope for a few years, some even for 20 and more, but eventually we
can't. Because having smaller, more-focused, and more (total count of) tools
that each do something really well, has been demonstrated to work better
everywhere, not only in IT."

 _(But I am aware this has been contended endlessly and it devolves into
belief and tribalism and science and historical evidence goes out the window
in the first three comments tops. Sad. Shouldn 't happen to techies. We should
be better than that.)_

> _Why would a tool that still does exactly what it used to do in the past not
> be able to also do more especially when it 's still within its sphere of
> relevance?_

Because it's NOT, and your parent commenter gave you an excellent example
which you immediately dismissed and THEN proceeded to ask your question again,
pretending he never gave an example. Are you aiming at being disrespectful on
purpose? Confusing.

> _And not the "out of principle" answer, not the handwaivy, tin foil hat
> answer_

You _are_ aware you are crossing into ad hominem and almost insults, no?

> _Times change and philosophies adapt to remain relevant._

Well that's a perfect example how the same sentence is viewed in two
completely different ways. This is EXACTLY the problem, dude: philosophies
don't adapt at all in our area, most of the time. I knew a sysadmin that
adopted those newer CLI tools (most written in Rust but some in Go, like
`fzf`) and he is now doing the job of 3 people -- especially after also
adopting Nix/NixOS. And I know people that will defend their 5000+ lines of
Makefiles with hacks around niche return values of `ls` with their dying
breath.

That's exactly the problem. Most people (and philosophies) don't adapt at all.
They prefer to fight very hard to mold reality to keep agreeing with their
philosophy. I am surprised you are not seeing this because it's literally
everywhere -- from your local airport with their laughable security checks to
your local cafeteria that likely still uses cash registers that can be hacked
with a simple cable and a laptop and running a script and the whole thing will
take about 7 seconds before the lady behind the counter even gets back from
the toilet.

> _" We must use time as a tool, not as a couch" \- JFK._

Again, your point?

\---

Don't take my comment as an attack. But you very quickly dismissed some pretty
good points and proceeded to [attempt to] hijack the discussion to irrelevant
-- or too abstract -- examples. Again, I'd change my computer and OS to
something more conservative, with one way of doing things and not hanging on
to ancient and obviously not-working-that-well-anymore philosophies, tomorrow
-- if it existed. But it doesn't.

The fact that now we have no choice doesn't prove your point at all.

~~~
dang
Please don't post in the flamewar style. It's unnecessary, evokes worse from
other commenters, and makes for tedious reading.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

~~~
pdimitar
Eh, I was indeed at the line. Apologies.

------
souprock
Ouch, the /usr/bin/tar team beat me. I only added 63 options to /bin/ps from
Slackware 3.1 (1996) to Ubuntu 12 (2015). They added 81 options. If I'd known
about the contest, I could have added enough options to win.

The true winner is probably gcc though, which was unfairly excluded from
competition. Over 1000 options is excellent work.

~~~
carterehsmith
Is there a better way? Can we make gcc with less options, while still catering
to all the use cases.

~~~
Paul-ish
That might not be possible. What if instead we created a configuration tool
that intelligently passed the correct options for you.

~~~
carterehsmith
Could that tool magically reduce the number of options? That would be quite a
tool.

~~~
bentcorner
You could subdivide what gcc does into other utilities each with a smaller
api.

~~~
TeMPOraL
This way lies Node.js. You'll end up with a complex config file just to
orchestrate all these smaller utilities.

------
p1mrx
It's strange that ls has grown to 58 options, but still can't output
\0-terminated filenames:

[https://unix.stackexchange.com/questions/112125/is-there-
a-r...](https://unix.stackexchange.com/questions/112125/is-there-a-reason-why-
ls-does-not-have-a-zero-or-0-option)

As an exercise, try to sort a directory by size or date, and pass the result
to xargs, while supporting any valid filename. I eventually just gave up and
made my script ignore any filenames containing \n.

~~~
jackewiehose
> I eventually just gave up and made my script ignore any filenames containing
> \n

I think we can all do that. Did anyone ever came across a legitimate reason
for filenames with newlines? All I can think of are intentionally bad-named
files to exploit bugs. Those filenames should be avoided in the first place.

~~~
strbean
Seems like a great argument for ensuring those are handled correctly, so you
don't get bit by a maliciously named file.

~~~
oblio
He is handling it correctly, he's ignoring those files.

~~~
why-oh-why
Maybe being ignored (i.e. hidden) _is_ the malicious intent.

What if the largest file on your drive has that filename and no tool can show
it?

------
dexen
_UNIX Style, or cat -v Considered Harmful_ ,
[http://harmful.cat-v.org/cat-v/](http://harmful.cat-v.org/cat-v/) (see the
linked PDF,
[http://harmful.cat-v.org/cat-v/unix_prog_design.pdf](http://harmful.cat-v.org/cat-v/unix_prog_design.pdf))

Another example: the modern Bash shell vs the Plan 9's Rc shell. The
difference in documentation is over tenfold, with the latter still being more
less laden with gotchas:

    
    
      $ man bash | wc -w
      48189
      $ 9 man rc | wc -w
      3264
    

Having said that, I still love me some sick _diff -burN_ -style mnemonics.

~~~
cmurphycode
Thanks for the new addition to my mnemonic collection!

perl -lane # process input by line, split into fields, print outputs with new
lines, execute code

netstat -planet # show process, extended info, all socket types

ss -pimento # basically every piece of info about live connections

iostat -txyz # extended, omit first report (the since-boot one), omit devices
with no activity, log datetime

~~~
dexen
Imagine an old PS/2 mouse with a mini Jack cord & plug. Non-standard but
works. I mean, on GNU/Linux:

    
    
      ps aux
    

Party time:

    
    
      rsync -rave ssh SOURCE DESTINATION

~~~
anthk
I use -avz

------
6gvONxR4sf7o
To some degree it reminds me of a change in Haskell that came about a few
years back. It used to be the case that many of the basic functions operated
on things like lists and functions. Tricky to learn but not too bad. It got
annoying when you wanted to use those tools on generalizations of lists, so
the language changed. Now those standard library functions operate on things
like the Foldable type class which is must follow some laws defined in terms
of endomorphisms. So simple functions like “any” or “concat” which used to
operate on lists and be super newbie friendly still can operate on lists in
the same way, but now do lots more, at the expense of the newbies (and
complexity).

I think the change was jokingly called “pulling up the ladder behind us.”

Learning some command line commands feels like learning a whole new framework.
A man page should rarely need to be a whole damned manual.

~~~
geofft
They still work on plain lists with the same syntax you could always use (that
is, lists are a member of the Foldable type class), right?

If so, then this primarily seems like an issue of how to present the
documentation and not the mechanics of the change itself. Instead of
autogenerating docs that say concat :: Foldable t => t (t a) -> t a (or
whatever) and making people think about Foldable, write docs that say that
concat is still [[a]] -> [a], with a note at the bottom saying, by the way,
this generalization exists, click here to see the more complicated version if
you've decided you care about Foldable. Probably the text of the docs remains
the same, in fact ("concatenate a list of lists together into a single list"),
you just display the real prototype and explain that "list" could also mean
something else. (There's a number of ways to implement this, from writing docs
by hand, to extending the doc protocol to support this specific case of lists
and Foldable, to extending the doc protocol to support documenting a fake
function prototype and then have a link to the real function at the bottom,
and using the type system to ensure that the fake prototype is just a special
case of the real function.)

A corollary is that documentation presentation is an inseparable part of
library / tool design.

~~~
kqr
The other problem are type errors, which would then require knowing about
Foldable to operate on lists.

The same type of solution that you propose for docs can work here too, though.
The compiler can trivially substitute the typeclass for any known instance.

------
lalaithion
I feel this way about optional function parameters. They have a tendency to
proliferate, because you don't break existing code by adding them, and they're
always optional.... until you have [https://scikit-
learn.org/stable/modules/generated/sklearn.sv...](https://scikit-
learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC),
where you have 15 optional parameters.

~~~
dgritsko
Relevant XKCD: [https://xkcd.com/1168/](https://xkcd.com/1168/)

~~~
chubot
The way I remember tar syntax is to use it more like a "normal" command. That
is, instead of the convenient but cryptic idiom:

    
    
        tar xvf foo.tar.gz  # extract, verbose, file
    

I use the longer but more readable stream idiom:

    
    
        tar -x -z < foo.tar.gz
        tar --verbose -x --xz < foo.tar.xz
    

(It does force you to specify -z for gzip or --xz for .xz, but I'm OK with
that redundancy in the name of readability.)

This makes it easier to remember the syntax for create and list too:

    
    
        $ tar --create e.txt k.txt  > out.tar
    
        $ tar --list < out.tar 
        e.txt
        k.txt
    
    

That's how I write the install instructions for Oil:

[https://www.oilshell.org/release/0.8.pre2/doc/INSTALL.html](https://www.oilshell.org/release/0.8.pre2/doc/INSTALL.html)

\-----

So basically tar can behave normally (at least GNU tar can) -- it's just that
there's an ancient idiom that most people have in their fingers, and that's
copied around in various instructions.

The nice thing about the verbose way is that the hyphen prefix enables
autocomplete.

~~~
jcelerier
I prefer my mnemonics :

    
    
        tar caf foo.tar.gz foo/ bar/ # Compress As Fuck
        tar xaf foo.tar.gz           # Xtract As Fuck

------
samatman
On balance, I prefer this. Extra command flags neither pick my pocket nor
break my leg.

The alternative (presuming the extra functionality is useful to someone,
somewhere) would seem to be more standard commands, and that choice would use
up more of the precious global namespace for small yet memorable lower-case
command names.

I prefer as much of that as possible be reserved for userspace. I'll admit my
mind boggles a bit when I ask myself what tar could possibly do with 139
flags, and it makes me a bit nervous about the surface area the program
exposes.

------
tclancy
I am with the author on the general point. But I think it ignores the fact
discussing how things were in the '60s when the commands were written just for
people ignores a fundamental change. To wit:

>the added options are convenience flags: setting the permissions of the new
directory and making parent directories if they don't exist.

Obviously completely unnecessary for normal users, but those have to exist for
automated processes. The "problem" is caused in part by the success of Unix,
that now computers are using computers.

~~~
oefrha
If you believe mkdir -p is completely unnecessary for normal users and only
exists for scripting, you probably want to think about how it might fit into
your workflow again. In fact, my mkdir is simply aliased to mkdir -p, and I
have a self evidently-named wrapper function called mkcd which I use even more
often than mkdir.

~~~
jmmcd
Wait, what does mkcd do?

~~~
Joker_vD

        function mkcd() {
            mkdir -p -- "$1" && cd -P -- "$1"
        }

------
dmitrybrant
ffmpeg says, "hold my beer"!

[https://ffmpeg.org/ffmpeg.html](https://ffmpeg.org/ffmpeg.html)

~~~
kqr
The same thing goes for ImageMagick, but that seems unfair because it's more
that they have created a command-line language for describing multimedia
transformation pipelines.

It's not as much that the arguments allow you to run a program under variant
configurations, but rather that you create completely new programs for each
run.

------
rb808
I think its just a reflection of how much more widely software is used, how
complicated our environments are, how many admins/developers there are
compared to the old days. Eg the aws command is hugely complicated, but I'm
not sure breaking it out into smaller commands easily really would make
anything easier, just pollute the command line namespace.

~~~
lotwxyz
In my web-based shell implementation [0], I use a library concept, such that
you have to "import" commands into the current execution environment. You can
either do this kind of thing:

    
    
      $ import fs
    

This imports all the commands in the 'fs' library, without any concern for
namespaces, and if there are any already existing commands with the same name,
they are not overwritten (the new ones are not imported).

But if you do this:

    
    
      $ import fs as myfs
    

Then, all the commands in the 'fs' library are invoked like this:

    
    
      $ myfs.vim
    

[0] [https://dev.lotw.xyz/shell.os](https://dev.lotw.xyz/shell.os)

~~~
mjcohen
Are you notified of the ones that are not imported?

------
mkchoi212
It seems like businesses are pushing towards releasing command line tools to
make their business "hacker friendly." One example I can think of is Github
and their new GitHub
CLI.[https://github.com/cli/cli](https://github.com/cli/cli)

~~~
mosaic-io
Super fan of the new Github CLI.

------
seemslegit
Command line options are a terrible surface for usability and more
importantly, discoverability. We are at least two decades overdue for
replacing them with a structured, typed, reflective API and yet every
generation of hackers persists to cargo-cult the cli utility, taking pride in
remembering copious minutia as if they were wizzards in command of arcane
incantations.

~~~
pm215
The lack of discovery is not inherent in a command line system. The usual
counter-example is something like TOPS-20
[https://www.bourguet.org/v2/pdp10/users/chap2#sec_2](https://www.bourguet.org/v2/pdp10/users/chap2#sec_2)
where the command line system was built such that for every command you could
interactively in the middle of typing your command line get help about what
the possible options were, what the argument for the option you'd just typed
was, etc (and this was built in to the system and commands, not a half-baked
add-on like bash tab-completion).

~~~
seemslegit
Didn't mean to imply that it was inherent, see the comment about powershell
bellow - just that that this is the standard we unfortunately came to accept
as adequate.

------
djsumdog
For the later years, it would be interesting to see the comparison for GNU
fileutils vs what's in the base of FreeBSD or OpenBSD vs what's offered by
busybox. The author mentions this briefly with Darwin's ls command (which
looks like it might just come directly from another BSD).

Also tar's increases might include new compressions methods like bzip2, XZ,
etc.

~~~
mprovost
Right, but what business does tar have compressing anything in the first
place? It's a tape archiver. You can pipe its input/output to compress or gzip
or bzip2. Early versions didn't have any ability to compress/decompress
archives and relied on external tools, but then that functionality was
absorbed, increasing the size of the codebase. And as you point out, every
time a new compression method is invented, someone needs to update a bunch of
tools instead of just installing one new utility.

~~~
dylan604
> but what business does tar have compressing anything in the first place?
> It's a tape archiver.

Most of the tape devices themselves had/have compression on them so that tar
did not need to do it. However, that compression was pretty weak (2:1), so
giving tar the ability to use bzip2 and send the compressed data stream to the
tape (bypassing the hardware compression) gave the user much more convenience.
Also, older systems probably were bad at doing the compression in real-time,
so it probably slowed down the write to tape. I'm just guessing, I have no
experience writing to tape from such an old system. I'm not really upset by
tar geting some extra features.

------
lancyH
This problem is universal. The solution is to use --words instead of -s single
letter options. Thankfully, git has a good manual that covers all the options:

[https://git-man-page-generator.lokaltog.net/](https://git-man-page-
generator.lokaltog.net/)

------
kstenerud
This is another reason why I'm developing an ad-hoc data representation with
strong typing. JSON was almost right, but failed because there are too many
fundamental data types that cannot be represented in a portable way.

A general purpose ad-hoc data format with all the fundamental types, that
doesn't require endless text parsing but can be converted to a 1:1 text format
for human viewing, would be helpful in so many situations, including
serialization for piping between unix tools.

[https://github.com/kstenerud/concise-
encoding#example](https://github.com/kstenerud/concise-encoding#example)

~~~
throwaway3157
Cool. When you evaluated all the alternatives, did you stash a pro/con list
somewhere on a blog? I’d appreciate reading it

~~~
kstenerud
I made a short list here [1]. I also kept a design document which explains the
choices I made [2].

[1] [https://github.com/kstenerud/concise-encoding#comparison-
to-...](https://github.com/kstenerud/concise-encoding#comparison-to-other-
formats)

[2] [https://github.com/kstenerud/concise-
encoding/blob/master/de...](https://github.com/kstenerud/concise-
encoding/blob/master/design.md)

~~~
kentonv
Hmm, I think your table is using a different definition of "zero-copy" than is
currently common when discussing serialization formats.

In your comparison you claim Concise is zero-copy, but it does not appear to
be so, at least in the sense that Cap'n Proto and FlatBuffers are. Concise
appears to use variable-width integers, tag-value sequences, and 8-byte
alignment, all of which make encoded messages unsuitable as in-memory data
structures; it looks to me like you have to parse the message into some sort
of AST before you could meaningfully operate on the content. "Zero-copy" in
the Cap'n Proto / FlatBuffers sense means that there's no need to do any
"parsing" because the whole message is efficiently traverseable as-is; e.g.
you can mmap() in a very large message and then find and access any one value
within it in O(1) time (or perhaps O(log n), if the data structure is nested).
This generally requires that primitive values have fixed widths and fields
have fixed offsets within their parent object.

Digging into your docs it looks like what you really mean by "zero copy" is
that individual strings or byte sequences can be used directly without copying
them out of the original message buffer. This is a fairly common property of
any binary protocol; e.g. Protocol Buffers can do this (albeit without NUL-
termination), but in your table you have indicated that it cannot. I would
maybe call this "zero-copy strings" to be clearer.

On another note, in your table you have indicated that Cap'n Proto doesn't
have a "String" type, but this is not true -- Cap'n Proto's "Text" type is a
UTF-8 string (and is even NUL-terminated on the wire).

(I'm the author of Cap'n Proto.)

------
hyperion2010
Anecdote from yesterday. Me staring at merge-pathnames and wondering why
everything was backwards from the physical layout of the paths. Ah, it is
because the default is optional because of the direction in which the function
is usually used. If I didn't take the 5 minutes to wonder about what I was
missing I would have implemented a macro to do it the way I thought was most
sensible. In the context of the op, I can only imagine that the impulse to
pause and reflect on the question of why, or what motivates the desired for a
new option has fallen victim to the fact that there are thousands of people
for whom the flag solves a real and tangible problem immediately.

------
socialdemocrat
Hmmm... maybe there is an opportunity here in replacing all the Unix tools
with simple versions.

I personally don’t use most of these options and their existence just clutter
up the manual.

------
pixelbeat__
GNU coreutils has been very careful when adding new options. We're also
careful to provide constructive feedback on why it might be best _not_ to add
an option, and have summarized many of these discussions

[https://www.gnu.org/software/coreutils/rejected_requests.htm...](https://www.gnu.org/software/coreutils/rejected_requests.html)

------
anonymousiam
The only command on the list (from 1996, not 1979) that still has no arguments
is 'clear'.

Back before clear, I used 'echo ^V<esc>c', but that's not portable to
different termcap/curses/ncurses display devices.

------
pcj-github
Seems like a natural progression given the growth in usage of these tools over
the last few decades. How many people were using these tools in 1979 compared
to today?

------
OskarS
This bums me out as well. One of the prime examples of this is tar: whenever
someone says "extract this tarball", they usually give the command as this:

    
    
       tar xfvz my-source-code.1.2.3.tar.gz
    

What does those flags mean? Who knows! I mean, I know because I've used it
enough, but for many people those are just a mysterious incantation (there's
even an xkcd about it! [0])

The reason is arguably that tar has too many responsibilities here. Tar should
only do two things: create tar streams and extract tar streams into files. You
do that with "tar c" and "tar x", and that's everything you should need to
remember (ok, fine, "tar t" for listing files is good to know as well).

Commands to create tarballs should look like this:

    
    
        tar c directory | gzip > directory.tar.gz
    

And untar commands should look like this:

    
    
        zcat directory.tar.gz | tar x
    

or maybe

    
    
        cat directory.tar.gz | gunzip | tar x
    

or whatever version you prefer with "tar x" at the end. This is composable,
super-easy to remember, and you can VERY easily swap out any parts for any
other parts (for instance, changing the source from a file to a url with curl,
or the destination for an ssh pipe or something).

One of the benefits of the "swapping out" parts are of course that you're not
restricted to what's built into tar: you can use ANY compression utility you
want. For instance, there's a thing called pigz[1] which parallelizes gzip
decompression for increased speed. It's trivial to swap into the pipeline when
you use tar that way:

    
    
        cat directory.tar.gz | unpigz | tar x
    

Congratulations, you now have much faster expansion of tarballs, without
needing to recompile tar.

We've taught a generation of programmers that "tar" is hard to use and you
have to look up a magic incantation every time you need it. Ironically, that
makes tar _less flexible_ , because now you think of it as a single-purpose
utility (creating and unpacking compressed tarballs) instead of a versatile
unix tool which can be used in all sorts of different situations.

[0]: [https://xkcd.com/1168/](https://xkcd.com/1168/)

[1]: [https://zlib.net/pigz/](https://zlib.net/pigz/)

~~~
intrepidhero
If I want to untar a file, odds are I also want to decompress it so I like
having both features in one tool. That said I also really appreciate your
composable approach. I wish my early unix education had leaned in pipes a lot
harder.

But what I came here to say is that I find the typical tar flags pretty
intuitive. X means extract, z means using zip/unzip, v means verbose, and f
for file I/O instead of stdin/stdout. For typical use cases, I don't find its
interface magical but logical. I'll freely admit to needing 'man tar' anytime
I stray from the typical use, but that's true of a lot of commands.

The bigger offender in my mind is rsync. I remember auvP as my typical flags
but don't ask me what they do. And good luck guessing whether my destination
or source path need a trailing slash on the first try...

~~~
OskarS
Yeah, it's true, rsync is a real nightmare when it comes to arguments.

I remember reading that xkcd comic though and I went "yup, that's me!", and
I've spent a bunch of time thinking about why that was. You list those flags
there and say there intuitive, but they're unique to tar, you have to memorize
them, and there's no reason that they need to be in tar when they're available
right there in the shell.

Why do I need to memorize that the "f" flag is "file output/input", when I can
just write "> dest-file" or "cat src-file |" (or "< src-file") in my shell?
That works for every command, not just tar. Why do I need to remember that the
z file is for gzip (what's the equivalent flag for bzip2?) when gzip/gunzip
exists and works for _anything_?

So many people simply don't understand what tar does or how to use it without
googling the magic formula they need, and it's because of all these damn flags
that everyone insists on using instead of doing it the "unix way".

~~~
jandrese
Backwards compatibility is a bitch mostly. Tar defaults to using the tape
drive because that's what it was originally designed to do. Using it to create
file archives was a hack added later, but nobody wanted to break backwards
compatibility with old backup scripts by changing its defaults, so we're stuck
with its oddball syntax today.

dd is another program where the commandline flags predate modern usage
(although it does at least behave nicely with pipes by default) and look
totally weird compared to everything else. But again nobody wants to touch
them because it would break millions of existing scripts.

------
ggm
a plague on getopt, and especially gnu getopt.

that said, I live in it.

------
jokit
I have wondered how it is that my Mac Plus became a machine with almost no
noticeable delays when I upgraded to 4mb ram, and now I have 8gb of ram, and a
less responsive system. I realize my system is far more dynamic today. I just
wonder if one day we'll be using systems with 4tb of ram that would be
snappier if upgraded to 8tb or 16tb.

Given the nature of open source, I'd be surprised if there isn't a alternative
for less dynamic, less resource req, systems. Is there a system out there that
still uses fewer commands?

I don't know how to ask this next question, so please bare with me.. Are there
any modern systems that don't require modern hardware, but are just more
efficient versions of past systems?

I have been happy with the way I could use my computer for the last 20 years
(probably earlier, too).. so are there any efforts to keep the simplicity of
older systems that were adequate for most, make them more efficient, and
either have "super hardware", or I suppose very cheap hardware?

Thanks for anyone who made it through my ignorant ramblings.

~~~
bear8642
>I just wonder if one day we'll be using systems with 4tb of ram that would be
snappier if upgraded to 8tb or 16tb.

Can't find but feel there's a quote like "Programs will grow to fill memory"

~~~
jokit
I know that some people are only as efficient as they're forced to be. I hope
there are efforts to take what is needed, and make it more efficient. There
are efforts like Commander X16 to make a new 8 bit computer, and others where
people are rewriting things in Rust..

BTW, I have a first gen iPad sitting around that worked very the first two
years or so of its existence, and it barely functions today. I know part of
that is Apple "protecting" older batteries by slowing down older stuff so we
go out and buy new things, but I would like to believe that we're capable or
creating an OS for that hardware today that would run better than it did
originally, rather than barely function at all.

