
Ambitions for a Unix Shell - luu
https://www.oilshell.org/blog/2020/01/ambitions.html
======
enriquto
This is some of the most exciting stuff happening in the unix world these
days.

My mind was blown after reading two posts on Oil's author blog, a few years
ago:

* Shell has a Forth-like Quality: [https://www.oilshell.org/blog/2017/01/13.html](https://www.oilshell.org/blog/2017/01/13.html)

and

* Pipelines Support Vectorized, Point-Free, and Imperative Style: [https://www.oilshell.org/blog/2017/01/15.html](https://www.oilshell.org/blog/2017/01/15.html)

Warning: these writings can change your view on the shell language. Now you
probably think, as many of us did, that shell is a cute little hack, useful
for one-off throwaway stuff, but not really fit for serious work. Afterwards,
you will start to see the shell as a glorious, essential element of our
civilization, worth of respect and deserving our careful attention. Pipes are
an essential programming tool that must be supported by the core language and
not by a library construct (as in python).

~~~
jstimpfle
I've written many, many shell scripts, back as a sysadmin, and because I
wanted it to not be a cute little hack. I know all the tricks described on the
blogpost and many more. I have an intimate understanding how the quoting works
and how we can work around its limitations.

Let me tell you: shell is sort of a local optimum for one-off throwaway stuff,
because it makes commonly needed thing accessible with so few keystrokes. But
also, shell is a cute little hack. It's literally impossible to write robust
programs in it.

~~~
chubot
_It 's literally impossible to write robust programs in it. _

Right, that's the whole point of Oil! You can make those programs robust
without rewriting them entirely. Run them with Oil, and then flip a few
switches to enable stricter behavior.

Here are a few examples:

 _Oil 's Stricter Semantics Solve Real Problems_

[http://www.oilshell.org/blog/2019/08/16.html](http://www.oilshell.org/blog/2019/08/16.html)

But there are a dozen more. The key point is that bash's codebase has kind of
"topped out" in the sense that it's hard to add anything without breaking
something else. On the other hand, the Oil interpreter is reasonable, small,
and you can add features to it, and fix things.

Anyone who knows about programming languages might be able to read the type
definitions in osh/runtime.asdl (and frontend/syntax.asdl) and learn a lot
about how shell works.

~~~
ashkankiani
At that point, just use an actual programming language instead of a shell...

~~~
coldtea
That misses the whole point - that there shouldn't be any difference.

Shells already behave like programming languages just bad ones.

The reason people use them over a "bona fide" programming language is their
command line editing, easy process handling, first class support for job
control, access to environment and pipes.

One could use a non-shell programming language, but then he'd have a much
harder time with all the later - even though he'll have a much better time
with abstraction, programming correctness, error handling, first class data
structures, and so on.

So we're in an either (shell+easy command line use+process handling/job
control/pipes etc) or (programming language+better syntax-cohesion-
correctness-better error handling-libraries-first class data structures-and on
on).

There are things in IT that can't be combined because one is a tradeoff to the
other. This is not such a case - this is just a historical accident: crappy
language features in shells and crappy job control features in programming
languages.

But there's no reason one can't have all those latter things in a proper
programming language.

Which is what Oil is.

------
jerf
"So the biggest change is that Oil will be based on strings (and arrays of
strings) like shell"

I would advise the author have a look at the internals of Tcl, not because you
should copy all the sophisticated features immediately, but because it
demonstrates a way to have both "everything is a string" _and_ "advanced data
structures" in a relatively principled manner. You don't have to start out
with all the optimizations from day one but I always find it helpful to have a
look down the road in cases like this, and make sure that if this sort of
thing interests you that you don't accidentally make it impossible later.

~~~
chubot
(author here) Yeah I appreciate a lot of things about Tcl, which is basically
Lisp + shell (I think Ousterhout even said that?) I read most of the thick
book about it even though I don't regularly program in Tcl.

A bunch of people have brough up Tcl, and as noted here [1], Oil looks
superficially like it because it has proc and set keywords.

The biggest difference is that Oil isn't homoiconic in the sense that Tcl is.
I believe that implies a lot of dynamic parsing which I'm staying away from
[2].

And of course Oil is a compatible Unix shell, which imposes a bunch of
constraints. Namely the 'proc' is "isomorphic" to a process -- it takes argv
and returns an exit code, and it can be transparently put in pipelines. I
wrote 3 years ago about this powerful form of composition [3] and I wish I had
finished that :-/

[1]
[https://www.reddit.com/r/oilshell/comments/esqyaw/ambitions_...](https://www.reddit.com/r/oilshell/comments/esqyaw/ambitions_for_a_shell/)

[2]
[http://www.oilshell.org/blog/2019/02/07.html](http://www.oilshell.org/blog/2019/02/07.html)

[3] [http://www.oilshell.org/blog/tags.html?tag=shell-the-good-
pa...](http://www.oilshell.org/blog/tags.html?tag=shell-the-good-parts#shell-
the-good-parts)

~~~
sansnomme
What are your thoughts on MSFT PowerShell?

~~~
chubot
Since a few people have asked, I might write a FAQ about this, but the short
answer is that I view PowerShell as one of the projects called "shell" that is
least like a Unix shell.

Part of that is on purpose, because it's designed for Windows. And what works
on Windows is very different than what works on Unix.

Windows is based around COM objects and binary formats like the registry,
while Unix is based around text files. That makes the design of an appropriate
shell extremely different.

I would say bash, zsh, and fish are pretty close (with the latter
concentrating on interactivity), but PowerShell is way out there.

PowerShell also appears to be coupled to the .NET VM and is pretty
inefficient. Some links and sharp comments here:

[https://lobste.rs/s/16djrz/type_safeness_shell#c_gtmqcl](https://lobste.rs/s/16djrz/type_safeness_shell#c_gtmqcl)

I think if you want a C# like programming language with shell-like syntax,
that integrates with Windows, then PowerShell is for you. But most bash users
don't want that! As mentioned, shell is used on tons of embedded systems now
(usually busybox ash, which is growing towards bash.)

~~~
ripley12
While I don't disagree with most of your points, NuShell's recent popularity
seems to suggest that there is a demand for shells that operate on structured
data – even in the Unix world. I dislike much of PowerShell's UX (syntax,
tooling, etc.) but I _love_ that aspect of it.

I see from the blog post that you've decided that Oil will operate on strings
instead – I'm really looking forward to the blog posts explaining that, I'm
sure it's not a decision you took lightly.

~~~
chubot
Here's some more color on that:
[https://news.ycombinator.com/item?id=22157587](https://news.ycombinator.com/item?id=22157587)

The summary is that I "left space" for types, but they won't be Python types,
for several subtle/philosophical reasons.

------
skrebbel
Mostly off topic, but I've never understood why shells are designed to have
this weird dual use case of both being easy to type into a command prompt, and
being easy to write 40+ line scripts in. Why does that need to have the exact
same syntax? This is the root cause of why bash (and batch and etc etc etc)
are so terrible to script in (and why powershell is so terrible to type into a
terminal). This weird shit like [ being a command, and having to think long
and hard about exactly when variables are replaced and when globs are executed
and how errors happen.. It's just pretty pointless.

Like, if you write a script using Python or Ruby or Perl or whatever, there's
easy ways to call into the shell. Often it's even built-in, like using
backticks `echo like this`. These languages aren't designed for shell
scripting but even they are often easier and cleaner than a bash script. It
could be much better thoughk I bet.

I'd love to have a language that is easy and familiar to type into a terminal
(and thus, sufficiently bash-like probably), while being significantly easier
to write scripts in. You could require the script to have a bit more syntax, a
bit more structure, but essentially have the same semantics as what you'd type
on the command line, and you'd have the best of both worlds. If well designed,
it'd still "feel" like the same thing (much like how all JS programmers
immediately grokked Coffeescript when it came out, because it really was just
JS); everything you'd learn on the command line, you'd be able to apply in a
script and vice versa. The script would just require more structure and thus
provide more comfort (good error messages, editor support, etc).

For example, globbing. The idea that the shell expands " _.foo " into a list
of files, before evaluating the command, is handy, but it's also an enormous
footgun. What if on the terminal it would do that, but when scripting, you'd
be required to write `glob("_.foo")` or else it'd error out? THat would force
you to clearly control when the expansion happens, it would let you assign it
to a variable in a readable way etc. Imagine similar stuff for argument lists,
escaping, variable substitution, etc - all the magic available on terminal,
but contained in the script.

Does this exist? I'd love to have it.

~~~
GSGBen
(Replied to op instead of comment):

Is your complaint about PowerShell the verbosity? Because I love its approach
to this: full verbosity is the default (great for scripts), then terminal
usage is improved by aliases (ForEach-Object is just %), great autocomplete
via psreadline, and automatic parameter shortening: -re and -fo are
automatically understood as -Recurse and -Force. Honestly makes every other
shell a pain to use now for me.

Also having everything be object based is just...I have no idea how I lived
before.

~~~
chubot
Here are my comments about PowerShell:

[https://news.ycombinator.com/item?id=22155943](https://news.ycombinator.com/item?id=22155943)

Oil is really aimed at bash users and not PowerShell users. There is some
overlap between the set, but only a bit.

------
keeganpoppen
not only is this an extremely exciting project that I've been following for
what feels like years, but there is such a tremendous richness of stuff only
tangentially related to the project that the author has collected as well (one
example:
[https://github.com/oilshell/oil/wiki/ExternalResources](https://github.com/oilshell/oil/wiki/ExternalResources)).
this is clearly such a well-organized, thoughtful endeavor that i have
absolute faith that great things will come of it. and he's an excellent writer
to boot!

~~~
chubot
Thank you! I'm trying to reign in the scope to make it useful sooner, and I'm
always looking for help:

[http://www.oilshell.org/blog/2019/12/09.html#help-
wanted](http://www.oilshell.org/blog/2019/12/09.html#help-wanted)

If you don't have time, maybe you know a programmer who does, and let them
know :)

------
techdragon
I’m looking forward to the post mentioned towards the end, detailing the
cutting of features. I’ve been following the work on this for ages and was
looking forward to types data as a way to guard against accidental mistakes,
so I’m keen to read the author’s reasoning for switching to strings and arrays
of strings. I’m sure there’s a non-trivial reason, his work has always felt
careful, measured and well though out to me.

~~~
chubot
Thanks for the support! Since I may not get to those blog posts in a timely
fashion, here's a little outline (and maybe this comment will form the seed of
them).

\-----

I should be more specific: I _am_ ruling out reusing Python data structures,
but I'm _not_ ruling out typed data forever. I'm leaving space for types.

Though given all the work I see in the next year, I don't see a ton of work on
them happening. If they happen, they won't not literally be Python types,
because they don't seem like a great fit.

Background: A major motivation for using Python was that I wanted JSON in
shell. JSON is a "versionless" textual interchange format that's ubiquitous on
the web, and I see the web as an extension of Unix.

And there's an obvious correspondence between Python data structures and JSON:
dicts, lists, string, float/int, bool.

But those structures are still a "model", and I've had many programming
experiences that led me to believe that "bytes on disk/network -> data
structure in memory -> bytes" is something of a fallacy.

Or at least it's a coarse, limited model, and not very shell-like. (For the
purposes of a blog post I should find a name for that fallacy.)

It very much relates to these comments:

\-
[https://lobste.rs/s/uizrgy/making_light_website_02_design#c_...](https://lobste.rs/s/uizrgy/making_light_website_02_design#c_xnagup)
\-- a different (old) way of HTML processing I encountered and used that's not
(text -> DOM -> text)

\-
[https://news.ycombinator.com/item?id=22156950](https://news.ycombinator.com/item?id=22156950)
\-- Making the analogy to the flawed Python 3 string type, which is (bytes ->
code points -> bytes)

\-
[https://news.ycombinator.com/item?id=22111403](https://news.ycombinator.com/item?id=22111403)
\-- Kragen's "memory models" post got me thinking about these things.

Another way to say it would be: instead of "dict, list, string" as the model,
I think a better model for a shell is "JSON, CSV/TSV, and HTML". In other
words: (1) tree shaped data, (2) table shaped data, and (3) documents. [1]

In a more literal and concrete fashion -- NOT really high level abstractions
for them. Maybe another slogan could be "Bijections that aren't really
bijections".

However I'm not promising to get to this in the next year :)

\----

But practically speaking, a lot of programming is spent shuffling between
representations. And a lot of the times that's not only inefficient, but also
incorrect. It relates strongly to this post:

 _How to Quickly and Correctly Generate a Git Log in HTML_

[http://www.oilshell.org/blog/2017/09/19.html](http://www.oilshell.org/blog/2017/09/19.html)

If you want a buzzword I would say shell is a language where you deal with
"concretions" \-- that is, you deal with reality rather than abstractions. But
a crucial point is that a language can still help you avoid mistakes when
programming concretely -- in a large part with principled parsing and
quoting/dequoting.

You're not going to "groveling through backslashes and braces one a time" as I
call bash's style. The language can provide sharper tools than (JSON ->
dict/list -> JSON) or (HTML -> DOM -> HTML).

\----

A related thought is that I noticed that procs and funcs don't really compose
(procs being shell-like functions, funcs are Python/JS-like functions).

As an analogy: If you've ever written a Python/C extension or a Node/C++
extension, you will also see that those models also compose pretty poorly.
Despite the similar syntax, there's a surprising amount of friction
semantically (types and GC being two main things).

So adding typed data and Python-like functions was almost tantamount to adding
a "separate" language within shell itself. The language loses its coherence
when the primitives don't compose. There can be too much shuffling.

\-----

So that's a bunch of thoughts that may go into a blog post. Hopefully at least
a few parts made sense :)

[1] This reminds me that there's a paper _Unifying tables, objects, and
documents_ that I think influenced C# ? I should go re-read it. An important
point is that objects are a often a poor fit for representing tables and
documents (e.g. ORM problem, awkward DOMs), and tables and documents are more
prevalent in the wild!

You can deal with tables and documents "directly", rather than awkwardly
manipulating object/tree representations of them. CSS selectors are a way of
doing this, i.e. you're not "shuffling objects". A comparison I would draw is
2004-style DOM API vs. jQuery / HTML5 DOM.

[https://scholar.google.com/scholar?cluster=12582942667964919...](https://scholar.google.com/scholar?cluster=12582942667964919771&hl=en&as_sdt=0,5&sciodt=0,5)

------
celeritascelery
> Replace the world's usage of bash.

I feel like this is hardest part. Bash is installed everywhere, and that
ubiquity is one of its greatest assets. Oil can’t replicate that (at least not
for a very long time).

~~~
fergie
And bash is also perfectly OK. It doesnt have all the bells and whistles of,
say, zsh- but the things that it does do, it does better (especially
autocomplete).

If you want to write scripts, bash is often a pretty good choice for anything
involving simple manipulation of text files. If you want something more
advanced then bash will let you edit, compile and run programs in the language
of your choice.

That said- there may well be a killer usecase for Oil. If so- what is it?

------
chadrs
This kind of reminds me of Ammonite: [https://ammonite.io/#Ammonite-
Shell](https://ammonite.io/#Ammonite-Shell) which definitely for a different
audience but kind of a blend of being a Scala repl and somewhat-usage shell
with several common commands being implemented as JVM functions with sensible
types.

------
Riverheart
Good luck Chubot! If you release in 2020 I look forward to using it.

~~~
chubot
I release all the time, you can try it now :) I'm looking for feedback -- a 5
minute first impression is valuable.

[http://www.oilshell.org/releases.html](http://www.oilshell.org/releases.html)

The home page has links on "where to send feedback" and what to try / "help
wanted":

[http://www.oilshell.org/](http://www.oilshell.org/)

------
boona
>What languages and tools does Oil aim to "replace"? >bash, awk, make, Python
2

What will in fact happen: It says here on your resume that you know bash, awk,
make, and python 2, but do you know Oil as well?

~~~
chubot
"replace" is in quotes for a reason :) The rest of the post gives some more
color on it.

