
Shell Has a Forth-Like Quality (2017) - pcr910303
http://www.oilshell.org/blog/2017/01/13.html
======
chubot
(author here)

Oil has made a lot of progress in the last 3 years, with a few releases every
month.

[http://www.oilshell.org/releases.html](http://www.oilshell.org/releases.html)

The goal of the project is to replace bash!

If you know shell and Python (or C++), I maintain a "help wanted" bug label:

[https://github.com/oilshell/oil/issues?q=is%3Aissue+is%3Aope...](https://github.com/oilshell/oil/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22)

It's almost there, except for speed, but it needs more testing from users. The
"should-run-this" label has some ideas:

[https://github.com/oilshell/oil/issues?q=is%3Aissue+is%3Aope...](https://github.com/oilshell/oil/issues?q=is%3Aissue+is%3Aopen+label%3Ashould-
run-this)

There a lot of project updates on
[https://oilshell.zulipchat.com/](https://oilshell.zulipchat.com/) too
(requires login)

~~~
zomglings
Very curious - what kind of people use OSH and how many actually use Oil in
OSH?

I am hesitant to spend the time ramping up on something like this because, as
much as possible, I try to write my shell scripts so that they can run without
any additional packages in an Alpine docker container.

Is this resistance that you have encountered before?

~~~
chubot
Right now it's mainly a few shell enthusiasts and early adopters, i.e. people
who want a better shell and want to influence the design of the Oil language.
[1]

I've definitely seen people who want to limit themselves to either bash or
POSIX shell on purpose, and that's fine.

For simple shell scripts, I might lean that way too (at least until Oil is
smaller/faster).

One major reason for Oil is to give you an upgrade path after you've used and
enhanced such a shell script for years. Then you will likely bump up into
shell's limitations, and the extra dependency will be worth it.

For example, all Linux distros are based on thousands of lines of shell
scripts [2], and I think those scripts desperately need a new language.
They're big and complex, and need a better language to be correct, robust, and
fast.

Here's a comment I wrote that might explain the design goals of the project
more:

[https://www.reddit.com/r/ProgrammingLanguages/comments/ddhmj...](https://www.reddit.com/r/ProgrammingLanguages/comments/ddhmj9/you_can_now_try_the_oil_language/f2htup3/)

[1] _Oil Language Design Notes #1_
[http://www.oilshell.org/blog/2019/08/22.html](http://www.oilshell.org/blog/2019/08/22.html)

[2] _Success with Aboriginal, Alpine, and Debian Linux_
[http://www.oilshell.org/blog/2018/01/15.html](http://www.oilshell.org/blog/2018/01/15.html)

~~~
zomglings
Thank you, your response was very informative. It got me to dive a bit deeper
into some of your essays and blog posts. I learned a lot particularly from
these:

1\.
[http://www.oilshell.org/blog/2016/11/06.html](http://www.oilshell.org/blog/2016/11/06.html)

2\. [https://github.com/oilshell/oil/wiki/Shell-
WTFs](https://github.com/oilshell/oil/wiki/Shell-WTFs)

------
dang
Discussed at the time:
[https://news.ycombinator.com/item?id=13477842](https://news.ycombinator.com/item?id=13477842)

~~~
macintux
Thanks, that's some quality discussion. Really impressed with the depth and
breadth of the commentary.

~~~
g82918
I really like oilshell and chubot for that. He has some really great posts on
the shell language and parsing.

------
Ididntdothis
When I see stuff like this I always wonder how people above about finding
these capabilities. I don't use bash every day so I tend to forget a lot of
"tricks". But then I get annoyed when I see people write about really neat
stuff that can be done in bash but couldn't figure out myself.

~~~
chubot
(author here)

I agree a lot of this stuff hasn't been documented well in existing shell
resources. I should write more about it, but I'm still busy working on the Oil
shell implementation nearly 3 years later :) I should have written a post
about xargs 2 years ago, which has nontrivial compositional patterns.
Hopefully next year!

I still recommend _The Art of Unix Programming_ (mentioned in this post),
which is more generally about Unix design, not about shell in particular.

I also got a lot of ideas by reading DJB's shell and C code, and his papers
(e.g. _Ten Years of qmail Security_ ). I probably learned about DJB 10 years
ago from Hacker News.

I would say there's a lot of "tribal knowledge" in people's heads, and online
forums are where it gets disseminated. For me those forums were Usenet, the
old JOS forum, and Hacker News and lobste.rs these days.

\----

Neil Stephenson has a good quote about this "oral history"

 _Unix, by contrast, is not so much a product as it is a painstakingly
compiled oral history of the hacker subculture. It is our Gilgamesh epic._

[https://en.wikipedia.org/wiki/In_the_Beginning..._Was_the_Co...](https://en.wikipedia.org/wiki/In_the_Beginning..._Was_the_Command_Line)

[https://en.wikipedia.org/wiki/Epic_of_Gilgamesh](https://en.wikipedia.org/wiki/Epic_of_Gilgamesh)

The next post has the "point-free" pipline pattern which is both useful and
hard to find out about!

    
    
        hist() { sort | uniq -c | sort -n -r; }
    

_Pipelines Support Vectorized, Point-Free, and Imperative Style_

[http://www.oilshell.org/blog/2017/01/15.html](http://www.oilshell.org/blog/2017/01/15.html)

------
haolez
I’ve done my share of shell tricks in the past and I love them, but I got very
disappointed when I noticed that the performance sucks for anything more data
intensive in the shell. At least in bash, which was the interpreter that I’ve
used. I ended up porting my code to Python and gained at least 10x in
performance.

But I prefer the shell code for tasks interfacing with system tools. I wonder
if there is a shell interpreter around that is really fast.

~~~
chubot
If you have some realistic shell benchmarks, please file a bug and link the
files:

[https://github.com/oilshell/oil](https://github.com/oilshell/oil)

Oil isn't fast at the moment, but I'm currently optimizing it (auto-
translating some code to C++, and eventually hand-optimizing other parts.) I
run benchmarks on every release, so speed is definitely a goal.

I haven't seen that many shell scripts that are "CPU bound" (as opposed to
waiting for processes or I/O). (Although now I recall a shell-script-library
bundler project that was.)

Of course, one big problem with shell is that you often have to spawn an
external process like sed or awk to do string manipulation that's just a
library call in Python. That could easily be a three order-of-magnitude
difference -- e.g. milliseconds vs. microseconds.

~~~
stephenr
I have a feeling the “bundler” lib you mentioned is mine - we discussed it a
bit on lobste.rs a few years ago? Good to see you’re still plugging away on
oil!

~~~
chubot
Yes! I remember it was a good workload because it was doing a lot of string
manipulation in pure shell.

I'm still interested in more feedback on Oil (e.g. does it still work on the
latest version of shell-script-library? :) )

It's changed a lot, and the Oil language now exists! Latest post: _You Can Now
Try the Oil Language_ \--
[http://www.oilshell.org/blog/2019/10/04.html](http://www.oilshell.org/blog/2019/10/04.html)

~~~
stephenr
The testing setup for the lib is much improved, with actual unit tests and a
built in capability for building/testing against a specific shell.

Next time I’m working on it I’ll grab a release of oil and see how it fares!

