
Oil Shell: Success with Aboriginal, Alpine, and Debian Linux - chubot
http://www.oilshell.org/blog/2018/01/15.html
======
detuur
Someone should push a patch to upstream bash that will make any script longer
than 50 lines abort. If you're building something that takes 2500 lines of
shell script, you're using the wrong tool for the job. You're trying to make a
program, but in a syntax (I was going to say language, but shell scripts
aren't a programming language) that is utterly unsuited for anything like it.

A shell script has a very simple function: execute a number of tedious but
common commands. That's it. Unfortunately someone decided they wanted control
flow, so they wrote the horrifying [ and [[ programs, which I still consider
to be a crime against humanity. Enter decades of write-once-read-never scripts
that run with abysmal efficiency, kill maintainability, and neuter innovation.
And somehow the entire Linux ecosystem depends on it and now every distro has
to ship a shell with cruft dating back to 1970, with probably as many hidden
vulnerabilities. And now every shell that tries to position itself on the
market has to implement the same quirks, bugs, and design flaws in order to
even be considered.

Let shells be shells, let shell scripts be dumb as bricks, and please use
python or even perl for the love of god if you're going to make a build
system.

~~~
effie
So much arrogance and negativity, so little evidence for the claims you make.

> A shell script has a very simple function: execute a number of tedious but
> common commands. That's it.

And why is that? What will happen to me if I write more complicated shell
scripts? The functionality is there and one can write very expressive and
readable programs in shell. I would definitely recommend shell for writing
long programs (for example various system administration and web backend tasks
which do not require complex libraries). Preferably bash since it is the most
common one.

> And somehow the entire Linux ecosystem depends on it and now every distro
> has to ship a shell with cruft dating back to 1970

The 'somehow' has quite a simple rationale, it isn't something that should
outrage you. The family of Linux-based OSes take inspiration and compatibility
target in unix systems on purpose. Unix systems are primarily operated via
shell.

The current standard of shell may be an old idea from 70's with some 'cruft',
but what language does not have cruft? The shell has proven to be usable and
worthy of continued use. The compatibility with the unix shell is a big part
of the early success of GNU/Linux and majority of users need it.

> And now every shell that tries to position itself on the market has to
> implement the same quirks, bugs, and design flaws in order to even be
> considered.

I have no idea what you are talking about.

> please use python or even perl for the love of god if you're going to make a
> build system

OK, so you think perl and python are better for making a build system. Have
you made one?

~~~
stephenr
> bash since it is the most common one.

Im pretty sure you meant to say "POSIX, because its rules are supported by
multiple shells, and you can't depend on bash being available or a recent
version on some systems".

~~~
effie
Actually, I meant bash, that is, if one works solely with Linux systems. Bash
has some very useful features and is the default on common Linux systems.
Those are not POSIX. It was frozen in time long ago and is very limiting, do
not wear that straightjacket if you do not have to.

For commercial unices or bsd's bash could be hard to use and using POSIX shell
may make more sense.

~~~
stephenr
From my experience, the extras Bash give you are likely to be bigger hints
that it _is_ time to move to a more appropriate language.

I'm definitely pro on shell scripts, but I'm also willing to use a different
language when posix SH isn't going to cut it.

------
jitl
Here's what the plan seems to be from reading the blog and the README:

1\. Re-implement Bash and similar shells in a modern programming environment
with an eye to extensibility

2\. Make good debugging tools because this is now easier because (1).

3\. Create a new language that extends Bash, similar to how ES6 extends ES5,
for a gentle migration of large shell scripts to a safer language

What, other than static parsing, should this endgame language offer? Where
does it fall, on the gamut from Bash to Haskell? We already have quite a few
languages that offer a souped-up scripting experience.

When my bash programs become too long, I usually reach for Ruby, which has
convenient back-ticks syntax for shelling out, and many convenience methods on
its built-in types for mashing strings and such. But like Bash, too much ruby
eventually becomes an unmaintainable mess without strict standards
enforcement. What will Oil offer for me, over Ruby?

Here's how I'd order existing languages on a scale from Bash to Haskell:

1\. Bash with `set -e`

2\. Perl

3\. TCL (has declared arguments, after all...)

4\. Ruby

5\. Python

6\. Golang

7\. Java

8\. C++

9\. Ocaml

10\. Haskell

~~~
chubot
Good questions. It's clear from this comment thread that I need to explain Oil
better on the blog (which I mentioned in the conclusion), but I'll respond
here briefly.

I somewhat agree with your spectrum of strictness, but the other dimension is
"how suited is this language for shell-like problems?".

C++ is strict, but obviously not good for writing "shell scripts" in.

Oil will be somewhere around Python in terms of strictness, but at least as
suitable as bash for shell-like problems (of course). The goal is for it to be
more suitable, but I haven't gotten there yet.

As far as what it offers, that is scattered over the blog, but this tag might
give an idea:

[http://www.oilshell.org/blog/tags.html?tag=oil-
language#oil-...](http://www.oilshell.org/blog/tags.html?tag=oil-language#oil-
language)

As a concrete example, Oil will be based more around arrays of strings than
flat strings. The "Thirteen Awkward Ways..." post hints at that. FWIW, the Ion
shell, part of Redox OS, was influenced by this way of thinking.

Also, I need to write a blog post entitled "Python Is Not an Acceptable
Shell". There was a nice blog post that translated shell to Python that
inadvertently proved this point.

I'll keep your questions in mind when writing future posts explaining the
project.

~~~
detuur
If you're going to solve the problem of shell scripting, one particular itch
that I have is that shell scripts, or at least the sufficiently complex ones,
are often strongly dependent on availability of local executables, sometimes
even specific implementations or versions.

I had a shell script on my desktop that would automagically extract any
archive I threw at it. Nice. So when I was working on android through a shell,
I wanted the same convenience and so I scp'd the script onto my phone. It
wouldn't run. Only worked with GNU tar, as opposed to any other tar, like bsd
tar or busybox's.

Having a language that provides system-independent abstractions of ls, tar,
top, or the other way, that clearly indicates upon which binaries it depends,
would greatly improve script usability.

It'd be nice if running an oil script would automatically warn me that I need
GNU tar >= version x.xx.

------
eltoozero
Sometimes - perhaps often - you are in situations where you have no control
over the shell, in these cases writing "portable" shell scripts is helpful.

I have been enjoying "Beginning Portable Shell Scripting"[0] by Peter Seebach
for this very reason.

[0][https://www.apress.com/us/book/9781430210436](https://www.apress.com/us/book/9781430210436)

~~~
majewsky

      $ cat run-bash-script.sh
      #!/bin/sh
      which bash &>/dev/null && exec bash "$@"
      source /etc/os-release
      case "$ID" in
        debian) apt install bash || exit 1 ;;
        redhat|suse) yum install bash || exit 1 ;;
        arch) pacman -S bash || exit 1 ;;
        alpine) apk add bash || exit 1 ;;
      esac
      exec bash "$@"
    

I don't know how to detect and handle the BSDs etc., but you get the idea. ;)

~~~
deadbunny
Now go try that on AIX boxes, in banks, where you don't get root/sudo.

~~~
oblio
You also don't get internet access, in case a bright bulb tries to go "curl
...".

Though in many of these cases I'd just advise leaving the premises unless the
salary is stellar (or there are other considerations such as family, etc).

~~~
deadbunny
On the DevOps side of things (Contracting in London) the day rate at banks is
about 1.5x everywhere else which can be nice. Just as long as you're happy
never getting anything done ever.

------
axlprose
Would be interesting to test this out on SourceMage[0], seeing as how its
package manager is written entirely in bash.

[0] [https://sourcemage.org/Intro](https://sourcemage.org/Intro)

~~~
chubot
Interesting, I didn't know about SourceMage. It appears to be a source-based
distro like Gentoo, and if it's package manager is written in bash, it's a
perfect use case.

Since I have my hands full with the distros I'm using, I may not be able to
try it for awhile, but I saved this page for further reading. I still think
distros are somewhat "broken", so alternative approaches are interesting to
me. (Just one problem off the top of my head: packages are old, distros can't
keep up with PyPI, npm, CRAN, etc.)

If you know the SourceMage code, please try it with OSH :) Or if you know its
developers, they might want to hear about a cleaner shell that's compatible
with bash (although it's nascent)

I spent some time last fall implementing some constructs that the Nix package
manager uses, even though I don't use Nix. Nix uses bash heavily as well.

~~~
igravious
Hey, love your project's ambition but I think it's unhelpful if you say things
like, “ I still think distros are somewhat "broken", so alternative approaches
are interesting to me.” (Think of all the work you've put into your shell
project, now think of how much integration work must be done to keep a distro
going.)

That said, I think you've highlighted _the_ problem that _all_ distros face
that would get me to change distros if someone solved it. That is to say,
“(Just one problem off the top of my head: packages are old, distros can't
keep up with PyPI, npm, CRAN, etc.)” That is a doozy of a problem.

One last thing–you mention four things:

    
    
      The trap builtin is unimplemented;
    
      alias is also unimplemented.
    
      set -h / hashall is a stub that does nothing.
    
      OSH builds are in a sense "shallow".
    

Is there some way to get OSH to be un-shallow? For instance if I've invoked a
shell script with OSH I'd probably like any shell script dialect that it can
handle to have OSH invoked as well. Also, surely it would help you out with
debugging OSH because you'd immediately way more lines of shell script!

Implementing `trap', `alias', `set -h' seem like no-brainers prior to 0.4

~~~
chubot
Hm I probably could have said "distros are suboptimal" and it would have meant
essentially the same thing. But I wasn't disparaging anyone's work, if that's
your concern. "Debian is broken" would be closer to that, but I didn't say
that. :)

But yes I do think distros are doing the best they can with the raw material
they have to work with (upstream sources, autoconf, etc.) I have some ideas
for a distro but I think I have my hands full with shell, and it is a big
problem because you can't do it by yourself. You need an army to help you
maintain packages to some standard!

\-----

For the shallow problem, I'll probably implement something like set -o hijack-
shebang. It's relatively easy for OSH to open scripts it runs and check for a
shebang that looks like a shell, and then prepend "osh" to the argv array.

It should really only be used for debugging OSH, because it wouldn't be good
practice to "lie" about your shebangs. That logic is fundamental and baked
into the kernel!

"trap" has come up a lot, although I'm not sure if it will be in the 0.4
release. I'm making releases pretty often -- every 6 weeks or so. "alias" is
interesting because it sits between the lexer and the parser, and no other
shell feature is like that.

~~~
igravious
Or symlinks to OSH of the various shells it can emulate?

~~~
chubot
Yes that's true, I should add the ability to symlink /bin/bash or ~/bin/bash
to the oil binary and have it work. It actually works in this "busybox" style
already, but it only supports a few names like "osh" and "sh".

I didn't mention that because it's not a good idea yet to make OSH the default
shell on any system :)

~~~
igravious
But for your test suite and testing OS build systems to wring out
incompatibilities and bugs it might work a treat (if it detects certain
environment variables).

Sounds great! You put so much effort into your blog posts, it's impressive…

------
sandGorgon
You should integrate with Docker Linuxkit

[https://github.com/linuxkit/linuxkit/issues/161](https://github.com/linuxkit/linuxkit/issues/161)

------
dx034
Hmm, relying on Python 2 for a program that's still in early stages and is
supposed to have a long lifespan?

~~~
chubot
I plan to cut the dependency on Python; the post called "the riskiest part of
the project" which I linked describes that.

[http://www.oilshell.org/blog/2017/04/08.html](http://www.oilshell.org/blog/2017/04/08.html)

There are some detail here in these posts:

[http://www.oilshell.org/blog/tags.html?tag=opy#opy](http://www.oilshell.org/blog/tags.html?tag=opy#opy)

FWIW, Oil was in Python 3 at one point! But I ported it back to Python 2.
There were a number of reasons, but unicode handling is one of them.

See this comment:

[https://www.reddit.com/r/ProgrammingLanguages/comments/7elxl...](https://www.reddit.com/r/ProgrammingLanguages/comments/7elxlv/python_3_and_firefox_57_an_observation/dq6ixqu/)

Summary: The shell deals with strings from file systems, and file systems
inherently have no encoding. So Python 3's model of unicode doesn't help with
such programs. It just makes things more awkward.

~~~
earenndil
Can you just use binary strings?

~~~
chubot
It's possible (see the reddit comment), but there's no advantage to doing so.

