
Why Create a New Unix Shell? - shadeless
http://www.oilshell.org/blog/2018/01/28.html
======
ComputerGuru
Lots of overlap in design goals with fish, except fish also places a premium
on users interactively using the shell (which means friendlier in-repl
experience but a balancing act when it comes to features). Fish’ auto
completions are incredible, too.

Best of luck to them. Another interesting shell to check out is elvish, lots
of new ideas there (even if awkward to use).

(Disclosure: I’m one of the core fish devs/maintainers. Edit: The entire team
is awesome and the others deserve virtually all the credit!)

~~~
chubot
(author here) Hm I've tried fish, and it seems very nice for interactive use.

However I don't see it being used AT ALL for the cloud/linux use case? Those
are the cases where you tend to get 1000+ lines of shell scripts.

For example, I mention Kubernetes/Docker/Chef, and I've never seen fish in
that space.

I also don't know of any Linux distro that uses fish as their foundation --
they all appear to use a POSIX shell like bash/dash/busybox ash. fish is "on
top".

See _Success with Aboriginal, Alpine, and Debian_
[http://www.oilshell.org/blog/2018/01/15.html](http://www.oilshell.org/blog/2018/01/15.html)
\-- these distros are built with thousands of lines of shell scripts.

Either 1) I don't know about such usage, 2) people don't know that fish can be
used this way, or 3) there is some problem with using fish this way.

I link to this post in the FAQ, which I think is a lot closer to Oil:

[https://ilya-sher.org/2017/07/07/why-next-generation-shell/](https://ilya-
sher.org/2017/07/07/why-next-generation-shell/)

It's basically the "devops" use case. (And as I mention the main difference
between Oil and NGS is that Oil is compatible / has an upgrade path from
bash.)

~~~
arghwhat
Back when I started programming, I wrote 1000+ lines of shell scripts.

Now, I quite seriously believe that a 1000 line shell script only exists out
of error. I still occasionally end up doing 2-300 line dense shell scripts,
but not without feeling very dirty along the way. Either split into small,
simple shell scripts (which is fine), or a different language.

In the cross platform build pipeline at work, I keep a strong discipline when
it comes to scripts: They _must_ be short (<=100 lines), and if their
complexity exceeds a certain threshold (parsing more than a "grep | cut" here
and there, total program states exceeding some low number), then a shell
script is no longer acceptable regardless of length. And, well, it's not safe
to assume the presence of anything more than a shell.

If you are writing and dealing with 1000+ lines of shell scripts, then
experience tells me that you are shooting yourself in the foot. With a gatling
gun.

(I used fish, btw. The interactive experience was nice, but the syntax just
felt _different_ without much gain, which was frustrating to someone who often
writes inline oneliners. Unlearning bash-isms is not a liberty I can afford,
as I need to be proficient when I SSH into a machine I do not own or control.
I can't force the entire company to install fish on all our lab servers, nor
is it okay to install a shell on another persons' dev machine just because we
need to coorperate.)

~~~
racer-v
> I quite seriously believe that a 1000 line shell script only exists out of
> error.

Perhaps the best use case for Oil is to provide a debugging environment where
you can figure out what your legacy shell scripts are doing, and rewrite them
in another language.

> Unlearning bash-isms is not a liberty I can afford, as I need to be
> proficient when I SSH into a machine I do not own or control.

This is a slippery slope. I have heard things like, "don't make your own
custom aliases / shell functions, because they won't be available when you SSH
to another machine." Forcing yourself to always use the lowest common
denominator of software is not a fun path.

~~~
rollcat
> Forcing yourself to always use the lowest common denominator of software is
> not a fun path.

Second this. Been there, and back, and there again, and recently back again.

Customize the hell out of your shell, make it your place, make it nice. You're
spending your day there, every day. Treat it like you'd treat your work desk.

If it's a stranger's machine, well, OK, suffer through the 10 minutes of
troubleshooting, an occasional session with busybox also helps keep "pure
POSIX" skillset fresh. If you're becoming a regular there, it's time to think
how to make your dotfiles portable. google.com/search?q=dotfiles

~~~
jsmthrowaway
While I appreciate your perspective, I think it’s more from the “occasional
SSH to a foreign server” mindset. Scaling “the way I work in a shell” to huge
fleets is a nonstarter, and as a working SRE, has been the biggest thing
holding me back from zsh, fish, and other alternatives.

OTOH, I’m also against, for example, installing zsh fleetwide in production
fleets to accommodate choosy folks. So I’m on both ends of the problem, and
know it.

~~~
rollcat
\- As soon as you have more than 5 boxes, you need different tools for your
everyday tasks, interactive SSH sessions just don't work. But you know that
already.

\- So you've found yourself SSH'ing into a particular box, because that's just
the best way to work on a problem. Either it's a pet box (in which case it
makes total sense to drop your dotfiles there), or it's a cattle box and you
don't care about the trash you left behind, because autoscaling will clean it
up for you.

------
jhillyerd
After using fish for 3 years, I'm finding there is very little reason to have
my login shell maintain backwards compatibility with bash.

The only time I run into issues is when a command expects manipulate
environment variables via bash syntax.

I think the fish documentation WRT to scripting could be much better, but the
language is more elegant than bash or PowerShell IMHO.

~~~
CGamesPlay
> The only time I run into issues is when a command expects manipulate
> environment variables via bash syntax.

And in my experience 90% of those are in the form `FOO=bar command` which can
be replaced with `env FOO=bar command` and works just fine in fish.

~~~
ComputerGuru
Both support for `&&` and `||` (instead of `and` and `or`) as well as
supporting `FOO=bar command` are under consideration for fish 3.0 to ease the
migration path. The former is pretty much going to happen, the latter if we
get around to it, DV.

~~~
geewee
This would be much appreciated. I know there are a few people on my team that
cam't use fish due to our npm scripts needing to be compatible with cmd

~~~
ComputerGuru
Why don’t your npm scripts specify /bin/sh as the interpreter?

------
simias
The idea to use a real programming language as a shell is a common one but I'm
not sure it's really a problem that can be solved without reworking the kernel
interface. Whatever you do pipes will still be byte streams, error handling
will always be using integer return values, you'll always have stdin, stdout
and stderr, job control and signal handling will always work pretty much the
same way. The kernel interface exposes an interface, the userland app expect
the shell to behave in the certain way, there's not a lot of wiggle room to
make things differently in between the two.

Not that the POSIX-like shell syntax is not all sorts of clunky and odd but I
almost consider it a feature, it's a deterrent to force you to move to a
"real" scripting language when the concepts become too complex to express in a
shell script.

~~~
effie
The whole point of these projects is to evolve the shell, go beyond the
current bash standard to make the work with the shell more fun and more
productive. Unix may seem to have limited default interfaces, but this was by
design - the idea was, the programmer/user knows much better what he needs so
let him build it. The system's role was shaped to provide robust universal
mechanisms. Also, the current standard shell is very limited when compared to
even existing unix interfaces. For example, the kernel provides select/poll
system calls but standard shell has no facility to use them, so simultaneous
processing of two or more data streams without blocking is not currently
possible. A new shell could finally provide them.

~~~
jklowden
Finally? You don’t need a new shell. Write select(1). Job done.

------
kevlar1818
> However, Python and Ruby aren't good shell replacements in general. Shell is
> a domain-specific language for dealing with concurrent processes and the
> file system. But Python and Ruby have too much abstraction over these
> concepts, sometimes in the name of portability (e.g. to Windows). They hide
> what's really going on.

Excellently put. POSIX shell languages have fantastic capabilities you just
can't get in most other languages. I would love to see a more safe, more sane
shell language gain enough popularity to change the narrative that "shell
scripts are dangerous and impossible to maintain."

The contrasts to Python and Ruby made me think of xonsh[1], a Python-based
shell that can dynamically switch between Bash-like syntax and standard
Python. It's not quite ready to become my daily driver, but I'm still excited
about it.

[1]: [https://xon.sh](https://xon.sh)

~~~
willghatch
Shell is my favorite domain-specific language. But many (including myself)
would argue that domain-specific languages are generally better embedded. Many
projects aiming to mixing shell with general purpose languages find a nice
embedded DSL for subprocess/pipeline management. Some others find a convenient
way to run shell commands or pipelines by mixing grammars and trying to
disambiguate them. [Shameless self-promotion] I've been working on a project
that aims to not only have a nice DSL for running process pipelines, but
focuses on making a syntax for using all functionality of the host language as
a command language, called Rash[1]. It is hosted in Racket, and can be
embedded in Racket at the module or expression level, and also normal Racket
expressions can be embedded in Rash (they can alternate arbitrarily deep). It
supports process pipelines, Racket function/object pipelines, and mixes of the
two. It's still alpha and has a TODO list a mile long, but I've been using it
as my daily driver interactive shell for months and have loved it so far.

[1]: [https://github.com/willghatch/racket-
rash](https://github.com/willghatch/racket-rash)

~~~
chubot
FWIW I have your project on my wiki page :)

[https://github.com/oilshell/oil/wiki/ExternalResources](https://github.com/oilshell/oil/wiki/ExternalResources)

I guess what you mean by embedded is that it should be an embedded DSL in a
full-fledged programming language? I don't quite agree, since there are at
least 20 projects like that on the wiki page, none of which is popular.

Probably the most popular one is eshell, in Emacs Lisp?

But if there's something I don't know about I'd be interested in hearing it.
This idea goes back at least 20 years, e.g. to scsh. And it hasn't taken off.

But certainly I don't begrudge anyone if their favorite environment is Racket
and they want to stay in Racket. That's a perfectly reasonable thing. It's
just not what I would expect anyone besides racket users to use.

One reason I'm interested in shell because it's the lowest common denominator
between different groups of programmers. C/C++ programmers, use it heavily,
Python, Ruby, JS programmers, Go, etc. Everybody uses it.

~~~
willghatch
Yes, actually I've looked over all the shells on that wiki page. I think most
of them haven't taken off because they either their host language has been
poor or unpopular, their design or implementation wasn't great, or they
haven't really solved the right problem. I think the idea of embedding a shell
into a general-purpose language still has a lot of merit. Most of those
projects trying to embed a shell DSL into a general-purpose language, scsh
included, are basically for programming and not interactive use. Shells that
are only interactive or only for programming end up fulfilling less than half
of the purpose of shell, in my view, because the interaction and scripting
feed back into each other in a virtuous cycle. The only ones on that list
aside from Rash that try to be embedded in a general-purpose language while
being good at both interactions and programming are Xonsh and eshell. Xonsh is
pretty new (I wasn't aware of it until after I had made Rash), and eshell is
in Emacs Lisp (which is not a very good programming language or platform to
build on for anything except extending the emacs editor).

Rash also tries to be better than the competition by adding object pipelines
(much like Powershell, it makes it much more reasonable to make system
administration commands in the host language, and have rich interaction and
inspection of command results), user-defineable pipeline operators, and
generally tighter integration with the host language while still having a
light syntax for basic commands.

I would like to be able to have my command language tightly integrated with my
programming language, and be able to define my system administration commands
(and interactive shell plugins and completion) as functions in the host
language (while still being able to naturally use external programs). And I
would like my shell scripts to be able to grow and potentially even turn into
full-fledged programs, or modules of some larger program. I think there are a
lot of benefits to the approach I'm using (which would be too long for a
comment here).

That said, I'm not holding my breath for it to catch on widely any more than
I'm holding my breath for Racket to take off as a popular programming language
(although I frankly wouldn't mind either one). I think a better Posix shell is
certainly a noble effort, because whatever better shell does become popular,
we certainly need one. And an automatic upgrade path for existing scripts
sounds great. So I salute you and wish you good luck with it. Also, as someone
looking at shells and their features to copy the best ones, your wiki is a
great resource. So thanks.

~~~
chubot
OK great, glad you have made use of the page. We had an "alternative shells"
thread about a year ago between the authors of Elvish, NGS, Oh, and mash.
Those were the main active/nascent shells I could find.

It might be time to start that again to see what ideas people have and what
has changed. If you're interested e-mail me at andy@oilshell.org.

I'm also interested in the Shill shell, which as I understand was written in
Racket, and then somehow they moved away from Racket? I'm not sure. I think it
was because of the runtime. I also saw some efforts to move Racket to Chez
Scheme.

I very much like Racket as an idea -- a meta language -- but I haven't gotten
a chance to play with it too much.

And I did experiment with femtolisp as a basis for Oil -- the lisp used to
bootstrap Julia -- but I decided against it.

------
jernfrost
I essentially solved these problems with fish shell and Julia programming
language.

For all sorts of interactive stuff I use fish, because it works the way you
want for the most common tasks. I can use it to really quickly match and get
back previous statements or do a completion.

Also much easier to configure and grok than bash, because it has saner syntax
and is a simpler shell language.

However when writing shell scripts I use a real language like Julia. It
integrates very well with the shell world so I don't find it problematic to do
this. It is very easy to read output from processes and pipe stuff. Much nicer
than say Python or Ruby.

You got built in syntax to deal with shell stuff, but then didn't make it
crazy so you end up with a mess like perl. Julia is actually a very clean and
nice language. Which also happens to blistering fast and have LISP style
macros.

------
castis
I see that this person has opted not to use python 3 because it is 'less-
suited to shell-like problems' than python 2.

In an effort to understand the reasons for actively choosing against 3, does
anyone know what problems those would be?

~~~
rocqua
The few times I tried to de shell like things in Python, the absolute pain of
actually running commands and getting to their output might be part of the
reason.

What is A=$(cmd) in bash is an absolute pain in python.

~~~
wutbrodo
That was my main pain initially after I finally switched away from writing my
scripts in Bash, but it actually turned out to be pretty easy. I wrote a quick
helper function that does 99% of what I need:

def _call(cmd_str): return
subprocess.check_output(shlex.split(cmd_str)).decode("utf-8")

Definitely a verbose monstrosity compared to doing it in bash, but more than
worth avoiding the garbage fire that is Bash. And it only works for simple
cases

~~~
_mhr_
Thank you for this.

------
zapita
Finally a modern shell that understands the importance of COMPATIBILITY! This
gives it a realistic chance of getting real adoption. Shells like zsh and fish
will never get mainstream adoption because they are not compatible with bash.

~~~
aspaceman
Why do people care about "COMPATIBILITY" so much w.r.t. shells? It's so easy
to use other shells to run your script.

> /bin/bash your_script.sh

And if your script is written with bash in mind, use a shebang:

> #! /bin/bash

And it will work perfectly fine on fish. As long as I have a bash binary, why
do I need COMPATIBILITY?

~~~
hrktb
One of the werid cases with fish as default shell is scripts and programs that
expect bash when executing “system” commands. For instance a php or Java
program running shell commands.

Having bash compatibility is a quality of life feature, no need to debug all
the werid cases where it breaks for purely syntax reasons.

~~~
chubot
Right, the system() function in C, which PHP probably uses, and is os.system()
in Python, is DEFINED to run /bin/sh. You never know when a program might call
it (although I agree it is almost always bad practice to use it -- use
fork/exec instead.)

So basically you should never make /bin/sh fish, because then you will no
longer have a Unix system (according to POSIX).

~~~
hrktb
PHP somewhat takes the user’s login shell, so I guess it’s yet a different
system. If it was taking /bin/sh it would have been fine actually.

~~~
TylerE
Don't blame the rest of the world for PHP's brain damage.

------
anderspitman
Love it. I think there's plenty of room for innovation in this space. I'm
currently in the process of porting a ~700 line bash script (I didn't write
it) to Python. Although longterm I think this will be much better for the
project, there are still tradeoffs. There are some things that are just so
easy to express in shell language, like composing transformations via pipes.
Sure, python can do it, but it feels clunky in comparison to me. I would love
to see a language like python (including package ecosystem) written with shell
use as a first-class citizen.

~~~
zanchey
There's definitely room for innovation - fish is for me but I'm always
impressed with the work being done on both new and old shells. One thing fish
doesn't do is get much into the semantics of how processes interoperate, and
I'm interested to see if there's a new idea that can gain some traction in
that regard.

------
dmix
I checked out Oil previously which looks nice but more of an incremental
improvement over Fish/ZSH rather than a significant evolution (it may have
changed since then, this was last year).

Im most excited about Elvish shell and the language that's being developed
around it. The shell is built with Go and feels super fast compared to my
plugin-heavy ZSH. The language design is quite nice too but still very alpha.
Looking forward to see what it evolves into...

[https://github.com/elves/elvish](https://github.com/elves/elvish)

~~~
chubot
FWIW, OSH is the incremental improvement, and Oil is the new language
(explained in the intro to this post.)

~~~
dmix
I'm aware of the difference, but they are both very much integrated into a
single UX. The language was what I'm most interested in because writing ZSH is
a giant headache even though I've been doing it for years, it' s still
painful. That plus the performance of the shell itself.

------
JepZ
@chubot: There is one use-case I come across every once in a while:

[https://stackoverflow.com/questions/356100/how-to-wait-in-
ba...](https://stackoverflow.com/questions/356100/how-to-wait-in-bash-for-
several-subprocesses-to-finish-and-return-exit-code-0)

So whenever you want to do things in parallel there is probably a limit to the
number of processes you would like to execute in parallel (e.g. the famous
compiler limit formula: number of CPU cores +1). It would be great if Oil
could support such a use-case out of the box, as easy parallelism without the
ability to artificially limit the number of parallel executions is often
useless.

~~~
chubot
Absolutely. In fact, the bash manual explicitly refers to GNU parallel for
this use case!

I use xargs -P all over the Oil codebase, which does what you want. The trick
is to end the file with "$@", and then invoke xargs -P 4 -- $0 my-func.

That way xargs can run arbitrary shell functions, not just something like sh
-c "..." ! I'm going to write a blog post about this. I also do this with find
-exec $0 myfunc ';'

[https://github.com/oilshell/oil/blob/master/test/spec-
runner...](https://github.com/oilshell/oil/blob/master/test/spec-
runner.sh#L220)

However I think Oil will have something built-in to make this parallel process
more friendly. I will probably implement xargs so it can run my own shell
scripts without GNU xargs, but then add a nicer syntax. (Probably "each",
since that's what xargs really does.)

This gets into your other question about standard utils, which I'll answer
now. Short answer: yes I would like something like that, it's just a matter of
development time and priorities. I agree with the problem you point out.

~~~
JepZ
Sounds pretty cool. My biggest problems with xargs is that I had constantly
some weird edge cases, so I try to avoid it. As GNU parallel doesn't seem to
be part of standard installations it would be an external dependency for a
script, which I try to avoid too.

So I ended up using the loop syntax:

    
    
      for i in {0..9}; do
        echo "$i" &
      done
      wait
    

It is not so Unix like, but I find it easier to debug. It would be great if
Oil would have a solution for limiting that kind of parallel execution too. I
am aware that this isn't simple as there are different options here how to
implement it (global limit vs. local limit vs. named limit).

Just an idea from the top of my head an idea for an optional named limit:

Lets call it 'flow':

    
    
      flow [options] [command]':
      -n number of max parallel processes
      -c (optional) identifier of the counter.
    
      Example:
    
      for i in {0..9}; do
        flow -c myCounter -n 4 echo "$i"
      done
    

Just an idea.

------
z3t4
I spent the day making a virtual terminal ... I think the terminal is an
unnecessary layer. All program UI is limited by this 40+ year old technology
that is the terminal. Instead of making a new shell, make a shell + _new_
user-interface.

~~~
executesorder66
You might be interested in this:
[https://github.com/withoutboats/notty](https://github.com/withoutboats/notty)

~~~
scandox
[https://github.com/withoutboats/notty/issues/67](https://github.com/withoutboats/notty/issues/67)

Dead project according to the author, though.

------
tambourine_man
A little tangential, but I keep wondering if Apple is developing its own shell
or will adopt one with a more liberal licence such as Oil or Fish.

I mean, they can't keep using Bash 3 forever, right? (Hope)

~~~
siteshwar
fish is released under GPLv2 and afaik Apple has a policy to not include any
new GPL software in macOS.

~~~
tambourine_man
I think GPLv2 is fine, GPLv3 is what stopped them from using new versions of
Bash. But I'm sure they would prefer MIT, Apache, BSD, etc

------
unixhero
Ignore the naysayers. I have encountered them when posting my shell
activities. You're doing great! Can't wait to try the final release. Maybe
even get it into my workflow.

------
jiggunjer
So many shells, but how come the virtual terminal hasn't got a revamp or
popular alternative? I'd like to see more control over keybinding (e.g. on my
machine pressing ctrl+del sends the same character as pressing
ctrl+backspace). I understand this would be more a kernel change than a
userland one.

Screen tiling and visual 'tabs' would also be welcome additions. Not everyone
needs a graphic environment, and I refuse to install X just for better
keyboard shortcuts on my terminal.

~~~
JdeBP
It has, a fair number of times.

* [https://unix.stackexchange.com/a/177209/5132](https://unix.stackexchange.com/a/177209/5132)

* [https://unix.stackexchange.com/a/196102/5132](https://unix.stackexchange.com/a/196102/5132)

------
JepZ
Sounds pretty cool and everybody who had to learn bash scripting at some point
understands why we need a sane language (my favorite are misplaced spaces in
if statements...; Disclaimer: I do and love bash scripting but while the
language has cool concepts, some things are just broken by design).

Nevertheless, there is one piece in this puzzle I am missing. There does not
seem to be a process which manages the 'core software set' across platforms.
So after decades we finally have a shell which is available on most operating
systems, but how long will it take before Microsoft, Apple, Oracle, etc. will
adopt a new shell?

So why don't the large OS corporations form a consortium to define something
like a 'cross platform run time environment' standard (maybe together with the
Linux Foundation and some BSD guys?). I mean its not so much about which shell
someone prefers, but more about a common set of interpreters and maybe tool
kits. And even more than that it is not about the state but the
process/progress.

What do you think, do we need such a process or is there another way to solve
the cross platform dilemma?

~~~
sedachv
What you are describing has been around since the late 1980s, IEEE POSIX:
[https://en.wikipedia.org/wiki/POSIX](https://en.wikipedia.org/wiki/POSIX)

~~~
JepZ
Well, POSIX is pretty similar to what I mean, but it has a lot of low level
stuff and I doubt that Microsoft has any ambitions to transform Windows into a
POSIX compatible OS.

I thought more about a higher level standard like adding Python, Lua or Qt to
every installation by default. As some of those things are pretty heavy I
doubt that it would be a wise choice to include them in POSIX.

Just imagine a world were you could simply write a small python script which
would start a complete GUI application on different platforms without any
additional installation procedures. To my knowledge that is not possible
today. AFAIK the only way today is to bundle the dependencies, but that has a
lot of negative effects.

~~~
jiggunjer
Isn't that what electron is popular for?

~~~
JepZ
Yes, that's what everybody uses electron for, because the use-case is
obviously there.

But everybody who has build an electron app and a Qt/GTK app, will agree that
the tool kits which are available to electron apps are not as sophisticated as
Qt/GTK (not talking about the ton of downsides of electron apps (huge app
size, outdated versions, etc.)).

I am completely pro PWA, but as far as I can see it it will take a few more
years (at least) before we will get to a state which will allow us to use them
in the same way we develop and use normal desktop apps nowadays.

------
XorNot
The pain of "not quite bash" ultimately is what put me off xonsh as my daily
shell. Also losing it whenever I SSH'd somewhere.

The latter problem could probably be solved with a wrapper which would pipe
and execute the shell (or a bytecode interpreter ala shuttle?) automatically -
but I've seen no alternative shell project take this part seriously for the
problem space.

------
fnord77
> Oil is taking shell seriously as a programming language, rather than
> treating it as a text-based UI that can be abused to write programs.

erm, there's a big difference between a command scripting language and a
programming language. These should be treated as different things.

I have years of experience using both, and I really don't want to be doing
shell tasks in a programming language and I don't want to write programs in a
shell language. Those sorts of hybrids are almost always mediocre. Horses for
courses and all that.

There's a reason bash keeps being used - it's mature, it's simple, it's easy
and people are productive with it.

------
aplorbust
This keeps showing up on the front page. No doubt it will have users.

Lets say there are two uses of a shell: 1. interactive and 2. non-interactive
(scripting).

Lets imagine the commandline user is learning about her OS. She learns it is
heavily reliant on shell scripts to build and (if desired) to automate
starting services.

She realises that to understand the OS she will have to learn the shell that
the OS developers used for scripting.

Then she realises that if she chooses another shell for interactive use, she
will have to learn _two_ shells.

Finally she realises that any script she writes in the "non-
interactive/scripting" shell _will also run under the interactive one_. But
not vice versa.

If she only has enough time in life to master one shell, which one should she
choose?

Over time I found I really cared more about the scripting aspect of a shell
than the interactive facet.

The scripting shell used by the OS authors might be an Almquist derived shell,
for instance.

Occasionally the ash Im using gets a new "feature" but not too often. I like
that it stays relatively small. The latest "feature" is LINENO.

But I also use a smaller version of this shell with no command line history,
no tabcomplete, etc. IMO, there is no better way to learn how to reduce
keystrokes. It has led to some creativity in this regard for which I am
thankful.

After "mastering" ash, I started using execlineb, pipeline and fdmove. I am
starting to use more components of execline and am continually replacing ash
scripts with execline scripts for more and more daily work.

I guess we will never see execline on the front page, which I think would be
interesting because I would like to hear whatever harsh critique HN can
muster.

Seeking a better _non-interactive /scripting_ experience, I have experimented
with many other shells over the years, and written simple execve "program
launchers", but in this vein, I have not found anything that compares to
execline.

The speed gains and resource conservation are obvious, but with the ability to
do "Bernstein-chaining" and the _option_ to use djb low-level functions
instead of libc, it is a rare type of project.

The speed and cleanliness of the compilation process is, compared to all the
other crud one routinely encounters in open source projects, "a thing of
beauty". Humble opinion only, but I think others might agree.

~~~
JdeBP
It was submitted once.

* [https://news.ycombinator.com/item?id=12600807](https://news.ycombinator.com/item?id=12600807)

Laurent Bercot no longer has xyr page about the compilation process. I have
since picked up some of the slack there. Although I don't go into things like
the way that M. Bernstein avoided autotools.

* [http://skarnet.org/software/compile.html](http://skarnet.org/software/compile.html)

* [http://jdebp.eu./FGA/slashpackage.html](http://jdebp.eu./FGA/slashpackage.html)

------
petre
Oil syntax looks pretty much like Tcl and Th[1] so the author could have
probably just used Th. It has saner ways to copy arrays than

    
    
      b = [ @a ]
    

which pretty much looks like Perl with added line noise. Why are the [] even
necessary when it's clear @a is an array?

1\.
[http://www.sqliteconcepts.org/THManual.pdf](http://www.sqliteconcepts.org/THManual.pdf)

------
desireco42
As a fish user, I didn't really get where the advantages lie as clearly. I am
for developing new shells, we really need this, innovation is important.

As for new language, I feel like if you want to script things, you can use
ruby or python, hell, perl will do and you could be fine. I don't want to be
unfair to this effort, I just feel that it is not for me and I am tinkerer.

------
tyingq
Applaud the idea, but you are fighting a ton of inertia. I have to wonder if
drawing a more clear line of when to go to Perl, Python, Lua, Ansible, Golang,
etc, might be more fruitful. Sometimes, a shell script solution is just
drawing any kind of shell too far outside it's core competency.

------
cestith
In your FAQ you decry Perl as having no ability to redirect around other
programs. Yet you don't explain how any of the following fail to meet those
needs.:

* [http://perldoc.perl.org/functions/open.html](http://perldoc.perl.org/functions/open.html)

* [http://perldoc.perl.org/IPC/Open2.html](http://perldoc.perl.org/IPC/Open2.html)

* [http://perldoc.perl.org/IPC/Open3.html](http://perldoc.perl.org/IPC/Open3.html)

* [http://search.cpan.org/~odc/IPC-Open2-Simple-0.01/lib/IPC/Op...](http://search.cpan.org/~odc/IPC-Open2-Simple-0.01/lib/IPC/Open2/Simple.pm)

* [http://search.cpan.org/~exodist/Child-0.013/lib/Child.pm](http://search.cpan.org/~exodist/Child-0.013/lib/Child.pm)

* [http://search.cpan.org/~rkrimen/IPC-RunSession-Simple-0.002/...](http://search.cpan.org/~rkrimen/IPC-RunSession-Simple-0.002/lib/IPC/RunSession/Simple.pm)

* [http://search.cpan.org/~trski/Proc-Forkmap-0.025/lib/Proc/Fo...](http://search.cpan.org/~trski/Proc-Forkmap-0.025/lib/Proc/Forkmap.pm)

* [http://search.cpan.org/~toddr/IPC-Run-0.96/lib/IPC/Run.pm](http://search.cpan.org/~toddr/IPC-Run-0.96/lib/IPC/Run.pm)

* [http://search.cpan.org/~ayoung/IPC-Run3-Simple-0.011/lib/IPC...](http://search.cpan.org/~ayoung/IPC-Run3-Simple-0.011/lib/IPC/Run3/Simple.pm)

* [http://search.cpan.org/~rjbs/IPC-Run3-0.048/lib/IPC/Run3.pm](http://search.cpan.org/~rjbs/IPC-Run3-0.048/lib/IPC/Run3.pm)

* [http://search.cpan.org/~djerius/IPC-PrettyPipe-0.03/lib/IPC/...](http://search.cpan.org/~djerius/IPC-PrettyPipe-0.03/lib/IPC/PrettyPipe.pm)

* [http://search.cpan.org/~xan/IPC-Pipeline-1.0/lib/IPC/Pipelin...](http://search.cpan.org/~xan/IPC-Pipeline-1.0/lib/IPC/Pipeline.pm)

* [http://search.cpan.org/~sscaffidi/IPC-OpenAny-0.005/lib/IPC/...](http://search.cpan.org/~sscaffidi/IPC-OpenAny-0.005/lib/IPC/OpenAny.pm)

* [http://search.cpan.org/~glai/IPC-Exe-2.002001/lib/IPC/Exe.pm](http://search.cpan.org/~glai/IPC-Exe-2.002001/lib/IPC/Exe.pm)

* [http://search.cpan.org/~zefram/IPC-Filter-0.005/lib/IPC/Filt...](http://search.cpan.org/~zefram/IPC-Filter-0.005/lib/IPC/Filter.pm)

~~~
chubot
Can you show me some code? How do you write this in Perl?

    
    
        f() {
          echo --
          ls /
          echo --
        }
    
        f > out.txt
        f | wc -l

~~~
gbacon
Assuming we have some sequence of commands whose output we want to capture and
eliminating any implicit use of the shell from perl, I’d define a sub along
the lines of

    
    
        sub output_of {
          my(@commands) = @_;
    
          my $pid = open my $fh, "-|" // die "$0: fork: $!";
          return $fh if $pid;
    
          for (@commands) {
            my $grandchild = open my $gfh, "-|" // die "$0: fork: $!";
            if ($grandchild) {
              print while <$gfh>;
              close $gfh or warn "$0: close: $!";
            }
            else {
              exec @$_ or die "$0: exec @$_: $!";
            }
          }
    
          exit 0; # child
        }
    

Call it as with

    
    
        my $fh = output_of [qw( echo -- )],
                           [qw( ls   /  )],
                           [qw( echo -- )];
    
        while (<$fh>) {
          print "got: $_";
        }
    
        close $fh or warn "$0: close: $!";
    

If implicitly using the shell is acceptable, but we want to interpose some
processing, that will resemble

    
    
        my $output = `echo -- ; ls / ; echo --` // die "$0: command failed";
        chomp $output;
    
        print "$0: lines = ", `echo '$output' | wc -l`;
    

This becomes problematic if the output from earlier commands collides with the
shell’s quoting rules. This lack of “manipulexity” that we quickly bump into
with shell scripts — that are otherwise great on the “whipuptitude” axis — was
a common frustration before Perl. The gap between C and the shell is exactly
the niche on POSIX systems that Perl occupies and was its initial motivation.

If all you want to do is redirect anyway, run

    
    
        system("{ echo -- ; ls / ; echo -- ; } > out.txt") == 0
          or die "$0: command failed";
    

Use the appropriate tool for the job. Perl was not designed to replace the
shell but to build upon it. The shell is great for small programs with linear
control flow. It’s hard to beat the shell for do-this-then-this processing.
The real world likes to get more complex and nuanced and inconsistent,
however.

Maybe I am missing your point entirely. Do you have a more concrete example in
mind?

~~~
chubot
See this thread for the real problem:
[https://www.reddit.com/r/oilshell/comments/7tqs0a/why_create...](https://www.reddit.com/r/oilshell/comments/7tqs0a/why_create_a_new_unix_shell/dtmq9o6/)

Sorry I got to this late -- I might do a blog post on it. I think your
response, along with the 3 or 4 others i got essentially proves my point:
"Perl is not an acceptable shell".

~~~
cestith
You're not saying why any of the above don't solve your problem. Dismissing a
solution because you refuse to understand it doesn't prove anything.

------
shmerl
I recently discovered Fish, which is actually not that recent, but also an
interesting shell.

------
wainstead
> Shouldn't we discourage people from writing shell scripts?

...people frequently ask this? Tip #21 of "The Pragmatic Programmer" states:
"Use the Power of Command Shells."

~~~
partycoder
Manipulating processes, exit codes and output is idiomatic in shells and
that's where they shine. I totally agree with you.

In a general purpose programming language there's a lot of overhead for doing
the same things.

For maintainability, there are now linters for shell languages that can help
making the job easier.

~~~
yjftsjthsd-h
> linters for shell languages

Obligatory in case anyone hasn't seen it:

[https://www.shellcheck.net/](https://www.shellcheck.net/)

Works as a web app or local tool.

~~~
partycoder
Please note that copying and pasting commands from a web browser into a
terminal can result in malicious code being executed.

A good idea to double check using a text editor.

[http://thejh.net/misc/website-terminal-copy-
paste](http://thejh.net/misc/website-terminal-copy-paste)

~~~
pbhjpbhj
Sounds like sandboxing of pasted code until confirmation would be a good
feature for a new shell ...

~~~
yjftsjthsd-h
There are terminals that do this, I believe.

------
zimbatm
What I would really like to see happening in Oil shell is a way to run a
strict subset of the bash language. Ideally with a tool to convert existing
scripts in that subset.

------
breatheoften
I would love an interactive shell that applies a type-checker to my input
before running it ... is that really an infeasible desire in 2018 ...?

------
fiatjaf
What kind of shell can I run on a server without filesystem access that I can
open to external untrusted users?

~~~
Spivak
Restricted bash? You will need the cooperation of your SSH server to fully
lock it down but it gives you the tools.

[https://www.gnu.org/software/bash/manual/html_node/The-
Restr...](https://www.gnu.org/software/bash/manual/html_node/The-Restricted-
Shell.html)

------
chrisvalleybay
Why is it called Oil? That name is so loaded. I think it actually will give it
less of a chance.

~~~
chubot
I explain the name here:

[https://www.reddit.com/r/ProgrammingLanguages/comments/7qn14...](https://www.reddit.com/r/ProgrammingLanguages/comments/7qn14p/oil_success_with_aboriginal_alpine_and_debian/dsqxgjh/)

Bizarrely (to me), more than one person thought the name was a play on the
company "Shell Oil". Is that the connotation you got from it?

That's unfortunate, but I think as people use it more, the name will take on a
different connotation. Guido was fighting "Python == snake" for a long time
too (it comes from Monty Python). There were a lot of people that said the
name Python was stupid and you couldn't convince your boss to use a language
with a name like that.

------
Sir_Cmpwn
I use fish but I hate it. My perfect shell is strictly POSIX sh but has a
better interactive experience (better tab completion and typeahead like fish
has).

------
fiatjaf
Wasn't there a in-browser Javascript shell somewhere a while ago? It could
fetch URLs and do cool stuff with APIs.

------
fanf2
No mention of plan9 rc or Tcl so it is hard to believe it is really breaking
the mould

------
psibi
Looks nice! Any reason why Apache was used as it's license ?

------
oleks
Why is Oil implemented in Python? IMO Python is a terrible language for
writing programming languages.

~~~
chubot
I will address that in part 2 of the FAQ, but the short answer is:

1) I prototyped it in Python; the dependency on the Python interpreter will be
removed [1]

2) Oil went through many implementation languages, and one incarnation was
2000-3000 lines of C++. But I realized I would NEVER finish that way. The goal
is to be compatible with bash, which is a tall order.

3) Oil is heavily metaprogrammed. It's only 16K lines of Python, compared to
160K lines of bash, and it can run some of the most complex bash programs out
there. [2]

It's more accurate to say Oil is written in Python + ASDL [3], i.e. somewhat
in the style of ML.

[1]
[https://news.ycombinator.com/item?id=16277358](https://news.ycombinator.com/item?id=16277358)

[2]
[http://www.oilshell.org/blog/2018/01/15.html](http://www.oilshell.org/blog/2018/01/15.html)

[3]
[http://www.oilshell.org/blog/tags.html?tag=ASDL#ASDL](http://www.oilshell.org/blog/tags.html?tag=ASDL#ASDL)

------
donatj
I was with the guy right up until the bile laden PHP hated started coming in
as a justification.

~~~
chubot
(author here) I didn't intend to criticize PHP, and I don't think I did.

I said that you can't convince people not to use bash or PHP by writing posts
on the Internet, which is true.

I also said that Facebook is replacing PHP, which is true. That's not a
criticism of PHP. The fact that huge companies like Yahoo and Facebook can be
started with PHP is amazing.

I think PHP is a good analogy for bash. It gets the core things right, and it
gets a ton of work done. I like languages you can get work done in! That's why
I use bash.

But both languages also evolved a lot of warts. That's inevitable when you
have so many users. They have diverse needs, and you need to preserve backward
compatibility, which leads to an awkward evolution.

------
devit
Seems an interesting idea, but it's implemented in Python, which means it will
never replace bash and probably not achieve any significant adoption unless
they rewrite it in Rust first (which they should have written it in to begin
with since they started in 2016).

The reasons for that are that shells must start very quickly (due to
subshells, local ssh, etc.), be fast, have no complex dependencies since they
are used to recover broken systems, be portable but also with full support for
OS semantics and be written in a language that allows rapid development of
robust software, none of which Python does well.

~~~
chubot
[https://news.ycombinator.com/item?id=16277734](https://news.ycombinator.com/item?id=16277734)

