
Shell Has a Forth-Like Quality - chubot
http://www.oilshell.org/blog/2017/01/13.html
======
djsumdog
I'm really excited about new shells. Fish has been mentioned here (which I've
used for 2+ years). It's amazing and adds a lot of good functionality.

I also like that the author mentioned runit. I wish there was more
documentation for runit, but I've been using it on Void linux and it's an
amazing, damn simply init system. If an init script is missing for any package
in Void, they are trivial to write. Most are less than three lines.

I also really like how the post author demonstrated how shell commands are
composed in ways that traditional functions aren't composed; that is commands
that change the environment for the command they run (environment variables,
processor scheduling, security, input/output redirection). This is the very
nature of an _implicit_.

~~~
wtbob
> Fish has been mentioned here (which I've used for 2+ years). It's amazing
> and adds a lot of good functionality.

What does it add exactly? As far as I can tell, it's needlessly incompatible
with POSIX shells, and doesn't do as much as e.g. zsh (or even bash?).

I'm not saying that zsh & bash are the end-all, be-all of shells, but I just
don't see that fish improves on them.

And honestly, for config files I think we could do a lot worse than an
s-expression language.

~~~
ridiculous_fish
fish dev here. We love zsh and anyone pushing on what shells can do!

Here's where I think fish fits in. The main goals of fish are:

1\. Thoughtful UI that just works

2\. Sane scripting

zsh does more than fish, and we're happy to let it. The tradeoff is that zsh
requires substantial configuration to realize that power - and once you get
into the weeds, you will learn which options are incompatible with each other.
zsh has 21 options controlling history, and fish has none, but you'll probably
be happier with fish's defaults.

fish has a smaller feature set, but that feature set just works out of the
box: tab completions, syntax highlighting, autosuggestions, etc.
Customizations (prompt, colors, etc) can be done with a web UI or a nice
command line interface, and not arcane syntax (PS1=...)

The hope is you can be more productive in your shell without needing to invest
tons of time configuring and learning it.

~~~
marklgr
> adds a lot of good functionality.

Parent's comment is misleading, then. As someone who already has a pretty
solid grasp of Bash, when I peeked at Fish it looked like there was nothing
that it would gain me -- though I understand that the preconfigured shell can
be useful for people who don't have the time to learn and tweak it.

~~~
ridiculous_fish
If fish has a hook (har har), it would be autosuggestions. fish suggests the
rest of the command as you type, like your web browser does for URLs. This
feels faster than tab completions, because you can see what command you'll get
ahead of time. It also lets you summon commands from history quickly, without
having to think about it - just start typing.

If you're a die-hard bash user, autosuggestions are the feature most likely to
pique your interest and make the command line more pleasant.

~~~
marklgr
Will take a look, but 'history-search-backward' fills most of my needs in that
regard.

~~~
djsumdog
You really should. Once I got use to the fish auto-complete history, the Bash
reverse search felt terribly primitive. There is no Ctrl+R. Just type part of
an old command and hit the up arrow to scroll.

------
xiaq
If you are interested in a shell with first-class function, list and map
values and pipelines that pass such complex data (instead of structureless
bytes), you might be interested in elvish:
[https://github.com/elves/elvish](https://github.com/elves/elvish)

Disclaimer: I am the author of elvish.

~~~
chubot
Hi xiaq I mentioned your shell here:

[https://github.com/oilshell/oil/wiki/ExternalResources](https://github.com/oilshell/oil/wiki/ExternalResources)

If there is a better summary, let me know, or feel free to edit the wiki page
yourself. (I think it should be editable.)

I've been in contact with the authors of the oh shell, NGS shell, and mash
shell, and exchanged some interesting ideas. The design of the oil language
should be more concrete in 2-3 months and I'm hoping to get feedback from
others who have thought about this space. I might ping you then :)

Are the elvish structured data pipelines actually separate processes, or are
they goroutines? Is there a format for piping them over say SSH? I have been
thinking about this a little bit for oil.

~~~
xiaq
Hi chubot, I didn't realize that you are developing a shell as well. It's
great to see competitions. Can you share some ideas that you learned from the
authors of said shells? I would love to hear those, designing shells is not an
easy task.

Structured data pipelines run in the same process as elvish, in goroutines.
For interacting with external commands you need to serialize and deserialize,
like this:

    
    
      ~> put [foo bar lorem ipsum] | to-json |
         python3 -c 'import sys, json;print(json.dumps([x.upper() for x in json.load(sys.stdin)]))' |
         from-json
      ▶ [FOO BAR LOREM IPSUM]

~~~
chubot
OK cool, that makes sense.

I had some good discussions with Michael MacInnis and recently Ilya Sher. I
had actually read Michael's masters thesis in 2011 or 2012 and found it to be
a very good historical overview of shells:

[https://scholar.google.com/scholar?cluster=99934401164441475...](https://scholar.google.com/scholar?cluster=9993440116444147516)

I agree with the goal of "taking shell seriously as a programming language",
which I think all projects share. I disagreed on the need to make shell a
variant of Lisp. I actually tried this a bit -- I hacked on femtolisp, which
is used to bootstrap Julia, and I thought I could bootstrap a good shell with
it. This didn't work out as nicely as I had hoped so I dropped the idea.

Ilya was having some issues with the "two modes" of parsing -- command mode
for shell-like constructs and expression mode for programming features
(function calls, math, etc.). I thought this would be an issue too, but as
mentioned in another comment, now that I've written a bash parser I think it's
straightforward to solve, and Ilya agreed.

The thing is that bash already has many different modes of parsing, which I
implement using "lexer modes" or lexical state. This same technique can be
applied to a new shell and then there should be a natural way of mixing
commands and expressions (quoted and unquoted words, etc.):

[http://www.oilshell.org/blog/2016/10/19.html](http://www.oilshell.org/blog/2016/10/19.html)

I'm also trying to make oil do what bash/dash/etc. do essentially syscall for
syscall. I believe that is a little hard with Go because it has its own
runtime. Although I think there is a way to pin a goroutine to a single thread
which can mitigate those issues.

I was planning to start a thread in maybe a month to review some of these
issues and exchange feedback. Let me know if you are interested!

------
sigil
Totally! Bernstein chaining is concatenative. So are shell pipelines, if you
squint at them just right: function composition mediated by special stacks of
records called stdin and stdout.

Reminds me, I have a pretty neat tail-recursive shell pipeline trick I need to
share with HN!

~~~
chubot
I'm interested in the tail recursive pipeline trick!

The next post mentions that pipelines are also concatenative, although there
is a slight difference and I think "point-free" is more accurate.

"Pipelines Support Vectorized, Point-Free, and Imperative Style"

[http://www.oilshell.org/blog/2017/01/15.html](http://www.oilshell.org/blog/2017/01/15.html)

~~~
sigil
Nice. I'm loving this series, keep it up!

------
jasonhansel
I think it would be interesting to have an init system and/or package manager
configured using a logic programming language, such as a Prolog dialect.

For instance, one could write "is_installed(X11) :- is_installed (KDE)" to
specify that X11 must be installed if KDE is installed.

~~~
iigs
I'm not sure how much of it is left today, but there is precedent for Prolog
in OS startup:

[http://www.redditmirror.cc/cache/websites/web.archive.org_84...](http://www.redditmirror.cc/cache/websites/web.archive.org_84624/web.archive.org/web/20040603192757/research.microsoft.com/research/dtg/davidhov/pap.htm)

~~~
jasonhansel
I wish that logic programming languages (and especially the one used in that
article) had more friendly syntax. Even just replacing ":-" with "<-" would be
nice...

~~~
OskarS
Actually, Prolog uses "\-->" for to define grammars[1] for parsing. Not
confusing at all! :)

(actually, parsing is one of the things i really love about Prolog. Writing a
simple grammars is much easier and much more pleasurable than using things
like regular expressions. The fact that it's far more powerful and readable is
just a bonus. )

[1]
[https://en.wikipedia.org/wiki/Definite_clause_grammar](https://en.wikipedia.org/wiki/Definite_clause_grammar)

------
gwu78
Easy way to do file descriptor passing ("Bernstein chaining") with any UNIX
programs, not just ones written by djb:

[http://skarnet.org/software/execline/pipeline.html](http://skarnet.org/software/execline/pipeline.html)

Execline resembles a "program launcher".

We can make a braindead launcher in a few minutes using the sh language. This
is just the first way that came to mind.

    
    
       #!/bin/sh
     
       test $# -ge 1||
       exec echo usage: $0 program arguments
       test $# -le 5||
       exec echo max 5 arguments
       
       {
       a=$#;
       b=$(command -v $1);
       
       echo 'main(){
       char *b['$((a+1))'];'
       
       a=$((a-1));
       test $a -ge 0||
       exec echo 'b['$#']=(void *)0;
       execve("'$b'",b,(void *)0);
       }'  
       echo 'b[0]="'$1'";'
       
       
       
       
       # repeat 
       
       
       a=$((a-1));
       test $a -ge 0||
       exec echo 'b['$#']=(void *)0;
       execve("'$b'",b,(void *)0);
       }' 
       echo 'b[1]="'$2'";'
       
       a=$((a-1));
       test $a -ge 0||
       exec echo 'b['$#']=(void *)0;
       execve("'$b'",b,(void *)0);
       }'
       echo 'b[2]="'$3'";'
       
       a=$((a-1));
       test $a -ge 0||
       exec echo 'b['$#']=(void *)0;
       execve("'$b'",b,(void *)0);
       }'
       echo 'b[3]="'$4'";'
       
       a=$((a-1));
       test $a -ge 0||
       exec echo 'b['$#']=(void *)0;
       execve("'$b'",b,(void *)0);
       }'
       echo 'b[4]="'$5'";'
       
       a=$((a-1));
       test $a -ge 0||
       exec echo 'b['$#']=(void *)0;
       execve("'$b'",b,(void *)0);
       }'
       echo 'b[5]="'$6'";'
       
       
       
       
       } #| exec gcc -xc - -o run -static  \
       && exec run
       
       
       
       
       
       
       
       
    
    

re: envdir, storing variables in the filesystem

"no stinkin loops"

or "stupid shell scripter" PoC, take 2

    
    
       #!/bin/sh
       # e is a dir of "environmental variables"   
       # e is mounted on tmpfs i.e., it's memory not disk
       # output:
       # ==> 0
       # ==> 1
       # ==> 2
       # ==> 3
       # ==> 4
       # ==> 5
       test -f e/$0||echo 0 > e/$0
       read n < e/$0
       test $n -le 5||exec rm e/$0
       echo ==\> $n
       echo $((n+1)) > e/$0
       test -f $0||exit # just in case
       $0

------
bloaf
As someone not well versed in OS history / design:

Why are terminals NOT just REPLs for regular programming languages with
OS_Convenience_Functions #include'd automatically?

~~~
yumaikas
From a design standpoint: Because the ergonomics of shells and regular
programming langauges are very different. 99% of your interactions with a
shell is going to be single commands, or short piped chains. Economy of
keystrokes is something that a lot of people want here, which is why cd and ls
are both that short to type. Most programming languages require at least () to
run commands, since not requiring that seems sloppy. This makes your two most
common interactions twice as long to type.

Shells have also been built to help glue applications together, and are more
concise for the easy versions than most programming languages, sometimes
heavily so. Most programming languages, (outside of maybe perl and ruby),
don't make this use case anywhere close to their top priority, but shells do,
trading off other things like a simple syntax for bitwise OR.

In another sense, shells are exactly what you describe, it just happens that
the most popular languages for them happen to be sh/bash/zsh/fish etc, which
make a different set of tradeoffs than the 'traditional' programming
languages.

I'm only partially aware of the history standpoint. My understanding is that
before awk, shells (sh, mostly, IIRC) were about the only scripting level
languages on various Unixen.

------
aidenn0
Is there a good place for discussing Oil with the author (mailing list, irc
&c.)? I have thought a lot about many things in the blog posts, but there's no
comment section, and I'd really like to get clarification as well as dive into
a bit more detail on some of these things.

[edit]

Found it here:

[http://www.oilshell.org/blog/2016/12/29.html](http://www.oilshell.org/blog/2016/12/29.html)

------
ajkjk
How hard would it to be to make an entirely new shell language that doesn't
feel like it's 30+ years old? Why isn't this more common?

I'm aware of, like, TermKit[0], but that didn't get finished and I don't
recall hearing about any newer efforts.

[0]: [https://github.com/unconed/TermKit](https://github.com/unconed/TermKit)

~~~
mamcx
Build it is not the "hard" part. Anyway you can say a lot of languages are
almost there (python, ruby, perl, forth, rebol).

Is that developer are THE WORST. Them will refuse to switch to anything better
if it means to a small change in his old ways.

That is why C/C++/Js/Bash rule the world...

~~~
hyperpape
It's somewhat difficult to come up with a good syntax. You need to support:
unquoted strings, executing commands, pipes, file redirects, etc.

Adoption would be even harder, but any resulting language is not going to look
like Python or any other "mainstream" language, if it's intended to actually
be usable.

~~~
chubot
Yeah this is a problem, but after writing a bash parser I think it's
straightforward to solve.

Oil is parsed line by line like any other shell, and there will be a simple
rule to determine whether you're in command mode or expression mode.

The "lexer modes" technique (previous referred to as "lexical state") should
handle the two sublanguages easily. You basically need it to parse bash, and
you will need much less of it to parse oil. My bash parser has 13 lexical
states (up from 8), but I expect oil to have somewhere between 4 and 8.

How it's used in the bash parser:
[http://www.oilshell.org/blog/2016/10/19.html](http://www.oilshell.org/blog/2016/10/19.html)

~~~
hyperpape
I'm not talking about implementation, but rather designing a clear and
consistent syntax. It sounds like your intent is to hew pretty closely to
existing shells, in which case you're probably doing more detail work.

~~~
chubot
The strategy is to break compatibility with bash but provide an upgrade path,
as mentioned here:

[https://news.ycombinator.com/item?id=13478291](https://news.ycombinator.com/item?id=13478291)

It is a design issue to combine command and expression syntax (unquoted and
vs. quoted literals). But I think I've figured it out ... we'll see in a few
months! The blog will have updates on these kinds of issues.

~~~
hyperpape
But are you maintaining POSIX compatibility? I thought I'd read that somewhere
on your blog, but I might misremember.

~~~
chubot
Yes, the osh language is basically bash, which is very compatible superset of
POSIX sh. The oil language is a totally different language, but you will be
able to upgrade to it automatically.

Once you upgrade it's of course not POSIX compatible anymore :)

------
JdeBP
As I pointed out at
[https://news.ycombinator.com/item?id=13397381](https://news.ycombinator.com/item?id=13397381)
, you forgot rtprio, idprio, chrt, and numactl; the s6, perp, daemontools-
encore, and nosh toolsets; and execline. And TCL and the Thompson shell.

* [http://wiki.tcl.tk/15088](http://wiki.tcl.tk/15088)

* [http://jdebp.eu./FGA/daemontools-family.html](http://jdebp.eu./FGA/daemontools-family.html)

* [http://v6shell.org/](http://v6shell.org/)

~~~
chubot
Thanks, I did add s6 in response to your comment, but it wasn't meant to be
exhaustive, so I left out the others. I didn't add Linux "perf" or a whole
host of other such tools either.

I'm not sure how TCL and the Thompson shell relate though.

~~~
JdeBP
Apart from the fact that you are re-using the Thompson shell's name, consider
things like its if command. Yes, command; not reserved word.

And my list is not exhaustive, either. But it makes the point to those that
would erroneously dismiss this (as you yourself do) as something that "doesn't
see much use today", that this, given how long the list of non-daemontools-
family commands that operate this way actually turns out to be when one looks
around, is not really a daemontools, or even a "Bernstein", thing.

------
GauntletWizard
In addition, rather than docker, try sometime running a process with cgexec.
It helps to know some of the underlying interfaces to get the full scope of
what advanced compositions of those interfaces are doing.

------
jacobolus
Calling anything “oil shell” seems like a pretty good way to invite a
trademark dispute from Shell Oil Company.

------
SixSigma
point free forth like does not make

