(author here) I don't think anything you wrote should be a surprise to regular readers of the blog. Many of my blog posts in the last year say "the scope of the project is too large" and similar stuff.
It might be a surprise if you're only reading what shows up on news aggregators. But to be clear, I basically write about what I'm thinking, what progress has been made, etc. And I do "documentation-driven development", which is IMO the best way to design a consistent and coherent language.
The main purpose is to get people to test out the shell, and attract developers who want to push in the same direction. I like this way of working, as I've gotten a lot of valuable feedback and contributions on both OSH and Oil. And I can use more.
Oil is NOT a "product" which you can download and use. (It doesn't even have a logo, partly for that reason.) Right now it's better for say people who have the desire and ability to make it through the quick start for contributing:
The "summer blog backlog" has some project status updates that aren't out yet, but I'll give the tl;dr version here.
There are roughly 5 equal sized parts of project, all large:
1. The compatible OSH language (mature, NOT buggy, but not fast yet)
2. The Oil language (this is 2 years old; there are bugs due to the way we reuse CPython)
3. The interactive shell (punting this to "headless shell")
4. Semi-automatic translation of the "executable spec" from Python to C++
5. Documentation
For context, last October I wrote a post saying that four features (that are DONE and polished) justify the OSH language and I can punt on the Oil language until the indefinite future.
So that was one way of cutting scope. But basically I changed my mind based on prior successes. So I worked on the Oil language a lot this year, and I'm pleased with how its coming together. (The summer blog backlog has sketches of new features. Big things left: the "missing declarative part" of shell for configuration/not-YAML; functions/coprocesses as opposed to shell-like procs)
I made a big list of alternative shells, and none of them is what I want, or what I think the open source ecosystem needs:
This is a very long argument and has to do with the "Perlis-Thompson Principle" and "narrow waists", which I'm about 25% through explaining on the blog.
-----
So the other way of cutting scope is punting the interactive shell into another process, with Oil being a "headless shell":
To clearly repeat what's in those posts: you won't get more than a basic interactive shell based on Oil unless someone else makes big effort. Join #shell-gui on Oil's Zulip, etc.
----
So the project is now focused on #1 #2 and #5: OSH and Oil and documentation. The translation to C++ passes about half the tests, but it has stalled. So I have to come up with some way of getting that done too, and it might involve soliciting donations to hire someone, some kind of two-sided market etc.
But right now I'm just concentrating on the Oil language design and explaining it. So you might see many more compelling posts about things you won't be able to immediately use. That is on purpose!
It all absolutely works -- download it now and try it -- but there is a gap to bridge to make it production quality, which involves documentation and translation to C++.
In addition to direct collaboration, I also encourage parallel efforts. Someone started porting Oil to Nim, and another person wanted to rewrite Oil only (not OSH) in Rust, etc. I encourage all those efforts.
The project is long, but the "hard part" OSH is quite mature (you should test it), and the "exciting part" Oil is less than 2 years old. And five years ago people said it was impossible to evolve bash into a good language. Now people seem to be taking it for granted that it should exist :)
Thank you for the extensive and thoughtful comment! This does help clarify your approach quite considerably. I wonder, since you are hoping to attract collaborators, whether there is some kind of formal spec for the language somewhere? For example, you mentioned parallel efforts: suppose I wanted to write a port to pure C; is there any way, short of reading every one of your posts and trying to contain the whole language in my head at once, for me to know exactly what I need to implement?
Something I've been trying to figure out: what is the exact relationship at present between OSH and Oil? When you say "OSH" do you mean the language, or the shell itself "oil shell"? If Oil is not something I can download, why exactly does that `const v = max(1, 2)` statement work in osh? It's clearly not just a Bash implementation, it's got other features. Is that a subset of Oil's features? Which subset?
Since you're also interested in other shells, you might have a look at pyp [1]. It captures a lot of the way I personally would like to use some future shell. If the features of pyp were integrated into the shell itself, you wouldn't need an external command, you could just (for example) pipe the output of one program into a python-like statement that mangles the incoming strings in some way, and pipe that out to some xargs-like program to use in a subshell. (The fact that you apparently can't use the pipe in what Xonsh calls "Python mode" is for me the central limiting feature of that shell.)
The algebraic data types alone give a huge leg up, but there's more than that. The spec is the code! That's why it's written in an unusual style with a high level language:
At the end of every release there are some links to key source files. At some point these could be nice HTML diagrams rather than raw source code, but the info is concisely encoded:
Also, I don't expect much of this to be intelligible at first... It's a bunch of pointers to start. Anyone who wants to go down this path this will have to ask a lot of questions on Zulip, which will motivate updates to the docs.
-----
I will write a FAQ about OSH vs Oil, since that has come up several times, and it has evolved over time.
There is only a single binary and single interpreter, but there are two entry points bin/osh and bin/oil, which basically correspond to the OSH and Oil languages.
So the interpreter is parameterized by parsing and runtime options. They are all under this group:
shopt --set oil:all # turn on everything as if you were running bin/oil
That is what this Tour doc refers to -- a shared interpreter data model, and a shared process model.
So we talk about them and document them as separate languages. But really it's a graceful upgrade of a single language. If you want you can even set individual options like shopt --set parse_paren.
-----
The syntax
const v = max(1, 2)
is valid in both bin/osh and bin/oil, since nobody has complained the taking 'const' breaks anything. (We try to minimize new keywords -- e.g. 'pass' was removed from the language based on feedback). Whether you want to consider it OSH or Oil is up for debate :) The meanings of OSH and Oil have slightly changed as the project evolves, but OSH means roughly "compatible features" and Oil means "new, incompatible features". It's a spectrum.
(I think I have written all of this in various places, but it does need to be put in a more central place on the site)
http://www.oilshell.org/why.html#why-not-use-it
It might be a surprise if you're only reading what shows up on news aggregators. But to be clear, I basically write about what I'm thinking, what progress has been made, etc. And I do "documentation-driven development", which is IMO the best way to design a consistent and coherent language.
The main purpose is to get people to test out the shell, and attract developers who want to push in the same direction. I like this way of working, as I've gotten a lot of valuable feedback and contributions on both OSH and Oil. And I can use more.
Oil is NOT a "product" which you can download and use. (It doesn't even have a logo, partly for that reason.) Right now it's better for say people who have the desire and ability to make it through the quick start for contributing:
https://github.com/oilshell/oil/wiki/Contributing (how to get a working bin/osh from git in 30 seconds on most Linux distros. You can then hack on it in pure Python and it will be fast)
----
The "summer blog backlog" has some project status updates that aren't out yet, but I'll give the tl;dr version here.
There are roughly 5 equal sized parts of project, all large:
For context, last October I wrote a post saying that four features (that are DONE and polished) justify the OSH language and I can punt on the Oil language until the indefinite future.https://www.oilshell.org/blog/2020/10/osh-features.html
So that was one way of cutting scope. But basically I changed my mind based on prior successes. So I worked on the Oil language a lot this year, and I'm pleased with how its coming together. (The summer blog backlog has sketches of new features. Big things left: the "missing declarative part" of shell for configuration/not-YAML; functions/coprocesses as opposed to shell-like procs)
I made a big list of alternative shells, and none of them is what I want, or what I think the open source ecosystem needs:
https://github.com/oilshell/oil/wiki/Alternative-Shells
This is a very long argument and has to do with the "Perlis-Thompson Principle" and "narrow waists", which I'm about 25% through explaining on the blog.
-----
So the other way of cutting scope is punting the interactive shell into another process, with Oil being a "headless shell":
See these two comments for details:
https://lobste.rs/s/xestey/opinionated_guide_xargs#c_cn4r2i
https://lobste.rs/s/sdum3p/if_you_could_rewrite_anything_fro... (yes I need to make blog posts out of these)
To clearly repeat what's in those posts: you won't get more than a basic interactive shell based on Oil unless someone else makes big effort. Join #shell-gui on Oil's Zulip, etc.
----
So the project is now focused on #1 #2 and #5: OSH and Oil and documentation. The translation to C++ passes about half the tests, but it has stalled. So I have to come up with some way of getting that done too, and it might involve soliciting donations to hire someone, some kind of two-sided market etc.
But right now I'm just concentrating on the Oil language design and explaining it. So you might see many more compelling posts about things you won't be able to immediately use. That is on purpose!
It all absolutely works -- download it now and try it -- but there is a gap to bridge to make it production quality, which involves documentation and translation to C++.
(This post is from a year ago but may give some color on these issues: https://www.oilshell.org/blog/2020/08/risks.html)
In addition to direct collaboration, I also encourage parallel efforts. Someone started porting Oil to Nim, and another person wanted to rewrite Oil only (not OSH) in Rust, etc. I encourage all those efforts.
The project is long, but the "hard part" OSH is quite mature (you should test it), and the "exciting part" Oil is less than 2 years old. And five years ago people said it was impossible to evolve bash into a good language. Now people seem to be taking it for granted that it should exist :)