(1) Languages have gotten worse as distributed back ends have gotten more powerful. The IBM JCL and XML references in this post were good!
(2) Workloads that currently run on such back ends could run on a single computer, or at least with many fewer resources. This is one reason I got into shell in the first place! I wrote some shell scripts that saturated 32 cores instead of using distributed systems. In other words I try to avoid the "COST" or "parallelizing your overhead" problem.
We're paying a huge productivity tax and in many cases not reaping the rewards. I think a better better UNIX SHELL can help in the following ways:
(1) We need to bring the interactivity of Unix back to distributed systems. We're still in the days of "IBM Job Control Language" with Kubernetes config and similar kinds of "declarative cloud configuration". We need a flexible and efficient analogue of Bourne shell.
(2) Unix shell is already how you set up local development environments: Docker embeds shell; virtualenv changes your shell state, Ruby's bundler, OCaml's opam switch, etc. We need to evolve this into first class and polyglot environments specified in shell.
Debugging distributed systems locally could be the norm, but it's not.
The local topology should simply be a configured variant of the distributed topology, but it's not. I used to do this at Google with a trick of generating a shell script with BCL (Borg Config Language).
(3) A better shell should be able express configurations that evaluate to JSON, and also statically validate them before pushing, to solve this problem.
XML, however, was universally rejected in favour of things like JSON, Yaml, HCL, Toml - all free of structure, with zero indication whether a computer would find your prose gibberish or the next Shakespeare play until you actually pushed your code to some test cluster.
I want https://www.oilshell.org/ to go in this direction, and there is already some progress. (Feel free to contact me if this resonates with you.) And I have a draft of a blog post about shell, distributed systems, and languages that I need to publish, based on these comments from a couple weeks ago:
Those comments talk about the other problem with the cloud: that it locks you in to APIs! We need the POSIX of distributed systems. Kubernetes was trying to do that, but it's not good enough.
Shell can solve this problem more economically: it expresses UNPORTABLE glue code to leave your application PORTABLE. I did that with Oil's continuous builds, and the Unix-y gg FaaS framework also appears to do that in a pretty nice way.
Yes, it is one of the most entertaining and insightful articles I have read on computing and programming language for a long time.
All the best for your new Oil shell, and I agree that we need to have similar to POSIX standard for distributed systems but it must put local-first software as the first class citizen [1].
Personally I think the local-first software cannot happen soon enough given that what we have endured for the past 30 years with regard to inefficiency and drudgery of the web based applications.
Just wondering about your thoughts on SDLang [2] as the type alternative to JSON, YAML etc, and lightweight alternative to XML?
So I think Oil will be very familiar to SDLang users, BUT it has programmability, which you need in cluster configuration and build configuration, e.g. to express duplication.
So I would divide the space into a few categories:
Very important: With categories 1 and 2, people add template languages like Go templates on top to solve the programmability problem, which we don't want!!! We don't want to move typed data back to the stringly typed realm!
Also, I don't think the "expression style" of HCL, Nix, and BCL is great. It's good for small things, but it starts to get hard to read for large configs. I'd prefer simple for loops and if statements, basically like the Starlark language (a subset of Python used in Bazel).
(1) Languages have gotten worse as distributed back ends have gotten more powerful. The IBM JCL and XML references in this post were good!
(2) Workloads that currently run on such back ends could run on a single computer, or at least with many fewer resources. This is one reason I got into shell in the first place! I wrote some shell scripts that saturated 32 cores instead of using distributed systems. In other words I try to avoid the "COST" or "parallelizing your overhead" problem.
We're paying a huge productivity tax and in many cases not reaping the rewards. I think a better better UNIX SHELL can help in the following ways:
(1) We need to bring the interactivity of Unix back to distributed systems. We're still in the days of "IBM Job Control Language" with Kubernetes config and similar kinds of "declarative cloud configuration". We need a flexible and efficient analogue of Bourne shell.
(2) Unix shell is already how you set up local development environments: Docker embeds shell; virtualenv changes your shell state, Ruby's bundler, OCaml's opam switch, etc. We need to evolve this into first class and polyglot environments specified in shell.
Debugging distributed systems locally could be the norm, but it's not.
The local topology should simply be a configured variant of the distributed topology, but it's not. I used to do this at Google with a trick of generating a shell script with BCL (Borg Config Language).
(3) A better shell should be able express configurations that evaluate to JSON, and also statically validate them before pushing, to solve this problem.
XML, however, was universally rejected in favour of things like JSON, Yaml, HCL, Toml - all free of structure, with zero indication whether a computer would find your prose gibberish or the next Shakespeare play until you actually pushed your code to some test cluster.
I want https://www.oilshell.org/ to go in this direction, and there is already some progress. (Feel free to contact me if this resonates with you.) And I have a draft of a blog post about shell, distributed systems, and languages that I need to publish, based on these comments from a couple weeks ago:
https://news.ycombinator.com/item?id=25343716
Those comments talk about the other problem with the cloud: that it locks you in to APIs! We need the POSIX of distributed systems. Kubernetes was trying to do that, but it's not good enough.
Shell can solve this problem more economically: it expresses UNPORTABLE glue code to leave your application PORTABLE. I did that with Oil's continuous builds, and the Unix-y gg FaaS framework also appears to do that in a pretty nice way.