So somehow this paper misses one of the greatest developments in this area, oil https://www.oilshell.org/ - which:
1. has a compatibility mode with sh and bash. it's the only new shell I know with a upgrade path from bash [0]
2. otherwise fixes tons of unix shell misfeatures and in particular fixes quoting hell [2]
PS: one fun quirk of the oil implementation is that it's written in Python.. but actually in a DSL that generates statically typed C++ code, ditching the Python runtime altogether [3] [4] - when you do compile Oil, you just get a C++ tarball (somewhat like fftw, a FFT package written in OCaml but which generates C code; fftw developers write OCaml, but fftw users compile C)
(author here) Thanks for nicely summarizing the project!
I didn't know about fftw, but another nice analogy is TeX by Donald Knuth! TeX is actually written in WEB / a subset of Pascal (H-Pascal I think), and then translated to C. That is, it's NOT typically compiled with a Pascal compiler (I wonder if anyone even maintains those?). This came up a few times on Hacker News and I got some interesting details.
I wrote yesterday that I want to hire a Compiler Engineer to help with the C++ translation:
It's already half done, passing ~1100 of 1950 spec tests, so I believe it's very feasible. But it needs someone who can concentrate on it full time.
I might as well post the draft of the "job listing" to get people thinking about this. I think the challenge is finding the right person rather than funding it (I'm applying for funding now)
One sentence description: write an ~10K line translator for statically typed Python, and a ~5K line C++ runtime, and get a free shell out of it! (over 5 years of painstaking work on the semantics!)
The input is ~36K lines of Python the output is ~100K lines of C++. This is a very doable project for an experienced compiler engineer. :) So if anyone knows a good candidate, send them my way at andy@oilshell.org.
----
Also, I (virtually) attended the panel where this paper originated:
I never thought I'd see a fftw reference again. I used it almost 6 years ago for a astronomy telescope data processing library, for convolution. I had no idea!
It seems odd that any paper on this topic (I read both TFA and the paper it references) would not mention PowerShell once, nor the broader issue of how the Unix shell uses text streams as the only intermediary format between pipeline components.
As someone who moved to Linux full-time, professionally and personally, there are times I do miss PowerShell. Never used anything like it. To tab complete "apis" and complex data structures was very enjoyable
For better or worse, it's what I and my team used to manage a decent size VMware cluster of a hundred hosts and 6k VMs ( I think, it was a very long time ago)
Powershell is available on Linux. Albeit it doesn't always play too nicely with existing CLI tools.
There are also other typed shells that appeal to yourself which aren't Powershell but do have behaviours similar to Powershell and which do play a lot more nicely with the wider Linux / UNIX ecosystem:
It seems odd that everytime there is a discussions about shells on UNIX, an ad to powershell comes in. Powershell might have its virtues but:
1)One cannot rely on it for system administration when limited resources are available.
2) even on windows is slow to start.
3) documentation is ... where ?
>everytime there is a discussions about shells on UNIX, an ad to powershell comes in.
The gp (Paul) is not "advertising Powershell" and he's not recommending people switch to it.
Instead, he's saying that it's strange that an academic paper talking about future concepts doesn't even have a cursory survey of what other popular shells have done.
Here's an example of that style of writing where other platform technologies are mentioned: Joshua Bloch is a ex-Sun employee and Java advocate proposing new syntax for Automatic Resource Management and in his note, he mentions what C# and C++ did: http://mail.openjdk.java.net/pipermail/coin-dev/2009-Februar...
It shouldn't have to be said that citing C# & C++ does not mean it's an "ad for C#".
When a new 2021 paper about future of shells has no mention of zsh, Powershell, etc by the authors, it's reasonable for readers to wonder if the team has overlooked what others have done. E.g. Powershell was designed by ex-UNIX admins.
I specifically and purposefully ignore anything Microsoft. I'm interested in open computing, which Windows and Microsoft exist to spite. I assume Microsoft does everything in bad faith, and that any technology they deploy is ultimately in service of their walled gardens. I refuse to participate; I won't let myself be walled-in, and I won't give them any credibility.
Nope, unix people like the fact that thirty-year-old shell scripts still work just fine. In ten years, powershell will be be replaced and it will be the new vbscript. No thanks.
To have a real discussion on the future of shells while ignoring the PowerShell is not an honest discussion. I'm not sure why you'd want to run a 30 year old script when the computing environment has changed so much around those scripts. Regardless of what future you see for PowerShell as a product, the concepts and implementation, is a great place to see what works well and what didn't, particularly when it comes to non-text oriented pipeing
Stability is an important thing. The fact that 30 year old scripts still work, and still can be run, is a testament to the staying power of the underlying technology. That staying power generally means some reduction in cognitive load: I can use the same semantics, principles, and idioms on my phone, laptop, and computer.
That's good.
That's powerful. Contrast that with the JavaScript framework du jour: React today, React with new shiny hooks tomorrow; Svelte a week ago and a week from now; while the boss just told you in an email that the meeting an hour from now regards porting everything to Vue. That's the kind of cognitive load that we're thankfully spared by long-lasting technology.
Now consider PowerShell. It is not an open standard. It is not an open technology. Its steward is Microsoft, of "Embrace, Extend, Extinguish" fame. It bucks against the long-established norm, so that all the established idioms are useless, while its semantics are just as tedious.
Why, therefore, should I bother putting pilot time into this proprietary flying rug whose engine is more-likely-than-not a cracked bottle of Microsoft semen?
As to whether PowerShell has anything to teach us about shell engineering: meh. It's not as good as Bash at doing Bash-like things and it's not as good as scripting languages at doing scripting things.
EDIT: I figured I'd look to see whether PowerShell is open. Surprisingly, it is! With an MIT license to boot. However, as expected with any Microscum code, it includes telemetry out-of-the-box.
As a matter of principle: I don't like it, I don't trust it, I don't want it. The semantics are ugly and it stinks of Microsoft.
> Nope, unix people like the fact that thirty-year-old shell scripts still work just fine.
No, they don't. Because even if a Unix from 30 years ago still does exist (Solaris and Aix) it and the programs used with it changed that much that almost all scripts are unusable by now. And the used shells (Korn shell anyone?) and the shells themselves changed too.
Same with Linux, it changed that much during the last 10 years, that almost any script had to be rewritten.
> the fact that thirty-year-old shell scripts still work just fine
With 30 years old bugs because bash can’t be written correctly. I really don’t understand why are we putting up with that — anything longer than #! line and one command invocation in bash should be a python script instead.
The title is promising, but when I read the paper half a year ago I found it disappointing, no practical value. Academic considerations to get a conference paper.
The shell is first and foremost a user interface. Its entire purpose is just to help users use the computer. A huge amount of its functionality is in dealing with making it convenient for people to do work - from looking at command history, to skipping back and forth along a text line, to dynamically changing a prompt, to auto-completion. Think of it as an IDE.
Shell 'scripting' is only supposed to be a way to automate that IDE. It's not supposed to be a programming language. Literally the purpose is that you're sitting there at your terminal and you think "boy, I would really love to rename 100 files in a certain way". At first you write a quick one-line series of commands with a loop. But the intention is to eventually write a real dedicated program (like the `rename` command) so you don't have to write a crappy line of "scripting".
Shell scripting should remain limited in functionality, and be tailored to helping someone use the IDE. If you want a systems programming language, then develop one. Go crazy and develop new ideas and push the boundaries of computer science. But please don't ruin the shell experience by trying to solve some unrelated problems with it and end up ruining the simple end user experience.
I wrote 75 lines of bash to do failure-resistant streaming parallel computation on heterogeneous distributed systems. "parallel" was very close to doing what I want, but it wasn't particularly hard to do it myself without it. I fire it up, peg every core on as many spot instances as AWS will give me, then shut them all down.
I regularly pass data structures through pipes, and "jq" does everything I have needed so far.
The tools can always get a bit better, but I don't need my duct tape to be any smarter, KTHX.
I’m also a bash expert but I disagree with your point.
Bash was good enough when being a sysadmin meant your structured data was largely just flat lists of plain text. However the industry moved away from that model years ago. And these days it’s more pronounced than ever with myself spending more time in YAML, JSON and tables thanks to modern DevOps tooling than I have done in the 2 decades of experience I had prior.
Sure, jq is amazing. But if I’m having to reach for jq in nearly every pipeline then one has to question why the shell can’t just understand JSON natively itself. And why should I have to learn a different tool with their own formatting language for working with YAML, CSV, XML, S-expressions and so on. Why can’t I use the same shell primitives regardless of the underlying data format is?
This is why I switched away from bash. I’m enough of an expert in bash to use it for DevOps everyday but I’m also pragmatic enough to realise that other shells actually do offer quality of life improvements. This doesn’t mean those shells have to be fully fledged programming languages - I’m really not suggesting that we should be using shell scripting for anything complicated. The problem is the stuff that isn’t complicated has evolved to become complicated in bash because the nature of our work has shifted away from what bash did well.
But each to their own. I wouldn’t want to impose my workflow on others anymore than I’d want others to lecture me about theirs. I just wanted to offer a counterpoint from someone who is also a seasoned sysadmin (and more recently “DevOps”).
I see your point, and you're right - it is a matter of personal taste. That's just where I draw the line as to what the shell should understand. I'm mostly in K8s these days where JSON and YAML are interchangeable so I just need jq (or yq, which is just a jq wrapper).
I'm curious, though, what other shells offer 'quality of life improvements' ... and can I convince my colleagues to install it everywhere?
> I'm curious, though, what other shells offer 'quality of life improvements'
To be honest most alt of them do. Bash has enough footguns to incapacitate a small army. Seasoned users like ourselves know how to mitigate them but using a shell that doesn’t require a conscious effort to support spaces in file names and other really basic things is one less piece of mental overhead I need to consider when working.
> and can I convince my colleagues to install it everywhere?
I’m not sure what you mean here. Are you trying to convince your colleagues to switch so you all use the same shell or saying you still spend a lot of time in remote SSH sessions?
Either way, it sounds like you’re probably best sticking with Bash if those are your requirements.
My remote workflow is largely API driven (serverless) and the few times I do need to SSH into a remote instance it’s usually just to perform basic tasks. The real meat of my console work is done locally these days.
What makes me an expert is the amount of computer books I have near my desk. Sort of like how the enchantment table works in Minecraft. I am also a bash expert because I own an O’Reilly book on the subject I bought at a Borders 20 years ago at retail price. :)
10 years ago my colleagues were astounded by my airtight bash scripts which formed the backbone of our deployment pipeline. I've come a long way since then, and I still learn new things about it regularly, so I guess I'm a bash expert.
Not OP, but I would say that we, regular Bash users, don't really write complex programs in Bash, even though it depends on what you call "complex". Some of our programs can be pretty long, and we can do more within Bash than some programmers know (eg. the arrays/associative arrays features, the vast parameter expansion options etc.), so no, we aren't bound to some 100-line-max rule, if that's what you had in mind.
Bash (and other shells) are very good at working with other programs and using the filesystem. It's true you have to know many idioms, but once you're there, you can be quite productive for these kinds of tasks.
For me, anything that I couldn't write without a guide is too complex. Like a function. Or a conditional.
Subjectivity of what is complex is the problem here, which means that regular Bash users regularly write programs that are complex to me. And I'd much rather use something with less footguns, even if the good parts of the shell become less easy.
So true. The shell encourages one to reason about and simplify workflows by leveraging programs that already solve the complex problems. I'm continually amazed at how many lines of code I can replace with a Bash one liner.
One's level of expertise will determine what is "too complex for $SHELL". When that point is reached, it's time to switch to something else, not time to add more features to $SHELL.
(If, when something becomes too complex for $SHELL, you switch to assembly, you might be too much of a $SHELL expert!)
What usually happens in my case is I start off writing a simple program in Bash, then it unexpectedly becomes complicated as I add more and more to it.
The fact that you can cobble together some parallel execution is missing the point.
Bash is an error-prone insane mess. It's not "a good system". Frankly it's so obviously terrible that I wouldn't trust the opinion of anyone that calls themselves a "bash expert"! It's like being a punch card expert.
Actually that's unfair to punch cards. They were well designed at the time. Bash was born deformed.
As a daily shell user, I want a shell that uses even less resources, and perhaps even does less. Otherwise I am going to use the default scripting shell. Today, that's an Almquist derivative as it has been for 20+ (at least, for NetBSD, Linux adoption was more recent). Never do I see proposals for a "next generation shell" which uses less resources than Almquist and can do all the same things. Anyways, Almquist shell isn't going away anytime soon, not for at least another 50 years. IMHO.
In order to use less resources, features will have to be removed.
The pdksh version of the Korn shell was available to the POSIX shell standardization committee. It was not chosen because it was too large, with too many features.
Another round of less-capable tools does not seem productive.
I dunno, I've never had any complaints about bash's performance ... but I do appreciate the idea of a simple performant shell, as long as it doesn't break anything. I was annoyed to find many #!/bin/sh scripts broken in Debian 11 until I update-alternatives'd it to bash. There goes my performance, but hey presto now things work!
Unix shell isn't usable by normal public. It's just a stash of random symbols everywhere with unintuitive semantics. You'd loose your brain if you begin writing one. There are so many quirky considerations you've to take in mind to handle for all cases just gets even more complex then remembering C++ rules.
The substitution and shell-quoting are the things that piss me off. Why in 2021 would ever want all that legacy cruft? I better write a Python script that is far more readable and intuitive than figuring out all the symbol combinations in shell programming. We deserve better, don't we?
Shell programming is like necromancing, one simple mistake leads to big disasters. Also, these shells never point out the actual mistake, like are very unhelpful.
I hope not. I was hoping that Unix and its shell would die out at some point. But it is a hydra that will live on and haunt our dreams.
FWIW, I have switched to Powershell & Python for most of my work. I only use Python when I have to share scripts with colleages, though. Powershell still beats Python for scripting.
Thanks to the author for posting this. Thoughts, in no particular order:
Repeatability:
- We already have a dataflow graph, it's called makefiles
- Going from shell commands to Ansible playbooks can be made easier
- Declarative, idempotent commands
- Integration with version control
User friendliness:
- GNU standards. If I cannot use --help and --version, I get angry
- Backward compatibility. Ripgrep, I'm looking at you
- Autocompletion is still in its infancy. Co-operative autocompletion is easy, and can be done by utility authors. What are the options (no pun intended) when the author would not update an autocompletion script? fzf and its kin greatly enhance user experience
- Performance. Accidentally quadratic stuff is a Cthulhu-scaled abomination
- Fancier prompts without needing to install extra software or add a bunch of PS1/PS2 scripts
'Novelties' (not so much, other OSes have had them before):
- Coprocs
- Easier parallelization
- Improvement in job control - freezing and thawing jobs, moving jobs to other hosts etc. K8s/kubeflow and friends
Security:
- Better integration with secret managers
- History cleanup when passwords are passed on the command line
> Backward compatibility. Ripgrep, I'm looking at you
Is that really that important? I do have a few POSIX incantations ingrained into my muscle memory, but it’s not a good thing imo. Is that 4 God knows what flag and strange order really that important? Maybe add a separate backward compatible mode when issued through the old tool’s name, but I would like at least a partial revamp of the frankly, not too ergonomic POSIX tools. While Powershell can sometimes be more verbose, I would prefer a similar convention.
It is a damper on gung-ho, brazen disregard for existing code/user base.
It prevents code base rot and saves money for corporations/governments.
It forces developers to adhere to the principle of least surprise, and ensures smooth user experience. Mind you, Powershell beats POSIX on readability hands down, and autocompletion would make longer, more readable options much easier to write. Still, the easiest way to increase adoption is to meticulously copy conventions from predecessors.
Compatibility mode would be enough for most purposes if it's maintained and not dropped three years later in the name of progress and limited maintainer attention span.
1. has a compatibility mode with sh and bash. it's the only new shell I know with a upgrade path from bash [0]
2. otherwise fixes tons of unix shell misfeatures and in particular fixes quoting hell [2]
PS: one fun quirk of the oil implementation is that it's written in Python.. but actually in a DSL that generates statically typed C++ code, ditching the Python runtime altogether [3] [4] - when you do compile Oil, you just get a C++ tarball (somewhat like fftw, a FFT package written in OCaml but which generates C code; fftw developers write OCaml, but fftw users compile C)
[0] https://www.oilshell.org/why.html
[1] https://www.oilshell.org/blog/2020/01/simplest-explanation.h...
[2] https://www.oilshell.org/release/latest/doc/idioms.html
[3] https://www.oilshell.org/blog/2021/01/why-a-new-shell.html#w...
[4] But actually I think there's still work to do to remove the CPython dependency at runtime, https://github.com/oilshell/oil/issues/636