GNU utilities have addressed this problem for a long time. Maybe he's on OS X? You can install them there though. You don't need anything like PowerShell.
These are options are easy to find with cut --help and xargs --help. Or 'man cut and 'man xargs'. (Though honestly it's weird that xargs doesn't accept --max-lines 1, only --max-lines=1 ? )
---
I plan to add a mode to http://www.oilshell.org/ to support longopts in all the builtins, e.g. instead of
read -r -d ','
instead of
read -raw -delimiter ','
I wrote a blog post last night that shows other benefits you can try now:
I think it's good practice to always use long arguments in (bash) scripts, for the same reason that a variable named `userLastName` is better than one named `str2`.
Because you mentioned it, PowerShell has a similar readability issue with positional parameters `Set-Content $file $text` vs `Set-Content -Path $file -Value $text`. In many cases the use and order is obvious, but it's not always (and will vary by person), and inconsistent style in code also makes it harder to read.
There's no silver bullet for this stuff. You have to be aware of context and future maintainers: If the vast majority of your code is bash and all maintainers are highly skilled in it, go nuts. If most of your product is in another language and you just have some ancillary install or build scripts in bash/PowerShell/whatever, you should assume people looking at it are closer to the beginner level, and it's better to make things as readable as possible.
This, plus the fact that a programming language and a command prompt are TWO DIFFERENT THINGS. One is made to be written "once" and then compiled or run many times. The other is a real time interface that should be thoughtful on the amount of effort required by the user to do things, and as such should not take a lot of typing to get things done (and secondarily support scripting).
This is not actually a natural distinction. It's an historical error dating back to OS/360 and JCL (Job Control Language). Fred Brooks, who ran that project, has since claimed it was a mistake.
For examples of languages that excel at quick interaction without the failures of shells, look at Rexx and Rebol. Or power up Smalltalk.
1) It's pretty natural to morph shell (OSH) into something like Python. bash already has anemic arrays and dicts -- might as well give them proper syntax and semantics.
2) Python pretty naturally morphs into something like Java (or even ML / OCaml) with MyPy. I statically typed the whole OSH front end recently and it worked well.
I don't believe in the artificial barriers between interactive and "compiled" languages. Those two things aren't mutually exclusive.
Langauges like C++, Zig, and Jai have Turing-complete interpreters in their compilers. OCaml is normally compiled but it also has a nice REPL that people often use in practice (the type inference helps).
So there is really no reason to think of interpreters vs. compilers as separate. Both of them should be interactive. Your compiler should be infinitely fast.
-----
I probably won't get to this for years, but I can definitely see a path toward making Oil an efficient compiled language. It almost falls naturally out of the odd "bootstrapping" I'm doing.
Yup, I forgot to mention the nice new example of Julia, which is used interactively, dynamically typed, compiles to machine code, and has introspection just like Lisp (e.g. SBCL afaik):
So I could read and understand the first example with no issue. At first I was like “yah!” and then as I read on I realized, “yup, your right it is gobbledygook” Then I read this post and said, “chubot you know, you are correct, but I am to old to learn your new fangled —- stuff”.
I cannot tell you the number of times I have mistyped - for — and —- for - over the years without thinking about it. Maybe we should make the command expect both and guess based on what’s after. That would make the CLI better.
Yup, Oil will be like Go's flag parser, and accepts one or two dashes!
The reason most tools don't is that they allow stuff like rm -rf and tar -xvf, rather than rm -r -f and tar -x -v -f. But that seems to be optimizing for the uncommon case now, given how many tools and flags there are. Nobody can memorize them all!
And its claims about #1 seems inaccurate to me. I haven't seen any tools that support only short options in a LONG time. For example, try pip --help or virtualenv --help. Or any tool written in Go, or node.js, etc.
The only case where you don't have long options if you're trying to limit yourself to POSIX, which not many people do, and which I don't recommend for this reason (among others).
-----
I agree with the gist of your post regarding #2 -- people should use long options more.
But there's very straightforward and actionable advice that can go at the very top. They don't have to wait for #1, because it already happened. You can and should just start using them in your bash scripts! You don't need to switch shells, switch languages, or even change the tools you use.
Based on this thread, it DOES appear that many people are confused by this. I'm tempted to write my own blog post, which I may do. But honestly I think a tool that translates short options to long options would be a better solution -- i.e. advocacy through tools rather than advocacy through words. (e.g. "go fmt" seems to have done wonders for formatting without making people read style guides.)
This should be pretty straightforward with the Oil parser [1] and I can use help if anyone cares enough :)
In the article he talks about the longer flag names. But the commands themselves are inscrutable. 'xargs?' 'cut?' in neither case can you accurately infer from the name what it does. definitely a good pattern to just name things what they do.
That's only the case for an operator that is rarely used. Once it becomes a primitive of your language, a heavier name gets in the way. You've integrated its meaning now, so you no longer need it explained, but the cost of the extra length continues forever. This is why we don't write, say, "x multiply y". Given this tradeoff, I'd say xargs and cut are good names for those commands: rich enough to convey a large portion of what they do, but short enough to be wieldy and—critically for shells—not to take up too much of a line.
I think that argument made sense years ago when people thought completion systems were slow or bloated, but on modern computers anybody who still thinks that way is antiquated or weird. The defaults shells are shipped with are a problem of course, but that could/should be addressed by distro packagers IMHO.
Consider the long identifiers used in various modern lisp dialects for instance, particularly scheme dialects. It doesn't bother people because Emacs and vim both have completion systems.
A bigger issue with renaming all these utilities is it would negatively impact people who've already learned them. Alienating your existing userbase to appeal to newbies might be in-vogue these days, but I think it's crap.
I don't think the argument has to do with speed of typing; that is a minor concern, since the bottleneck is thinking of what to type in the first place. The issue is the intelligibility of the resulting program. That probably sounds counterintuitive, because people always say that long explicit names make code more readable—but this is only true of code locally. It is not true of programs as a whole. Using long names makes programs longer, as does factoring out subsections into functions and naming those functions. A longer program is harder to understand than a shorter one, and the delta expands non-linearly as system size grows. There's a deep tradeoff here, much deeper than the popular discourse around software development has ever accounted for.
You can see this come up in the OP where the author introduces functions (listAttachment and attachmentInfoLineToAttachmentId) into the example without including their definitions, which would probably double the size of the example.
If you have a rich set of orthogonal, composable primitives, you can do powerful things in a line or two that would require 10x or more code in other environments. The tradeoff is that you have to actually learn the primitives to the extent that you become like a craftsman in a workshop whose tools are all at hand and who can reach for each one as needed without thinking. If you don't know the primitives, of course that line or two will be inscrutable.
IMHO CamelCase is the offender, not the length of the symbol. (CamelCase kills local readability, and therefore global as well) lisp-style-hyphenation is readable to me, even when long. If it helps the code read like prose, all the better.
Completion fast enough to be faster than a human's reaction time has been feasible for many years now. Furthermore there are completion systems that let you type discontiguous parts of your target word, essentially giving you the best of both worlds.
To truly be fast enough not to break the flow of thought->text, it has to (for me) be ~2 tokens ahead of where the cursor is, not completing the current one. And even then I doubt it'd work completely, because it requires rapid context switching to check the completion and pick the right one.
True, but personally, I think this a cost well worth paying for intelligibility and self-documentation. In my opinion, documentation and clarity is more important than maintaining programmer's flow and being economical with command length (and therefore typing time), especially with newer tools that manage a command-line history better. Clear command names pay dividends that outweigh the cost of repeated typing. The dividends come when another person inevitably picks up the shell script you wrote or the documentation you've written to ramp up - and they might not have yet achieved the same level of fluency with the command line. For them these commands might not come pre-integrated.
I would never accept a function in a code review named 'xargs' no matter how much of a short-hand convention it became. I think many company styleguides enforce similar rules for code. Why should command line tools be any different?
Only sort of, and cut much more than xargs. in fact, to a newcomer, seeing xargs in a command pipeline would I imagine be completely unintelligible until reading the man pages, as it was to me the first dozen or so times I encountered it.
This seems to be primarily an argument that "CLIs should support GNU-style double-dash options". I agree with that argument, and in most cases it's easy to add support for them. Most option-processing systems support double-dash (long option names), and I find them much easier to explain later.
The problem is that at least historically many of the BSD developers, and thus the POSIX standards community, strongly resisted standardizing support for double-dash. So while utilities allowed* to have long option names, the POSIX standard doesn't directly support them and doesn't standardize them. That means that if you want to write portable invocations of common tools, the only way that always works are the short (single-letter) names.
It may be time to revisit this decision. If the *BSD developers (and so on) are now willing to accept long names, and the POSIX standard is finally modified to support it, that would eliminate at least one roadblock to actually using them.
I think that's a good idea, but even if BSD developers accept them and POSIX changes, we'll still have deployed systems for decades that support only short options.
This situation reminds me of the dictatorship of the minority:
I think a more deployable solution is to write shell function wrappers for all POSIX utils that support GNU-style options. I just filed this as an issue for Oil:
edit: As mentioned on the issue, busybox and toybox (Android) are probably even less likely to change than POSIX/BSDs. So I do think the solution realistically belongs in the shell.
Busybox has good reason not to change. It's focused on being lightweight and embeddable, not being user-friendly. It would have to catch both types of flags to maintain compatibility, and parsing for long flags is definitely more complex. Coreutils are different, but busybox is fine with short flags.
By the way, oil does look interesting. One key issue is you really need to move to python 3 asap. 4 months and change before it's no longer supported. Plus, people want fancy unicode stuff in shells these days, and that's much easier in python3. Out of curiosity, do you think python vs c will have a major impact on performance? On the other hand, I figure stricter semantics means more room for optimization and slow-down on corner cases. Are there any performance numbers I can look at?
> busybox can be kept the same, and the parsing can happen in the shell.
Busybox includes a shell, ash.
Thanks for linking the FAQ; that clears a lot of things up. I can see how the incredible opinionated of python 3 on the whole unicode mess might be an issue. You're maintaining what you need (the big concern) and have clearly done your homework. Clearly you're pretty invested in it, which makes the project look much more interesting. I will be so happy if some one finally gets shell scripting right.
In no small part this is because parsing long opts can be significantly more complicated than short ones. It is by contrast very easy to parse short, single-letter flags. Not saying it's not worth doing, but there is a reason this has not been standardized.
Parsing long opts is trivial; just determine if the string exactly matches. E.g., "--version" only matches "--version". Short opts is what takes slightly more effort, because you have to accept both "-ld" and "-l -d". We already handle the "hard" case, we just need to support the easy case.
One thing the author's missing is a consideration of how often a system is used vs. how often it's learned. Let's just take one example from the article. Would it have been better if `sed` had been named `streamEditor` from the start?
sed was written in 1973. Think about that, it's been around for half a century. It's regularly used by millions across generations. Imagine how many countless man-hours have been saved from naming it "sed" instead of "streamEditor"? And I'm not just talking about typing code, but also reading code.
Yes "sed" is pretty inscrutable if you don't know Unix. But pretty much any Unix developer or sysadmin instantly recognized "sed" the same way they would a common English word. Very few people just use the Unix command line tools a few times, then forget about them. Once you learn Unix, chances are you'll keep using it for decades. And for those people, their brains almost certainly parse "sed" much faster than they would "streamEditor".
So, I think here's the lesson. The utility of terseness is very much related to the relative balance of expert users vs novice leaners. If you expect your system to mostly be repeat power-users, then terseness is very good. Systems that are around for decades, used widely by thousands, and general-purpose frameworks tend to fall in this category. In contrast, a high percentage of your user's man hours may just be people getting acquainted with the system. Maybe it's an application that solves some niche corner case, and most people use it once or twice then forget about it. Or maybe it's some internal software, where corporate turnover means that most devs working on it at any given time are neophytes. In this case verbosity and explicitness is a very good thing. You're not trying to cater to power-users. You want things to be as clear as possible, even if it means a lot of typing and spelling errors and wasted screen space.
The challenge here is that you mostly need to make these decisions at the development stage. You have to try to predict what your user-profile will look like, before you actually have any users. I doubt the original authors of sed could have ever imagined how popular and lost-lasting it would become. But given that it has, there's no doubt they made the right choices.
On another hand, imagine if the original computers on which unix commands were designed weren't restricted to 80 characters terminals with a few kilobytes of memory and almost no storage nor CPU. I'm pretty sure the author would have favored much more expressive commands.
I think design of those command lines were done first and foremost because the underlying system constraints were huge. Now that we have tabs autocomplete on shell and IDEs, the length of a command pretty much doesn't have any kind of importance.
> I'm pretty sure the author would have favored much more expressive commands.
The very first compiled computer language was designed to be very expressive and able to be read almost as English, and be simple to read and understand. Would this be what you wanted? Would you change your mind when I tell you its name is COBOL?
well, I'll walk that back just a touch: I can see someone plausibly arguing that it may take longer to read longer commands. In that case, I'd argue maybe we need multiple views of the same source: one view for novices and one for experts.
The first time i encountered really long names (on purpose, as opposed to java-gone-wild frameworks) was with objective-c. It shocked me at first, but since the parameters are spread along the method call, just like in a regular sentence, it really didn't matter, and actually made the code really nice to read. This style is nowaday pretty common with named parameters in function calls.
On another hand, imagine the original computers on which unix commands were designed for weren't restricted to 300 baud teletype terminals. Are you really willing to wait 3x or 4x the time to read something as the print head goes WHACK WHACK WHACK across the page. Going through print ribbons like crazy? Wasting paper?
I disagree. Unless you are talking about lets say "real" experts e.g. people literally living in the shell for years and doing all stuff in it, I think it is pretty much impossible for less regular users to remember command flags or deduce their meaning looking at them. I use ssh and scp daily but for the life of me I can't remember which one uses -P or -p for port and I still constantly guess wrong. Some common stuff like ps or ls - who the hell knows what those obscure flags mean? I just remember "magic" set of flags for each command and use them every time. Unpack tar or tgz? Impossible without man first. And the list goes on. Again - I'm using shell daily, I remember "some" stuff and I can deduce meaning of small part it, but still general rule is - shell is arcane magic and has to be googled or looked up in man.
> Unpack tar or tgz? Impossible without man first.
tar has inbuilt format detection nowadays. Just specify extract (either with an x or --extract). GNU tar has supported it since before 2010, and bsdtar from around the same time frame. Debian lagged a bit, but can also do it.
I think that's largely due to inconsistency more than anything else. As with a lot of other things, you don't need to memorise all the options, just the ones that you use the most.
Been using UNIX for over 2 decades, and as my only OS at home for over 15. And I use the vast majority of UNIX tools (including sed) only occasionally. For me it's just a lot easier to write it in Python (and these days, even easier in Xonsh). Because even when I read the man page of some of these tools, I have to experiment multiple times till I get the command right. And if I ever have to revisit it months later, it's inscrutable, whereas my Python code isn't.
In my experience, the following are the types of people who know UNIX tools really well:
- People in IT who need to do a lot of shell scripting
- People who are forced by the company/environment to do a lot of work in a constrained UNIX environment (e.g. embedded)
- People who really want to be UNIX command line gurus (extreme minority).
The majority of UNIX/Linux users do not fall into any of these categories. They may be overrepresented on HN, though.
Agreed. I only just started using Unix commands for the job I started a year ago. It's quite possible I won't need them for my next job/team and I will certainly forget most of the commands I have memorized.
Not saying I'm the standard here, but us passable Unix users exist.
But pretty much any Unix developer or sysadmin instantly recognized "sed" the same way they would a common English word.
That's the key point. We talk about grepping and piping and sedding and sudoing as if they are English words, because they've effectively become part of the vocabulary.
This is a good way of thinking about this. I think the ideal solution may involve multiple views of the same source code. Experts and novices can switch to different views depending on their level of familiarity.
I actually think the same thing could exist in non-bash contexts.
That bash code he posted as ‘unreadable’ is perfectly clear to me, and I don’t even know anything about the jira command line interface.
I think the real problem here is people always have preferred tools for jobs. The author obviously spends more time with JavaScript and finds that to be clearer and easier to write. I spend more time with bash and for me, it’s very clear. It doesn’t make it ‘inscrutable’
You have to admit the bash code is less readable to someone who hasn't used those particular CLI arguments or flags in a while (or ever), as compared to someone trying to read the JavaScript version without knowing much about JS or the library that the code used.
You might argue that this is an acceptable trade-off to make to in order to gain terseness and ease of typing. But this is different than saying it's just matter of tool familiarity.
I think you're missing part of the author's point. In the bash script, each command and flag being used does something and what they do is not obvious to anyone who is not well-versed in those commands and flags.
The JS example is closer to english or pseudo-code, which makes it inherently more readable. It is not as terse, but it is also not esoteric like the bash script. Someone who doesn't know bash or JS will probably be able to understand the JS version more quickly.
I have no idea why you're being downvoted -- the person who wrote the code shouldn't be deciding how readable it is, as they're clearly biased.
I had to deal with this the other day. I approved a PR, with a comment that 5-7 lines of the logic were unreadable and convoluted. I left a more concise way to write the code as a suggestion.
When I saw the merged code in master a few days later, I noticed that the convoluted logic was still there. When I asked them about it, they just said "well, to me, it's readable, so I decided to keep it".
¯\_(ツ)_/¯
Lesson for me: Don't approve PRs if I don't feel good about the quality of the code being committed.
Lesson for my coworker: If someone strongly disagrees with you, get a second opinion, don't just merge your code into master.
(Arguably?) Powershell seems to take this approach. I have to admit, I like it philosophically, but whenever I try to use it there's some part of it that feels icky to me. It might just be microsoft's ConventionOfOverusingCapitalizationEverywhere though.
This is all an example to show style, not real DCL.
Part of what makes DCL great is that you use a single, consistent API to add commands and flags to the system, connect your code into the system, and then interpret them when your code gets called at runtime.
This even leads to common verbs for different purposes, making the whole system feel more unified.
As other comments have pointed out, it seems like the author is looking for PowerShell. I have used a fair amount of PowerShell and it always leaves me feeling like I should just fire up Visual Studio and learn C#. I think that shell languages should prioritize terseness at the expense of readability, allowing experienced users to accomplish their goals as quickly as possible. Any code that you expect others (or yourself) to maintain in the future does not belong in a shell script.
>Any code that you expect others (or yourself) to maintain in the future does not belong in a shell script.
That is a gross generalization that is not true in many, many circumstances. Creating a full-blown application requires several more steps and much more maintenance due to compilation and packaging than writing interpreted scripts. Whether it's PowerShell, Python, or something else, scripts solve an accessibility problem: not everyone knows how to, is comfortable, or has the time to create compiled applications.
Additionally, PowerShell in particular has excellent self-discovery and help facilities built into the language. The Get-Help cmdlet can be used on any script or cmdlet to retrieve its associated documentation. Tab completion means I don't have to guess what classes, namespaces, members, parameters, etc. are available and I can even be lazy and not use Get-Help to discover them.
Any code that you expect others (or yourself) to maintain in the future does not belong in a shell script.
Why? A shell script is often the best tool for the job. If your reason is that shell scripts aren't maintainable, then this is a circular argument basically amounting to "we shouldn't make shell scripts maintainable because we shouldn't ever maintain shell scripts because shell scripts aren't maintainable".
allowing experienced users to accomplish their goals as quickly as possible
Having both short and long versions of each flag seems to give you the best of both worlds.
I agree that ideally shell scripts should be maintainable, and that designs which allow both succinctness and maintainability – like supporting both short and long flags – are preferable. But there will always be cases where you have to choose between the two, and in those cases I think a shell should choose succinctness. For all of the Unix shell’s weaknesses as a programming language, it’s an incredibly powerful tool for interactive programming, writing one-liners at the shell prompt. And when you’re trying to program as close as possible to the speed of thought, every keystroke counts.
And when you’re trying to program as close as possible to the speed of thought, every keystroke counts.
And when is that ever important? I've been in development and sre roles for decades, and I've never been in a situation where I have to program at the speed of thought.
The only time I've seen milliseconds matter is watching movie depictions of hackers.
People here in these comments are claiming that terseness in CLI commands is a good thing.
Perhaps there is a way to do one better? I could imagine an interactive shell that can show you the relevant portion of the man page for that specific flag.
Man pages never have sat right with me; I would love to have a shell that can show me (intellisense?) the most common flags and options commands are run with.
The command line is powerful but the barrier to entry is high. Anybody who does not fathom complexity hidden behind pages of manuals is doomed to be overwhelmed with the sheer ignorance and lack of discoverability a shell provides.
> Man pages never have sat right with me; I would love to have a shell that can show me (intellisense?) the most common flags and options commands are run with.
While not intellisense, the fish shell [0] might be what you're thinking.
> Man pages never have sat right with me; I would love to have a shell that can show me (intellisense?) the most common flags and options commands are run with.
Apple's A/UX (their Unix port for the Mac, before OS X) had a cool thing called Commando that did something like this. If you entered "..." in place of an argument to a command in the shell, it would invoke Commando to get the command arguments, and then replace the "..." with them.
Commando would bring up a dialog for the command that would list the various options and flags. There's a screenshot of the Commando dialog for ls here [1].
More complicated commands, like the C compiler, had tabbed dialogs with different tabs for things like language settings, optimization settings, debug settings, and so on.
PowerShell can do that first part for PS cmdlets rather than general man pages; Get-Command searches for available functions, loaded module exports and executables in the path. It takes a parameter -Module which limits that to things exported by just one module. You can use Get-Help to find out about just that parameter, e.g.:
PS C:\> Get-Help -Name Get-Command -Parameter Module
-Module <String[]>
Specifies an array of modules. This cmdlet gets the commands that came from the specified modules or snap-ins.
Enter the names of modules or snap-ins, or enter snap-in or module objects.
This parameter takes string values, but the value of this parameter can also be a PSModuleInfo or PSSnapinInfo
object, such as the objects that the Get-Module, Get-PSSnapin, and Import-PSSession cmdlets return.
You can refer to this parameter by its name, Module , or by its alias, PSSnapin . The parameter name that you
choose has no effect on the command output.
Required? false
Position? named
Default value None
Accept pipeline input? True (ByPropertyName)
Accept wildcard characters? false
Or in a shorter version, help gcm -param mod*
> I would love to have a shell that can show me (intellisense?) the most common flags and options commands are run with.
It can do some of that for PS cmdlets as well, intellisense can look into the cmdlets, find the names of exported parameters and show them all to you (not in most common order) and then for some commands you can do get-executionpolicy -Scope {tab} and it will cycle through the three available scopes. That autocomplete can be extended to do things like interrogate a database engine and complete table names, by the author, or wrapped by the user to do that. Again not really for system commands unless you wrap around them.
>Perhaps there is a way to do one better? I could imagine an interactive shell that can show you the relevant portion of the man page for that specific flag.
I've had a similar idea before. I think this is a great way to think about the problem.
Just like so many other comments in this thread, you're describing powershell. There's autocomplete for parameter names in the basic powershell.exe prompt, but VS Code, VS, and PowerShell ISE (integrated script environment) all show argument types and docs through intellisense.
I think explainshell [1] is a halfway measure for what you're looking for.
My current approach is simply to have a notes folder with the title of a man page as the filename, I put a 'tl;dr', common commands section, and write down any quirks or complexities you only tend to remember when you've just been digging through things - am working on something better at the moment.
I think Powershell tried to do something like this article suggests. I appreciate the efforts, but do agree with the common criticism of verbosity.
Also, it goes to show that no matter how bad aspects of bash are, the power accumulated over the years is hard to match, and a lot of that power is strongly coupled to the arcane and terse conventions.
Powershell is almost good, yet it falls off the mark in about a dozen ways.
A lot of very simple things are harder to do (text is easier than objects up until a point) since Powershell treats everything as objects and LinuxCLI/Bash does everything as text being piped around to specific commands.
The biggest complaint I have with Powershell is that it is so obnoxiously slow at parsing text files, that it just can't do basic scripting tasks on even medium sized files ~100MB without tons of StreamReader[] overhead or without calling a pre-compiled C# binary directly. This is the biggest fail. Nearly all the obvious ways of appending to files or parsing them stop working once the file size gets bigger than tiny. This isn't a problem with Linux commands that just pipe text to highly optimized C binaries. I'd drop Python for 90% of my scripting tasks on Windows if Powershell were faster in this area, but it is just painfully slow (if all the obvious ways of doing something in Powershell take ~5min when even Python can do it in ~4sec, you have a problem).
However, you can usually write some very beautiful and easy to read code by using the built-in objects once you learn them. I also like how powershell's functions allow you to create commands with parameters in a super easy way so you can easily build DSLs in a way. Everyone feels differently, but calling my commands in this manner (InvokeCommand -param1 "blah") reads a lot more naturally to me than (Object.Method(param1)). Also, Powershell can handle things like filesystem interaction in a way that doesn't feel like a bolted on afterthought as Python's does.
I agree with you that the "obvious things" are frustratingly slow, but you can get significant (>10x) improvements with small changes, even if not the fastest. If big.txt is a random test file of 100MB and 700k lines then I get this:
PowerShell 7 is working on performance improvements throughout, but early design decisions mean this particularly is not likely to change.
> This isn't a problem with Linux commands that just pipe text to highly optimized C binaries.
Python can do this easily as well, and PowerShell can't, and that is one thing I think doesn't get enough discussion and mention. For however good C#/.Net can be, it's still another layer between PowerShell code and low level OS stuff, a layer other shells and high level languages don't have, and a shell is supposed to be "OS Script" first, not "C# script". Python ctypes -> C library is just a ton easier than PowerShell -> C# -> P/Invoke -> Win32 API.
First of all, thank you for the detailed reply with examples.
I didn't know that slightly modifying "Get-Content" could speed it up so drastically. As we've both stated, it isn't very obvious or intuitive without knowing the detailed inner workings of the cmdlet. Do you know why the "Get-Content" can't be more like the $x = ${'path'} option by default? Also, what is going on in that example and how is it so much faster than "Get-Content"?
The examples where you call out to .NET directly aren't too bad (well at least the first one is terse enough), but there are a lot of users like myself who don't write much C# and therefore don't know which namespaces to call for .NET voodoo. The reason I learned Powershell was so I wouldn't have to use C# for scripting as it really doesn't Excel there being statically typed and not having an interactive REPL. If I have to constantly write C# for performance, I might as well just write C# lol.
I'm glad to hear Powershell 7 will get performance improvements and am sad it won't change anything here, but at least I can try out your examples!
As far as Powershell performance is concerned, I still don't understand how some cmdlets can't call out to something pre-compiled. Does all the powershelly parsing and dynamicism really hurt that bad?
Haha that's OK, I can rant or ramble on PS for ages ;)
For the cause of the slowness, try a Get-Content file.txt | format-list * -Force on a single line file and have a look at what comes out; you'd think it would be a System.String with only a Length property but surprise, there is a lot of metadata about which file it came from, and which provider the file was on. A small addition to one string, but it adds up for tens of thousands of lines. That is the big reason Get-Content is so slow. PowerShell has Usermode filesystem / virtual file equivalents called PSProviders, so there is some overhead for starting up a cmdlet at all and then binding its parameters and then dropping through to the FileSystem provider, but the added string properties are the big overhead for file reading.
They should allow you to do convenient things further down the pipeline, e.g. filter the strings, then move the files which contained the strings you have left, although I'm not aware of any genuinely useful use case. I expect there is one somewhere. This is baked in from v1, so removing that would be a breaking change on a very heavily used cmdlet, so it won't stop happening.
The variations like Get-Content -Raw were the intended workaround, by outputting a single string, you skip much of the overhead. The other approaches, ${} and .Net methods, avoid get-content altogether. ${} is the syntax for a full variable name, and PowerShell has PSProviders which are a little like virtual drives, usermode filesystems, or /proc and so you can ${env:computername} for an environment variable, ${variable:pwd} for another way to get the working directory, and ${c:\path\to\file.txt} fits that format so they made it do something useful.
> Does all the powershelly parsing and dynamicism really hurt that bad?
The language is interpreted, but loops which run enough times get compiled by the Dynamic Language Runtime (DLR) parts of .Net, swapped in while the code is running, and run even faster. Parts which are a thin layer over .Net like @{} is a System.Collections.Hashtable and the Regex engine is just the .Net regex engine with case insensitivity switched on, they run pretty fast.
Function calls are killer slow. Don't use good programming practises and make lots of small dedicated functions and call them in a loop, for speed inline everything. Because functions can "look like" cmdlets with Get-Name -Param1 x -Param2 y, there is a HUGE overhead to parameter binding. You might have configured things so -Param2 gets the values from the metadata on the strings from Get-Content, there's a lot of searching to find the best match in the parameter handling.
The pipeline overhead is quite big, code like Get-Thing | Where { .. } | Set-Thing runs pretty slow compared to a rewrite without any pipes or with fewer cmdlets. (Get-Thing).Where{} is faster, and $result = foreach($x in Get-Thing) { if (..) { $x } } even faster.
The slow bits IMO are where PowerShell does more work behind the scenes to try and add shell convenience rather than programming convenience; the great bit is "it's an object shell", the downside of that is "making objects has costs". Compare SQL style code "select x,y from data" describes the output first, then the database engine can skip the data you don't want. Shells and PowerShell have that backwards, you "get-data | select x,y" and the engine has no way around generating all the data first only to throw it away later. Since PS tries for convenience, its cmdlets err on the side of generating more data rather than less, which makes this effect worse.
I don't want to end up writing C# for everything either :)
Thank you for the insight. I don't do a lot of powershell daily, but the last time I had a big batch job I was disappointed in how clunky it was to make a foreach-object -like pipeline utilize a sensible number of processes. I have forgotten the details but I think I needed some sort for add-on that split jobs across processes. Also, the pipeline wasn't pipelined in the parallelism-sense.
No problem. I'm certainly not an expert, but that is mainly due to the deficiency I mention above. I'd use it for a lot more use cases if it could handle those scenarios.
I think Powershell "jobs" can help run things in parallel, but I've never done more than just read about it. The Powershell in Action book (3rd edition) covers that functionality I think. The first chapter to that book is pretty awesome.
My dream as someone forced to use Windows would be for Microsoft to use some dollars to make Powershell a one stop shop for data science, simple games (think commodore sprites), numerical computing...etc. That sounds a little wacky I know, but I think people would appreciate every computer shipping with a powerful environment like that. Since Powershell is built on .NET, surely someone could make a library for Powershell that abstracts away some of the .NET complexity to where I can just do something like:
Not having to install Anaconda and import a ton of libraries on the target computer would be helpful. Note that I'm not saying to reproduce all of Matlab in Powershell, but putting some of the most popular routines as built-in commands would be pretty awesome.
I like the sound of your dream, but I don't think there's much commercial return on making a fun to play with 8-bit home computer environment. That's what Raspberry Pi wanted to be, and instead it became "cheap web browser host" and geek paperweight. The PS team is small, their real goal seem to be to encourage you to Azure through managing it with PS from any OS.
It would be up to "the community" to make modules to do this kind of thing, which would be possible if anyone wanted to, but unlikely to ship with Windows.
> putting some of the most popular routines as built-in commands would be pretty awesome.
Microsoft's push is to avoid more "bloat" by leaning more heavily on optional modules installed from the PS Gallery. But `group-object` is a builtin for doing a kind of SQL GROUP BY operation, and Measure-Object just gained a -StandardDeviation option, and there's now ConvertTo/From-Json, and ConvertFrom-Markdown. If there is a basic routine that would be popular and fit many use-cases, you could request it at https://github.com/powerShell/PowerShell/issues/
There is a GPL licensed scientific computing framework called Accord.NET, it's not wrapped for PS but can be used almost directly, and some of these others might be viable as well:
Thanks for your thoughts. You're probably right about the pipe dream.
I agree that there isn't much commercial incentive, but I honestly don't see why major OS providers can't include some basic graphics primitives in the OS. I'm not talking about embedding the unreal engine or anything. It could be used for a lot more than primitive games btw.
I understand the desire to avoid more bloat. I can't believe Windows 10 requires the storage and RAM that it does.
Which should help at least some cases. (PoshRsJob is a reasonably easy / fast way to use multiple processors inside the same overall Powershell process (with RunSpaces).
I think we may be on the edge of that transformative moment though.
I had spent 2 hours yesterday trying to sort CSV lines in bash. Decided to do so even though I have a nice strongly typed framework to manage tuples, which is slower than bash.
I always forget that the -k argument requires 2 numbers if you want to sort by only one column.
Then LC_ALL.
These are not the things that are so easy to debug against.
So in the end it would have been faster to use Python.
There’s xsv-rs, but it doesn’t provide the same semantics or power I have in the framework.
Point is, I think there’s some space for novelty in UNIX tools. Especially given the research in DSLs for defining the transformations of tuples.
Sad thing is, I am not sure at this point if my framework is more powerful than this or not. I think mine preserves the semantics of “sql execution plan” better than CSVKit, but I might as well be wrong.
I did not know about CSVKit three years ago; but I don’t think it would take a prof. developer to build it too much time either.
The problem is that the shell is frozen in time. have not evolved and it show.
The shell is just BAD. Anti-USERS and even anti-APPS!. composing commands in shell is just BAD. Is the worse API you can cobble in a hurry.
---
I love the interactivity of it, but frankly FoxPro DOS 2.6 have a MUCH better experience. And in fact exist many others experiences of interactivity much better (Smalltalk, JS Dev tools, Jupyter notebooks, etc).
I have dreamed about this and think will be nice a shell that:
1. Instead of just BLOB of binary-hopefully-is-text-maybe it interface with SHAPES and TYPES:
enum Data {
Stream(Data),
Lines(Scalar), //utf8, bytes, ints, bools, etc...
Table(Header, Data),
Tree(BTreeMap(Data, Data)),
Graph(Nodes< Data >)
}
This abstract interface will support most stuff we need. Then with mime-types you can request "Tree as JSON" and get that.
This is similar to HTTP headers: But instead of html and that you ask for a shape and the type of it. Of course some utilities will only support some and can reject. And at minimum can return BLOB for interoperatibility with old shell utils.
When returning you get the mimetype/shape so will know that the blob is really a JPG and use that fact for displaying purposes and type inference.
With types and shapes you will get some static type validations.
2- Rich output (images, graphs, etc).
3- Uniform help and API discoverability alike swagger.
4- More rich and uniform API for the language that glue all.
For example is nuts that each utility must reimplement sorting, filtering, etc.
Instead if the shell have filter/map/group/etc in-built commands together with shapes and types:
You mean using something like typescript as a shell language? That could work and it would leverage the javascript runtime and yet provide the rich type system and benefits.
I don't focus in the language itself but in how interface between utilities. The shell is a place for a light scripting layer and make talk apps in many langs.
Instead in what the utilities talk. Like in a REST interface you declare the kind of data (table, tree, lines of text, a binary blob) and this unlock what combinatory commands are available (so for table/lines you get relational commands for example) that are global. So the utilities are mostly concerned about their input/outputs.
Probably for efficience the utilities get a REQUEST call alike:
GET /list: WHERE name = "hello"
to have the opportunity of do the processing inside.
"Maybe we just haven’t rethought CLIs since their inception in an environment".
And there is a reason for that: backward compatibility. CLIs get used in scripts, and the worst that can happen is to see the script breaking because the command changed.
Let's take the example given in the article: it uses sed (created in 1974), cut (created in 1985), and xargs (created before 2001). All these tools are older than Jira and I'm quite sure that using a version of sed from 10 years ago or the latest one with basic flags will give the same result. CLIs are often ancient pieces of software that started as small projects and grew because features were requested while the way it was working shall not be broken as millions of developers use sed in their bash scripts and expect them to work on any kind of Linux machine, old and new.
For context, I'm working on a CLI for an enterprise product and we got in trouble just because we were changing error outputs. You just cannot do like on a UI where as long as there is a way to perform the same action as before users will not complain.
The second issue is that a CLI is used in two very different contexts: as an application by engineers and as a parts of scripts by those same engineers. This makes it tricky as a CLI developer to know what to do in terms of naming as a daily user will use shortcuts quite fast but a new developer reading a script using our CLI might not understand the purpose of a command. IMO if a CLI has a good documentation (in app and online) you should focus on making your daily users happy as someone seeing the command in a script can check the docs online and hopefully understand what's happening in less than a minute.
Random historical note -- xargs dates back to 1977 or 78 -- https://groups.google.com/forum/m/#!msg/comp.unix.questions/... has a post in 1988 from the original author saying they wrote it at Bell Labs a decade before, and that it actually predates the Bourne shell itself.
Humble suggestion: use the terse options when writing iteratively and the long form in scripts that you save to a file. The latter will likely be read by others or yourself in a few months from now, and it could be nice as immediate documentation if you did something fancy with a tool or option that you seldom use.
For instance, I know the handful of options that I personally encounter and use for the "grep" program, but the man page lists plenty that I do not immediately recognize. So typing them out in the script helps a lot with readability.
Going overboard with terseness or verbosity is obviously not helpful to the (human) reader, regardless of the direction.
I think there is another post on HN right now about how computers are more like appliances today and no longer as readily programmable as they were before.
And looking at that problem, you realize that a shell is a huge part of the tinkerability of an OS and bash has dropped the ball in that regard. It's like it's frozen in 70s.
We should be moving towards something like a LISP machine where the whole system is exposed through your shell itself. Perhaps, we can start using Chrome dev tools console something like a shell, a good first step might be using typescript for the shell.
Short options make it quick and easy to use commands you use all the time.
Your little bashism was quick for me to read because it was compact, scannable, and used popular programs I use frequently (cut, sed etc). Yes, you need to look up the options for programs you don't use often but you need to look them up anyway because you don't use them frequently.
Conversely the longer example you used had longItentifiersThatNeedParsing (and are too long to fit in your fovea) and worse, are easy to accidentally confuse with say longItentifiersThatNeedNoParsing -- an error you are likely not to see when reading the code and an error you might especially use when your tooling has a pop-up of suggested completions.
The gnu approach of long + short options can help; it would be better if getopt() would write a completion grammar into a section of the ELF file that could be mmap()ed and used by readline(), similar to the TOPS-20 jsys that inspired it.
Perl allows this. Perl allows easy string stream manipulations, file manipulation, and runs external programs easily with backticks or qx(). As soon as a shell script has more than a single loop and conditional, you can rewrite it in Perl instead, just as terse, more readable and way faster. Just use Perl, the Swiss Army Chainsaw.
Yes! Perl is what I turn into when I have a Bash script that is becoming too big or too complex. It's easy to to turn Bash code into bad Perl code that is still better than Bash. From there, it's easy to create a maintainable Perl script that is more elegant, concise, and maintainable. It's a fantastic glue language that will tie all the components of your project neatly.
And if you are thinking "yeah, but then I need to learn Perl", don't worry. The amount of Perl you need for translating Bash into Perl can be learned in a few afternoons.
I've been a huge fan of PowerShell ever since I started using it (over two years now) and it's my go-to scripting language on my workstation. You can write integrations in the form of modules - VMware has Power CLI, Azure and AWS both have great PowerShell modules.
Why would you say it's not a real programming language? It's as powerful as Bash and seems to do the job quite nicely in most cases for scripting.
Oh. It's certainly better than Bash, but it's still better to use Python or C#. Powershell makes so many weird decisions and it's so verbose and wordy that it becomes unreadable in the opposite way of Bash. I don't think the shell scripting space is desirable. Either you should be typing in commands manually or you should be writing a program. Powershell and Bash exist in a weird space in between them and wind up being crappy programs rather than readable shell commands.
I remember sitting down at a VMS terminal for the first time (a university library's terminal had trouble with its OPAC and dropped me into a logged in VMS shell). In fifteen minutes I had learned how to work with it, had the machine show me the university's network, and started to feel comfortable. I cannot imagine a similar experience if exposed to a Unix shell for the first time.
The poster is holding a line of bash to the standard of code and is illustrating that readability should be the goal and a way of bringing bash commands to a standard of readability for something like a PR. Readability is really there to show _intent_
I would say though that if you are bringing this to the code standards of today then this should really be wrapped up in some kind of unit test (https://github.com/sstephenson/bats )for it to pass the PR. That would make the code a bit more maintainable and can be integrated as a stage in your CI/CD pipeline.
If we do that then the intent would be clarified by the input and the expected output of the test. Then then the code would at least be maintainable and the readability problem becomes less of an issue when it comes to technical debt.
I've done this plenty of times with my teams and its certainly helped.
Sounds like the author wants the VMS shell with its concept of command tables and attributes that provide fully consistent syntax and standardized online help. It even makes it nearly trivial for any utility to provide its own command line that follows all the same syntax, help mechanisms, and conventions.
Which is fine with me. I have always hated how people seem to want computers to be more human-like. It's a computer, not a human being. I like that this stuff takes some skill to learn/use
Completion helps, but pushing too much in that direction (automated tool assistance) hits up against another thing: You want a CLI which is usable on a slow system or over a slow network connection, and those kinds of things are still very real, especially when you're trying to fix a system which is running a runaway process and therefore can't devote much CPU time to running a helpful CLI.
There are other ways to fix that specific problem. None of them are as generally applicable as having a terse CLI.
I think the larger issue is that shell scripts are intrinsically bad for writing readable code.
Consider the "cut" line in the shell script vs the "map" line in the JS script. The map line is easy to understand because it uses a chain of simple functions (map, split, array access, and lastly trim). The "cut" line has to cram all of this functionality into a few command line arguments instead of composing basic building blocks together.
I mean look at this: What does "only delimited" mean, what does "fields" mean, I might just be an illiterate boor but I don't remember what delimiter means off the top of my head (something like seperator, I assume).
cut --only-delimited --fields 2 --delimiter '|'
The JavaScript version is easier to read because it uses fundamental building blocks in the language that are
1. Super intuitive. (Split seems pretty easy to understand, especially when you know the input is a string, similarly everyone knows [1] means array access).
2. Widespread. You have probably encountered all of those JS functions before, I'd say the chance of encountering the "cut" utility is a bit less.
Summary: Shell just fundamentally sucks because it outsources a lot of functionality to various executables, each which have their own command line arguments. The ideal method is to use a programming language with consistent, composable, building blocks.
I feel like in my time we wanted to learn everything. These days people don't find as much value learning everything and prefer as little new knowledge as possible.
It's a skill that will get you out of trouble, allow you to do things others can't and because of that it might get you promoted which will get you hired.
In my experience, while what you say is true, if I were to rank the skills that "might get me promoted", there are far too many that are way higher than this.
Conversely, look around you at those who are at your "level" in the company, or at tech people who are higher than your level. How hard is it to find someone who does not use sed, awk, cut, etc?
I've always been a senior developer at the highest level at where ever I've worked. No one I've worked with over the years has ever used those tools. I do and it gives me an edge in response time.
But if you want to move up.
Everyone role above senior developer involves dealing with people/managing projects. Those skills get people promoted. More tech skills don't.
Better IDE support would also help here, if it could annotate what each flag meant for example and provide inline help and autocompletion. The nice thing about shell is the commands are very short and easy to type on the command prompt, PS is too verbose and is cumbersome for that use case and more full featured languages have all the upsides plus better tooling.
Powershell has abbreviations (aliases) for common commands and fantastic autocompletion for flags and arguments passed to cmdlets. Where-Object|Format-Table is just ?|ft for example.
Also remember that in PowerShell there's very little munging - half of any serious shell script is sed/cut/awk/tr and so on to munge the output of one command into the input of the next. That more than makes up for PowerShell's individual commands or cmdlets having more characters in.
I get that some people don't like or invest in it, but I'm fine with the way the default shells work on macOS, GNU/Linux with coreutils or BSD.
Sure, you can make a version that is usuable by people who aren't normally using it, but then you get the inverse later on where you need a special shell for people that use it a lot and want shorter commands...
There is no one size fits all, and instead of trying to bend what's there to suit a larger group you could simply do what others have been doing and make more 'entry level' shells for those that want/like/need it. There is no reason we can't have both.
p.s. the examples on the webpage are super easy to read, but the javascript ones are probably much more prone to aging and breakage than the bash ones... I'm pretty sure it won't work everywhere, wheras the cut/sed/bash/xargs ones will work anywhere with the utils in de base systems for the past 20 years.
I'd say the counter example to this is Powershell - if you've ever used it, it's quite verbose and can be less than pleasant to write. I think having both options is the best, when you're getting started it's nice to have verbose, highly self explanatory commands. As you get more familiar with the environment, it's nice to move to something more concise.
The bad part is when writing "good" powershell the verbosity is exponentially worse and it turns into
try
{
if (-not [string]::IsNullOrEmpty($_.Column8))
{
$fullPathName = Join-Path -Path $dstDir -ChildPath $_.Column8
$pathTestResult = test-path -LiteralPath $fullPathName -ErrorAction Stop
# This is a hashtable of the parameters to a cmdlet
# the only purpose of this 'splatting' is that
# powershell commands get too long
# and can't be line-wrapped in any good way
$exportParams = @{
Encoding = 'ASCII'
NoTypeInformation = $true
Append = $true
LiteralPath = $fullPathName
Verbose = $true
}
Export-Csv $exportParams
}
}
catch [whatever]
{
}
1. Quicker to type repeated variations of commands at a CLI.
2. Memorization of flags isn't as much a concern since you're at the CLI, you can often easily look up the meanings of options and flags with -h or man pages.
It may be slightly more difficult in the beginning, but once you're familiar, terseness saves significant time.
I would relate shell languages more to vim or emacs in that regards. They have a higher learning curve but pay off in efficiency in the long term.
I neglected to think about tab-completion, but I see you mentioned it at the end. I've never used it for bash. It really depends on whether I can guess or remember the first letter(s) to the option. If it's a command I am new to, I'd still need to print out the available options anyways.
> Programs must be written for people to read, and only incidentally for machines to execute.
That's true of programs that are written to be used and maintained over a period of time. Shell programs like that are mostly system configuration and installation scripts; they are not written that often compared to throw-away one-liners that most shell users write every day. I'd hate to use even a regular programming language for the daily tasks, let alone a verbose one.
I'd buy the argument that using single-letter flags is confusing, except...
the huge popularity of hot keys in GUIs, etc, which have the additional demerit of not being particularly discoverable or guessable. We accept non-discoverable hot-keys etc in GUIs, so this argument probably results from unfamiliarity.
Anyone else put off by camel case at the command line?
Jumping back and forth between JS files and functions in camel case and Unix standard snake case is more confusing for me than memorizing the couple highly used flags.
I sknow I’m being subjective, but we’re talking style, so it’s all kind of subjective.
I happen to like the short options though (at least it is my opinion). (Of course, some programs can do both, but some programs only have long options.)
the cut, sed, xargs, utilities are available on all major unix derivatives and have a pretty consistent set of command line flags... I don't see why we need to suddenly slap camelCasedParameters on everything because the author needed to read some man pages...
This might be one of the core reasons why Linux never took off as a Desktop solution. Also looking at the comments here just shows how out of touch people are with the real world...
If you relish in terse commands that don't mean anything to anyone who hasn't been spending their last 20 years working on the command line, then go for it. Just realize that most of us (talking about the already small population that is software developers) are not interested is spending anytime on dealing with this junk. This is not to mention the other 99% of the population who doesn't even know what Linux or Bash is, and good riddance for them...
The JavaScript code the author posted is immediately intelligible to anyone who has been keeping in touch with programming language developments over the past years. It doesn't even matter if you come from C#, Python, JS, Java, etc. It should be familiar to all of them. You might not know exactly what it does, but it seems oddly familiar.
I may be making a leap here, but I would go as far as saying that this very attitude of command-line elitism is the primary reason why Linux has had zero chance to take hold in the Desktop space. The developers have no clue why "normal" people would not want to use the command line (and I am including myself in it) and their efforts of creating a GUI (just look at the junk that is KDE, Gnome and Unity... Proudly sold off as GUI) have been so out of touch with reality, that I can't even.....
Anyway, each to their own. I am mostly using the command line to start a build process. While I am able to do a lot of more advanced stuff, including properly using VIM, I am actually regretting spending the time on learning it. These days I wrote most of my "bash" scripts in Python instead (just to clarify: I write scripts in Python, for which it seems well suited. I don't see Python as a viable choice for creating actual software). There are some scenarios in which Bash is still a better choice, but thankfully they are far and few between.
The Linux Desktop has nothing to do with the command line.
And most of the desktop environments are significantly less of a mess than Windows 10's mishmash of five separate GUI styles. Even if I don't like Unity either.
His
can be written Likewise can be rewritten These are options are easy to find with cut --help and xargs --help. Or 'man cut and 'man xargs'. (Though honestly it's weird that xargs doesn't accept --max-lines 1, only --max-lines=1 ? )---
I plan to add a mode to http://www.oilshell.org/ to support longopts in all the builtins, e.g. instead of
instead of I wrote a blog post last night that shows other benefits you can try now:Oil's Stricter Semantics Solve Real Problems http://www.oilshell.org/blog/2019/08/16.html