Hacker News new | past | comments | ask | show | jobs | submit login
Creating a bash completion script (iridakos.com)
169 points by pelsio on Sept 24, 2018 | hide | past | favorite | 41 comments

Shell programmable completion highlights how slow the OS community can be to change/improve core tools (shell, coreutils, ttys). Completion should be specified in the binary, and compatible with all shells. A similar interface could be designed for all scripting languages. It was obvious ~20 years ago that it makes no sense for each shell to specify their own completion system, and yet here we still are.

This makes sense for some completions (e.g., vanilla lists of options). But often the completion depends on other context that the binary doesn't need to know about. For example, I complete `git grep` patterns based on ctags. There's no reason git should know about ctags; it's only my personal completion that brings the two together.

That is probably a very rare case, and any shell-agnostic-completion-protocol could be extensible to handle such a case. But, I think 99% of completions that people want/use on a daily basis could be covered by a machine-readable version of the usual "--help" option (with support for placement of paths, pids, strings, numbers etc.).

If it doesn't work reliably for all cases, there would soon be inevitable need for the kind of scripted completion we have now.

You will be a fan of argcomplete, which makes some effort to standardize an approach for in-script completion: https://pypi.org/project/argcomplete

Of course, it still has to be bootstrapped using the conventional bash approaches, but it's a first step.

> Completion should be specified in the binary, and compatible with all shells

This is somewhat like the approach that DEC took with VAX/VMS in the late 1970's.

The command line and arguments/parameters were declared external to the executable and known to the the shell (in this case, DCL) via the SET COMMAND command. It had its own little data description language. The shell did the parsing user interaction (recall, completion, etc) and provided utility functions to the binary to call to obtain params. So while the completion was not technically in the binary, a standard api was used to obtain it and the definitions were created side by side.

Worked very nicely and was extensible by 3rd parties and could be used to incorporate "foreign" binaries as well.

I'm optimistic about binaries providing completions, I think we'll see more of that as tools adopt frameworks like https://github.com/clap-rs/clap or https://pypi.org/project/argcomplete/ which can provide completions for your favorite shell.

But I do think many important/useful binaries will never provide completions, or provide sub-par completion that people will want to improve, so we can't ever count on completions from binaries.

Recently I've been thinking some about how the LSP (https://microsoft.github.io/language-server-protocol/) might be useful for a CLI. A readline implementation could use something like a LSP for syntax coloring and completion.

Imagine having a consistent editing in your favorite shell, repl, and editor.

I'm working in this area right now if you want to help :)


I took a survey of what people use here: https://lobste.rs/s/z96uyr/survey_what_shell_completion_scri...

That produced a link to this old project which tried to do something like what you want: a completion system that works for multiple shells (in this case, by code generation from a DSL)


It's easy for you to say "completion should be specified in the binary" without actually specifying HOW. That's not straightforward, although I agree it's a good goal.


Right now I'm implementing a bash-compatible completion API, and I'm actually getting pretty far without too much code. git-completion.bash is 3000 lines of code, so it's actually easier to emulate bash (with all its warts) than to write an entirely new completion system AND THEN write 3000+ lines of git completion in that hypothetical system.


FWIW git is also the biggest completion script in the zsh source tree by a factor of 2 -- ~7000 lines IIRC with the next largest being around ~3000 lines.

Eventually Oil should have something better than bash completion, but the first step is taking advantage of ~40K lines of existing code in the bash-completion [1] project. I also looked at emulating zsh but that would be a lot more work than emulating bash.

[1] https://github.com/scop/bash-completion

Are you building something to support multiple shells, or just yours?

Well, since I'm emulating bash to start, those completion scripts will support multiple shells :)

However, I recently learned that zsh also emulates bash, e.g. the complete/compgen/compopt builtins, along with some global variables. So that is already technically true.

Still, I'd like to define some kind of "nicer" protocol that binaries can implement to get completion.

But that also sort of exists: I learned that when you do "ls --<TAB>" in bash right now, the script actually dynamically greps "ls --help" for flags! This is in contrast to how ZSH works -- it has a bunch of canned completions, which presumably suffer from a version skew problem.

Someone out pointed out "npm completion" in this thread, which is new to me (try it; it prints a bash script).

git is also adding 'git --listcmds' to help with completion. So the logic is partially in the binary, and partially in the shell completion script.

I think I've seen other ad hoc mechanisms along those lines too. Ah yes I recall that there was XML output for the Google flag parser [1], which is meant to be used for completion, although I'm not sure how commonly it's used.

So basically there are a lot of different systems. Hoping that someone will come along and produce a universal solution is probably wishful thinking. It's another "boiling the ocean" problem.

[1] https://github.com/gflags/gflags

I'd like to contribute to a project that would help me be able to write a CLI tool and have tab completions work across shells. The problem is that I don't have a good enough understanding of how tab completion currently works to even know where to start. Is it fair to say that you're having the same problem?

Yes I think that is a good problem to solve. I'm working in this area right now, although I wouldn't say cross-shell compatibility the specific problem I'm solving now.

I think what would be nice is to collect examples like 'git --listcmds' and 'npm completion' (previously mentioned), i.e. places where the binary itself helps out with completion.

In those cases, more of the logic is shell-independent. It's not 100% and will never be, but it's nice to share as much as possible.

Also, it would nice to see other ways that commands support multiple shells. For example, git has both bash and zsh completion in its tree.

Though I'm pretty sure the zsh developers wrote their own completion that is much richer than the bash-like one in upstream.

I believe the problem is that "completion" in zsh has a higher standard. You can make a lowest-common-denominator solution already. I think that is OK for now, but zsh users might disagree.

I think the other popular interactive shell is fish (based on my survey), so I'd be interested in learning more about it. However I wouldn't underestimate the fact that probably 90% of people use bash.

Feel free to e-mail me if you want to talk about it more (address in profile), or you can chat with me here:


PowerShell does this.

as a friend of mine uses to say in these cases: PR or GTFO ;)

I noticed a while back that Debian 9 introduced some strange breakage with bash completion. Previously, if you were typing 'dd if=/dev/...' and started using tab for completion, it would behave as expected. Under Debian 9, typing 'dd if=/dev/' and hitting tab will turn the command into 'dd /dev/'.

I'm sure this was an oversight in the march of progress, but it sure is annoying to see a regression.

If this is still an issue, would you mind reporting it?

Slowly but surely, I'm replacing half my job with bash scripts. Anything even remotely tedious or error prone, I find that the small investment of time I'm putting in pays off remarkably quickly.

This article? Amazing. My scripts are about to become even more useful and easy to use.

I'd recommend trying to move anything fairly complicated into a "real" language - you'll get a lot of libraries, more well-defined and familiar syntax and also won't have to deal with cases like "do I want double quotes here"

Or, have your cake and eat it too!


I tried writing Bash completion handlers for a Lisp dialect. I couldn't figure out how to get Bash to stop closing quotes, so I gave up on it.

Concretely, here is the behavior:

  $ ./txr -e '(list[Tab]
  (list         (list-carray  (list-str     (list-vector  
  (list*        (listp        (list-vec     
  0:[0924:142129]:zelenka:~/txr$ ./txr -e '(list-vector'
I continued with -e '(list-v[Tab] and it then produced -vector' including the closing quote!

Clearly, it is oriented toward the idea that each token of the language being completed is a separate command line argument, such as a path name or whatever; it is not clear how to get it to complete syntax that is packaged as a single argument in a quote.

The bash code is:

     local target=$2
     local context=$3
     COMPREPLY=($(txr ./txr-bash.tl "$target" "$context"))

  complete -F txr_complete txr
The ./txr-bash.tl file is:

  (tree-bind (target context) *args*
      ((m^$ #/-[etpP]/ context)
       (iflet ((match-rng (r$ #/[^ ()\[\]]+/ target)))
         (let* ((pref [target 0..(from match-rng)])
                (suff [target match-rng])
                (funs (if (plusp (len pref)) (find [pref -1] "[(")))
                (boundfn (if funs (fun fboundp) (fun boundp))))
           (mapdo (op pprinl `@pref@1`)
                    [andf boundfn
                          (opip symbol-name
                                (starts-with suff))]
                    (package-symbols :usr))))))))

The "real" screenshot, which was done with transparency turned on and displayed parts of what was underneath the shell, was a bit difficult to read.

Thanks! I only knew about this linux journal article: https://www.linuxjournal.com/content/more-using-bash-complet...

I used it for coding a simple bash completion for Kafka. Code is almost self-explanatory. https://github.com/igponce/kafka-bash-completion - just in case someone might find it useful.

Nice write up. If you want it down to the point I can recommend this StackOverflow/AskUbuntu answer:


If you care about having nice completions and are able to switch your shell, use zsh. The completion engine in zsh is much more featureful.

I created once a completion for a bash function I wrote. Hint: launch npm completion and steal code from there.

I would like completion to automatically work on aliases in bash. I use bash over zsh in order to become better at bash programming, but completion of aliases is the main thing that makes me miss zsh. Well that and * * globbing syntax.

(How do I cause two consecutive asterisks to be rendered as such in HN?)

> I use bash over zsh in order to become better at bash programming

That's masochism. It's perfectly reasonable to use different interactive and programming shells. I mean, I write most of my would-be shell scripts in Python these days, but I wouldn't want to use it as my main command line (although xonsh is actually quite nice).

It's perfectly reasonable, but what's relevant is whether it's optimal. In today's world, with the ascendancy of cloud services and containerization, shell scripting continues to be just as important a skill for software engineers as it was in 2008, 1998, or 1988. And today, the language of production shell scripting is bash. Therefore, using bash in one's personal system is good training for the skills you will make use of in production systems.

E.g. see https://google.github.io/styleguide/shell.xml

> bash is the only shell scripting language permitted for executables.

I'm surprised you choose to write would-be shell scripts in python. I think that a major reason that shell scripting endures is that handling process management and process output is much more work in real programming languages than in dedicated shell languages.

EDIT: But thanks for the pointer to xonsh; that looks fun. Something innovative like that could definitely tip the balance in favor of using that for local shell work and leaving bash as a language one has to know for production work.

I write plenty of shell scripts where appropriate, but my main point was really that you don't need to couple your interactive environment to your scripting environment. Sure, write your shell scripts in bash. That doesn't mean you need to use that as your regular shell, though.

It's true. My thinking was that there are some techniques that I use in both interactive shell usage and in scripts, and that using the same language for both would teach me them better. I'm trying to think what the best examples are, having run this experiment on myself for a couple of years now. I think one is process substitution with <() and io redirection stuff like <<<.

Also IIRC correctly bash and zsh differ in whether piped commands run in different subshells.

Stuff like that; I'd like to just have firmly in my mind for interactive shell and scripting and not have to know two similar languages.

But I miss tab completion for aliases.

Bash version 4 supports * *.

Thanks, updated.

This is tangential do the OP but what would it take to send external input to a completion script?

The idea would be to have a different interface to do the auto-completion.

I'm thinking of creating a visual UI through service discovery and providing an easy shortcut sheet to various points of a system.

I really wish they would make completion work in BusyBox/Ash. Would make Alpine a lot more useful without having to install Bash.

I'm guessing this works with macOS ancient version of Bash, right?

It's been on my todo list for a long time.

You should update by installing from homebrew.

Yeah, but replacing something as basic as the shell is not a comforting thought. My days of fiddling with the OS are long gone.

If you're nervous about replacing your default login shell you can do it just for tmux in your .tmux.conf like:

    set -g default-shell /usr/local/bin/bash
Having competitions on my Mac really makes my day, and every time I complete on a remote host like below, I am still thrilled.

    scp somehost:<tab>

That sure is enticing.

Do you mind sharing the needed incantation?

    brew install bash bash-completion@2
The SSH magic lives in: /usr/local/Cellar/bash-completion@2/2.8/share/bash-completion/completions/ssh in a function called _scp_remote_files().

You aren't really replacing it. Brew installs in another directory than the one where system tools are.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact