This is a WIP (hence why there are drafts and todos). This is being done by the folks at http://mywiki.wooledge.org/BashGuide which is definitely a valuable resource if you're trying to learn Bash and its idiosyncrasies.
The first few chapters look very good, best of luck with the rest!
In case anyone here is interested in more reading material, I recently wrote a small book about Bash that could be helpful: https://gumroad.com/l/datascience
To make sure it didn't read like a manual, each chapter is an "adventure", where I show how to use only command line tools to answer questions such as: What's the average tip of a NYC cab driver? Is there a correlation between a country's GDP and life expectancy? etc
This is an interesting project (I'm all for approachable learning) but it seems to be missing almost every chapter...maybe not ready for the spotlight?
Bash scripting and its array of tools is a poorly designed language. Writing a non-trivial program, even for an experienced developer, is a painful process. The syntax is uneven, hard to read, and easy to get horribly wrong. I would say mastering Bash has diminishing returns past the intermediary. Any time you need to write a non-trivial program, you will save time and life expectancy from stress management by using ANY other language, even Perl or C. Writing complex shell-modifying code in my .bashrc has been one of the more tedious and non-rewarding parts of my life.
I find shell to be a great part of my computers, ideal for making little inter-operating data-processing tools, system control programs and other hacks; it has enhanced my interaction with OS greatly. Its syntax is a little unusual, but to me it is very concise and readable. It's just that it requires some time to get used to it. After you've made your few tens-hundreds useful programs, you'll get the vibe shell has and begin to see even advantages to, say, python (in some task domains).
Maybe it's my lack of experience with Bash, but I feel the same way. While I generally find most interpreted languages easy to parse, bash just looks like Greek to me. I'd be curious to know what advantages a Bash script would have over, say, a Python script (aside from the obvious Python dependency).
I use bash pretty regularly in the devops/sysadmin world pretty often. It can do some wonders, but it has some /serious/ issues.
That said, the reason it beats Python and all is simply because it's almost guaranteed to be on any *nix server you'll log into. Solaris, BSD, some networking equipment - it'll be there. Python? Maybe. Python 2.7? Sorry, your sysadmin didn't add that yum repo and we're stuck on 2.6 at best.
You'll have similar problems if some packages aren't installed on a machine, but as long as your scripts are using the usual gnu packages, you should be good (with some exceptions of course).
It's a crappy problem to have, but it's probably not going to go away, especially in prod environments. It's just easier to run bash than to get your sysadmins to do some damn patching (or management to approve it..) than to try/potentially fail with anything else.
Then again, I'm all for using /bin/sh for everything. :)
In bash simple things are simple to write, get and read. And you don't need to deal with imports to do them, it's out of the box. Execute any ex. file? Just write down its path. Don't want to pause the main program? Add & after that path. Write output to file? Just append > filename after the command. Regex matching? Just use if [[ string =~ pattern ]]; ... lots of useful stuff, few keystrokes away. Python can do these things and more, but there is often a lot of tiresome overhead with recalling library names, dealing with local namespaces, method parameter positions, type casts just doing the most elementary stuff. This high-level complexity is useful for big and monster-big collaboration projects, but is wonderful to avoid when possible. Shell makes that often possible and is fun to use as well.
Not only just the environment dependency, but also human dependency. Being able to script in bash is considered a skill that any sysadmin would have. If you write bash scripts you can pretty much assume any sysadmin can read/write it. They may or may not know python.
This is much needed. We essentially have all this software we deal with daily and many people don't know basic things about it, not just bash or zsh... and that is funny, people install zsh because it's the thing to do, but you see that they don't know why.
Thanks for making this! I wish I had a guide like this when I was starting out. Does anyone know if there is something like this for zsh? I'd imagine there would be a lot of similarities, but some notable differences.
Bash is incredibly useful, and I think more people should use it as a cross-platform default scripting language. That said, for the most compatible shell scripting, learn Bourne shell scripting. https://en.wikipedia.org/wiki/Bourne_shell
It drives me nuts when people say this. /bin/sh is not even Bourne in some distros (dash in Debian for instance), and bash is installed everywhere (unless you're running AIX, I think ksh is still default). There are some bash-isms that are good things and people shouldn't be afraid of. I think it is actually counter productive to tell people to write things in other languages if you need more complexity. If you're writing a script I think you should be weighing instead whether or not you should fully automate what you're doing. If you decide that's the case of course bash isn't going to be your go-to tool, but if you're writing a one-off script what's the point of telling people to use a more robust language if you can do the job with bash?
I don't understand why people are so opposed to bash in general when it's so powerful for how straightforward it is.
Dash is Bourne-compatible! And Bash is not installed everywhere. I can think of at least five systems i've used in the past five years that either didn't have Bash, or Bash was not the default shell, and even the path to Bash isn't the same on all systems (since it is technically an add-on). Anyway, I did not say use Bourne for more complexity, I said for more compatibility.
People are opposed to learning yet another tool. But here's the really funny thing about the "use a more robust language" argument: not only do they introduce more dependencies, they can introduce more bugs (due to increased complexity), and are almost always a maintenance headache. The one thing that's easy to maintain after 10 years is a shell script. Perl is a nightmare to maintain after 10 years (mostly due to the lack of good Perl programmers) and awk is just barely better.
Right I was thinking about that after I replied to your comment, that I let my point get away from me. I read so many other comments putting down bash I was pleading my case to them, not you, haha.
I did not realize dash was bourne compatible. Also, I'm curious what systems you've used didn't have bash installed by default, as I haven't used anything in a long long time that didn't have bash.
Solaris, BSD, HPUX, hell even stripped-down Linux installs (though sometimes it exists and is just in a weird path). Occasionally I need to bootstrap an OS installer and they never have Bash.
By default I write Bourne, and if I need the extra complexity/speed/etc i'll switch to Bash, but it's pretty rare that I need to use Bashisms.
Something else to consider is that many embedded Linux systems do not have bash, only sh (aka Bourne shell). It is often easier to use only Bourne shell-isms to get things to work in a cross-target fashion. That said, I prefer bash for most cross platform (as in cross OS platform) tasks that need a simple shell script of some kind.
If you are writing functions in Bash, your task is probably sufficiently complex that it would benefit from being written in a language other than Bash.
Bash functions are very useful when you want to manipulate the state of the shell in some way. That is, you write a function for it and stick that function in your .bash_profile. (Unless ofc an alias would be able to perform the job in which case one should just make an alias instead.)
Beyond that, I have been in many situations where standalone shell scripts make the most sense for a task and in which case bash functions have helped me organize the code such that it is both more readable and also so that I don't have to repeat sections of code multiple times within a single script.
It all comes down to personal experience and preference, I think.
I wrote sh for this very reason http://amoffat.github.io/sh/ This lets you use a programming language, instead of a shell language, to solve programming tasks.
Actually, this is exactly the sort of guide I've been meaning to read, since I'm inclined to take the opposite view.
Too often, I think, my task is probably sufficiently simple that it could be a nice portable obviously-a-script Bash script rather than Python/whatever.
Agreed. Bash (and similar scripts) are particularly useful in tasks involving filesystem and process management, where they offer capabilities that are more complicated in other languages. Functions are a helpful way to organize your use of those capabilities.
Zsh has quite a nice option "auto_cd". If the first word you type doesn't resolve to a command, it tries to change the current working directory to it instead. So just typing ~ will get you to $HOME, and .. will take you up one.
I love this option. I type "/tmp" when I'm about to create some short-lived scratch files. Those saying "you can just alias that" are missing the point. Where do you stop with the aliases?
Yes. Although you get it for free with Zsh where it works with any directory you type. Not just the ones you think to make an alias for beforehand. :-)
Even if that is true, it doesn't mean you should not cleanup and organize the bash you have "today" and will port "tomorrow". In the meantime your bash will be better organized and more maintainable.
About 7 years ago I had a job where we supported almost every Unix all the way back to AT&T SVR3 (including OpenServer, UnixWare, HP-UX, AIX, Linux, OSF/1). With the exception of SVR3 every single one had binary packages for Perl 5.
Your point is valid, that bash should be avoided as processes become increasingly complex. But there is still room for bash. How much room depends on what you are trying to do.
In hopes that the creator does read this, it looks like a great resource but I can't read the text, my eyes are very strained and I got a headache very quickly.
>My life has gotten so much better since I [switched to rc].
How long ago was that? I switched to rc in 2001 because I was unhappy with bash, and wish I had not bothered.
If the entire world switched from bash to rc when I did, then my switching to rc would have been a minor improvement, but since of course the rest of the world did not switch, my switching to rc just made me slightly incompatible with the rest of the world, and those incompatibilities more than cancel out the relatively minor advantages of rc relative to bash.
For example, file-name completion (the tab key) in shell mode in Emacs has been programmed to use the backslash character to escape, e.g., spaces in file names (and OS X ships with files whose names contain spaces). But rc does not grok the backslashes: it uses a different convention to quote characters like spaces, and no one as far as I have been able to discover has modified shell mode to use rc's convention (and none of fish, zsh, csh, tcsh, etc, share rc's convention on quoting).
rc is different enough from bash that about a third of the time, I cannot just copy a command line from a web page and paste it into rc without manually editing the command line first. for example, the two common ways of doing command substitution in Unix shell command lines are `like this` and $(like this), but rc recognizes neither convention, and I have to change it to be `{like this}. (Being different from the way the rest of the world does things for little or no actual benefit, by the way, is common in software from Plan 9.)
When I say that rc has only minor advantages over bash, I mean that I still find it almost as annoying to write anything involving a looping or a branching construct in rc as I do in bash. And I still miss not having access to the data structures I have access to in, e.g., Emacs Lisp. And for those data structures, such as the string, that rc and bash do have, I miss the API I have access to Emacs Lisp.
I still use rc as my interactive shell (mainly because I do not yet want to rewrite my 570 lines of customizations -- mostly rc functions that wrap commonly used commands like ls and du) but plan to phase out my reliance on rc gradually over the next few years.
> For example, file-name completion (the tab key) in shell mode in Emacs has been programmed to use the backslash character to escape, e.g., spaces in file names (which are part of the base system on OS X). But rc does not grok the backslashes: it uses a different convention to quote characters like spaces, and no one as far as I have been able to discover has modified shell mode to use rc's convention.
It's quoting is it's power!
And I fixed shell-mode in emacs for myself with the following:
The remaining function to fix is comint-quote-filename - of which you'll see commented out there. It's slightly different from the shell-quote-command override in that it needs to only be enabled during shell-mode and it needs to know when to skip quoting all together.
Don't fear, I'll try to get it working and send it to you (and on my github). I truly enjoy rc and emacs simultaneously, so hopefully any discoveries I make can help you if you'd like them. If not - it sounds like you want to switch to eshell anyways, which could be great (albeit will have bad things as well as it doesn't support i/o redirection).
And you've guessed wrong about my wanting to switch to eshell.el: I took a fairly long look at eshell.el, but adopted shell.el instead. After about 5 years of using shell.el, I wrote my own mode for running and capturing the output of command lines, which I have been happily using for the last 9 months.
Like shell.el, my mode relies on a traditional shell like bash or rc to parse the command lines entered into it.
Although I do want to learn how to use Emacs Lisp code to parse the command lines I write, when I do, I'll probably just turn most of those parse trees right back into command lines (i.e., strings) and pass them to rc, bash or dash.
Once I have the ability to quickly modify my Emacs to intercept the occasional command line before it is passed to a traditional shell, in other words, I will probably feel like I have all the control I need over my relationship with the somewhat unruly beast that is the traditional Unix shell and will not feel the need or desire for the code I wrote to actually stop sending command lines to it. I'm not one for making larger changes to my software environment than necessary.
If I could figure out which of the over 13,600 lines of code that is eshell.el is responsible for parsing command lines, I'd use that, but my experience with eshell.el leads me to believe that it will probably be quicker and easier for me just to write code from scratch.
The main advantage is that it's a smaller dependency. If you don't need the functionality, there's a bigger benefit to being portable. For example, running scripts on a router with embedded linux.
yet not full of bad habits like the "defensive bash programming" blog post that was on the front page earlier today.
It's the continuation of http://mywiki.wooledge.org/BashGuide
Omg a whole 3 months ago???!! This repo is soooooooo stale. Bash is constantly changing every day right? I don't want to learn from an antiquated resource!
As opposed to improper bash? Get off your high horse.
There is very much a bad way of doing bash. When I first started doing bash scripts, most of them looked like this:
cat file | grep string
cat file | wc
cat file | while read line
Multiple problems there. Then there was my initial attempts at finding files in a directory:
for i in `ls *`
This is when I learned about globbing.
There is enough variance in how things can be done in Bash with varying degrees of effectiveness that Google even has a Shell Style Guide:
https://google.github.io/styleguide/shell.xml
I'm not so sure it is so useless. Consider when you want to "grep something access.log". Then you want to look at yesterday's so it's "grep something access.log.1". Then you want to look at the log before that, but it's compressed. "gunzip -c access.log.2.gz | grep something". (Ignore zgrep for the moment. Let's say I want to use grep's -A and -B options to show context of the lines that it finds.) To switch between the compressed and uncompressed files, you have to move the filename to a completely different part of the command, which requires more re-typing. You would have been better off starting with so-called "useless" cat. Separate out the concern of "putting a stream into the pipe" from "doing something with the stream". gunzip -c is just one command that might produce output.
https://google.github.io/styleguide/shell.xml
my fx doesn't render the site properly it's just one huge blob of continuous text. impossible to read. why would they do it as an *.xml?
It's entirely possible to learn improper bash when cobbling stuff together over the years. I've certainly done a few things that I've since found much better alternatives for.
Another good resource is http://wiki.bash-hackers.org/.