I may be a bad person to ask. I like history a lot. A lot of this stuff was originally picked up by secretaries and non-computer people working on manuals for Bell Labs. If you understand it, it will save you time. It will also make you use and understand Linux/Unix better, which will bleed into any other programming you do (assuming you write other code). The skill also comes up a lot in strange places like Dockerfiles, CI Pipelines, and a lot of that kind of surrounding infrastructure for applications.
The video referenced in the article is actually pretty interesting, as is the book "The Unix Programming Environment". Both will refer to outdated technology like real terminals, but I think they help understand the intent of the systems we still use today. Any of the Bell Labs books are great for understanding Unix.
Also do a minimal install of Linux or FreeBSD and read all of the man pages for commands in /bin, /usr/bin. Read the docs for the bash builtins.
But writing real scripts is probably the only way to learn. Next time you need to munge data, try the shell. When you read about a new command, like comm, try to use it to do something.
Also force yourself into good practices:
- revision control everything
- provide -h output for every script you write (getopts is a good thing to be in the habit of using)
- use shellcheck to beat bad habits out of you
- try to do things in parallel when you can
- assume everything you make is going to run at a much larger scale than you intend it to (more inputs, more servers, more everything)
- after something works, see where you can reduce the number of commands/processes you run
- write things like a moron is going to use them who will accidentally put spaces or hyphens or whatever in the worst place possible, not provide required arguments, and copy/paste broken things out of word
- don't let scripts leave trash around, even when aborted: learn trap & signals
The two things you should probably understand the most, but on which I'm the least help on are sed and awk. Think of sed as a way to have an automated text editor. Awk is about processing structured data (columns and fields). I learned these through trial and error and a lot of staring at examples. Understanding regular expressions is key to both and is, in general, an invaluable skill.
Oh and if you are a vi(m) user, remember that the shell is always there to help you and is just a ! away. Things like `:1,17!sort -nr -k2` (sort lines 1-17, reverse numerical order on the second field) can save a ton of time. And even putting some shell in the middle of a file and running !!sh is super-handy to replace the current line that has some commands with the output of those commands.
You’ve left some good tips. sed, awk, and grep. getopts. Read the man pages. Try to automate even simple series of commands.
Don’t be shy about running a whole pipeline of commands (in an automation script) just to isolate one field within one line of one file and add one to it, and store the result in a shell variable.
The video referenced in the article is actually pretty interesting, as is the book "The Unix Programming Environment". Both will refer to outdated technology like real terminals, but I think they help understand the intent of the systems we still use today. Any of the Bell Labs books are great for understanding Unix.
Also do a minimal install of Linux or FreeBSD and read all of the man pages for commands in /bin, /usr/bin. Read the docs for the bash builtins.
The old Useless Use of Cat is great: http://porkmail.org/era/unix/award.html
But writing real scripts is probably the only way to learn. Next time you need to munge data, try the shell. When you read about a new command, like comm, try to use it to do something.
Also force yourself into good practices:
- revision control everything
- provide -h output for every script you write (getopts is a good thing to be in the habit of using)
- use shellcheck to beat bad habits out of you
- try to do things in parallel when you can
- assume everything you make is going to run at a much larger scale than you intend it to (more inputs, more servers, more everything)
- after something works, see where you can reduce the number of commands/processes you run
- write things like a moron is going to use them who will accidentally put spaces or hyphens or whatever in the worst place possible, not provide required arguments, and copy/paste broken things out of word
- don't let scripts leave trash around, even when aborted: learn trap & signals
The two things you should probably understand the most, but on which I'm the least help on are sed and awk. Think of sed as a way to have an automated text editor. Awk is about processing structured data (columns and fields). I learned these through trial and error and a lot of staring at examples. Understanding regular expressions is key to both and is, in general, an invaluable skill.
Oh and if you are a vi(m) user, remember that the shell is always there to help you and is just a ! away. Things like `:1,17!sort -nr -k2` (sort lines 1-17, reverse numerical order on the second field) can save a ton of time. And even putting some shell in the middle of a file and running !!sh is super-handy to replace the current line that has some commands with the output of those commands.