
Testing shell commands from Python - pplonski86
https://blog.esciencecenter.nl/testing-shell-commands-from-python-2a2ec87ebf71
======
theamk
Note that this recommends "sh" package, which made an _interesting_ decision
to run processes in tty by default. This goes against unix conventions that
batch processes should not be using tty, and breaks a number of utilities like
git and systemctl. This is so bad that even their example from the front page
is broken(!):

    
    
        $ python3
        >>> import sh
        >>> sh.git.show('HEAD').count('\n')
        49
        # Now resize your terminal window to be 10 lines tall
        >>> sh.git.show('HEAD').count('\n')
        9
    

Yep, it truncated "git show" output to the height of your terminal window. Do
you want _this_ in your code?

While you can append special flag for this, it is very easy to forget to do
so, and the documentation does not help either. Their FAQ states that they are
aware of the issue and won't fix it [0].

So my advice is: stay away from "sh" package, as it makes it very simple to
generate unreliable code. Instead, use built-in tools which are pretty nice
and just a bit more verbose:

    
    
        >>> import subprocess
        >>> subprocess.check_output('git show HEAD', shell=True).decode().count('\n')
        49
    
    

[0] [https://amoffat.github.io/sh/sections/faq.html#why-is-tty-
ou...](https://amoffat.github.io/sh/sections/faq.html#why-is-tty-out-true-the-
default)

~~~
anarcat
it's not "just a bit more verbose", subprocess is much longer and complex
module than sh! i constantly need to go back to the manual (is it check_output
or call? or maybe i need to create a pipe for that one? never remember).

sh provides clear semantics for certain use cases - that it doesn't match your
expectations on how a program shold behave is understandable, but there are
knobs to change that behavior, so I don't know if that is a strong enough
argument to completely "stay away from" it altogether.

i know i've been happy it's there and used it in a few projects with good
success, particularly projects that need exactly that: replace a shell and
have the terminal passed down correctly.

~~~
theamk
It is obviously each person's preference, but I especially hate non-
reproducible, context dependent bugs. I think having a reproducible,
consistent runs is key to having nice software that everyone enjoys
developing.

In this light, I find the tty decision particularly bad, as it is very prone
to introducing subtle changes -- for example, do you need the special flag for
"git log"? what about "git for-each-ref"? what about "ls"? what about "gcc"?

For this reason, I find it way easier to just ban "sh" outright altogether.
Our organization policy say "do not use sh module". When new member joins, and
they way to shell stuff out, they will find out that they need to use
subprocess module, grumble a bit, search the web, but then produce a working,
if verbose, code.

An alternative would be to say: You can use sh, but never use it as sh's own
front page recommends. If you find an article on the web which mentions sh, be
aware that you cannot just copy-paste code from it, as it can be broken
depending on which command you run. You need to audit which command you run to
see if it cares about tty. Currently, you have to include the flag for many
(but not all) git subcommands, "ls", "apt", ... but that list can change at
anytime.

> i know i've been happy it's there and used it in a few projects with good
> success, particularly projects that need exactly that: replace a shell and
> have the terminal passed down correctly.

You have been lucky so far, but every time you use "sh" you are walking on
landmines. I can give you a ton of plausible examples where sh breaks. Here is
a shell replacement gone wrong:

    
    
        $ git show HEAD | wc
             49     273    2202
        $ python3 -c 'import sh; print(sh.wc(sh.git.show("HEAD")))'
             19      97     812
    
    

edit: have you seen "subprocess.run" in python 3.5 [0]? It gives nice all-in-
one API, with readable options like `check=True' or `capture_stdout=True`

[0]
[https://docs.python.org/3/library/subprocess.html#subprocess...](https://docs.python.org/3/library/subprocess.html#subprocess.run)

------
jaimebuelta
A good framework to test command line is cram

[https://bitheap.org/cram/](https://bitheap.org/cram/)

You can define in a small test file what is the shell code to execute and what
is the expected result, including error codes

I like sh very much in Python to generate scripts, but in terms of pure
testing shell commands, I think cram is brilliant.

Here is an example on tests for a shell utility I created:
[https://github.com/jaimebuelta/ffind/tree/master/tests](https://github.com/jaimebuelta/ffind/tree/master/tests)

~~~
glandium
To put credit where credit is due, as far as I know, cram was derived from the
test framework used for Mercurial unit tests.

~~~
black-tea
It says that in literally the first line on the page.

------
laktak
#!plumbum

[https://plumbum.readthedocs.io/en/latest/](https://plumbum.readthedocs.io/en/latest/)

------
bow_
Some time ago I wrote a pytest plugin[1] that helps me test my long running
shell scripts.

The idea was to treat the shell process as a fixture, and conditions before
and after the test (input file checksums, output files, generated
stdout/stderr, etc.) would be checked. It was mostly for data analysis
pipelines (hence the name).

I don't use it as much these days as my days of writing long running pipelines
are behind me. But for that time, it did its job pretty ok. It was also a nice
way to peek into pytest's internals.

[1] [https://github.com/bow/pytest-pipeline](https://github.com/bow/pytest-
pipeline)

~~~
SonicSoul
Interesting! If you were doing this today, would you still use your tool or
are there other better ways to test?

~~~
bow_
It depends.

Another commenter mentioned Cram, which I would look into if checking
stdout/stderr (mostly) is enough for my use case.

If I am already using a specific framework to write the pipelines, I would
also look into whether it already provides ways for testing.

I am not aware of other tools that test for pre-/post- conditions, so I would
probably start with mine and see if need to tweak things along the way.

One thing I am not too happy about the plugin is that sometimes setting up the
process as a fixture can be quite verbose. It feels like this part can be
improved with some more thought.

------
ianbicking
For a simple helper for testing scripts, I might also suggest
[https://scripttest.readthedocs.io/en/latest/](https://scripttest.readthedocs.io/en/latest/)

------
kqr
It's also possible to do this using shell scripts, with e.g. BATS:
[https://github.com/bats-core/bats-core](https://github.com/bats-core/bats-
core)

------
im_down_w_otp
We did a fair bit of this kind of thing at my last company.

More specifically, we were using the PBT-like `hypothesis` framework in Python
for the purposes of more exhaustively exercising the CLI tools we were
creating.

It worked pretty well.

------
ClutchBand
Maybe I am off here, but I do something a bit more simple

``` def issue_command(self, command):

    
    
            if not str(command):
                
                response = None
                
                logger.error('Command argument must be string')
            
            else:
                
                response = os.system(command)
                
                if response != 0:
                    
                   logger.error('Command returned non-zero status: {}'.format(command))
                    
                   exit(1)
    
            return response

```

------
dkural
So cool to see CWL used in the wild.

