
Lessons learned from writing ShellCheck - r4um
https://www.vidarholen.net/contents/blog/?p=859
======
regnerba
> discovering late in the interview process that Apple has a blanket ban on
> all programming related hobby projects

O_o There must be more to it then that. What the hell does anyone do in their
spare time at home for fun then?

~~~
kwhitefoot
I suspect that that would be unenforceable in the EU, UK, and Scandinavia.

Is there a lawyer with experience in the field reading this?

~~~
lostmyoldone
Not a lawyer, but I _have_ previously contracted one to evaluate and discuss
something closely related, which are the Swedish laws about non-compete
clauses in employment or contractor contracts.

In summary. They are allowed to some extent, but if a court would consider a
non competition paragraph as being too broad, the paragraph becomes void in
its entirety, as if it was never entered into the contract at all.

For non management employees the exclusivity must be quite specific, and
unless the law has changed recently without my knowledge, and entire industry,
or a broadly specified skill would fail the test in any case I can think of.

Applying the most basic levels of those principles, I would assume that
attempting to limit a programmers ability to engage in programming in general
would lead to nothing but a complete dismissal of any claims related to the
non-compete clause if someone brought it to court.

Effectively, if you try to prevent someone to apply their trades and/or
talents to broadly, they could essentially moonlight for your worst
competition without much of a risk as long as they don't convey what is
clearly trade secrets as they are differently protected.

Suffice to say that companies with decent legal departments tend to write
specific non competition clauses especially outside of the realm of management
positions, as they get _no value at all_ from too broad non competition
clauses.

------
ice3
I want to thank the author for an amazing tool.

I've used ShellCheck to catch/fix numerous bugs on some mission critical
financial systems (that depend on shell scripts!).

Added it to pipelines and as a mandatory pre-commit hook in git, so all teams
started using it (whether they like it or not).

It even caught some places where a rm -rf / could happen because default
parameter substitutions were not set ( Same as the Steam bug where it would
remove your home directory, but on some big financial systems :) )

All in all - an amazing tool that saved me from a lot of grey hairs throughout
the years.

------
acqq
ShellCheck's unique error/warning codes and wiki descriptions for every code
with the explanations and example are the work of genius.

It seems so "obvious" in hindsight but not many other tools are so good.

Also, tangentially, this is also so perfect:

[https://github.com/koalaman/shellcheck/wiki/SC2154](https://github.com/koalaman/shellcheck/wiki/SC2154)

and, unless somebody proves otherwise, makes me believe that with ShellCheck
and using the described conventions a shell script could be better diagnosed
for mistyped variables than a Python script can (namely at "compile time" and
not during the run).

Vidar, your work significantly improved the life of people confronted with
shell scripts. Thanks!

~~~
mkesper
Hmm, this can be done with Python code too. And there are already tools doing
so.

~~~
acqq
> this can be done with Python code too

Can it? That's what I would like to know, how to know without executing the
code with the argument 4 that there's an error in this Python code:

    
    
      import sys;
    
      def f( x ):
        if x == 4:
          x += q
        print( x )
    
      f( int( sys.argv[1] ) )
    

That will "work" until 4 is passed at the runtime:

    
    
        $ ex-undefined.py 4
        Traceback (most recent call last):
          File "/tmp/ex-undefined.py", line 8, in <module>
    
            f( int( sys.argv[1] ) )
          File "/tmp/ex-undefined.py", line 5, in f
            x += q
        NameError: name 'q' is not defined
    

With this shell code

    
    
      #!/bin/sh
      f() {
        x=$1
        if [ "$x" = 4 ] ; then
          x=$((x+q))
        fi
        echo $x
      }
      f "$1"
    
    

and shellcheck I get:

    
    
       Line 5:
          x=$((x+q))
                 ^-- SC2154: q is referenced but not assigned.
    

even if I have never executed the code.

~~~
Ded7xSEoPKYNsDd
The standard python linters can do it:

    
    
        $ flake8 foo.py
        foo.py:6:14: F821 undefined name 'q'
    
        $ mypy --check-untyped-defs foo.py
        foo.py:6: error: Name 'q' is not defined
    

(They also have some other complaints about your code that you'd either have
to disable or adjust to.)

~~~
acqq
My note: invoking the first on the given example produces 17 lines of
different complaints, most totally irrelevant to the validity of the code (of
course it complains about the "style" \-- it was written as a "Tool For Style
Guide Enforcement"). Invoking the second without the magical switch --check-
untyped-defs produces:

    
    
        Success: no issues found in 1 source file
    

and additionally produced a .mypy_cache folder of 2 MB at the place where the
script was.

So the style of "unreasonable" defaults of Python alone (not detecting the
error, speaking from the point of view of a user of Perl) propagates to the
"unreasonable" defaults of the checkers.

Still thanks Ded7xSEoPKYNsDd, I really wasn't aware of these! Yet, even if I
haven't formally specified that, I was looking for the way to do it with the
Python as the language and its default interpreter alone, as for Perl nothing
additional has to be installed:

    
    
        use 5.010;
        use strict;
    
        sub f {
            my $x = shift;
            $x += $q if ( $x == 4 );
            say $x;
        }
    
        f( $ARGV[ 0 ] );
    

Gets me:

    
    
      Global symbol "$q" requires explicit package name (did you forget to declare "my $q"?) at ex-undefined.pl line 6.
      Execution of ex-undefined.pl aborted due to compilation errors.
    

I am aware that for shell I'd need an additional shellcheck but Python is
many, many times bigger than the shell binary alone (or even the sum of the
shell and shellcheck binaries), and actively changed, whereas the shell
semantics is standardized and effectively frozen in time, and from the shell
interpreter alone a very low startup overhead is expected.

------
endgame
> Converting them to a cleaner ReaderT led to a 10% total run time regression,
> so I had to revert it. It makes me wonder about the speed penalty of code I
> designed better to begin with.

I have heard from other Haskellers that you can sometimes get good performance
by hand-rolling an application monad, and then writing out the MonadFoo
instances so you get good ergonomics:

    
    
        -- Instead of this:
        newtype App a = App { runApp :: ReaderT AppEnv (ExceptT AppError IO) a }
          deriving (Functor, Applicative, Monad, MonadIO, MonadReader AppEnv, MonadError AppError)
        
        -- Try this:
        newtype App a = App { runApp :: AppEnv -> IO (Either AppError a) } deriving Functor
        
        instance Applicative App where ...
        instance Monad App where ...
        instance MonadIO App where ...
        instance MonadReader AppEnv App where ...
        instance MonadError AppError App where ...

~~~
arianvanp
Your two datatypes are representationally equal. So would be very surprised by
handrolling the datatype maling any difference. The instance however might
make more sense

~~~
endgame
Indeed. The folklore I heard is that you can get performance gains because
with the handrolled version, GHC is able to inline the instance dictionaries.

------
travmatt
I had commented in a previous discussion regarding Haskell that a common
criticism of Haskell is that it’s lazy evaluation can lead to problems
reasoning about runtime performance, a criticism which is repeated here. I
wonder if there any resources, whether language specific tooling, or general
theory, that could help developers struggling with this. I suppose that flame
graphs would be a useful tool to see where your time is being spent.

~~~
rotifer
Over the years I've collected various links on topics related to Haskell
performance. I'll spare you the whole list, but here are a few that may be of
some use.

Haskell Performance Patters - Johan Tibell (2012)
[http://johantibell.com/files/haskell-performance-
patterns.ht...](http://johantibell.com/files/haskell-performance-
patterns.html#\(1\))

Performance (Haskell Wiki)
[https://wiki.haskell.org/Performance](https://wiki.haskell.org/Performance)

Detecting Space Leaks - Neil Mitchell (2015)
[http://neilmitchell.blogspot.com/2015/09/detecting-space-
lea...](http://neilmitchell.blogspot.com/2015/09/detecting-space-leaks.html)

The Haskell performance checklist - Chris Done, et. al. (2017 - 2019)
[https://github.com/haskell-
perf/checklist/blob/master/README...](https://github.com/haskell-
perf/checklist/blob/master/README.md)

ThreadScope (Haskell Wiki)
[https://wiki.haskell.org/ThreadScope](https://wiki.haskell.org/ThreadScope)

Top tips and tools for optimising Haskell - Will Sewell (2015)
[https://making.pusher.com/top-tips-and-tools-for-
optimising-...](https://making.pusher.com/top-tips-and-tools-for-optimising-
haskell/)

~~~
srik
I’m in a Haskell learning phase and honestly can’t get enough recommended
study pointers. Share as many as you feel comfortable sharing, there’s always
some like me who’d benefit from the curation.

~~~
Eugeleo
Not the OC and (mostly) not related to Haskell performance, but the following
could be useful to you:

State of Haskell Ecosystem: [https://github.com/Gabriel439/post-
rfc/blob/master/sotu.md#c...](https://github.com/Gabriel439/post-
rfc/blob/master/sotu.md#compilers)

Guide to Haskell (2018): [https://lexi-lambda.github.io/blog/2018/02/10/an-
opinionated...](https://lexi-lambda.github.io/blog/2018/02/10/an-opinionated-
guide-to-haskell-in-2018/)

Haskell learner milestones:
[https://stackoverflow.com/a/1016986](https://stackoverflow.com/a/1016986)

GHC optimizations (rather advanced, but interesting):
[https://stackoverflow.com/questions/12653787/what-
optimizati...](https://stackoverflow.com/questions/12653787/what-
optimizations-can-ghc-be-expected-to-perform-reliably)

Compilation of various Haskell tutorials on more advanced topics:
[https://markkarpov.com/learn-haskell.html](https://markkarpov.com/learn-
haskell.html)

Despite the popular saying, there’s a wealth of knowledge about Haskell beyond
just academic papers, and in digestable form, but it’s scattered around
little-known blogs. I’d recommend you to check out r/haskell from time to
time, or subscribe to the Haskell newsletter - at least for me, those are the
places where I got to know most of Haskell blogs.

------
thundergolfer
> I’d also put serious consideration into how well the language runs on a JSVM

What's a JSVM? I'm not getting anything from Google that I think makes sense
in the context of post.

~~~
fulafel
JavaScript virtual machine, you know the thing that's the compilet target for
ClojureScript, TypeScript, ES5, asm.js etc.

