
Do-nothing scripting: the key to gradual automation - thameera
https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-the-key-to-gradual-automation/
======
dusted
We use this at my place, though in bash, and we call it executable
documentation. Stuff like:

NAME="$1"

# Create directory for the code

mkdir "$NAME"

# Checkout the source

cd $NAME && git something

# and so on

Sometimes with an echo "You need to read and understand this before just using
it"; exit 1 thrown in somewhere for good measure.

These files will have no logic, and make very little use of variables. They
will also have a .txt filename to ensure that people understand, that first of
all, this is a set of instructions for how to do something, which just so
happens to be a valid bash script.

~~~
qznc
You could prefix the script with `#!/bin/bash -vn` and make it executable. It
will just print itself.

~~~
thunderbong
Thanks. Didn't know about this.

------
victormours
This idea is great. You could even argue that these scripts actually do
something: they allow storing and updating the _state_ of a process. By having
these scripts run in a shell rather than in your head, the current state is
stored in a machine, not in an unreliable human brain.

~~~
empath75
also they allow you to gradually script stuff as you figure it out.

~~~
rakoo
This should be read in conjunction with this other great article about how to
gradually automate your process:
[https://queue.acm.org/detail.cfm?id=3197520](https://queue.acm.org/detail.cfm?id=3197520)

~~~
mturmon
This is a great article, providing even more context for why this stepwise
automation is smart.

------
heinrichhartman
Very nice idea, Dan! This reminds me of the power of checklists that has been
discussed here a few times: Simple but highly effective.

It allows for a pretty natural transition:

\- (Markdown) Manual

\- Do-nothing-script (printing out the manual in steps)

\- Partial automation

\- Full automation

~~~
praptak
The reference article for the power of checklists:
[https://www.newyorker.com/magazine/2007/12/10/the-
checklist](https://www.newyorker.com/magazine/2007/12/10/the-checklist)

Another article, with HN discussion:
[https://news.ycombinator.com/item?id=20293246](https://news.ycombinator.com/item?id=20293246)

------
andrewguenther
I have a script in our automation setup called "be-the-change" which takes
either markdown or a wiki url as a parameter. It will output the markdown or
wiki content, wait for the user to confirm they've completed the steps and
then print:

Doing manual steps sucks, wouldn't it be great if this was automated for the
next person? Be the change you want to see in the world.

------
swiley
Is anyone else bothered by how they were handling ssh keys in the example?

The whole point of public keys is that the private key never has to leave the
box it’s generated on. Once you go sharing them you might as well just use a
random passphrase.

~~~
deg4uss3r
Yes, very much so. In addition stop using RSA[0] and use ed25519 with `ssh-
keygen -t ed25519`

[0]: [https://blog.trailofbits.com/2019/07/08/fuck-
rsa/](https://blog.trailofbits.com/2019/07/08/fuck-rsa/)

~~~
viraptor
Unless you're using a hardware token via gpg-agent. In that case RSA is the
only way most of the time.

~~~
xemdetia
Yeah hopefully when FIPS 140-3 finishes we'll start seeing hardware tokens
that can do elliptic curves.

------
darkerside
I love things like this. Mindbendingly simple ideas that rely purely on
overturning preconceived notions about what things are "worth" a certain
amount of effort. There's a tautological, cart-before-horse paradox in many
cases.

------
beat
I'd go farther... you can have steps that _literally_ "do nothing". They can
be comments, or simple echo statements. They're placeholders to remember to
write the code for that step later. This is particularly important when you
need to deal with integration that isn't necessarily easy - handling auth
credential safely, making sure it works in your pipeline, etc.

But all this gets back to something I used to say a lot, and the more refined
version I say now. I used to say "Have humans do what humans do well; have
computers do what computers do well". The key part of that was the idea that
computers can't do what humans do. Now, I say "Anything a computer can do
well, a human cannot do well". Which includes those pesky twenty step manual
processes.

~~~
dkarp
But these steps already do nothing except say echo what should be done. How
does that differ from your "literally do nothing" steps?

~~~
beat
In a truly minimal system, there isn't even a prompt. A code comment is about
as minimal as I know how to get. That's just a placeholder to remember to
automate a step.

When automating a complex manual process, the first thing I do is write
comments describing the sequence, and any parameters needed. Parameters in
particular can be enlightening, because when manual processes grow
organically, naming consistency becomes a source of suffering - humans can
just sort of figure out the right thing to type, computers can't intuit these
things and they must be made explicit. It's worse when it _can 't_ be explicit
without updating the process or creating some sort of data store for mapping.

This is why naming things is one of the two hard problems in computer science.

~~~
cyphar
Printing means you can run the script as-is, and any bits which aren't done
automatically are prompted for. Just putting comments describing the steps
would be like writing a README except in a more esoteric format and would
defeat the primary benefit of the approach outlined in TFA.

~~~
beat
It kind of boggles me that a real analysis of the problems with naming
conventions in manual process got massively downvoted. But no one ever got
rich counting on the analytical sophistication of HN.

I'm not saying the original approach is bad. I'm saying it's not the simplest
thing that can possibly work. I'm talking about how to move to automation, not
how to reduce errors on the manual process. Both are needed.

------
nerdponx
I get it, but this script looks a lot like a checklist to me. Seems like a
checklist is a good place to start, no?

If you want more "interest" during the process, make it a point to refine the
documentation for your process (clarify a step here, split one command into
two steps). Then you have a very clear spec by the time the process needs to
be automated.

~~~
wtetzner
I think the value of using a script as your checklist is that the script can
handle some of the book-keeping for you, including making it harder to lose
track of where you are.

~~~
maaaats
It also documents the process, which is often lacking for these one-offs
tasks.

------
sourceless
This seems like a surprisingly powerful and should-be-obvious kind of idea...
I think I'll start making use of it!

------
Ididntdothis
We do a similar thing for tests that are hard to automate. Instead of having
the test protocol in a document it’s in a script that prompts the user to do
things. The observations are then recorded and stored.

------
phoe-krk
They are _do-nothing scripts_ but it isn't equivalent to say that _the scripts
do nothing_. That is a very important distinction to me.

These scripts organize tasks into flows in a way that makes it possible to
split work by logical checkpoints. The tasks between these checkpoints are
dependent on one another in the overall flow, but have independent
implementations that can be delegated to software or done by humans.

------
kostarelo
It seems to me that all you want is to document a manual process. Having them
documented in dump scripts is no different than having them documented as a
guide in your knowledge base .e.g in Confluence or as product requirements.

I can see the usage in an isolated team of a few people that will write these
scripts for their own benefit.

I am opposed in general with having scarce information around. Better have
everything a single place, whether that's a guide on how to provision a user
or is how to add a new endpoint to your API or write a new end to end test.

~~~
TuringTest
There is one advantage of this script over an equivalent document: it provides
the instructions in the same context that it's being used (the command shell).

Unless we have script shells that can universally launch a web document side-
by-side from the shell prompt where it needs to be used, having it being
displayed from the shell itself step by step will reduce the mental load
required to follow the instructions. Also, the benefit of having an already
semi-automated process in order to build a script in the future is not
negligible.

~~~
pojzon
I think dual screen solution provides exactly that. Working only on a
corporate laptop is very inefficient in this regard.

------
marshmellman
An additional benefit of this approach is that your checklists now get code
reviews, git hooks, history, etc. for free.

I mention this because I worked at a company where checklists were typically
simply added to a wiki and required separate processes for things like peer
review.

------
gtf21
This is a really great idea, and appealing in its simplicity -- I think we
might try to implement this in my company for some of our tasks which are
really very manual, but for which automation is a daunting task.

------
orestis
I like the checklist nature of this. I approached this problem from another
side, which involved a very powerful REPL, but lacked the structure this
shows: [https://orestis.gr/clojure-unlocking-incremental-
automation/](https://orestis.gr/clojure-unlocking-incremental-automation/)

------
peterwwillis
That's actually a runbook. But it's a pretty good idea to make your runbook
into code! This is a nice, gradual way to begin automating. You automate one
step at a time, easiest tasks first, until eventually all of it is automated.
The more your steps are reusable, the easier it is to automate other tasks.

An example is approving new users to access a service. Say your list of users
is a YAML file. Someone edits the file, fires off a PR, waits for someone to
approve it, merges it, and applies the config. You might think it's so easy to
do by hand, automating it would be a waste. But if you actually add up the
time it takes for humans to do those steps, plus the lead-time from creation
to completion, plus the occasional typo, wrong username, or Norway bug, it can
take from hours to days to actually complete the task. By automating steps one
at a time, you gradually move toward the entire process being a single script
that takes a username, and completes the task in about 30 seconds. This is a
small win, but added up over all your operations, reduces hundreds of hours of
toil _and_ reduces human error. If every engineer automates one step of one
runbook per week, over a year you'll have hundreds of reusable automation
steps.

To the OP, I'd add the following: 1) use the least-complex way to accomplish
the task. If you don't _need_ to write Python, don't. Bourne shell is your
friend. Re-use existing tools and automated features. 2) Add documentation to
the code that can be auto-generated, so you can use the script as long-lived
static documentation. Documenting somewhere else will lead to stale docs. 3)
Run all your runbooks from a Docker container. 4) Plan to run these runbooks-
in-containers from a CI/CD system using paramaterized builds. 5) Organize your
reusable automation steps by technology, but have product-specific runbooks
that call the automated features. 6) In a big org, a monorepo of automation
features is an efficient way to expose automated features to other teams and
ramp up on shared automation quicker.

------
m463
I do the same sorts of things for my scripts.

I start with my "template" script and fill it out.

totally over-engineered for most tasks, but then as the tasks mature they have
a firm foundation.

very simplified example:

    
    
        #!/usr/bin/python
        import sys,os,argparse
    
        parser = argparse.ArgumentParser(argument_default=None)
        parser.add_argument('-d', '--debug',     action='store_true',  help='debug flag')
        parser.add_argument('files', nargs='*',  default=[],           help='files to process')
        arg = parser.parse_args()
    
        def die(errmsg,rc=1):
            print(errmsg, file=sys.stderr)
            sys.exit(rc)
        
        if not arg.files:
            die('filename required')

------
fredley
I can't help looking at this and working out how to build a tool that can
generate a Do Nothing script straight from your histfile, making the business
of archiving the commands you just executed as easy as possible too...

~~~
jasonjayr
If you're about to do a sequence of commands into a terminal, you can create a
transcript with the 'script $filename' command ('script' command is in the
util-linux package', according to the man page), and it will capture
input/output into the transcript. This would probably make this task easier,
since you could see the output data, and how the output data is transformed
into arguments into the next command ...

~~~
vfinn
Came to say this.
[https://www.youtube.com/watch?v=tweyWNr6X18](https://www.youtube.com/watch?v=tweyWNr6X18)

------
nmca
This is a checklist. Checklists are incredible, and I quite like this version.

------
vagab0nd
I did something very similar just recently. I have a calendar event to backup
all personal data every 3 months, based on a checklist. I'd always wanted to
automate it but every time I tried, it just seemed such a daunting task that I
never ended up doing it.

Last time, I figured I'd start going at it gradually. So what I did was I
turned the checklist into a bash script, telling me what to do at each step. I
also implemented the ones that are easy to automate. The plan is to knock out
two or three items each time I do the backup, until it's fully automated.

------
dragonwriter
That's not “do-nothing” scripting, it's orchestration scripting for a process
where the steps are executed by a human.

It's basically the minimal version of a standalone workflow automation; it's
got crappy state storage and is really only good for a single human execution
unit per script (or perhaps multiple swapped sequentially rather than in
parallel), but it's an instance of a fairly well-known class of do-something
software.

------
nlawalker
Suggested upgrade: Have such scripts ask for a unique name/"instance
identifier" when run so they can log to simple text files in a well-known
directory and potentially pick up where they left off. For when you invariably
use these for long-running tasks and have 7 of them in-flight, and your power
goes out or Windows forces a reboot.

------
mbar84
Interesting idea! I'll wrote some boilerplate that I think is a bit nicer:

[https://gist.github.com/mbarkhau/60fc4bbe505914369ebd2fec1ab...](https://gist.github.com/mbarkhau/60fc4bbe505914369ebd2fec1ab6d21b)

------
2T1Qka0rEiPr
Nice idea! I'm actually in the process of automating a task bit by bit.
Unfortunately it's with the intention of moving it into a CI pipeline, so the
"success" there is rather binary, but I will most definitely adopt this
process in future.

------
chousuke
In situations where using full-blown configuration management has too much
overhead I often do something like this by just copying the commands I run
into a script to begin with. It allows me to easily generalize a sequence of
actions into a do-nothing template and doubles as documentation.

I'm very much against documenting concrete steps in a wiki if it is something
that can be scripted. Such documentation becomes something that you must
maintain, and faulty documentation is worse than no documentation.

I think that wiki documentation should be used to summarize workflows and
provide context, but if you're writing step-by-step instructions, often you
can just write a script with exactly the same amount of effort.

------
AstroJetson
Why do I think that writing a script in Expect
([https://en.wikipedia.org/wiki/Expect](https://en.wikipedia.org/wiki/Expect))
would take as long as doing this and then you would be done?

~~~
moonbug
..because you've never tried using expect in anger?

~~~
AstroJetson
Nope, I like Expect, it's my friend and I'm able to make scripts with it. So,
no anger.

------
dradtke
Anyone interested in this idea should check out
[https://github.com/braintree/runbook](https://github.com/braintree/runbook)

------
pgt
This is a good idea. I recently wrote down some ideas about "How to Grow a
Program," in a symbolic programming environment:
[https://github.com/theronic/hoist/](https://github.com/theronic/hoist/)

I call it Hoist - from "hoisting up" concrete, hard-coded values into abstract
symbols for reuse. It's complete vaporware at this point, just a place to
collect my thoughts for a future essay.

~~~
teddyh
See also: In TDD, there are defined steps you go through as you gradually
refine your code to actually do more and more things; from returning nothing,
to returning one value, to returning a list, etc.

------
grecht
Great idea! From personal experience I can say that there are some things ops
teams never get around to automate, either because there is always something
more important to do, or because it would be too expensive to have a developer
spend time automating them. This is a simple, quick solution that solves the
problem of accidentally skipping a subtask, thus screwing something up and
feeling stupid for doing so.

------
abathur
This feels like a useful concept.

I stumbled into doing roughly the same in my macOS system bootstrap script. I
found it from the other side, though--usually only after failing to find a
good way to automate a step, or realizing automation would be vastly more
complex/fragile than just prompting myself to do something manually at the
right time.

Probably a more useful concept when it's conscious from the getgo. :)

------
manas666
if you are working at a small shop or startup - then whatever makes you happy,
even these classes are OK.

if you are working at enterprise - then these processes gotta be implemented
according to ITIL in a specialized IT Service Management system - like
ServiceNow, which allows building process flows like directed acyclical graphs
and automation of necessary steps

~~~
oblio
I'm not sure if this is ironic. I've been numbed by too many years of internet
usage, I guess :-)

------
timwis
What is the value of this over a checklist?

~~~
crispyambulance
I would argue that it _is_ a checklist.

The difference is that it's a more gradual entry to automate the task.

One could, for example, replace any item with a script/program invocation,
while leaving the others untouched. It's an intelligent, piecemeal step
towards automation.

I would also add that in really critical tasks, one could automate _either_
the item, or it's verification. Sometimes it's easier to check that something
is correct than to do it-- or vice-versa.

------
persephonee
raw_input is python 2.x. Dont use it.

------
js8
Seems basically to be a checklist.

However, I don't think it's a good idea. Checklists are not bad, but they are
to be executed by humans.

I think trying to automate an existing human process directly is a mistake.
Human processes often have features that only humans can do, such as pattern
recognition or adapting to small difference. It's often easier to create a
computerized (automated) process from scratch than to try to account for all
the edge cases that humans might have to deal with (and do without problems).

(The opposite is also true, processes that computers have no issues with can
make trouble for humans. For example, humans have imperfect memory. So if the
process asks humans to keep track something for extended periods, it can be
difficult.)

~~~
jpitz
>Checklists are not bad, but they are to be executed by humans.

I disagree. People employ checklists precisely in the situations where it is
most critical that every step be executed, in order, as written. The airline
industry flies on checklists.

>I think trying to automate an existing human process directly is a mistake.
Human processes often have features that only humans can do, such as pattern
recognition or adapting to small difference. It's often easier to create a
computerized (automated) process from scratch than to try to account for all
the edge cases that humans might have to deal with (and do without problems).

Now, you're letting perfect prevent you from ever attempting the good. TFA is
explicitly about building a checklist that can morph the automatable steps
into automated ones.

~~~
js8
> People employ checklists precisely in the situations where it is most
> critical that every step be executed

I am not arguing against checklists per se. At work, I work on (a rather old)
software product which also largely runs on checklists. (Various installation
and maintenance procedures are described by a checklist.)

> Now, you're letting perfect prevent you from ever attempting the good.

No. I just, based on my experience with checklists, argue that they are often
poor starting point. Often they rely on abilities of humans to execute them
properly.

For example, the checklist might read "edit this text file and add this and
this line after line that is like this". This is difficult to automate
correctly - the proper way might be perhaps to generate or keep a correct copy
of the text file so automation can pick it up.

Humans are good at adapting checklists for the purpose at hand, computers
aren't. That makes (human-oriented) checklists somewhat difficult to automate.

------
punnerud
When I can (often):

1\. Write SQL to automate

2\. Put SQL with output in Metabase

3\. Have it manually checked alongside day-to-day operation

4\. Iterate 1-3 (you often will get a lot of feedback)

5\. Replace the manual work, now that the SQL it checked on a lot of cases in
production

------
banku_brougham
Implementation question:

Whats a good way to generalize such a python script so that windows and nix
users can run it?

~~~
alexis_read
The best way is probably a bootstrap script ie. ps1 or sh which supplies the
platform-specific parameters, and can pip install any script dependencies. So,
3-file solution. Additonally, this can be generalised to a CI platform as
python can subprocess-run anything there's not a native lib for. Lastly, you
could do autodoc via the script for a complete solution.

------
crispyporkbites
Are there any good slack plugins out there that can do this kind of workflow/
state management?

~~~
NateEag
I don't understand why you would want to introduce Slack to this.

What benefits would that give?

~~~
crispyporkbites
Slack is like the shared OS we never had

~~~
NateEag
I'm sorry, I still don't understand.

The only interpretation of that I'm coming up with is that each step of the
process would be kicked off in Slack.

Wouldn't that mean people could easily stomp on each other's toes? Wouldn't
there also be tons of needless noise as each of us does our workflows?

I use Slack daily, but we don't have any automation hooked up in it. I realize
some teams drive deploys, alerts, and other stuff through it. That makes sense
to me - this does not.

Am I missing something?

------
jsilence
Combined with the concept of literate programming this would also help capture
the "why"s of the slog, resulting in inline documentation. Jupyter and org-
mode for the win!

~~~
ben509
Running such a checklist in Jupyter seems like a very good approach.

The big advantage is someone else can pick up from where you started.

And reporting is built in as you can save the notebook and review it later.

The only thing it lacks is control flow. A good operations script has exit
points: if an operation fails, you often want to rollback and abort. Though
even there, you can save state in a variable and have a check.

------
lazyant
Checklists as Code

------
X6S1x6Okd1st
It's a checklist

------
probablyfiction
This just seems like checklists with extra steps.

~~~
rusticpenn
but this can be gradually automated with less effort than a checklist.

------
xtat
I do this!

------
klmr
I know this isn’t the point of the article but the code shown there is the
archetypal case where writing classes is an anti-pattern, as argued in “Stop
Writing Classes” [1]. All the classes in that code can — should! — be
functions. There’s no downside, and it’s less code (and otherwise identical).

[1]
[https://news.ycombinator.com/item?id=3717715](https://news.ycombinator.com/item?id=3717715)

~~~
naringas
maybe he was a Java programmer?

all joking aside, the reason he's doing classes seems to be an intention
towards making a FooStep (and/or a BarStep) which could then maybe handle
generic Foo stuff (authentication comes to mind) in the future?

~~~
epr
"maybe"..."in the future" is the root of much object oriented evil

~~~
shareIdeas
I don't think there is a correct answer, but I like that a developer has
foresight.

~~~
mattnewton
The problem is you get more information in the future, but all this
boilerplate is has been written before that information so the first instinct
/ seemingly least-resistance thing to do is to fit the new information into
the old boilerplate / hierarchy.

Lots of people have OO scar tissue from getting weird impedance mismatches on
newish OO systems designed this way.

