
Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool - robertelder
http://blog.robertelder.org/don-libes-expect-unix-automation-tool/
======
seiferteric
Expect is cool, I have used it in the past as well as the expect perl module,
but in my opinion is always a suboptimal way to automate things. It's always
better and more robust to have a proper interface to script things instead of
relying on matching brittle text/regex responses and timers and such. Of
course sometimes it's not possible, so expect comes in handy.

~~~
wyldfire
I 100% agree. If I can use paramiko/fabric or something similar, I prefer it
way more than expect.

If you have a slave device over RS232, expect is probably your only option.
But if that remote item is running sshd, you are way better off without
expect.

~~~
crististm
what's wrong with expect spawning ssh??

~~~
jstimpfle
I think expect is often used to have it enter username and password
"interactively". But it's easier and more robust to just use the public-key
auth facility that is built-in to ssh.

~~~
Karunamon
Not always possible. The problem is that ssh lives in the mythical land of
perfect security, where people never need to do unsafe things in order to get
their jobs done.

Hence keys silently not working without "correct" permissions, passwords not
passable on the CLI, inconsistent type-yes-to-override behavior, and other
user-hostile design choices.

I love openssh, but it's a real pain in the ass sometimes.

~~~
jstimpfle
I don't have any of these problems. Just don't use passwords non-
interactively, but use public key auth. Also, as any sane program, (open)SSH
does not ask you to type anything when running non-interactively in a script
(more precisely, when no terminal is present).

~~~
Karunamon
If you don't have any of those problems, you're very lucky. Again, ssh is
designed with the assumption that people never have a reason to do unsafe
things.

------
skywhopper
Expect is a last ditch tool that becomes less and less necessary as automation
libraries and remote APIs become more and more common. The examples given are
cute but hardly representative of what expect is really for.

The only times I've needed to use expect was to drive weird non-scriptable
telnet-based remote systems and to automate unpackaging of intentionally-
broken vendor-supplied self-expanding software archives that require some
level of interaction.

The point being if you want to create shortcuts to jump to specific locations
in a man page or drive outdated appliances or open user-hostile zips, expect
is the tool for you!

~~~
robertelder
Do you have any links to examples of what it is really for? I've had a hard
time finding code examples because the reasons listed at the bottom of the
article.

~~~
dekhn
Sure, let's say you want to remotely control a Cisco router with a serial
interface. expect makes it really easy to remotely connect to the serial
interface, wait for a prompt, send a command, and take a conditional action
that depends on the response.

~~~
robertelder
I was thinking more of code examples. Expect is capable of doing much more
than just the basic 'expect', then 'send' pattern.

For example, this article: 'Most programmers don't know what Expect can do for
them'

[http://wiki.tcl.tk/3913](http://wiki.tcl.tk/3913)

~~~
lilyball
The article mentioned it's used for testing gcc and LLVM. If you want an
example you can actually look at, it's also used for testing the fish-shell
(fish-shell has other tests too, but expect is specifically used to actually
test user-visible behavior, which makes expect the perfect tool for the task).
You can find those tests in [https://github.com/fish-shell/fish-
shell/tree/master/tests](https://github.com/fish-shell/fish-
shell/tree/master/tests). The files of the form "foo.expect" are the tests
(the "foo.expect.err" and "foo.expect.out" files are how it interacts with the
rest of the testing infrastructure), and the file interactive.expect.rc
provides a bunch of helper functions for the expect scripts, and in particular
a standard way of looking for prompts (because the shell will redraw the
prompt at various times, this function, coupled with the fish code in
interactive.config, lets the expect scripts tell when it's the old prompt
redrawn versus the new prompt after a command has been executed).

------
setq
Expect is great for persuading things, particularly poorly written Java
monoliths that use text interactive installers, to play nicely with ansible.
On more than one occasion it has been tasked with handling one.

You run autoexpect first to record what you want it to do on a dry run, then
tidy up the generated expect script, then plug it into ansible as a raw task.

Granted this shouldn't be a problem in 2016 but some vendors are idiots. But
thanks to expect my life is a little more comfortable :)

------
zwischenzug
I use pexpect as the basis of an automation framework I use a lot:

[https://github.com/ianmiell/shutit/blob/master/README.md](https://github.com/ianmiell/shutit/blob/master/README.md)

Recently I used it to migrate an etcd cluster within OpenShift reproducibly:

[https://medium.com/@zwischenzugs/migrating-an-openshift-
etcd...](https://medium.com/@zwischenzugs/migrating-an-openshift-etcd-
cluster-a7e43e861d61)

~~~
zwischenzug
PS I also wrote autopexpect, which is based on another underrated tool
(autoexpect)

[https://github.com/ianmiell/autopexpect](https://github.com/ianmiell/autopexpect)

[http://expect.sourceforge.net/example/autoexpect.man.html](http://expect.sourceforge.net/example/autoexpect.man.html)

~~~
SEJeff
Oh this is really nice! I've used pexpect (and pxssh) extensively along with
expect in a previous lifetime. I'd always meant to write an "autopexpect" or
"autopxssh" and never did.

Thanks!

~~~
zwischenzug
Glad to hear it! It's very raw, so please do contribute if you can.

------
RHSeeger
Long ago, I had to write a utility to backup the configurations settings from
all the network devices in a given network. At the time, the network consisted
of many, many different types of routers, switches, and the like.

For many of the devices, there was a way to fetch the config but it required
logging into the device (generally telnet) and navigating a menu to "send" the
config somewhere. Using expect to do that was a fantastic way to handle it.

There's also a number of cases where you want output from a command line
program "as if run by a user at a terminal". For some programs, they change
the output (ie, removing color codes, etc) if the output is not going to a
terminal. Expect does a fantastic job of pretending to be a terminal.

Also, it's Tcl, and I love Tcl. It's (possibly tied) at the top of my list for
languages. So much fun and more flexible than anything that doesn't support
lisp level macros.

~~~
EvanAnderson
In case somebody's interested in a tool that does the kind of thing you
describe re: network devices after reading your comment I thought it might be
helpful to include a link RANCID:
[http://www.shrubbery.net/rancid/](http://www.shrubbery.net/rancid/)

Silly backronym name aside, RANCID is a very handy tool and uses expect to do
a lot of the 'dirty work'.

------
teddyknox
Don Libes' daughter went to my high school, and I was working as the
"webmaster" for the high school newspaper's website at the time. He would
ruthlessly criticize via email the product our self-taught and earnest
2010-era PHP web development. I am not a fan of Don Libes. For those
interested, the website was/is:
[http://theblackandwhite.net/](http://theblackandwhite.net/)

~~~
clifanatic
Looking at the link I'm inclined to agree with Don.

~~~
GFK_of_xmaspast
While there are plenty of reasons to get salty towards high school kids,
'unfortunate web design' is probably not one of them.

------
hughw
A friend taught me about expect by giving me this expect script for logging
into our vpn. It automates login using Cisco Anyconnect on a Mac.

    
    
      pw=c3-f3-341a
      spawn /opt/cisco/anyconnect/bin/vpn connect vpn.example.com
      expect "Username:*" { send "myname\r" }
      expect "Password:*" { send "$pw\r" }
      interact

------
bxrxaxdx
OMG I have been burned by this so many times. Expect itself is a cool piece of
technology and works as well as anyone could hope. However, every time I've
tried to use it for anything the solution ends up being super flaky. I think
trying to automate text interfaces like this is just fundamentally unreliable.

~~~
chriswarbo
I think of it in the same way as screen scraping Web pages, or writing buffer-
heavy Emacs Lisp:

\- If there are alternatives, they'll usually be better

\- It'll still be around when there are no alternatives

\- Some tasks (e.g. "toggle foo mode") are inherently easier than others (e.g.
"press up five times and hope that's enough")

\- Some tasks need frequent maintenance and tweaking (e.g. as i/o formats
change, program features are added/rearranged, shell environments are tweaked,
etc.)

\- Some tasks work so well that they're taken for granted for a decade, and
it's only when you want to tweak some part that you remember how truly
horrifying the implementation is ;)

------
michaelsbradley
Don Libes' _Exploring Expect_ [1] is a devops classic (if there is such a
thing). It would be nice if O'Reilly would make the text (completely) freely
accessible[2], as they have done with some of their other older titles[3].

See also the Caius framework[4].

[1]
[http://shop.oreilly.com/product/9781565920903.do](http://shop.oreilly.com/product/9781565920903.do)

[2] [https://www.safaribooksonline.com/library/view/exploring-
exp...](https://www.safaribooksonline.com/library/view/exploring-
expect/9781565920903/)

[3] [http://www.oreilly.com/openbook/](http://www.oreilly.com/openbook/)

[4] [http://caiusproject.com/](http://caiusproject.com/)

------
wangchow
Tcl (upon which Expect is built) is under-appreciated too. It's a neat
language how strings behave as a sort of polymorphic data structure (can be
dicts, lists, etc). Under the hood they get optimized based on usage.

------
hydandata
Expect can be a lifesaver when you cannot use your usual go-to tools like
Ansible etc. especially with legacy setups where you cannot tweak stuff much.
I have successfully used it combined with tmux for automated multi-term like
functionality, deployment verification and other chores to save a lot of $ for
my company. I do try to keep my Expect scripts as simple as possible though,
just so I do not have to remember too much since I do not work with TCL
anywhere else.

------
sheraz
I remember using expect to make my US Robitics modem to dial-up and get
online, and to authenticate past RADIUS servers.

good to see this old util get some new light here.

------
kazinator
Things only 80's admins and hackers might remember: similar to Expect, some
terminal emulation + serial communication programs for PC's came with
languages for automation, resembling Expect. Expect might have been inspired
by that sort of thing.

I had experience coding in two of them for DOS PC's: ProComm Plus's ASPECT
language ("Pascal-ish" flavored), and a C-like language called Salt built into
Telix.

I mostly used the very popular Telix for my personal use, but a law firm for
whom I did some IT contract work used Procomm Plus for connecting to various
databases available over dial-up (e.g. local land title registry). I wrote a
bunch of ASPECT code to automate their scraping tasks: log in, look up this
and that, save in a file, that sort of thing.

[https://en.wikipedia.org/wiki/Telix](https://en.wikipedia.org/wiki/Telix)

[https://en.wikipedia.org/wiki/Procomm_Plus](https://en.wikipedia.org/wiki/Procomm_Plus)

------
danso
I'm going to try this out on my own because what I read in the comments makes
me _think_ I will find use for it, but if I had to be honest, I did so because
of the HN comments. The OP was clearly written and nicely formatted, but as
someone with moderate CLI/Bash experience (I do almost all of my data
processing via Bash programs and pipes), I didn't really understand what
`expect` does. Maybe I needed a simpler example, because I don't uss most of
the other parts that are in the example, such as `interact` and `send`. As a
Pythonist/Rubyist, I have the added baggage of associating `expect` with
unit/line testing.

Maybe expect is not for me, but given the title, I would've loved to see maybe
a couple paragraphs of _why_ `expect` is underappreciated...that is, what do
typical Unix programmers use instead, because they're ignorant of expect. Much
easier for me to understand the perspective of ignorance :)

~~~
qwertyuiop924
Expect is a TCL-based command-line scripting tool. It's designed for scripting
things that don't have a scripting interface, and weren't designed to be
scripted.

Essentially, expect scripts run a program, send it some input, look for some
regexes on stdout, and then send more input, which may change depending on
what the regex matches. And so on.

This is useful, say, if you have a package management tool that requires you
to explicitly confirm every update. Or if you want to automatically drive a
shell on a remote machine over SSH, and you can't use ssh -c for some reason.

You don't need it often. But when you do, it's very useful.

------
chriswarbo
> Many of them are actually dead links, or projects with less than 40 stars on
> GitHub.

Off topic, but how long has "stars on github" been an indicator of software
quality/longevity/usefulness/etc.?

How many "stars on github" does Expect have?

~~~
jquast
pexpect (python variant of the same) has 750 stars, so its popular.

people have just dropped tcl in favor of perl and later python and others ...
I don't personally like tcl anymore, but I programmed tcl/expect in the 90's
and enjoyed it a lot. I now co-maintain pexpect.

------
daenz
Shameless self promotion: if you want to automate subprocesses with python in
an intuitive way, give sh a try:
[https://github.com/amoffat/sh](https://github.com/amoffat/sh)

~~~
spangry
Man, I was so happy when I discovered your library. It's how I always felt
subprocess _should_ be. With split('\n') and basic string parsing there's
little that can't be automated with sh. Keep up the good work!

~~~
daenz
That makes me happy to hear, thanks!

------
chrismealy
I gave up on expect in 1990s. It always almost but not quite worked. Is it
better now?

------
znpy
Speaking of surprisingly unappreciated unix tools, I recently started reading
the documentation of the gnu info package, and I must say, that is really
impressive.

if you didn't already, I highly recommend you type 'info info' in your
terminal and spend half an hour learning how to use it.

Plus, if you are an emacs user, the emacs manual is bundled with emacs in info
format, and you can obviously read info documents in emacs too.

Reading the gnu emacs manual inside emacs is a game-changer.

~~~
andlrc
`info` also contains a lot of historical notes, i.e.

    
    
        $ info Groff | awk '$2 == "History"{f = 4};f && f--' RS= ORS='\n\n'
        1.2 History
        ===========
        
        'troff' can trace its origins back to a formatting program called
        'RUNOFF', written by Jerry Saltzer, which ran on the CTSS (_Compatible
        Time Sharing System_, a project of MIT, the Massachusetts Institute of
        Technology) in the mid-sixties.(1)  (*note History-Footnote-1::) The
        name came from the use of the phrase "run off a document", meaning to
        print it out.  Bob Morris ported it to the 635 architecture and called
        the program 'roff' (an abbreviation of 'runoff').  It was rewritten as
        'rf' for the PDP-7 (before having UNIX), and at the same time (1969),
        Doug McIllroy rewrote an extended and simplified version of 'roff' in
        the BCPL programming language.
        
           In 1971, the UNIX developers wanted to get a PDP-11, and to justify
        the cost, proposed the development of a document formatting system for
        the AT&T patents division.  This first formatting program was a
        reimplementation of McIllroy's 'roff', written by J. F. Ossanna.

~~~
kazinator
I'm well familiar with Awk RS="" paragraph mode and all, but I'm scratching my
head why f = 4 to fetch three paragraphs.

[Edit]: Indeed, I ran your command verbatim and obtained one more paragraph:
_When they needed are more flexible language [...]_.

You sneaky devil!

~~~
andlrc
> You sneaky devil!

He he. Never copy/paste shell code found on the internet :-)

------
ChuckMcM
This tool keeps the command line interpreter (CLI) alive. Sysadmins everywhere
can glue hundred different systems from a hundred different vendors together
if they have CLI and the sysadmin has access to a machine that runs expect(1).

There are many stories of complex infrastructure being taken down by new
firmware on routers or switches that change the order of parameters in a
command :-)

~~~
kazinator
Rather, this tool keeps alive systems that do _not_ have easily scriptable
command interfaces, and must be driven at the character-level through their
interactive visual interfaces.

If the system has a CLI, you write a script in that CLI; you don't write an
expect script.

Sometimes systems have CLI, but only behind a remote access wall (telnet,
serial, ...); then you might use expect---because there is no way to upload a
script and dispatch it.

~~~
ChuckMcM

       > Sometimes systems have CLI, but only behind a 
       > remote access wall (telnet, serial, ...); then 
       > you might use expect-
    

All managed switches, all routers, and all PDUs pretty much.

~~~
kazinator
Well, perhaps not quite _all_ routers, like this cheap consumer device:

    
    
      $ ssh root@router uname -a
      root@router's password:
      Linux router 2.4.20 #1 Sun Jun 27 20:13:35 PDT 2010 mips GNU/Linux
    

The only use case for Expect here is typing in the password, which we can
eliminate by administering some SSH keys.

~~~
mypalmike
And then there are the "enterprise switches" from at least one major vendor,
whose crappy ssh implementation only supports a single tty channel (no exec
channels), so you have to run expect anyhow. :-(

$ ssh root@router uname -a

Connection closed by shitty ssh implementation.

$ ssh root@router

Welcome to router!

>

------
jashmenn
For folks like me who read this and want to use `autoexpect` on OSX:

    
    
        brew install homebrew/dupes/expect
        # find the expect/tcl path with this command
        otool -L /usr/local/bin/expect
        export TCLLIBPATH=/usr/local/Cellar/expect/5.45/lib
        autoexpect

------
joshaidan
Reading the examples in the man page brings me back to a different era of
modems and BBSs:

"The name 'Expect' comes from the idea of send/expect sequences popularized by
uucp, kermit and other modem control programs."

"Cause your computer to dial you back, so that you can login without paying
for the call."

"Start a game (e.g., rogue) and if the optimal configuration doesn't appear,
restart it (again and again) until it does, then hand over control to you."

"Connect to another network or BBS (e.g., MCI Mail, CompuServe) and
automatically retrieve your mail so that it appears as if it was originally
sent to your local system."

"Carry environment variables, current directory, or any kind of information
across rlogin, telnet, tip, su, chgrp, etc."

------
airesQ
I've used expect to automate distro upgrades at some point, as we needed to
upgrade a couple hundred machines.

During testing I kept finding edge cases. Then I eventually accounted for most
of the edge cases, and gave a default answer for "unknown" edge cases. It sort
of works, but I would like to have more confidence in the process.

Then again, I never felt that the problem was expect, expect does its job, and
does its job reliably. The problem usually boils down to unexpected prompts
(pun unintended).

I've also used pexect in Python for testing, no complains there. In fact at
some point somebody tried to refactor this particular python program to use
Paramiko, but decided to keep pexpect after running into some problems.

------
stevenhuang
Expect was great when I had to automate ssh'ing into a bunch of different
machines with generated passphrases. Not the best for security, but the
machines were in a private network and key auth was not used at the time.
Saved a lot of typing!

------
kazinator
I did an "expect like" thing with my TXR language for scraping documents.
Namely IMAP authentication:

[http://www.kylheku.com/cgit/tamarind/tree/auth.txr](http://www.kylheku.com/cgit/tamarind/tree/auth.txr)

All that's missing is the pseudo-tty manipulation and whatnot to do that sort
of thing over TTY-based sessions.

Timeouts are handled by setting up timeouts at the socket level. These turn
into an exception, which we catch and convert to a pattern matching failure.

------
geff82
I used expect heavily at several clients in the past and have never seen
someone else use it. I never used it as a precise tool, rather as a
Sledgehammer. I think I remember I used Expect to script remote machines
(expect was literally the "programmer") when everything else failed and was
"prevented" due to policy. I guess my script was also non-compliant, but noone
complained.

~~~
ianmcgowan
I've done the same thing, and on the windows side of things AutoIT
([https://www.autoitscript.com/](https://www.autoitscript.com/)) is similarly
flexible when all else fails. It's not pretty, but arguably is a good example
of the hacker ethic.

~~~
EvilTerran
For Windows, I like AutoHotKey - IIRC, it started out as an open-source
alternative to AutoIt, but has since grown a significant amount of
functionality of its own:

[https://autohotkey.com/](https://autohotkey.com/)

------
kazinator
I'm calling expect dynamically out of CGIT (the git repo serving CGI program).
The expect script runs Vim in pseudo-terminal, to load a source file and
syntax-colorize it via :TOHtml. That HTML is then integrated into the file
view generated by CGIT.

This is not done for every file type; a master shell script dispatches
different coloring strategies based on suffix.

~~~
Somasis
You know, cgit does come with two example syntax highlighting scripts... :)

~~~
kazinator
That's correct. (I think the old version I'm using maybe only with one.) That
script is what I hooked my custom logic into.

------
jquast
I wrote a brief article about pexpect, driving into the difference of why we
chose to use a pseudo-terminal in some special cases of automation,
[https://jeffquast.com/post/terminal_1/](https://jeffquast.com/post/terminal_1/)

------
cafard
Expect is the reason that I first learned Tcl. I haven't used it quite a
while, though.

------
jquast
Noah Spurrier did a great job with pexpect, which i helped port to python 3
and add unicode support.

I've been in love with pty's in general since the mid 1990's, and naturally
found myself maintaining a highly visible python project about pty's.

love this stuff.

------
pron
Many years ago I used Expect to write a multi-host shell:
[https://github.com/pron/sysman](https://github.com/pron/sysman)

------
a3n
I used to use expect to automate testing a medical device over telnet.

------
tluyben2
I have used this a lot in the past but don't have the need for it anymore. It
was incredibly useful and I would've died of overwork if I did not have it
back then...

------
erikb
I used pexpect (a Python expect like library) in my master thesis. Was far
from perfect but much better for what I wanted than all the other libraries
I've tried.

------
joe563323
The lesson learned from expect is to not name a software with generic name, it
is very hard to search on internet regarding the usage problems and in forums.

------
pmoriarty
The worst thing about expect is that it doesn't have an IRC channel on which
you can find knowledgeable people to help you out when you're stuck.

~~~
cmacleod4
Expect questions do get answered on Stackoverflow though -
[http://stackoverflow.com/questions/tagged/expect](http://stackoverflow.com/questions/tagged/expect)

------
a3n
[https://en.wikipedia.org/wiki/Don_Libes](https://en.wikipedia.org/wiki/Don_Libes)

------
jtcond13
Any reason Expect isn't included on many of the common Linux distributions?
That kind of limits its use, for many of us.

------
danielweber
Every once in a while I find I need to pull out Expect and it's like using an
old friend.

------
digi_owl
I find myself thinking about autohotkey.

