Expect is cool, I have used it in the past as well as the expect perl module, but in my opinion is always a suboptimal way to automate things. It's always better and more robust to have a proper interface to script things instead of relying on matching brittle text/regex responses and timers and such. Of course sometimes it's not possible, so expect comes in handy.
Expect is cool, I have used it in the past as well as the expect perl module, but in my opinion is always a suboptimal way to automate things.
When I was a graduate Comp Sci TA in the 1990's, it was the perfect way to automate grading the Unix shell that the Systems Software class was supposed to write. Effectively, I was also writing a regression/functional testing system for the class. I explained that the Expect/Tcl script was going to be used to grade their project, and I provided the code.
One of my students said I was the best TA ever! There were a few students who complained that they shouldn't be marked off for a feature breaking, because it had worked a week or two ago. There were a few students who never figured out that the script was going to be used to grade their project, even though I provided weekly email updates explaining this.
> I explained that the Expect/Tcl script was going to be used to grade their project, and I provided the code.
If it's deterministic, is there a risk of someone just writing a second Expect script (instead of a solution to the programming exercise) that interacts with yours and gives the replies that you want?
I had a C class in college where we submitted homework with a program on a server. The program would compile the C code, run it with a bunch of inputs, and then report back if it passed (i.e. the output matched the expected output). The output was expected to be in exactly the correct format, so the submit program would helpfully print your output and the expected output so you could fix any formatting issues.
I got about 2 assignments into the class before I realized that I was being a chump. I wrote a C program that simply printed out the expected output, and submitted it. I received a passing grade on the assignment, so I took it further. I wrote a Perl script which would submit "int main(void){return 0;}", read the expected output, generate a C file that printed out that output, and submit it again.
Apparently the TA for the class never checked the submitted assignments and relied entirely on the submit program, so I was uncaught for a few weeks. My downfall was trying to make the Perl script as efficient as possible (I had extra time on my hands because I didn't have to do the C assignments!). I had run my Perl script a few dozen times while optimizing it, and apparently the TA received an email every time an assignment was submitted. He brought it up to the professor and I was called into his office. Luckily he was lenient and let me just redo the assignments :)
I've taken online classes where they give you unit tests that use one set of inputs/outputs, when grading they'd use the same tests, but have a different set of inputs/outputs.
Early on there was one or two cases where subtle problems came up because the public set and private set accidentally tested things slightly differently, but the TA / prof was quick to respond.
Nice use of expect. I expect (pun intended, though realized after the fact) that you used it to simulate sending user input to their shell? and checking that the shell's response was as expected? (not doing the second on purpose either, honest, it just seems the right word to use).
I 100% agree. If I can use paramiko/fabric or something similar, I prefer it way more than expect.
If you have a slave device over RS232, expect is probably your only option. But if that remote item is running sshd, you are way better off without expect.
I've used paramiko + expect to automate distributed benchmarking + results gathering. I can say that while it ultimately worked, Curse That Thing In Particular. It was so so so fiddly to get everything working, and very fragile + output-dependent.
> while it ultimately worked, Curse That Thing In Particular
"Things Only 90s SysAdmins Will Remember"
Expect is mostly a tool of last resort when a utility has silly prompts that you can't bypass by hitting it with command line options like `-y` and stuff.
Paramiko is a Python API for SSH client, so it's not comparable to either of
the rest.
Fabric itself is a poor man's badly implemented RPC for sysadmins: you send
commands and receive output and result codes. Fabric wraps this in a Python
API, so you can react to how your operations run.
Salt and Ansible are similar to Fabric (and "poor man's badly implemented RPC"
comment applies to them as well), but they also define some higher level
tasks, like "have X package installed". All three have the operations defined
at the call initiator and return synchronously, when all the work is done.
Then there are CFEngine, Puppet, and Chef. Every managed server runs an agent
that periodically executes some rules. CFEngine and Puppet have their own
DSLs, and Chef uses general purpose language (Ruby), what I consider a bad
fit. These work asynchronously, as you don't push the rules to the servers,
but change them in their distribution point and wait for servers to pick up
the update.
expect scripts in general are brittle IMO. The general design is: "look for this and then that and then this. If at some point you stall along this path for more than X seconds, something must have gone wrong, abort."
Well, I maintain that the characters in the stream aren't as good success criterion as the stack of exit codes among the tasks. So if I "ssh remotefoo /tests/the_test.sh" it is the_test.sh's job to return 0 if it succeeded and nonzero otherwise. ssh will return 0 if the remote script returns zero, so I can count on that. But if the host is busy sometimes and the_test.sh takes 91 seconds on a heavily-loaded day but 37-45 seconds normally, I might encode the wrong timeout in my expect script. Things like /etc/issue or /etc/motd or PS1 should have no bearing on my test results. But if someone writes the text "Password:" or "Username:" or some other similar content in there, my expect script might choke on it.
Indeed, which is why in ShutIt I used some tricks with the prompt to determine whether a command was finished, and then (optional, but defaulting) checks on exit codes to determine success/failure. It seems to work pretty well, and no need to specify what it expected.
Aside: the bare except clause used here [1] is a bad habit. You should try to avoid it. Python raises things like NameError and TypeError for simple design problems like mistyping variable names. It's extremely rare that you can handle errors like that. Your "except:" would attempt to handle and effectively mask bugs like this.
I think expect is often used to have it enter username and password "interactively". But it's easier and more robust to just use the public-key auth facility that is built-in to ssh.
Not always possible. The problem is that ssh lives in the mythical land of perfect security, where people never need to do unsafe things in order to get their jobs done.
Hence keys silently not working without "correct" permissions, passwords not passable on the CLI, inconsistent type-yes-to-override behavior, and other user-hostile design choices.
I love openssh, but it's a real pain in the ass sometimes.
I don't have any of these problems. Just don't use passwords non-interactively, but use public key auth. Also, as any sane program, (open)SSH does not ask you to type anything when running non-interactively in a script (more precisely, when no terminal is present).
If you don't have any of those problems, you're very lucky. Again, ssh is designed with the assumption that people never have a reason to do unsafe things.
there is also sshpass for supplying password, as well as using a languages library like paramiko for python which I believe allows you to supply a password. It would take more than password auth to force me to use expect these days :)
This isn't expects' fault. If you did not want to record the password in the script then ssh manually and then continue the script from there (e.g. with 'interact' command)
Why would you mix expect with ssh? Most of the time, if you can ssh, you can use an exec channel ("ssh somehost somecommand"), so you shouldn't need to pretend to be typing characters and waiting for prompts.
Yes, expect is really cool. I haven't used it heavily, but had used it some earlier, to automate small command-line tasks to good effect.
Also:
1. I had read much of Don Libes' Expect book. He had a unique style of writing in that book. Hard to describe, partly since it was some years ago, and partly because I can't find the words for it. But I think anyone reading the book would notice the style after reading a few pages. It was not a bad style by any means, just ... something different.
2. I've read that expect was (maybe still is) used heavily in the electronics/EDA industry, though I am either not sure or don't remember how exactly they use it, or rather, for what. And also maybe it was used heavily by GNU software for testing. Maybe someone here can confirm.
Expect is a last ditch tool that becomes less and less necessary as automation libraries and remote APIs become more and more common. The examples given are cute but hardly representative of what expect is really for.
The only times I've needed to use expect was to drive weird non-scriptable telnet-based remote systems and to automate unpackaging of intentionally-broken vendor-supplied self-expanding software archives that require some level of interaction.
The point being if you want to create shortcuts to jump to specific locations in a man page or drive outdated appliances or open user-hostile zips, expect is the tool for you!
I agree, expect is less and less useful, many vendors provides ways to automate deployment now. But "unautomatable" deployment is still happening.
In some cases you can deal with it with some "printf 'line1\nline2\n'| vendor_command", but the vendor command may ignore stdin and you have to use expect.
Having to automate that way is extremely brittle, stdout and stdin are very poor and unstable APIs. For example I remember some internationalized installers which gave stdout outputs according to your locales (lesson learned, export LC_ALL=en_US.utf8 before an expect script). And if one message is added or is changed, it can break...
Personally, I tend to repackage badly shipped software (yay installanywhere crap...) in deb or rpm files, at least that way, I encounter issues before I deploy in production and I can update/remove this software easily and completely. However, it only solves part of the problem, if the configuration steps are interactive only, you've to resort to expect in production.
Expect is less and less useful, it's a last resort solution, but when you need it's extremely helpful.
Do you have any links to examples of what it is really for? I've had a hard time finding code examples because the reasons listed at the bottom of the article.
Sure, let's say you want to remotely control a Cisco router with a serial interface. expect makes it really easy to remotely connect to the serial interface, wait for a prompt, send a command, and take a conditional action that depends on the response.
The article mentioned it's used for testing gcc and LLVM. If you want an example you can actually look at, it's also used for testing the fish-shell (fish-shell has other tests too, but expect is specifically used to actually test user-visible behavior, which makes expect the perfect tool for the task). You can find those tests in https://github.com/fish-shell/fish-shell/tree/master/tests. The files of the form "foo.expect" are the tests (the "foo.expect.err" and "foo.expect.out" files are how it interacts with the rest of the testing infrastructure), and the file interactive.expect.rc provides a bunch of helper functions for the expect scripts, and in particular a standard way of looking for prompts (because the shell will redraw the prompt at various times, this function, coupled with the fish code in interactive.config, lets the expect scripts tell when it's the old prompt redrawn versus the new prompt after a command has been executed).
Expect is great for persuading things, particularly poorly written Java monoliths that use text interactive installers, to play nicely with ansible. On more than one occasion it has been tasked with handling one.
You run autoexpect first to record what you want it to do on a dry run, then tidy up the generated expect script, then plug it into ansible as a raw task.
Granted this shouldn't be a problem in 2016 but some vendors are idiots. But thanks to expect my life is a little more comfortable :)
Oh this is really nice! I've used pexpect (and pxssh) extensively along with expect in a previous lifetime. I'd always meant to write an "autopexpect" or "autopxssh" and never did.
Long ago, I had to write a utility to backup the configurations settings from all the network devices in a given network. At the time, the network consisted of many, many different types of routers, switches, and the like.
For many of the devices, there was a way to fetch the config but it required logging into the device (generally telnet) and navigating a menu to "send" the config somewhere. Using expect to do that was a fantastic way to handle it.
There's also a number of cases where you want output from a command line program "as if run by a user at a terminal". For some programs, they change the output (ie, removing color codes, etc) if the output is not going to a terminal. Expect does a fantastic job of pretending to be a terminal.
Also, it's Tcl, and I love Tcl. It's (possibly tied) at the top of my list for languages. So much fun and more flexible than anything that doesn't support lisp level macros.
In case somebody's interested in a tool that does the kind of thing you describe re: network devices after reading your comment I thought it might be helpful to include a link RANCID: http://www.shrubbery.net/rancid/
Silly backronym name aside, RANCID is a very handy tool and uses expect to do a lot of the 'dirty work'.
Don Libes' Exploring Expect[1] is a devops classic (if there is such a thing). It would be nice if O'Reilly would make the text (completely) freely accessible[2], as they have done with some of their other older titles[3].
Don Libes' daughter went to my high school, and I was working as the "webmaster" for the high school newspaper's website at the time. He would ruthlessly criticize via email the product our self-taught and earnest 2010-era PHP web development. I am not a fan of Don Libes. For those interested, the website was/is: http://theblackandwhite.net/
OMG I have been burned by this so many times. Expect itself is a cool piece of technology and works as well as anyone could hope. However, every time I've tried to use it for anything the solution ends up being super flaky. I think trying to automate text interfaces like this is just fundamentally unreliable.
I think of it in the same way as screen scraping Web pages, or writing buffer-heavy Emacs Lisp:
- If there are alternatives, they'll usually be better
- It'll still be around when there are no alternatives
- Some tasks (e.g. "toggle foo mode") are inherently easier than others (e.g. "press up five times and hope that's enough")
- Some tasks need frequent maintenance and tweaking (e.g. as i/o formats change, program features are added/rearranged, shell environments are tweaked, etc.)
- Some tasks work so well that they're taken for granted for a decade, and it's only when you want to tweak some part that you remember how truly horrifying the implementation is ;)
Tcl (upon which Expect is built) is under-appreciated too. It's a neat language how strings behave as a sort of polymorphic data structure (can be dicts, lists, etc). Under the hood they get optimized based on usage.
Expect can be a lifesaver when you cannot use your usual go-to tools like Ansible etc. especially with legacy setups where you cannot tweak stuff much. I have successfully used it combined with tmux for automated multi-term like functionality, deployment verification and other chores to save a lot of $ for my company. I do try to keep my Expect scripts as simple as possible though, just so I do not have to remember too much since I do not work with TCL anywhere else.
Things only 80's admins and hackers might remember: similar to Expect, some terminal emulation + serial communication programs for PC's came with languages for automation, resembling Expect. Expect might have been inspired by that sort of thing.
I had experience coding in two of them for DOS PC's: ProComm Plus's ASPECT language ("Pascal-ish" flavored), and a C-like language called Salt built into Telix.
I mostly used the very popular Telix for my personal use, but a law firm for whom I did some IT contract work used Procomm Plus for connecting to various databases available over dial-up (e.g. local land title registry). I wrote a bunch of ASPECT code to automate their scraping tasks: log in, look up this and that, save in a file, that sort of thing.
I'm going to try this out on my own because what I read in the comments makes me think I will find use for it, but if I had to be honest, I did so because of the HN comments. The OP was clearly written and nicely formatted, but as someone with moderate CLI/Bash experience (I do almost all of my data processing via Bash programs and pipes), I didn't really understand what `expect` does. Maybe I needed a simpler example, because I don't uss most of the other parts that are in the example, such as `interact` and `send`. As a Pythonist/Rubyist, I have the added baggage of associating `expect` with unit/line testing.
Maybe expect is not for me, but given the title, I would've loved to see maybe a couple paragraphs of why `expect` is underappreciated...that is, what do typical Unix programmers use instead, because they're ignorant of expect. Much easier for me to understand the perspective of ignorance :)
Expect is a TCL-based command-line scripting tool. It's designed for scripting things that don't have a scripting interface, and weren't designed to be scripted.
Essentially, expect scripts run a program, send it some input, look for some regexes on stdout, and then send more input, which may change depending on what the regex matches. And so on.
This is useful, say, if you have a package management tool that requires you to explicitly confirm every update. Or if you want to automatically drive a shell on a remote machine over SSH, and you can't use ssh -c for some reason.
You don't need it often. But when you do, it's very useful.
pexpect (python variant of the same) has 750 stars, so its popular.
people have just dropped tcl in favor of perl and later python and others ... I don't personally like tcl anymore, but I programmed tcl/expect in the 90's and enjoyed it a lot. I now co-maintain pexpect.
Man, I was so happy when I discovered your library. It's how I always felt subprocess should be. With split('\n') and basic string parsing there's little that can't be automated with sh. Keep up the good work!
Speaking of surprisingly unappreciated unix tools, I recently started reading the documentation of the gnu info package, and I must say, that is really impressive.
if you didn't already, I highly recommend you type 'info info' in your terminal and spend half an hour learning how to use it.
Plus, if you are an emacs user, the emacs manual is bundled with emacs in info format, and you can obviously read info documents in emacs too.
Reading the gnu emacs manual inside emacs is a game-changer.
`info` also contains a lot of historical notes, i.e.
$ info Groff | awk '$2 == "History"{f = 4};f && f--' RS= ORS='\n\n'
1.2 History
===========
'troff' can trace its origins back to a formatting program called
'RUNOFF', written by Jerry Saltzer, which ran on the CTSS (_Compatible
Time Sharing System_, a project of MIT, the Massachusetts Institute of
Technology) in the mid-sixties.(1) (*note History-Footnote-1::) The
name came from the use of the phrase "run off a document", meaning to
print it out. Bob Morris ported it to the 635 architecture and called
the program 'roff' (an abbreviation of 'runoff'). It was rewritten as
'rf' for the PDP-7 (before having UNIX), and at the same time (1969),
Doug McIllroy rewrote an extended and simplified version of 'roff' in
the BCPL programming language.
In 1971, the UNIX developers wanted to get a PDP-11, and to justify
the cost, proposed the development of a document formatting system for
the AT&T patents division. This first formatting program was a
reimplementation of McIllroy's 'roff', written by J. F. Ossanna.
This tool keeps the command line interpreter (CLI) alive. Sysadmins everywhere can glue hundred different systems from a hundred different vendors together if they have CLI and the sysadmin has access to a machine that runs expect(1).
There are many stories of complex infrastructure being taken down by new firmware on routers or switches that change the order of parameters in a command :-)
Rather, this tool keeps alive systems that do not have easily scriptable command interfaces, and must be driven at the character-level through their interactive visual interfaces.
If the system has a CLI, you write a script in that CLI; you don't write an expect script.
Sometimes systems have CLI, but only behind a remote access wall (telnet, serial, ...); then you might use expect---because there is no way to upload a script and dispatch it.
And then there are the "enterprise switches" from at least one major vendor, whose crappy ssh implementation only supports a single tty channel (no exec channels), so you have to run expect anyhow. :-(
Reading the examples in the man page brings me back to a different era of modems and BBSs:
"The name 'Expect' comes from the idea of send/expect sequences popularized by uucp, kermit and other modem control programs."
"Cause your computer to dial you back, so that you can login without paying for the call."
"Start a game (e.g., rogue) and if the optimal configuration doesn't appear, restart it (again and again) until it does, then hand over control to you."
"Connect to another network or BBS (e.g., MCI Mail, CompuServe) and automatically retrieve your mail so that it appears as if it was originally sent to your local system."
"Carry environment variables, current directory, or any kind of information across rlogin, telnet, tip, su, chgrp, etc."
I've used expect to automate distro upgrades at some point, as we needed to upgrade a couple hundred machines.
During testing I kept finding edge cases. Then I eventually accounted for most of the edge cases, and gave a default answer for "unknown" edge cases. It sort of works, but I would like to have more confidence in the process.
Then again, I never felt that the problem was expect, expect does its job, and does its job reliably. The problem usually boils down to unexpected prompts (pun unintended).
I've also used pexect in Python for testing, no complains there. In fact at some point somebody tried to refactor this particular python program to use Paramiko, but decided to keep pexpect after running into some problems.
Expect was great when I had to automate ssh'ing into a bunch of different machines with generated passphrases. Not the best for security, but the machines were in a private network and key auth was not used at the time. Saved a lot of typing!
All that's missing is the pseudo-tty manipulation and whatnot to do that sort of thing over TTY-based sessions.
Timeouts are handled by setting up timeouts at the socket level. These turn into an exception, which we catch and convert to a pattern matching failure.
I used expect heavily at several clients in the past and have never seen someone else use it. I never used it as a precise tool, rather as a Sledgehammer. I think I remember I used Expect to script remote machines (expect was literally the "programmer") when everything else failed and was "prevented" due to policy. I guess my script was also non-compliant, but noone complained.
I've done the same thing, and on the windows side of things AutoIT (https://www.autoitscript.com/) is similarly flexible when all else fails. It's not pretty, but arguably is a good example of the hacker ethic.
For Windows, I like AutoHotKey - IIRC, it started out as an open-source alternative to AutoIt, but has since grown a significant amount of functionality of its own:
I'm calling expect dynamically out of CGIT (the git repo serving CGI program). The expect script runs Vim in pseudo-terminal, to load a source file and syntax-colorize it via :TOHtml. That HTML is then integrated into the file view generated by CGIT.
This is not done for every file type; a master shell script dispatches different coloring strategies based on suffix.
I wrote a brief article about pexpect, driving into the difference of why we chose to use a pseudo-terminal in some special cases of automation, https://jeffquast.com/post/terminal_1/
I have used this a lot in the past but don't have the need for it anymore. It was incredibly useful and I would've died of overwork if I did not have it back then...
I used pexpect (a Python expect like library) in my master thesis. Was far from perfect but much better for what I wanted than all the other libraries I've tried.
The lesson learned from expect is to not name a software with generic name, it is very hard to search on internet regarding the usage problems and in forums.
