

Gremlin.js: Throw a thousand monkeys at your page - sam152
http://dailyjs.com/2014/03/11/gremlins-customsync/

======
mitchellh
For those who don't know: this is PROBABLY named after Palm OS Gremlins. Palm
OS had a debugging tool called "gremlins" that would randomly tap and type and
gesture all over your Palm OS app. This debugging tool would often find edge
cases that would end up crashing your app.

As someone who used to program for Palm OS a lot, I was pretty happy to see
this name. :)

~~~
apinstein
Back in the very early days of Palm OS, Palm had a "Quality Certification" you
could apply for, and if your app passed, you'd get a bunch of marketing
benefits. So we went for the highest-level certification on our products,
which required surviving 1,000,000 gremlins.

At first I was very skeptical that it would find any meaningful bugs, but in
fact it was very good at finding memory leaks and edge cases. In the end it
took us several weeks to finally be able to survive the 1M gremlins without a
crash. A single run of 1M gremlins would take about 24 hours.

In later years they added the capability to supply your own dictionary of
values to apply to certain inputs so that gremlins could get past validation,
and it would also save the entire state of the device every 10K events so that
you could easily reproduce errors that Gremlins found.

I could definitely see this as a good way to stress-test certain parts of your
app, though it takes a ton of time, and frankly without surrounding tooling
(like resource monitoring, saving "state") it wouldn't be practical.

------
quesera
The 1984 Macintosh had a similar testing tool called The Monkey.

Andy Hertzfeld:
[http://www.folklore.org/StoryView.py?story=Monkey_Lives.txt](http://www.folklore.org/StoryView.py?story=Monkey_Lives.txt)

------
xhrpost
I think this is essentially "Fuzz testing"[1]. Something I read about in a
book years ago but never saw done in real-life. The term may be antiquated at
this point. [1]
[http://en.wikipedia.org/wiki/Fuzz_testing](http://en.wikipedia.org/wiki/Fuzz_testing)

~~~
billyhoffman
I wouldn't call this fuzz test because it focuses on simulating legitimate
actions. Do random clicking. Do random scrolling. Do random mouseovers. Do
random [any DOM event]. Yes, they are doing this at a massive scale, but the
key here is that germlin.js can only do things a user can do. In short,
Gremlin is just automating actions; basic legal, legitimate actions. Still
cool, and great for stress testing or looking for race conditions in your UI
code, but not fuzz testing.

Fuzz testing, on the other hand, is about modifying the data, to interacting
with the UI. To use an analogy, fuzz testing Microsoft word might involve
corrupting various structures in the DOC or DOCX file or in an OLE embed with
malformed data and seeing how the parser/program reacts. The Gremlin.js
equivlent would be just clicking on a bunch of buttons in the Word UI really
really fast.

Both are helpful, but they are testing different things

~~~
yeukhon
Not entirely true.

Fuzzing _is not_ about modifying data. I don't think that's the right way to
describe fuzzing.

It's about testing the durability of a program by trying all kinds of data, as
greedy as possible. This includes known problematic inputs and random inputs
to match some expectations. By random it can either be totally random (any
length, any pattern) or protocol-aware.

Random clicking is a form of random data testing because you are trying random
input stream to a program. Your argument is not entirely wrong either. By
fuzzing his UI he may trigger the browser to crash. He may trigger his monkey
to crash. Fuzzing is a very general technique. I can write a fuzzer that fuzz
Firefox's UI Australis. What will I look for? Maybe after opening 100 tabs and
closing the 45th tab the titlebar disappeared. Or resize the browser from
range s to range w I will find some range will cause the UI to look ugly
(style overflow, etc). Or after clicking on the scrollbar several times
consecutively the browser crashed.

Barton Miller, the "Father of Fuzzing" did UI fuzzing by simulating actual
keystrokes and mouse clicks. See
ftp://ftp.cs.wisc.edu/paradyn/technical_papers/fuzz-nt.pdf

In his Forwards page, he even mentioned "Monkey"
[http://pages.cs.wisc.edu/~bart/fuzz/Foreword1.html](http://pages.cs.wisc.edu/~bart/fuzz/Foreword1.html)

~~~
Mutjake
One nice way of describing fuzzing is that it is running a search problem
(searching for bugs) using more or less guided Monte Carlo method for the
program's input space. Generally the testing should also be automated (test
case injection and testing oracle), so that one can beat the program with
enough test cases for the fuzzing to be viable.

Relating to the practical example: if one is trying to find bugs from the UI
code (or it is the only way to feed inputs to the program), monkey method of
fuzzing is the way to go. But if one tries to test the deeper layers of the
program, it is beneficial to try to find the lowest layer of inputs we can
access, since it enables faster input of test cases and thus makes the fuzzing
more effective.

One way of the other, my opinion is that both are fuzzing, by the definition I
gave for it :-)

~~~
yeukhon
I like the way you treat fuzzing as search program.

~~~
Mutjake
It's a nice way of summing up the general principle. Unfortunately I can't
take the credit for coming up with that one :-)

------
sergiotapia
Netflix also has a famous division of monkeys that they unleash to take down
servers in the hive, and generally muck about.

They want to break as many things as possible while keeping uptime and that
sounds freaking awesome! :D

~~~
madeofpalk
More info (the open sourced it): [http://techblog.netflix.com/2012/07/chaos-
monkey-released-in...](http://techblog.netflix.com/2012/07/chaos-monkey-
released-into-wild.html)

~~~
bhhaskin
That is pretty darn cool!

------
fzaninotto
Hi! I'm one of the authors of Gremlins.js. Glad to see that some of you find
it interesting.

I wrote a blog post a while ago about the purpose of the lib, it's called
"Completing the JavaScript Test Stack". Worth reading if you wonder what this
is all about.

[http://redotheweb.com/2014/01/07/completing-the-js-test-
stac...](http://redotheweb.com/2014/01/07/completing-the-js-test-stack-
introducing-greminsjs.html)

------
imkevinxu
lol I'm gonna call this "Twitch Plays the Internet"

~~~
wildpeaks
You read my mind, that totally looks like anarchy mode applied to to the web
:)

------
presty
For Android:
[http://developer.android.com/tools/help/monkey.html](http://developer.android.com/tools/help/monkey.html)

------
atom-morgan
The GIF in Github makes me think TwitchPlaysYourApplication.js would be a
better name :)

------
Houshalter
That looks really cool, though I think it would have a hard time navigating
past the first few menus, and could hit features that are only rarely meant to
be touched (e.g. "delete everything".) Perhaps a searching algorithm that
intentionally tries to explore every menu and feature, navigate to them and
then test them. Or intentionally tries things it knows are more likely to find
bugs.

~~~
fzaninotto
All the gremlins are configurable, you can specify which elements they can
interact on (and therefore exclude "dangerous" elements).

Relevant doc: [https://github.com/marmelab/gremlins.js#configuring-
gremlins](https://github.com/marmelab/gremlins.js#configuring-gremlins)

------
taybin
Now have this play the 2048 game.

------
toolslive
So the person who named this is old enough to have seen (and liked)
[http://en.wikipedia.org/wiki/Gremlins](http://en.wikipedia.org/wiki/Gremlins)

I guess 40-something.

~~~
reubenmorais
The gremlin is older (and broader) than that movie:
[https://en.wikipedia.org/wiki/Gremlin](https://en.wikipedia.org/wiki/Gremlin)

------
ianbicking
A similar library that I wrote:
[https://github.com/ianb/walkabout.js](https://github.com/ianb/walkabout.js)

------
espeed
Not to be confused with the Gremlin graph traversal language
([https://github.com/tinkerpop/gremlin/wiki](https://github.com/tinkerpop/gremlin/wiki))
or the JavaScript version of Gremlin.

------
metaedge
I just give my laptop or iPad to 2 year old son and it does the same trick!

