
Polly.js – Record, replay, and stub HTTP interactions - zwentz
https://netflix.github.io/pollyjs/#/
======
stephen
Looks well done (it uses unicode art, so it must be amazing) but I have a
fundamental distrust/dislike of record/replay frameworks...just seems like
you're papering over an inherently bad testing approach.

E.g. sure, when replays work, they're great, but:

a) you have to do a manual recording to create them the first time (which
means manually setting up test data in production that is just-right for your
test case)

b) you have to manually re-record when they fail (and again means you have
manually go back and restore the test data in production that is just-right
for your test case...and odds are you weren't the one who originally recorded
this test, so good luck guessing exactly what that was).

In both cases, you've accepted that you can't easily setup specific, isolated
test data in your upstream system, and are really just doing "slightly faster
manual testing".

So, IMO, you should focus on solving the core issue: the uncontrollable
upstream system.

Or, if you can't, decouple all of your automated tests from it fully, and just
accept that cross-system tests against a datasource you can't control is not a
fun/good way to write more than a handful of tests (e.g. ~5 smoke tests are
fine, but ~100s of record/replay tests for minute boundary cases sounds
terrible).

~~~
jasonmit
> you have to do a manual recording to create them the first time (which means
> manually setting up test data in production that is just-right for your test
> case)

Your test invokes the the recorder. There isn't anything manual outside of
writing & running your test.

> you have to manually re-record when they fail

Again, nothing manual. It would require running your test again with Polly in
record mode if you want to "refresh" the recording with a newer set of
responses.

> In both cases, you've accepted that you can't easily setup specific,
> isolated test data in your upstream system, and are really just doing
> "slightly faster manual testing".

This is by no means a replacement to E2E testing. It is a form of
acceptance/integration testing where you're testing your application against a
point in time that you verified all systems were talking correctly with your
application. E2E tests are much slower, difficult to debug, and intended to
capture those breakages in contracts.

It's a tool for your toolbox, reach for it when needed. We plan to release a
tutorial/talk to should clear up any misconceptions. There are also other
applications for Polly such as building features offline or giving a demo
using faker to easily hide any confidential data.

~~~
stephen
> Again, nothing manual

Yes, sorry for being inexact/overusing the term--I understand the tests drive
the recording.

What I meant by manual is getting the e2e system into your test's initial
state.

E.g. tests are invariably "world looks like X", "system under test does Y",
"world looks like Z".

In record/replay, "world looks like X" is not coded, isolated, documented in
your test, and is instead implicit in "whatever the upstream system looked
like when I hit record".

Which is almost always "the developer manually clicked around a test account
to make it look like X".

This is basically a giant global variable that will change, and come back to
haunt you when recordings fail, b/c you have to a) re-divine what "world looks
like X" was for this test, and then b) manually restore the upstream system to
that state.

If no one has touched the upstream test data for this specific test case,
you're good, but when you get into ~10s/100s of test, it's tempting to share
test accounts, someone accidentally changes it, or else you're testing
mutations and your test explicitly changes it (so need to undo the mutation to
re-record), or you wrote the test 2 years ago and the upstream system aged off
your data.

All of these lead to manually clicking around to re-setup "world looks like
X", so yes, that is what I should have limited the "manual" term to.

~~~
ryanbrunner
But in the case we're talking about, where you're reliant on an external
service that can change underneath you, "world looks like X" is genuinely not
under your control. It feels like pretending that it is will lead to just as
many failures as acknowledging it's inherent volatility.

~~~
stephen
Agreed! And, to me, record/replay is still pretending like it's controllable,
b/c even if you decouple for replays, records will always be a PITA.

My depressing solution is to just not even try to automate tests against the
upstream system and instead invest in test builders/DSLs that make mocks/stubs
on both sides as pleasant as possible.

And when bugs slip through, make sure to update your stubs/mocks on both sides
to prevent the regression.

To me this gets the most agility and reliability, and will be a test suite
that developers don't hate 1-2-5 years down the road.

------
burger_moon
I would love to hear from people involved in projects like this, what kind of
work/ how much, was done to get it ready for and approved to be open sourced
by the company.

Especially at large corps like Netflix I'm sure there's a lot of hoops to jump
through.

~~~
staticassertion
Netflix seems to also have a really strong culture around this though, so I
wouldn't be surprised if it's a lot less hoopy than you'd imagine.

~~~
error54
Not to shamelessly plug but if you're in the Bay Area on June 28, we're giving
a talk that's a bit about performance, a bit about Netflix engineering
culture:
[https://jstalks2018.splashthat.com/](https://jstalks2018.splashthat.com/).

~~~
MaxLeiter
If I’m a student can I sign up to attend?

~~~
wesleytodd
Sure! (at least I don't see why you wouldn't be able to)

------
jorisd
This is very cool, solving some issues I no doubt many people have when
writing tests against a (fast-)moving target. I'll definitely give it a try in
my next project.

I looked through the codebase, and noticed that this uses a custom data format
to persist HTTP requests and responses in local storage. I'm not sure if it's
technically possible in all circumstances, but I think it might be valuable to
have requests and responses be stored as HAR 1.2 [1] when possible, so that
the trace can be used by other tools [2] to aid in debugging, verifying and
analyzing behaviour as well as perhaps automated creation of load/performance
tests.

[1] -
[http://www.softwareishard.com/blog/har-12-spec/](http://www.softwareishard.com/blog/har-12-spec/)

[2] - e.g.
[https://toolbox.googleapps.com/apps/har_analyzer/](https://toolbox.googleapps.com/apps/har_analyzer/)

------
nitinreddy88
There is already famous Policy package for .NET with same name

[https://github.com/App-vNext/Polly](https://github.com/App-vNext/Polly)

------
vaidik
I used to use nock which would work very well in node environments. But this
works in the browser as well. So I guess this can be fairly helpful while
writing tests post development. If you are doing TDD, then recording/replaying
doesn't fit anywhere in the development cycle.

I like the API of this library and the browser support that was missing in
nock. So thanks Netflix! Although it would have been nice to see nock add this
support. Which is what I wonder - why not just contribute to existing
libraries.

~~~
pimterry
If you're looking for Nock but not just node, try Mockttp:
[https://github.com/pimterry/mockttp](https://github.com/pimterry/mockttp).

It lets you create & configure mock HTTP servers for JS testing, but with one
API that works out of the box in Node and in browsers. This avoids the
record/replay model too, so you can still take the TDD approach and define
your API behaviour in the test itself.

(Disclaimer: I'm the author of Mockttp)

------
yawz
I love the name! "Polly" repeats everything... and wants biscuit every now and
then :).

~~~
silentguy
And this polly would love some cookies too.

~~~
yawz
"cookie" sounds even better. I wasn't sure if it was "cracker" or "biscuit"
:).

------
bicubic
Related to that, is there anything that allows to completely save the state of
a modern website with all of the fetch requests and websocket related stuff it
fired off?

I just want an ability to save and reopen exactly what I'm looking at. There
are some cool websites which will eventually go down and I want to preserve an
interactive snapshot of them.

~~~
Mandatum
This won't work for WebSockets, really websites that use WebSockets require
some interaction to generate the transmitted message which is often dependent
on the servers response. Private websites, or websites that require a login
are hard - but it can be done. Would suggest HTTrack.

~~~
reymus
But it's not impossible to have some tool that records all of those
interactions to reproduce later. A smart enough tool could record everything
since you open the site until you click save. It would not reproduce the
functionality that is backend dependent, but iy sure can replicate the dom,
etc. Am I missing something?

~~~
Mandatum
Yeah - but there's a LOT of variables that come into play for something like
that. It'd likely be easier to either record it with something like
BugReplay.com or video.

------
isuckatcoding
Unrelated but There are so many things called Polly that it gets confusing

~~~
youdontknowtho
Yeah. I was going to bring up the library for .net that provides policy based
retries.

~~~
swyx
i went to the slackbot that has a cute parrot logo. we at CodingBlocks love
our Polly.

~~~
xelfer
I thought of Amazon Polly.. converts text to lifelike speech.

------
biznickman
So VCR gem for javascript. Great! Personally I stopped using VCR gem a while
back as it blocks edge cases. However for larger projects where things can get
unwieldily this makes a lot of sense. Local test suites should never hit
external APIs so it's much better to have mocks/stubs than to have no tests at
all.

However on smaller projects I've found that just clicking through to make sure
things work and then letting my error reporting system catch bugs to be much
more effective :)

It's a hard line to walk and I surely haven't perfected it. I'll give it a
shot on a future project!

------
bookofgreg
I personally replace all stubbed HTTP interactions between my services with
contracts with good success. [https://docs.pact.io/getting-
started](https://docs.pact.io/getting-started)

------
textmode
Alternate URL (no Javascript required):

[https://github.com/Netflix/pollyjs](https://github.com/Netflix/pollyjs)

~~~
tills13
a little ironic, no?

------
mygo
I know this is slightly different, but I wish more people knew about Chrome /
Safari / Firefox’s “network” console tab. Great for debugging. Can look at all
requests, headers, responses, timespans, etc. Some will even let you copy a
given network request as a cURL command, capturing all headers, body, query
strings, etc.

~~~
dudul
And out of curiosity, what makes you think that people don't know about it?
I've never met a web dev who didn't know about it in the past few years.

~~~
thijsvandien
New people are introduced to web development every day. Assuming some things
are just common knowledge is not very beginner friendly.
[https://xkcd.com/1053/](https://xkcd.com/1053/)

------
ascendantlogic
I'm curious what the application could be for load testing? Tools like locust
and gatling are nice but are still synthetic. I'd love to capture X minutes of
traffic, then dupe it Y times and replay it as a more accurate representation
of traffic patterns for load testing. Is that a thing?

~~~
jmckenz
not had a chance to properly try yet, but
[https://goreplay.org/](https://goreplay.org/) does exactly what you are
asking. Alternatively, in the container world, tools such as Istio
([https://istio.io/](https://istio.io/)) allow traffic shadowing - you can
duplicate traffic and route it somewhere else

------
xab9
I did something similar, but as an interim proxy (can record, replay, there
are modifier hooks, can slow down requests). You have to point towards a
backend api and on the frontend you use the proxy url instead of the original.
But it's mostly for debugging, so the scope is much more limited.

------
gnufied
Also check out [https://github.com/code-
mancers/interceptor](https://github.com/code-mancers/interceptor) which as
well uses browser APIs to enable users to mock http responses via a chrome
extension.

------
john838
How does it hook into the browser APIs? I can't seem to find it. By what black
magic would it know how to hook into my puppeteer instance? Or I'm I not
understand this?

------
badrelmers
Why another tshark/tcpdump? all this can be done with a simple script with few
lines. Today we need javascript recorders, traffic recorders are a kid game
and using a certificate to touch https is a dangerous way (but every project
there is doing same). Tshark and sslkeylogfile is the only safe way... but I
like this project I don't know why! I feel something.

------
invisible
This seems like it will be especially useful with apollo-client for graphQL
requests.

------
Osiris
I've used mitmproxy + proxychains to do this. How is Polly different?

~~~
jonknee
[https://netflix.github.io/pollyjs/#/README](https://netflix.github.io/pollyjs/#/README)

Why Polly?

Keeping fixtures and factories in parity with your APIs can be a time
consuming process. Polly alleviates this by recording and maintaining actual
server responses without foregoing flexibility.

* Record your test suite's HTTP interactions and replay them during future test runs for fast, deterministic, accurate tests. * Use Polly's client-side server to modify or intercept requests and responses to simulate different application states (e.g. loading, error, etc.).

~~~
curun1r
Not sure that tells me how it's different from replay or the other half dozen
npm modules that do the same thing. It'd be nice for them to contrast their
tool with existing ecosystem options considering some of them are pretty well
established.

~~~
jasonmit
Can you share which libraries you know that achieve the same thing? I'm happy
to go through and respond to the differences.

~~~
curun1r
I only have experience with replay, but an npm search turns up:

replay, replayer, http-record, talkback, sepia, mitm-record, fetch-vcr, tape-
nock, jest-playback, eight-track, axios-vcr, replayer, node-vcr, mocha-vcr,
mockyeah, yakbak, nine-track, dkastner-replay, node-nock

At which point I stopped looking...recording http requests isn't exactly new
territory.

------
swiezy2
Interesting but would be more useful with support for streaming

~~~
zwentz
On the roadmap, depending on your definition of "streaming" (e.g. buffer
streams, websockets).

------
ksejka
is it a port of [https://github.com/App-vNext/Polly](https://github.com/App-
vNext/Polly) ?

------
incadenza
so this will make the actual http request the first time, then keep a
recording? I’m not entirely clear how this works from the docs.

------
sidcool
What would a use case of this library look like?

------
ikeboy
So is this basically selenium in javascript with some neat features?

~~~
gleenn
Sounds more like the VCR gem from Ruby land.

~~~
zwentz
Yes exactly, that's the idea anyway. Has a few nice features on top such as
controlling the network latency and expiring recordings (useful when working
on a project supported by a big team).

------
vaidik
Also, what's up with the in-your-face hiring pitch right in the documentation?

[https://netflix.github.io/pollyjs/#/README?id=we39re-
hiring](https://netflix.github.io/pollyjs/#/README?id=we39re-hiring)

~~~
anchpop
I assume that devs read the documentation, and they want to hire devs, and
it's their tool, so they put their hiring pitch in their documentation for
their tool to try and hire devs

