
Puppeteer: Headless Chrome Node API - uptown
https://github.com/GoogleChrome/puppeteer
======
tholman
One of the biggest wins here is this little tidbit:

> When you install Puppeteer, it downloads a recent version of Chromium (~71Mb
> Mac, ~90Mb Linux, ~110Mb Win) that is guaranteed to work with the API.

A lot of the chrome interface libs about at the moment require you to maintain
your own instance of chrome/chromium and launch the headless server with your
command line, or require a pre compiled version, that can quickly get out of
date ([https://github.com/adieuadieu/serverless-
chrome/tree/master/...](https://github.com/adieuadieu/serverless-
chrome/tree/master/chrome)). Having this taken care of is a blessing.

~~~
fake-name
There is complexity here you may not be seeing, namely if you are on a
platform without X, it will not work.

Normal chromium requires a bunch of X libraries be present. It doesn't _use_
them, but for things like headless testing, it's a massive pain, since the
apt-get (or equivalent) is generally many hundreds of megabytes.

~~~
DonHopkins
X: The First Fully Modular Software Disaster

[http://www.art.net/~hopkins/Don/unix-
haters/x-windows/disast...](http://www.art.net/~hopkins/Don/unix-
haters/x-windows/disaster.html)

~~~
rbobby
> If you're using the Motif self-abuse kit

hehehehehe

------
rcarmo
This is great, but it's sobering to see how hard it is to get a nice,
"complete" PDF screenshot out of a modern site.

Here's a quick hack:
[https://gist.github.com/rcarmo/cf698b52832d0ec356c147cf9c9ad...](https://gist.github.com/rcarmo/cf698b52832d0ec356c147cf9c9ad898)

I'm using The Verge for testing because it lazy loads images, and am being
clumsy about the scrolling, but it mostly works - I can get 90% of the images
to show on the finished PDF.

What I can't seem to get right, though, is creating a single-page PDF where
the page height matches the document height perfectly - it always seems to be
off by a bit, at least on this site (mine works fine, _sometimes_).

Anyone got an idea of why this is so?

~~~
Ded7xSEoPKYNsDd
If I'm reading [1] right, you want to use getBoundingClientRect() instead of
clientHeight, which does not account for borders and margins.

1: [https://developer.mozilla.org/en-
US/docs/Web/API/CSS_Object_...](https://developer.mozilla.org/en-
US/docs/Web/API/CSS_Object_Model/Determining_the_dimensions_of_elements)

~~~
rcarmo
Well, no, really. Seems like this is due to the way Chrome deals with print
CSS and PDF generation rather than any kind of calculations I do prior to
invoking PDF generation...

------
bauerd
So first there was Selenium's JSON Wire protocol, then came the W3C WebDriver
spec and now we're back to browser-specific implementations? As someone who's
tried/is trying to automate Firefox/Chrome/Safari/IE in a consistent fashion,
my only question is: WHY?

~~~
gervase
Based on a quick read of the API, my interpretation is that this is not
targeting people who are trying to automate _every_ browser, but those who
need to automate _any_ browser.

In that context, it's dead-simple to use, and someone with very little
experience should be able to get a working prototype in under 5 minutes.

For my use case, it's closer to "wget/curl with JS processing" than
"automating a user's browsing experience". I don't particularly care which
browser is doing the emulation, with the ease-of-use of the API making the
biggest difference.

It seems very similar to PhantomJS, but to be honest, it's more attractive
from an ongoing-support standpoint simply because it's an official Chrome
project.

~~~
Rezo
Exactly, this looks perfect for taking a screenshot of a page[1], or
converting a page to a PDF[2] in just a few lines of code.

If you have an existing web service, this appears suitable for actual
production usage to deliver features like PDF invoices and receipts, on-demand
exports to multiple file formats (PNG/SVG/PDF) etc., which has quite different
requirements compared to an automated testing framework.

[1]
[https://github.com/GoogleChrome/puppeteer/blob/master/exampl...](https://github.com/GoogleChrome/puppeteer/blob/master/examples/screenshot.js)

[2]
[https://github.com/GoogleChrome/puppeteer/blob/master/exampl...](https://github.com/GoogleChrome/puppeteer/blob/master/examples/pdf.js)

------
VeejayRampay
I've found Chromeless to work great for that task. But then again, if this is
from the Chrome DevTools team, it's a good guarantee to have.

------
simonpure
There's also a docker image for headless chrome automatically updated from the
trunk if you're looking to test against the latest -

[https://hub.docker.com/r/alpeware/chrome-headless-
trunk/](https://hub.docker.com/r/alpeware/chrome-headless-trunk/)

Full disclosure: I'm maintaining the image.

------
bluepnume
I'm really loving headless chrome so far. I have around 650 tests which are
mostly dealing with iframes and popup windows, and they run flawlessly. The
first release seemed to have a memory leak which wasn't present in non-
headless chrome, but that seems to have been fixed in version 60.

Even sped things up a little versus phantom.

~~~
kensoh
Hi bluepnume, if you are ok, can you share how you manage iframes and popups?
I'm using DevTools protocol through websocket directly to communicate with
Chrome. Have to do a lot of context handling (frames) and using
Target.sendMessageToTarget (popup windows) to deal with them. These 2 features
seem to be the harder to handle parts of the interaction layer when doing
automation with DevTools Protocol. Thanks in advance!

------
cestith
I'm a little concerned about the confusion in the market of another product in
the automation space called "Puppet"-something.

The project itself looks exciting.

~~~
brlewis
I can imagine using Chrome Puppeteer in conjunction with Google Puppeteer
[https://github.com/google/puppeteer](https://github.com/google/puppeteer)

The latter having been renamed from "puppet" when Google open-sourced it.

~~~
reificator
I believe they're referring to [https://puppet.com](https://puppet.com)

EDIT: Left the tab open too long, beaten to the punch by 20 minutes.

------
chis
It's funny how open the market for browser automation is. Everyone needs to
switch off phantomJS in the next few years and is looking for the easiest
option.

Sadly, there's probably no money on the line. But you will get your buggy
software used by huge corporations for years to come!

~~~
pimlottc
I'm not super familiar with this space, can you explain further why everyone
needs to switch from PhantomJS?

~~~
adamb_
Core developer stepped down:
[https://news.ycombinator.com/item?id=14105489](https://news.ycombinator.com/item?id=14105489)

~~~
kensoh
There was still some commits in early July but haven't seen new updates till
date. Maintaining a browser project is mammoth task. Even for an established
startup such as Segment you see that NightmareJS has lots of issues unreplied
for months. I guess it's primarily because of a potentially large user base
and all kinds of edge cases requirements from different users. And add on to
that, suppose to work on all OSes. Nightmare.

------
nailer
The important bit:

> Who maintains Puppeteer?

>

> The Chrome DevTools team maintains the library

------
borplk
Does anyone know if it's possible to take a screenshot of a specific DOM
element? (instead of the entire page)

~~~
paulirish
Yes using the `clip` option of screenshot(), you can get these dimensions from
an element's getBoundingClientRect().

~~~
ebidel
full example of screenshoting a DOM element:

[https://github.com/GoogleChrome/puppeteer/issues/306#issueco...](https://github.com/GoogleChrome/puppeteer/issues/306#issuecomment-322929342)

------
ohitsdom
The API looks nice and clean, but I'm puzzled by this from the FAQ:

> Puppeteer works only with Chrome. However, many teams only run unit tests
> with a single browser (e.g. PhantomJS).

Is this true? Do teams write unit tests but only test them in a single
browser? With test runners like Karma and Testem, running tests concurrently
in multiple browsers is easy. You'd be throwing away huge value if for some
reason you only decided to test in one vendor's browser.

~~~
hobofan
From my experience most teams only test against a single browser, yes. It's
however nice to have the option to switch to another browser when debugging a
browser-specific bug.

From what I've seen in practice, for a lot of teams and projects, running
tests concurrently across multiple browsers isn't easy as you claim. Once you
have a database involved and not a perfectly clean setup, concurrent testing
can become a hassle and is rarely worth the effort of fixing.

~~~
bsimpson
It's also quite easy to run Chrome and/or Firefox for free in CircleCI or
TravisCI. Safari and Edge are non-trivial to test in those environments. (I
think you can get Safari if you tell CircleCI you're an iOS app and hack your
way over to Safari from there, but I don't know if you can even test Edge in
the popular CI stacks.)

------
samwillis
Has anyone found something similar but for Python? The few I found all seemed
to be abandoned or too limited in capability.

~~~
ghetzel
I've been working on something similar (Headless Chrome via DevTools protocol)
called Webfriend. It is a Python wrapper to the DevTools protocol, as well as
a simplified imperative scripting environment which is specifically built for
ease of use by people with a technical-but-not-programming background
(lovingly called Friendscript).

It's by no means done, but it is functional and I'm hoping to see the project
grow beyond a toy if there's interest in the community.

Things to note:

\- Documentation is there, but there are gaps (especially w.r.t. the Python
API.) I'll eventually get around to wrestling with Sphinx, et. al., but have
not as of yet.

\- Targets Google Chrome / Chromium 58.x - 60.x. No testing outside of those
versions has occurred.

\- Is Chrome only for the moment, but may evolve to work with WebDriver and
other browser APIs in the future.

[https://github.com/ghetzel/webfriend](https://github.com/ghetzel/webfriend)

Also, PRs and issues are welcome, but my time to work on this is limited at
the moment (which does speak to the point made elsewhere about corporate
backing vs. individual maintainers, but this was built largely to scratch an
itch.)

~~~
kensoh
Thanks ghetzel for sharing, nice work! I saw that you've made your own
scripting language to make it user-friendly :)

Just went through your Friendscript syntax, the amount of work going into
defining that language is impressive....

------
jotto
> Crawl a SPA and generate pre-rendered content (i.e. "SSR").

I've been maintaining (thanks to this team and Headless Chrome) a convenience
API based on this feature. Some additional features:

    
    
      * React checksums for v14 and v15 (v16 no longer uses checksums)
      * preboot integration for clean Angular server->client transition
      * support for Webpack code splitting
      * automatic caching of XHR content
    

[https://www.prerender.cloud/](https://www.prerender.cloud/)

and for the crawling or to delegate your single-page app hosting and server-
side rendering entirely [https://www.roast.io/](https://www.roast.io/)

------
jakozaur
Great move! One of the biggest advantage of PhantomJS was easy to use high
level API.

------
equasar
How can this work on Cloud Functions/Amazon Lambda/Azure Functions without
installing the dependencies each time it has to run?

Haven't done anything before with those serverless approaches.

~~~
peterbaumgarten
It probably will not run on AWS Lambda as the version of Node required by
Puppeteer is too recent.

From puppeteer readme[1]

> Puppeteer requires Node version 7.10 or greater

From AWS website[2]

> AWS Lambda supports the following runtime versions:

> Node.js – v4.3.2 and 6.10.3

[1]
[https://github.com/GoogleChrome/puppeteer/blob/master/README...](https://github.com/GoogleChrome/puppeteer/blob/master/README.md#installation)

[2] [http://docs.aws.amazon.com/lambda/latest/dg/current-
supporte...](http://docs.aws.amazon.com/lambda/latest/dg/current-supported-
versions.html)

~~~
equasar
sight... I checked Azure Functions and Cloud Functions and they are running
6.x versions of node as well.

------
bofia
Are there any new python drivers for headless chrome? Selenium seems to still
be lacking several features exposed by the DevTools protocol (e.g. print PDF)

------
bdcravens
Has headless Chrome enabled file downloads yet? (I don't mean navigating to a
known url and saving content, but when a site pops open a save dialog)

~~~
kensoh
This issue is being tracked here and moving along fine -
[https://bugs.chromium.org/p/chromium/issues/detail?id=696481](https://bugs.chromium.org/p/chromium/issues/detail?id=696481)

Default behavior by design is to block automated headless downloads for
security reasons. But above issue tries to address this important use case.

~~~
kensoh
This's the Puppeteer issue tracking downloads -
[https://github.com/GoogleChrome/puppeteer/issues/299](https://github.com/GoogleChrome/puppeteer/issues/299)

------
ontouchstart
Using puppeteer to call google seach the keywords "chrome puppeteer will
return link to this HN page itself. :-)

This github page was generated by a markdown file created by a test.js running
in a Puppeteer docker container:

[https://ontouchstart.github.io/170817/home/puppeteer-
test/](https://ontouchstart.github.io/170817/home/puppeteer-test/)

------
martinald
Is there an equivalent of this for C#? I use Selenium a lot and find it
annoying. Be interested to see if this is much better.

~~~
paulirish
Under the covers this uses the DevTools protocol which has a number of
projects around it: [https://github.com/ChromeDevTools/awesome-chrome-
devtools/#c...](https://github.com/ChromeDevTools/awesome-chrome-
devtools/#chrome-devtools-protocol)

A C# binding was added just recently: [https://github.com/BaristaLabs/chrome-
dev-tools](https://github.com/BaristaLabs/chrome-dev-tools)

~~~
martinald
Yes, I understand it's just a protocol but it's a pain writing it by hand
especially when I just want to test it out on some existing projects Vs
selenium.

Thanks for the binding link. Will check it out this weekend.

------
commandertso
Pretty psyched to see this. A lot of my automated testing headaches came from
the intersection of using PhantomJS with transpiled code - which makes sense,
since the Phantom team was always forced to play catch-up with the browsers
being emulated.

------
boozang
Awesome! This will make the Boozang/CI integration much smoother. I was using
npm-headless-chromium before, but the Xvfb dependency was error-prone and
difficult to setup properly. Thank you very much!

------
barbolo
Would be nice if it was possible to change the proxy settings for each request
or for each session. Last time I checked it was only possible with the C API.

~~~
paulirish
The request interception API might be able to satisfy your usecases:
[https://github.com/GoogleChrome/puppeteer/blob/master/docs/a...](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pagesetrequestinterceptionenabledvalue)

Outside of that, Chrome doesn't currently support changing a network proxy
dynamically, let alone for individual (and overlapping) requests.

------
stephen123
Im looking forward to seeing the pdf printing features develop. They are
lacking a few of the formatting features of wkhtmltopdf.

------
lanceloth
Has anyone found something similar than Selenium Grid, but for managing
headless versions of Chrome? Thanks a lot!

------
borplk
This is really nice. Thanks for your work.

------
kusmi
Has anyone been able to get headless mode to work with socks5 proxy, yet?

------
qualitytime
Does anyone know if it supports screen capture for webgl?

------
boozang
Awesome! This will simplify the Boozang Jenkins integration significantly (I
was using npm-headless-chromium before, but there were many manual steps, and
the Xvfb dependency was annoying). Thank you very much!

------
hartator
I wonder if anyone know a Ruby equivalent?

------
whipoodle
Very cool. In a previous job we ran our JS unit tests on PhantomJS, using a
tool that allowed arbitrary code to be piped to Phantom for execution and
redirected Phantom's console to stdout. (We did all this so our tests ran on
something resembling a real browser, as opposed to jsdom.) Something similar
should be possible with this.

------
Akujin
I think simple-headless-chrome is more far along
[https://github.com/LucianoGanga/simple-headless-
chrome](https://github.com/LucianoGanga/simple-headless-chrome)

~~~
paulirish
We're fans of LucianoGanga's project (and of Chromeless, Doffy, Chrominator,
Chromy, Navalia). I can tell you from personal experience that dealing with
the raw DevTools Protocol isn't ideal for a developer writing an automation
script, so it's clear there's demand for libraries with this higher-level API.

Would love to know if there's a feature parity concern you have or what you'd
like to see from puppeteer (or any of these projects). (My personal feature
unicorn: generate not a static screenshot but a proper video of a page
session)

~~~
malsheikh
Chromeless allows for the setting of cookies. I noticed in an issue for
Puppeteer that it wasn't seen as a high priority. Our use-case (session based
cookies) rely on a cookie being set in order for our PDF screenshots to work.

Any chance you could give us some insight on when this may be available?

~~~
paulirish
One of the puppeteer devs tells me he started on cookies last night. You'll
see a PR soon. :)

~~~
malsheikh
Awesome! So stoked on this. After some Phantom headaches we're really excited
to transition to Chrome Headless.

------
megamindbrian
Can you help with one of the existing API's that does the same thing instead
of inventing a new one?

~~~
kensoh
Actually the release of Puppeteer is a really exciting development. I've been
waiting for some time for something like this to happen. We've seen what
happened to PhantomJS (almost 2k open issues and main maintainer stepping down
without a successor), NightmareJS (lots of unreplied issues for months,
probably the project is not a strategic part of Segment) and so on. In theory
it is great for an individual or an established startup to drive a web browser
automation project. But in reality, the scope of web browser automation simply
gets out of hand very quickly. There are just too much edge cases to support
for a fast-changing domain.

Being driven by a large commercial entity actually has a chance of making it
work out. With the browser automation tool and the browser dev team being one
team, there can be synergies not possible otherwise. When I spoke to CasperJS
creator some time ago, I can understand why there will be burnout. Referring
to the popular Chromeless project launched less than a month ago, there are
already 150+ new issues and 100+ still open, and they already have enough
pipelines for a few releases ahead. It can be a nightmare to manage.

There's just too many needs from a large user-base for such projects. I'm
speaking from the context of test automation and general browser automation.

~~~
chrisweekly
Yep. Pat Meenan's herculean / ultramarathon support of WebPageTest is a
remarkable exception. (Speaking of WPT, after just a cursory glance at this
thread on my phone prior to thumbing this comment, it surprisingly hasn't been
mentioned yet? shrug)

~~~
kensoh
I just recently got to know WebPageTest. It even has scripting abilities! I'm
just surprised why the project didn't enter into mainstream (in the sense that
an average test automation guy like me will know).

