> When you install Puppeteer, it downloads a recent version of Chromium (~71Mb Mac, ~90Mb Linux, ~110Mb Win) that is guaranteed to work with the API.
A lot of the chrome interface libs about at the moment require you to maintain your own instance of chrome/chromium and launch the headless server with your command line, or require a pre compiled version, that can quickly get out of date (https://github.com/adieuadieu/serverless-chrome/tree/master/...). Having this taken care of is a blessing.
Normal chromium requires a bunch of X libraries be present. It doesn't use them, but for things like headless testing, it's a massive pain, since the apt-get (or equivalent) is generally many hundreds of megabytes.
That said, headless still has a mountain of bugs open against it, so most parties will be best suited by sticking with ChromeDriver for now.
Here's a quick hack: https://gist.github.com/rcarmo/cf698b52832d0ec356c147cf9c9ad...
I'm using The Verge for testing because it lazy loads images, and am being clumsy about the scrolling, but it mostly works - I can get 90% of the images to show on the finished PDF.
What I can't seem to get right, though, is creating a single-page PDF where the page height matches the document height perfectly - it always seems to be off by a bit, at least on this site (mine works fine, _sometimes_).
Anyone got an idea of why this is so?
In that context, it's dead-simple to use, and someone with very little experience should be able to get a working prototype in under 5 minutes.
For my use case, it's closer to "wget/curl with JS processing" than "automating a user's browsing experience". I don't particularly care which browser is doing the emulation, with the ease-of-use of the API making the biggest difference.
It seems very similar to PhantomJS, but to be honest, it's more attractive from an ongoing-support standpoint simply because it's an official Chrome project.
If you have an existing web service, this appears suitable for actual production usage to deliver features like PDF invoices and receipts, on-demand exports to multiple file formats (PNG/SVG/PDF) etc., which has quite different requirements compared to an automated testing framework.
headless-chrome would be used for the functionality of a server-side microservice, rather than for automated testing of UI/UX, there are already more appropriate projects to achieve that.
As someone with a bunch of tooling around the dev-tools API, it's a huge pain in the ass to not be able to tell what functions the remote browser supports. There's a version number in the protocol description, but it's literally never been incremented as far as I've seen.
It should be possible to get the running chrome version and to do feature detection over this protocol.
If it isn't, it's a bug
For an example of an impossible task, try to retrieve request headers using Selenium. Browser automation getting more and more complicated the more cases you're trying to cover and in my impression WebDriver is definitely not enough. Who knows, perhaps some new version of WebDriver that I never heard of it will catch up once the functionality gets properly defined.
Given that the selenium leadership is apparently uninterested in improvements, and it's many limitations, trying to improve there is more effort then it's worth.
Basically, they're still stuck in this idea that they're ONLY for emulating user-available input (and the dev-tools don't exist).
In reality, there is tremendous interest in more complex programmatic interfaces, but apparently they're unwilling to see that, and are instead only interested in their implementations "user-only" ideological purity.
Selenium can't do that itself but Selenium can drive a browser that uses a proxy and you can retrieve everything about the request from that. It's a lot more challenging if you're testing things that use SSL but it's not impossible with a decent proxy app (eg Charles).
Due to the nature of browser plugins, they could only access information that the browser made available. Also, the dev's have consistently been of the opinion that they only care to simulate an end-user experience. (End user's only care about the webpage's presentation, not which headers were present per transaction.) Although, I suspect that early restrictions influenced that viewpoint.
It wasn't until much later that browsers started implementing their own automation channels; Chrome's Automation Extension and Debugging Protocol, Firefox's Marionette, etc. At the same time, browsers started putting additional security measures around plugins making it even more difficult to have consistent features across Selenium's drivers.
Which is why the WebDriver became an open specification instead various driver implementations. I believe, Microsoft was the first to implement their own driver, InternetExplorerDriver, for IE7+. Then ChromeDriver (powered by Chrome Automation Extension), GeckoDriver (Firefox translation to their Marionette driver), and SafariDriver is now baked into Safari 10.
I think for most developers ("devs", "qa", "scrapers", etc.) there's very little appeal in moving away from Selenium because it would require maintaining multiple test suites. It gives consistent results and just works. If you want lower level information, it's fairly simple to either 1) just use a CLI client (curl, wget, etc.), or 2) a library libcurl, Requests(Python), Net::HTTP(Ruby), etc. etc. etc., or 3) setup proxy server. I do all of the above, each has it's own downsides clients and libs don't do any rendering themselves and proxies tend to rewrite transactions (ex. strip compression and add/remove/alter headers 'Content-Type: gzip').
But resonated with your point that there are just so much codebase / automation assets already written. Usually exploring new tools happens when a new project happens, rather than recoding entirely an existing project. For the existing code base, unless contributors from the community writes a parser to translate those to Puppeteer API?
Even sped things up a little versus phantom.
The project itself looks exciting.
The latter having been renamed from "puppet" when Google open-sourced it.
EDIT: Left the tab open too long, beaten to the punch by 20 minutes.
Sadly, there's probably no money on the line. But you will get your buggy software used by huge corporations for years to come!
> Who maintains Puppeteer?
> The Chrome DevTools team maintains the library
> Puppeteer works only with Chrome. However, many teams only run unit tests with a single browser (e.g. PhantomJS).
Is this true? Do teams write unit tests but only test them in a single browser? With test runners like Karma and Testem, running tests concurrently in multiple browsers is easy. You'd be throwing away huge value if for some reason you only decided to test in one vendor's browser.
From what I've seen in practice, for a lot of teams and projects, running tests concurrently across multiple browsers isn't easy as you claim. Once you have a database involved and not a perfectly clean setup, concurrent testing can become a hassle and is rarely worth the effort of fixing.
Full disclosure: I'm maintaining the image.
It's by no means done, but it is functional and I'm hoping to see the project grow beyond a toy if there's interest in the community.
Things to note:
- Documentation is there, but there are gaps (especially w.r.t. the Python API.) I'll eventually get around to wrestling with Sphinx, et. al., but have not as of yet.
- Targets Google Chrome / Chromium 58.x - 60.x. No testing outside of those versions has occurred.
- Is Chrome only for the moment, but may evolve to work with WebDriver and other browser APIs in the future.
Also, PRs and issues are welcome, but my time to work on this is limited at the moment (which does speak to the point made elsewhere about corporate backing vs. individual maintainers, but this was built largely to scratch an itch.)
Just went through your Friendscript syntax, the amount of work going into defining that language is impressive....
I've been maintaining (thanks to this team and Headless Chrome) a convenience API based on this feature. Some additional features:
* React checksums for v14 and v15 (v16 no longer uses checksums)
* preboot integration for clean Angular server->client transition
* support for Webpack code splitting
* automatic caching of XHR content
and for the crawling or to delegate your single-page app hosting and server-side rendering entirely https://www.roast.io/
Haven't done anything before with those serverless approaches.
From puppeteer readme
> Puppeteer requires Node version 7.10 or greater
From AWS website
> AWS Lambda supports the following runtime versions:
> Node.js – v4.3.2 and 6.10.3
If you are familiar with Docker you can think about Lambda image as a small Docker image, while real work will be done in instances (Docker containers) created from this image.
Usual scenario is to provide ready to go image (with all source code, npm packages being installed, with Chrome Headless plugin, etc). Then AWS/Azure will run VM instances based on the image for almost every function request. Most of the time spinning such lite VMs takes no more than a couple of seconds.
This github page was generated by a markdown file created by a test.js running in a Puppeteer docker container:
Default behavior by design is to block automated headless downloads for security reasons. But above issue tries to address this important use case.
A C# binding was added just recently: https://github.com/BaristaLabs/chrome-dev-tools
Thanks for the binding link. Will check it out this weekend.
Would love to know if there's a feature parity concern you have or what you'd like to see from puppeteer (or any of these projects). (My personal feature unicorn: generate not a static screenshot but a proper video of a page session)
And here is the thread I'm gonna use to discuss this:
If someone has an idea of what the "ideal implementation" would look like, I'd really appreciate it!
Any chance you could give us some insight on when this may be available?
Outside of that, Chrome doesn't currently support changing a network proxy dynamically, let alone for individual (and overlapping) requests.
Being driven by a large commercial entity actually has a chance of making it work out. With the browser automation tool and the browser dev team being one team, there can be synergies not possible otherwise. When I spoke to CasperJS creator some time ago, I can understand why there will be burnout. Referring to the popular Chromeless project launched less than a month ago, there are already 150+ new issues and 100+ still open, and they already have enough pipelines for a few releases ahead. It can be a nightmare to manage.
There's just too many needs from a large user-base for such projects. I'm speaking from the context of test automation and general browser automation.