
Running feature specs with Capybara and Chrome headless - TimPetricola
https://drivy.engineering/running-capybara-headless-chrome/
======
thibaut_barrere
Earlier this month I wrote an ETL extractor using Capybara & headless browser
(to work-around a lack of API - PS: only do that as a last resort!).

One thing I learned and wanted to warn about here is that Chrome headless
currently doesn't support file downloads [1].

PhantomJS won't work either (unless you use a custom build) [2].

I also tried with capybara-webkit, but no luck either [3].

The only driver which ultimately allowed me to download files at this point
was Selenium + Firefox (with some tweaking on profile options
browser.download.dir, browser.download.folderList = 2,
browser.helperApps.neverAsk.saveToDisk set to the MIME type of files to be
downloaded).

[1]
[https://bugs.chromium.org/p/chromium/issues/detail?id=696481](https://bugs.chromium.org/p/chromium/issues/detail?id=696481)

[2]
[https://github.com/ariya/phantomjs/issues/10052](https://github.com/ariya/phantomjs/issues/10052)

[3] [https://github.com/thoughtbot/capybara-
webkit/issues/691](https://github.com/thoughtbot/capybara-webkit/issues/691)

~~~
vadansky
>One thing I learned and wanted to warn about here is that Chrome headless
currently doesn't support file downloads [1].

Not true. At least with C# You can use

    
    
                var client = new WebClient();
                client.Headers[HttpRequestHeader.Cookie] = cookieString(driver);
                client.DownloadFile(reportURL, savePath + fileName + ".xlsx");
    

Along with:

    
    
                string cookieString(IWebDriver driver)
                {
                    var cookies = driver.Manage().Cookies.AllCookies;
                    return string.Join("; ", cookies.Select(c => string.Format("{0}={1}", c.Name, c.Value)));
    
                }
    

Edit: Just wanted to emphasize that this ensures all the steps you did to
login won't be lost since you're using the logged in cookie. The idea is to
use Selenium to get the download url and then use a regular download method
using the selenium cookie.

~~~
thibaut_barrere
Yes - I could have done that on other cases, but not in the specific case I
tracked, where I don't know the report URL at all, nor I can construct it
(it's generated by some enterprise app / javascript code). I think the bug
report I mentioned relates to that.

(otherwise you could use pretty much whatever you want to download the file,
indeed).

------
mabbo
Chrome headless migration has been my bane for the last two weeks. It's
_almost_ ready, but not yet.

The short version of what's wrong: chrome headless does not support
extensions. The chromedriver used by selenium works via starting chrome with
an automation extension that gives it certain needed functionality. So things
like setting the window size, taking screenshots, and a few other important
features just cause chromedriver to crash or hang.

There's bugs filed and some clever googlers are trying to fix it all- but it's
not ready for our purposes yet. I don't want to rewrite dozens of tests that
work with phantomjs to get around chrome's bugs.

~~~
TimPetricola
Indeed, some features are not supported yet (setting size, alerts, ...). We're
lucky not to use too many of them in our actual test suite!

~~~
MikeKusold
Just implemented headless testing for a rails app today, and discovered that
alert support has been worked around in the latest Capybara (2.14.4).

[https://github.com/teamcapybara/capybara/commit/12c100597093...](https://github.com/teamcapybara/capybara/commit/12c10059709346bfeadb2cea18f510174d842303)

------
mrskitch
I hear a lot of folks mentioning the pain working with headless chrome. This
was the inspiration for Navalia, which is inspired by the API from nightmare.
It also handles the case where you want to run simultaneous scripts by using
new tabs (and even multiple chrome instances as well). It's still early in
life, and I'm about to do another breaking release, but here is the repo
[https://github.com/joelgriffith/navalia](https://github.com/joelgriffith/navalia)

------
wingerlang
Kind of off topic, but. Is running automation as complicated as this? I
recently wanted to log in to a website, click some page and download a .csv
file. I saw that chrome can be run headless, nice.

So I opened it headless and there's this REPL, nice I can do JS directly into
the website. Now how do I automate this.

This leads me into all sorts of things that doesn't seem related at all (I is,
but still). Selenium stuff, automation setups, drivers, language bindings,
chrome-api-stuff and no end in sight.

All I want is something like "chrome -headless -js script-flow.file
[http://URL"](http://URL")

Am I to overwhelmed or is there no simple way without buttloads of 3rd party
tools and setups required?

~~~
Dirlewanger
What does headless Chrome provide that a web scraper in any given language
can't?

~~~
wingerlang
I was hoping it would let me automate a few tasks with a "user" flow (i.e.
enter details, click, click). With e.g. curl or python I didn't even get
through the login screens because they seem to require some special dealing
with cookies, request/response cookies, smfd, itc, id_ado, _ip_xat and so on.

Basically auth on websites seems to require a whole bunch of stuff and I just
started looking at simpler forms of automation instead. Still not sure which
one I'll continue looking into.

(If anyone sees this, I am trying to log into iTunes Connect and Fabric and
download metrics)

------
sparaker
"It is also supposed to improve memory usage and stability"

Did you do a comparison of memory against phantomJS?

~~~
royka118
Just this morning my team migrated from phantomJS to chrome headless for our
Karma plus Jasmine tests. Did a bit of researching before the move and we
found tests ran faster plus chrome used less memory than phantomJS. ~ 2000
specs compiling html and stuff

~~~
rgbrgb
Any guess at how much faster the tests are running?

------
waffle_ss
I tried switching from poltergeist to Chrome headless last week using a very
similar setup, but it was much, much slower. I assumed it was Selenium's
fault, but didn't dig any deeper as it seemed to be expected from the guide I
was using[1] ("I can anecdotally report that Capybara-Webkit seems
significantly faster").

[1]: [https://robots.thoughtbot.com/headless-feature-specs-with-
ch...](https://robots.thoughtbot.com/headless-feature-specs-with-chrome)

------
lloeki
Wow, I completely missed the announcement that the maintainer and sole
developer of PhantomJS is stepping down.

------
jayflux
Poltergeist had a network_traffic mode which we found quite useful. (Seeing if
requests were made etc)

Does Chrome headless have an equivalent? It's the only thing preventing us
from moving across. And if not where should I raise it?

~~~
rberdeen
Headless Chrome is controlled using the Chrome DevTools protocol, which has
quite a few options for tracking and modifying network requests.

Everything you get in the Network inspector in Chrome DevTools should be
available in Headless Chrome.

[https://chromedevtools.github.io/devtools-
protocol/tot/Netwo...](https://chromedevtools.github.io/devtools-
protocol/tot/Network/)

------
zimbatm
When specs are failing you actually want to see what's happening in the
browser. Xvfb + normal Chrome + ffmpeg screen capture. This can be run in
Docker and thus on Macs as well.

------
Kiro
> To prevent some issues in PhantomJS when elements would overlap, we had a
> lot of calls like this:

I don't understand. How would you do it when it's not overlapping?

