
A list of headless web browsers - fanf2
https://github.com/dhamaniasad/HeadlessBrowsers
======
DonHopkins
Shouldn't Chromium Embedded Framework be on this list? [1] That seems to be
robust, well maintained and widely used by many projects [3], uses a recent
version of Chrome, and it supports off-screen (non-GPU-accelerated) rendering.
[3]

I haven't used it myself, but I'm considering it, so I'd appreciate hearing
other people's thoughts about it.

[1]
[https://bitbucket.org/chromiumembedded/cef](https://bitbucket.org/chromiumembedded/cef)

[2]
[https://en.wikipedia.org/wiki/Chromium_Embedded_Framework#Ap...](https://en.wikipedia.org/wiki/Chromium_Embedded_Framework#Applications_using_CEF)

[3]
[https://bitbucket.org/chromiumembedded/cef/wiki/GeneralUsage...](https://bitbucket.org/chromiumembedded/cef/wiki/GeneralUsage#markdown-
header-off-screen-rendering)

~~~
to3m
It's a strange omission given that CEF is free and seemingly in rude health
while Awesomium, number 1 on the list (if only by dint of its name), looks
dead, even though you have to pay for it.

I'm using it for off-screen rendering. It's pretty straightforward, once past
the inevitable initial hump. The code is sensible (if a little smart pointer-
happy), and you're provided with a good range of hooks. Documentation is OK.
Nothing you can't figure out if you've got a debugger...

(I used Visual Studio with the prebuilt Spotify builds. This addin really
helps:
[https://visualstudiogallery.msdn.microsoft.com/a1141bff-463f...](https://visualstudiogallery.msdn.microsoft.com/a1141bff-463f-465f-9b6d-d29b7b503d7a))

~~~
wyldfire
I made a PR [1] based on your endorsement. EDIT: it's merged. Yay, we did it
HN!

[1]
[https://github.com/dhamaniasad/HeadlessBrowsers/pull/16](https://github.com/dhamaniasad/HeadlessBrowsers/pull/16)

~~~
to3m
You can add "open source contributions" to your CV now ;)

I'm going to add "thought leader".

------
joefarish
It's obviously not a headless browser but xvfb can be very useful depending on
what exactly one is trying to test:
[https://en.wikipedia.org/wiki/Xvfb](https://en.wikipedia.org/wiki/Xvfb)

~~~
bryanlarsen
Xdummy is another headless X display that has a more modern, more complete
stack.

[https://xpra.org/trac/wiki/Xdummy](https://xpra.org/trac/wiki/Xdummy)

~~~
reacweb
TIL. I did not know Xdummy. I have just followed your link. It is trivial to
launch. It avoids installation of vnc, has a better compatibility and is
already present everywhere.

------
VeejayRampay
I really wish we had a good set of scriptable browser engines (i.e. everything
but the graphics part) for all browsers. That would make testing so much
easier.

Even simple tasks like taking a screenshot of a given web page involves
PhantomJS, which seems really heavy-handed.

Thanks for compiling this list but the issue I'm having with this is the lack
of a rating system based on usage, Github stars, downloads, repo activity,
project creation, corporate support, etc. to get a measure of how reliable
those tools are.

~~~
icebraining
The problem is that to take a screenshot of how the page actually looks, you
need almost everything from the browser. Rendering of HTML/CSS/images/videos,
Javascript execution, networking, cookies... you end up with 90% of the
browser, and then removing the last 10% is not worth it.

------
mbrock
Here's a thing I made to script WebDriver from the terminal or shell scripts:

[https://github.com/mbrock/wd](https://github.com/mbrock/wd)

~~~
jalfresi
That's really cool! Like it a lot!

~~~
mbrock
Thanks! I made it to automate some tasks involving saving receipts to files
and so on. Using the shell as a REPL and being able to use the shell's
functions for working with files and texts is why I like it.

------
riston
Most of the tools listed use PhantomJS or QT core browser as the base. The
problem is that PhantomJS doesn't support many new API-s or the rendering is
different.

There is also headless chrome browser in development, not sure if there are
any prebuilt binaries out there?

~~~
sebcat
I've been using chromium-headless for certain automated tasks, and I've been
getting a lot better results than from PhantomJS in terms of correctly
rendered resources, resource consumption and execution speed. If your business
depends on rendering engines and you're using PhantomJS, chromium-headless is
probably something you want instead.

I build from chromium src, and my code is more or less an extension with its
own BUILD.gn with a deps on //headless. Once you've built the initial tree
(took a night for me), builds will be really fast.

Most of the interaction with the browser is by the chrome debugging protocol,
as with "normal" chromium. Generally speaking, you want to target that,
instead of the internal APIs.

There are some tweaks you need to be aware of. Chromium needs a homedir, so
you want to configure that. There's state stored between sessions, so if you
want process invocations stateless, consider spending some time hacking or run
the process in a fresh namespace/chroot/docker/jail/whatever each time. I had
to write my own protocol handler and URLRequestJobs because I render pages
from archives on disk and that was the easiest way to get that started at the
time.

I don't know if vanilla chromium supports it, but I'm pretty sure I've seen a
--headless flag somewhere... If not, headless_shell is pretty easy to build
following the official build instructions[1]. I'm running on
RHEL/Fedora/CentOS. There's also unofficial images on docker hub if you want
to go that route.

[1]:
[https://chromium.googlesource.com/chromium/src/+/lkgr/headle...](https://chromium.googlesource.com/chromium/src/+/lkgr/headless/README.md)

~~~
boundlessdreamz
What about startup time? phantomjs is the only headless browser I have tested
which starts up relatively fast.

~~~
sebcat
Can't test now, but around a few tens of milliseconds for about:blank

------
dom96
Anybody else seeing something similar to this?
[https://i.imgur.com/Dz1FLGQ.png](https://i.imgur.com/Dz1FLGQ.png). It's as if
GitHub was embedded in an iframe on this repo's page.

~~~
dom0
Ditto. Probably someone messed up some piece of dynamically loaded server-side
rendered stuff in the repository info bar. Since not everyone seems to have it
it's probably not on full-roll-out, but just a small part of the user base.

------
konradzikusek
Chrome will officially support headless mode:
[http://crbug.com/546953](http://crbug.com/546953)

------
k__
The problem I found with this was mostly that there are bugs that only show in
one browser (only FF, only Chrome, only Edge, etc) so these tools get their
own share of bugs.

At the moment I use Ghostinspector (a SaaS) which works okay. Their UI is a
bit clunky, but recording stuff via Chrome extension and modifying it later
works okay. Also they let me run the tests with Firefox later. So I develop
with Chrome and run tests with Firefox.

------
duke360
i've tried some of these for a project ( i had to _cough_ automate download of
files from a site made in old asp.net and with a lot of new js on it), only
awesomium was able to do the job, but, man, it was overwelming difficult! i
had to inject fiddler as proxy to catch exactly what was happening, awesomium
by itself just can't do it, we are very far from a real scriptable headless
browser.

~~~
DonHopkins
Unfortunately, Awesomium seems to have been in "lights on, nobody home" mode
for over a year now, and uses an outdated version of Chrome. [1] The nail in
the coffin is a predictably unfulfilled promise (see Yannou's rule #1 and #2)
followed by radio silence: [2]

"The product is very well alive. Version 1.7.6 will be released in a few days
with many new features and an updated Unity plugin as well. Meanwhile, we are
also working on our next major version (v2.0)." Perikles C. Stephanidis, Jun
15

The other big problem is that it's not free. So nobody is going to pick it up
and run with it.

[1] [http://answers.awesomium.com/questions/6880/does-the-
project...](http://answers.awesomium.com/questions/6880/does-the-project-is-
still-supported-developed.html)

[2] [http://answers.awesomium.com/questions/7255/is-awesomium-
dea...](http://answers.awesomium.com/questions/7255/is-awesomium-dead-
now.html)

------
matthewhall
What about Electron? Multiple people, including myself, have used it to make a
web browser

~~~
m0th87
fwiw, nightmare is just a thin layer above electron's browser.

~~~
OtterCoder
I'm currently using Nightmare for a little automation. It's fantastic. A few
small gotchas, but so easy to set up.

------
natch
Missing: lynx, links (both have headless mode), curl, wget, nscurl.

~~~
skinnymuch
Pretty sure being able to render JavaScript is needed to be on the list. So
not most of what you listed.

------
nathancahill
It's missing the browser engine and version that each of them use. Even latest
PhantomJS only uses Chromium 40. Most important metric IMO.

~~~
recrof
isn't PJS based on webkit?

~~~
nathancahill
Same same

------
equasar
Would be nice to include if is it maintained or their last update. There are a
lot of them that aren't updated in years.

------
tzaman
I know it's somewhat a vanity metric, but this list should have GitHub stars
listed for each project. Would save us some work :)

~~~
r3bl
Unless someone creates a bot that will automate the process of updating the
number of stars, it's a cat and mouse game.

~~~
jpalomaki
Maybe something like this would be an easy start:
[http://githubbadges.com/](http://githubbadges.com/)

