A shameless plug: https://github.com/scrapinghub/splash is an open source tool similar to Phantom JS, and it has an interactive web editor built-in.
The concurrency model in Splash scripts is not the same as in Phantom JS scripts: Splash allows to use loops/ifs/functions in async code without recursion tricks or things like "casper.thenOpen". See http://splash.readthedocs.org/en/latest/scripting-tutorial.h... for more info.
We haven't done a public announcement, and Splash is not yet feature complete, but it is used in production for more than a year, there are hundreds of unit tests and it is in active (though a bit sporadic) development - so give it a try if you want something different :)
Looks nice! I created a service called Ghost Inspector last spring that runs PhantomJS under the hood: https://ghostinspector.com/
It's not open source, but there's a free tier and lots of nice features built in like video recording, screenshot comparison and a GUI editor.
If anyone is planning to write PhantomJS scripts directly, I strongly recommend checking out CasperJS: http://casperjs.org/ -- It gives you a really nice layer on top that makes Phantom much more wieldy.
There's also Automately (http://automate.ly/) it's a pretty interesting platform and it also allows for use of browser automation but differently than PhantomJS it's a little confusing to get setup at first but once you're there it's easy from there on.
We will actually be open sourcing our core very soon! We will also be including the source code for our SDK and Automately Pi (unreleased Raspberry Pi integration).
It's actually more than that, our API doesn't only provide access to the Browser module. We provide an SDK that can help people build real-time applications that can connect with almost anything. We're getting ready to release our Raspberry Pi integration. We are also going to be going open source fairly soon.
We've basically created a connected platform that makes it very easy to connect things. You can have your web application talk to your server-side automately scripts. We've made it as simple as including some JavaScript on your web page. Our server-side APIs make it very easy to communicate with APIs and WebSites. Your Automately scripts can store data via the DataBus and they can talk to other execution jobs via the MessageBus.
interesting idea! however, I am struggling to see when and how someone could use this, because almost always a developer such as myself would simply create an endpoint for something like this directly on the server.
It would be interesting to see what type of customers or use cases your solution might be of use.
I am not fully sure what you mean but if you mean running the automately stack on your own servers then you would be interested in automately-core our open-source core. Shoot us an email at open@automate.ly.
I was under impression PhantomJS 2.x is rather active, but of course Splash contributions are welcome, that'd be great!
Splash Web UI is quite similar to PhantomJS Pad UI - by default it shows a CodeMirror text editor (with a simple autocompletion) and a screenshot of the webpage. Also, by default it shows information about requests/responses (like a "Network" tab in Firefox/Chrome development console) and a textarea with rendered html.
PhantomJS Pad UI ideas are good; tabs for images, an editor window which is open by default and a field with console logs make a lot of sense.
Since the formal semantic failed to become massively used, everyone is trying to make sense of the Web using scraping. It's something most viable sites should want to get behind, from ecommerce where you can find the best price faster to education. But I suspect if a very good shared scraping system were created we'd see countermeasures pretty fast, from timid content producers, as well as the Googles who want to have exclusive access to this level of information. Yet it's a natural layer for a shared commons, and what can be built on that, when finally get past the mess of arbitrary representations, is very exciting. Anyone know of any standard to create interoperable scrapers? In Javascript, of course. Here's one submission: https://github.com/zotero/translators
Neat. Did you have a real use-case for this, or just fun? I'm curious, what dependencies does it have that would make setup difficult?
In the Yelp example, I was confused for a second by the variable name of 'pizza' when doing a Yelp search on seafood. Renaming that would make the example more clear. :)
What does the browser-view show? I didn't try, but if the code traverses multiple locations, is it a screenshot of the last step?
I was curious with this example, as I have had zero luck getting webfonts to render using PhantomJS so far. Have you had any success rendering webfonts?
Hi. Small CodeMirror-related nitpick. If you set your editor to be the height of the panel it is in, and let it take care of its own scrollbar, it'll look better (the gutter will span the whole panel) and be more efficient in case of big documents (it'll render only the visible part).
The concurrency model in Splash scripts is not the same as in Phantom JS scripts: Splash allows to use loops/ifs/functions in async code without recursion tricks or things like "casper.thenOpen". See http://splash.readthedocs.org/en/latest/scripting-tutorial.h... for more info.
We haven't done a public announcement, and Splash is not yet feature complete, but it is used in production for more than a year, there are hundreds of unit tests and it is in active (though a bit sporadic) development - so give it a try if you want something different :)
Installation is just "docker pull scrapinghub/splash" - see http://splash.readthedocs.org/en/latest/install.html.