For anyone looking for a non-Python way of doing this, PhantomCSS will achieve nearly all of this. The main difference between the two is that Huxley seems to listen and re-perform actions, instead of needing them to be scripted.
We've had PhantomCSS running on a living style-guide/style tile on check in and can even fail the build on TeamCity. It's resemble.js under the hood to do the pixel-by-pixel comparison and will spit out a percentage difference, fail screenshots etc.
Just to clarify, it's written in Python but you don't have to write any Python code to use it. Just a teeny config file that contains a test name and a URL.
Nifty project. It'd be nice to add in automated analysis and make it a full Web Consistency Testing tool (disclaimer: a project I work on a lot): http://webconsistencytesting.com/
The three major problems with screenshot based tools are: 1) hard to adapt to acceptable levels of element shifting; 2) hard to extract semantic understanding of what broke; and 3) if it's not automated and you test a lot of pages in a lot of browsers, they simply won't get compared. But adapting this shouldn't be too hard.
Looks quite awesome! It would be helpful to include some installation instructions for those of us unfamiliar with python libraries... do I use pip? do I use easy_install? do I git clone and then do something with setup.py?
Would anyone pay for this as a service? i.e. you enter a bunch of urls and screen sizes, then get sent nice emails with request stats, screenshots and pdiffs with easy integration into your deployment workflow.
It writes a .huxley dir that contains screenshots and a log of user actions. It can replay these user actions programmatically and if the screen shots have changed it will save new ones to disk. You can use a tool like Kaleidoscope to compare the old ones with the new ones (we use git to keep the old ones around).
When you commit, the tests will be updated and the new screenshots will be part of the code review.
Additionally, you can run it in continuous integration mode which will break the build if screenshots are out of date.
Just to clarify, if I want to run this as part of travis then I should have the screenshots under source control and when I intentionally change the UI I should run it locally and push the new screenshot as part of the commit but when I don't intend to change the UI it will fail the build if something changes.
We've had PhantomCSS running on a living style-guide/style tile on check in and can even fail the build on TeamCity. It's resemble.js under the hood to do the pixel-by-pixel comparison and will spit out a percentage difference, fail screenshots etc.
EDIT: Link added https://github.com/Huddle/PhantomCSS