
Show HN: Appraise – Visual test automation - adzicg
https://github.com/AppraiseQA/appraise
======
adzicg
Author here - Appraise is a new opensource tool for visual test automation.
It’s like FitNesse, but for visuals, and working from Markdown that shows up
nicely in GitHub repositories.

Appraise can help you:

\- Automate acceptance/regression tests for visual look and feel in a visual
language, rather than xUnit style code

\- Review and approve changes to web pages, visual layouts and browser
components quickly through visual inspection

\- Publish easily maintainable/verifiable developer docs with visual examples
to Github easily (markdown) or as a static site (html)

\- Start Spec by Example/BDD from a sketch (hand-drawn, wireframe, or from a
graphic tool), easily compare actual outcomes, then just approve the final
result to make a regression test

In the near future, Appraise will also help you:

\- Speed up visual exploratory testing by making it easier to rebuild visual
components from example data/configuration

\- Run visual tests quickly/in parallel using AWS Lambda

This is another tool we built for MindMup, and we used it for the last few
months internally, to assist with visual checks and test automation. It helped
us move away from tedious layout tests that were very difficult to maintain
for small changes, but that would be visually easy to approve or reject.
Appraise makes visual tests easy to compose and maintain, by letting people
describe what they want something to look like (with a drawing, photo,
screenshot or a wire-frame), instead of describing how to test it (eg 10
pixels left, this css color). Here it is in action, preventing me from
introducing a stupid layout bug: [https://gojko.net/assets/appraise-
screenshot.png](https://gojko.net/assets/appraise-screenshot.png)

Under the hood, Appraise uses headless Chrome to take screenshots, so it can
inspect anything that Chrome can render.

The initial release of Appraise is alpha-quality. It covers our key usage
scenarios, so we think it is ready for some nice community feedback. It’s
useful to us, let’s make it useful to you together.

We’ve designed it to be extensible and pluggable, so other ways of executing
fixtures, taking screenshots or processing results should be easy to bolt on
to it. We’re releasing it under the MIT license, and the code is on GitHub.

~~~
chrisweekly
Bravo! Sounds fantastic. Curious whether you’ve considered possible paths to
integration with tools like WebPageTest.

------
nathan_f77
Nice! I built a very similar prototype a few years ago when I was working on a
mobile app. My test suite would upload screenshots, and I would have a gallery
where I could see all the screenshots that had changed. It was incredibly
useful, and I wanted to turn it into a side-project or startup. But I was
always worried that CircleCI or TravisCI would add the feature and make my
service obsolete. I think I was wrong to worry about that though. Percy
launched a little while ago and seems to be doing really well:
[https://percy.io](https://percy.io)

My new startup has a very large number of competitors, and I've learned the
lesson that competitors are generally a good thing.

Appraise looks much more polished and ready to use. I'll definitely be trying
it out for my current project. I had a bad UI bug recently where I forgot
about scrollbars on Windows (I used offsetWidth instead of scrollWidth.) I
would like to take a lot of screenshots on different browsers and platforms,
to make sure I don't have any regressions. I'm going to look into using
Appraise for that.

~~~
hobofan
I've actually started building a side-project (imagetest.io, the landing page
is still pretty bare-bones) in the visual regression testing space, because I
didn't couldn't find percy.io or Applitools (the two main services right now)
after quite some searching. Once I discovered the others, I was already too
into the project to abandon it, and I also think that there is quite some room
for improvement in the space.

If you are interested in testing it out, you can reach out to me at the e-mail
in my profile. I'm especially interested in making it work for use-cases that
a lot of developers have to solve, but are somewhat underdocumented right now,
like testing on Windows/IE10 (I have the exact same requirements at a current
freelance client).

------
bcherny
This looks pretty neat! A few questions:

1\. Are there plans for a multi-browser (especially mobile web) test runner?

2\. Are there plans to detect change provenance (eg. if I change my heading
height on a page, all the content below it will shift down. The change I
really want to be notified about is that the header's height changed, not that
everything was pushed down because of it)?

3\. There are a lot of similar tools (eg. back in the day I used
[https://github.com/facebookarchive/huxley](https://github.com/facebookarchive/huxley))
- where does Appraise succeed where those tools failed, or what does Appraise
do better?

~~~
adzicg
> 1\. Are there plans for a multi-browser (especially mobile web) test runner?

the screenshot engine is designed to be pluggable, so theoretically this
should be easy to do with some kind of mobile-test-farm runner.

> 2\. Are there plans to detect change provenance...

You can set the relevant clip region to just the header, so it will only
notify you if the header changed and ignore the rest. check out
[https://github.com/AppraiseQA/appraise/blob/master/examples/...](https://github.com/AppraiseQA/appraise/blob/master/examples/controlling-
screenshots.md)

>3\. There are a lot of similar tools...

appraise is designed to be used like fitnesse (so example+fixture=> result)
instead of record/replay tools that end up being fragile. it makes the
examples easy to understand, splitting what is being tested and how it will be
tested clearly so both of those aspects are easy to maintain.

it's also designed to work well with github markdown, so your tests become
executable API/component specs that are easy to read and also easy to verify.

I'm working on an AWS Lambda runner now, which will also make it quick and
cheap to run UI tests in parallel, so it will provide reasonably fast feedback
for large test suites.

------
revelation
This is diffing PNGs? That sounds like a recipe for disaster.

Here is what this looks like in the much denigrated Java Enterprise world:

[https://eclipsesource.com/blogs/tutorials/rcp-testing-
tool-r...](https://eclipsesource.com/blogs/tutorials/rcp-testing-tool-rcptt-
basic-tutorial/)

~~~
adzicg
>This is diffing PNGs? That sounds like a recipe for disaster.

Not necessarily, if you use it for what it was intended -- visual layout tests
that are easy for a human to approve/reject when there is a difference. It
doesn't know if the change is right or wrong, just helps humans be more
effective by showing the difference clearly and aiding in comparisons.

------
sarthakjain
Would be good to have prebuilt rules for eg material design guidelines.

Would it be even possible to test the material design rules across and entire
page without having to define what individual components across the entire
website look like?

~~~
adzicg
not currently, but that's definitely an interesting idea.

Appraise is designed to make it easy for a human to approve/reject current
state or changes, with a smart visual diff and a good way to show actual and
expected pictures side-by-side or as an overlay, but it currently doesn't have
any smart logic beyond that. Theoretically it could do something smarter as a
comparison.

As it stands now, you could run it once to capture the current state, and get
someone to approve it, then it becomes a regression test pack and the tool
would alert to any changes with a smart diff.

------
alkonaut
Could this be used to diff pdf's?

~~~
adzicg
haven't tested it myself, but I assume headless chrome can render PDFs, so the
answer should be yes. Try it and if it doesn't work, submit an issue on github
and we'll make it work together. You'll probably need to use a fixture
producing a local file, check out
[https://github.com/AppraiseQA/appraise/blob/master/examples/...](https://github.com/AppraiseQA/appraise/blob/master/examples/fixture-
types.md#fixtures-returning-a-local-file-name)

------
finnn
That's a nice domain, didn't realize .qa was a ccTLD

~~~
adzicg
yep, qatar :)

------
lunarcave
Very Nice! Can you tell me how this could be different to a tool like niffy
([https://github.com/segmentio/niffy](https://github.com/segmentio/niffy))?

~~~
adzicg
I have not used niffy so this is based on their github description, so might
be wrong

1) niffy seems to be designed to use UI interactions to lead you to the
result, appraise works like fitnesse, separating what is being tested from how
it's being tested, so you could go several levels deeper than UI interactions
and eg test how your models and views produce outcomes, without the http
serving and web pages in between. this makes tests faster and easier to
maintain. appraise is more for visual unit testing.

2) niffy seems to be optimised for developers, appraise is intended for cross-
functional collaboration between developers, testers and designers. with
appraise, the test is a markdown page that renders nicely in github and shows
expected results and the conditions under which they should happen.

------
felipellrocha
How is this different than happo?

------
iamaelephant
Oh good I look forward to deleting mountains of these kinds of tests in a
couple of years when the shine comes off the apple and people realise the
effort of separating the flake from the real failures isn't worth it.

~~~
dang
It's always possible I've misread your comment, but if not: please don't
dismiss other people's work in an assholish way on HN. It breaks both the site
guidelines and the extra rules for Show HNs:

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

[https://news.ycombinator.com/showhn.html](https://news.ycombinator.com/showhn.html)

Substantive critique is welcome, of course, but in this case that would mean
talking about problems with visual testing and making it clear you're not
picking on this particular implementation.

~~~
atomical
Is calling someone an asshole okay under the guidelines?

~~~
dang
I was careful not to. Would you please stop this now?

~~~
atomical
Could you stop with the incredibly arrogant attitude? You wouldn't act like
this in real life -- at least I hope not. You would be shunned by everyone
around you.

