
Show HN: Spotbot, a Slack bot that monitors your website for visual bugs - jmtulloss
https://spotbot.qa
======
jmtulloss
Hi Hacker News! We've been working hard on Spotbot for the past couple months
and are looking forward to your feedback on what we've built. @mgc and I will
be around all day to answer any questions.

~~~
spenserhale
Justin,

People cannot try free, your servers are responding with 503s. I tried to
email you from your contact link on website (hi@spotbot.qa) and was
undeliverable.

~~~
jmtulloss
Well the email thing is pretty embarrassing, just fixed. I wrote
"hi@spotbot.qa" and set up an email for "hello@spotbot.qa". Face palm.

We are experiencing a lot of load (or we were until we went down). We scaled
up and things seem to be working better now.

------
reustle
I understand slack integration for some tools, but is this slackbot-as-a-
product thing going to become the new fad? What if I want to use your tool
without using slack?

~~~
jmtulloss
We plan on supporting other workflows soon, we just started with Slack cause
that's what we use.

Next stop is probably email. Any other integration you'd like to see?

~~~
rhplus
I've worked with many systems like this and know that notifications are too
easy for individuals to ignore. Your primary action should be to open an issue
in a bug tracking system, and let that system trigger the email/instant
notification. This helps teams get a better picture of bug volume/impact/area
over time, allows for prioritization of work and ensures issues don't get lost
during major bug storms.

Also, being able to link a bug conclusively to a changeset, build or a
deployment ID would be very valuable. Rather than building lots of deep CI
integrations, you could do this by allow users to configure your monitor to
look for version numbers in server hints (headers, hidden fields, status
pages, etc).

------
dkopi
I can see this working for static pages, not sure about pages that require
input, user login, or display constantly changing data.

~~~
jmtulloss
We're really excited about tackling those problems.

@mgc and I founded Spotbot with the thought that models trained with ML
techniques can be immensely useful to developers (novel, I know). So much of
coding, debugging, and firefighting is pattern matching, and machines are
getting really good at pattern matching. That's how we're going to tackle the
user login and dynamic data issues in the long term.

We have some short term solutions in the works as well.

For dynamic content, we'll have a way to ignore changes that are expected and
don't mean anything significant changed.

For login, we may implement a token API that will allow us to log in with a
bit of work on the developer's side. We can also script logins without too
much effort if we are provided with test account credentials.

Thanks for giving me an opening to talk about all this :)

~~~
giovannibajo1
My app is a social network. A social network means that the two most important
pages (global stream and user stream) continuously mutate because there is
always new user-generated content.

Now, humans are very good at catching bugs there; if a the name of the user
goes above the avatar picture next to it, humans will see it immediately.

But how can a tool like Spotbot not to be confused by all the normal changes
in user-generated content, and to just spot structural differences?

~~~
jmtulloss
Right now we're not great at it. However, the way Spotbot works means that we
will be decent at it soon.

We do a hash of the structure, so we know exactly what kind of changes there
are. For something like your page, the image sources, the text, the names, the
votes, and other content change all the time. Soon we'll have a way to ignore
those changes while still recognizing that other changes need to be vetted.

So, for instance, if the padding between the avatar and the user's status
changes, we'll still catch that even though we're ignoring the avatar source
and the text content.

Soon!

~~~
giovannibajo1
Thanks! I'm a little hesitant because it sounds a little like magic, but I'm
happy to help with testing it, should you need testers. Any place I can
subscribe for updates?

------
juecd
Super excited to try this — as an early start-up (and the founder in charge of
UI), I KNOW our site has UI issues but I don't have the time to do full
inspections.

What kind of "changes need attention" will Spotbot tell me about?

~~~
jmtulloss
Right now it checks your site every 24 hours, and we try to alert on things
that might actually need a person to verify (ie, not animations or videos).

We're in the process of building moderation tools so that you can refine
what's important to you, and we're really excited about building training a
model to be able to highlight changes that are probably broken. We're starting
at square one though :)

------
Gazoo101
I can't really say as to whether this is due to load, or some underlying
problem, but it seems that - currently - if you just wait around for the
system to get to your requested (free) screenshot, it ticks down to 4
Screenshots ahead of yours, and... Nothing.

Not a single one of the boxes seem to update anymore. They all just say 4
screenshots ahead of yours. It seems, I spoke a bit too soon. Something is
happening now. I guess my cynicism got the better of me when they all hit 4
and just... Stopped :)

------
nir
Shameless plug: [http://shoov.io/](http://shoov.io/)

~~~
jmtulloss
Looks pretty cool!

I like that the tool itself is open source.

We found that pixel diffs were pretty unwieldy for a lot of different
scenarios. How do you deal with, say, a div inserted at the top that pushes
all the content on the page down 50px?

~~~
amitaibu
Shoov uses webdriverCSS under the hood, so you can simply remove/ hide/
exclude that div.

------
5vforest
We use Percy.io for this and can't recommend it enough. It integrates with
your CI workflow, which makes a whole lot more sense to me than waiting until
production. (Or to be fair, staging.)

~~~
jmtulloss
Percy is awesome, no doubt. We're a little different than Percy though.

We don't require any code. Setup is just entering a URL. Percy doesn't really
require code either, but it relies on you having already written integration
tests.

We do structural diffing, not pixel diffing. Pixel diffing can be challenging
to view as a developer since innocuous changes can cause massive visual pixel
diffs. With structural diffing we can have a better idea of exactly what
changed and highlight those regions.

We're definitely going to be integrating with CI workflows and non-production
sites as well :)

------
ldehaan
this is cool, but I just wrote something like this at work using hubot and
some node libs, and there are a couple extra options you can setup.

if they let me I'll release it OSS, but it was super easy to write, and it
pulls diffs from the github project when sees a change.

I was actually working on a reverse of that, so if it sees a repo get pushed
to it will let you know if the site changed or if it didn't.

hubot is so cool, but coffee script is just gross. meh.

other than that, it's been incredibly useful.

------
aroc
Interesting, but this needs to be integrated into CI, not Slack, to be taken
seriously by teams larger than just a few people. And arguably, I tend to set
up CI of _any_ serious project before it goes to production.

~~~
jmtulloss
What's your CI system of choice?

~~~
aroc
I really like Circle CI, but I've used (and enjoyed) most of the other
services as well.

------
DocG
Hey It seems that spotbot does not cover for delayed page transistion effects
in CSS. I have multiple objects appearing in row and pictures show currently
each page in different stage and not fully loaded page :(

Other than that, awesome :)

~~~
jmtulloss
Thanks!

We do try to deal with CSS transitions but I don't think we have every
situation covered yet. What page were you trying it on?

------
tekklloneer
I tried it with our product and it took a snapshot before all of the (wall
time intensive) api calls finished. Is there a way to configure it to wait a
few seconds?

~~~
jmtulloss
Not currently, but this is a common request.

What would you prefer:

\- A javascript object being injected into your page that you could call to
indicate the page was done

\- A setting within the UI that would allow you to configure a wait time

~~~
alxndr
Instead of injecting something into the page, what about having the
snapshotter running an is-it-done-yet callback periodically to see whether it
returns true? That would allow some basic checks like "is this particular DOM
element visible yet" or "how many entries are in this list" without needing to
modify the production code in the page.

------
jenniferDewalt
This looks very useful! I'm wondering if it's possible to see a timeline of
changes, like the history of changes for the last 30 days?

~~~
jmtulloss
It is, although it's not the best UX at the moment. In the details about any
page that you're tracking, the past snapshot runs will be listed on the left
side. You can go to any point in time, view it, and compare it to the previous
time something actually changed.

~~~
jenniferDewalt
Got it! I can see that being helpful when looking at other site metrics.

------
xerophyte12932
What if my website isn't published yet? Is it possible to run this locally for
a locally deployed website/webapp.

------
rocky1138
Why would I choose this over Browserstack?

~~~
jmtulloss
We actually use Browserstack for some of our devices.

Spotbot monitors your site over time, so it'll alert you when things change.
With Browserstack, you have to manually run your site in all the devices you
care about each and every time you want to check something. Or you can write a
Selenium test, but that's a lot more painful than just entering a URL and
letting us take care of it for you!

~~~
rocky1138
Got it, thanks!

------
alecdibble
How will this work with a split testing tool like Optimizely?

~~~
jmtulloss
Great question.

If you're willing to do a little work on your side, you can craft a URL that
will guarantee that the user ends up in a particular bucket.

One of the features that we're thinking about is doing visual diffing on A/B
test variations. Then you can page through your variations and quickly see how
they're different from the control, or if they're broken on any browsers.

To do that we would need a deeper integration, either with the user or with
the A/B service. I think it'd be pretty cool though!

