Hi Hacker News! We've been working hard on Spotbot for the past couple months and are looking forward to your feedback on what we've built. @mgc and I will be around all day to answer any questions.
People cannot try free, your servers are responding with 503s. I tried to email you from your contact link on website (hi@spotbot.qa) and was undeliverable.
The site I'm currently maintaining doesn't change very often, but it's approx. 1,500 pages if spidered, plus a large search index. I'm looking at the Fancy plan, but we've got multiple subdomains or internal versions of the site. How are these covered if it's not all under one domain? Are sub-domains part of the same domain? :)
Is it possible to integrate with a CI system for on-demand runs, perhaps by supplying a list of changed pages as a hint on where to expect changes and where not? The idea being that to lower your costs, a subset of pages could be run on-demand?
No specific CI in mind, maybe just Github integration or a webhook to start. I'm currently looking at options to modernize and add tests to a rather manual, SVN-based process.
I'll email for sure, I'm interested to see how this will work. Thanks! :)
I understand slack integration for some tools, but is this slackbot-as-a-product thing going to become the new fad? What if I want to use your tool without using slack?
Nope, although we do have a good relationship with Slack.
We started with Slack cause it's really easy to get started with and a lot of people, including ourselves, use Slack as the central repository of what's going on in the organization.
We realize that it's not for everybody though, more options will be coming soon.
I tried setting it up to post to my Slackbot but hesitated at the permissions (giving away everyone's email in particular, when I just want to notify myself right now.)
I feel like a shitty tirekicker for complaining, but I really would rather just get email alerts and then optionally integrate with Slack directly or via webhook/Zapier. The service feels a bit inaccessible.
We would also prefer not to ask for that permission. In fact, the only reason we do it is be able to call Slack's `users.info` API on your user. Which... we're only doing in order to be able to show a profile picture and your real name. Those things are nice, but not must haves.
I was thinking the same thing. I'm interested in trying it out, but do I really feel like setting up slack for a side project when I'm the only one that works on it?
No I mean a personal side-project that I have no slack set up for because I'm literally the only person that works on it. Not just the only person on a project in a company.
I've worked with many systems like this and know that notifications are too easy for individuals to ignore. Your primary action should be to open an issue in a bug tracking system, and let that system trigger the email/instant notification. This helps teams get a better picture of bug volume/impact/area over time, allows for prioritization of work and ensures issues don't get lost during major bug storms.
Also, being able to link a bug conclusively to a changeset, build or a deployment ID would be very valuable. Rather than building lots of deep CI integrations, you could do this by allow users to configure your monitor to look for version numbers in server hints (headers, hidden fields, status pages, etc).
We're really excited about tackling those problems.
@mgc and I founded Spotbot with the thought that models trained with ML techniques can be immensely useful to developers (novel, I know). So much of coding, debugging, and firefighting is pattern matching, and machines are getting really good at pattern matching. That's how we're going to tackle the user login and dynamic data issues in the long term.
We have some short term solutions in the works as well.
For dynamic content, we'll have a way to ignore changes that are expected and don't mean anything significant changed.
For login, we may implement a token API that will allow us to log in with a bit of work on the developer's side. We can also script logins without too much effort if we are provided with test account credentials.
Thanks for giving me an opening to talk about all this :)
My app is a social network. A social network means that the two most important pages (global stream and user stream) continuously mutate because there is always new user-generated content.
Now, humans are very good at catching bugs there; if a the name of the user goes above the avatar picture next to it, humans will see it immediately.
But how can a tool like Spotbot not to be confused by all the normal changes in user-generated content, and to just spot structural differences?
Right now we're not great at it. However, the way Spotbot works means that we will be decent at it soon.
We do a hash of the structure, so we know exactly what kind of changes there are. For something like your page, the image sources, the text, the names, the votes, and other content change all the time. Soon we'll have a way to ignore those changes while still recognizing that other changes need to be vetted.
So, for instance, if the padding between the avatar and the user's status changes, we'll still catch that even though we're ignoring the avatar source and the text content.
Thanks! I'm a little hesitant because it sounds a little like magic, but I'm happy to help with testing it, should you need testers. Any place I can subscribe for updates?
Couldn't you point it at a test environment that would be loaded up with non-changing content so that UI only changes could be tested against the same data always?
You could create a page which uses as much of the real production code paths as possible, but which returns the same "stories" every time. It takes a little dev work, of course, but it would almost certainly be worth it.
Super excited to try this — as an early start-up (and the founder in charge of UI), I KNOW our site has UI issues but I don't have the time to do full inspections.
What kind of "changes need attention" will Spotbot tell me about?
Right now it checks your site every 24 hours, and we try to alert on things that might actually need a person to verify (ie, not animations or videos).
We're in the process of building moderation tools so that you can refine what's important to you, and we're really excited about building training a model to be able to highlight changes that are probably broken. We're starting at square one though :)
I can't really say as to whether this is due to load, or some underlying problem, but it seems that - currently - if you just wait around for the system to get to your requested (free) screenshot, it ticks down to 4 Screenshots ahead of yours, and... Nothing.
Not a single one of the boxes seem to update anymore. They all just say 4 screenshots ahead of yours. It seems, I spoke a bit too soon. Something is happening now. I guess my cynicism got the better of me when they all hit 4 and just... Stopped :)
We found that pixel diffs were pretty unwieldy for a lot of different scenarios. How do you deal with, say, a div inserted at the top that pushes all the content on the page down 50px?
We use Percy.io for this and can't recommend it enough. It integrates with your CI workflow, which makes a whole lot more sense to me than waiting until production. (Or to be fair, staging.)
Percy is awesome, no doubt. We're a little different than Percy though.
We don't require any code. Setup is just entering a URL. Percy doesn't really require code either, but it relies on you having already written integration tests.
We do structural diffing, not pixel diffing. Pixel diffing can be challenging to view as a developer since innocuous changes can cause massive visual pixel diffs. With structural diffing we can have a better idea of exactly what changed and highlight those regions.
We're definitely going to be integrating with CI workflows and non-production sites as well :)
Interesting, but this needs to be integrated into CI, not Slack, to be taken seriously by teams larger than just a few people. And arguably, I tend to set up CI of any serious project before it goes to production.
Hey
It seems that spotbot does not cover for delayed page transistion effects in CSS. I have multiple objects appearing in row and pictures show currently each page in different stage and not fully loaded page :(
I tried it with our product and it took a snapshot before all of the (wall time intensive) api calls finished. Is there a way to configure it to wait a few seconds?
Instead of injecting something into the page, what about having the snapshotter running an is-it-done-yet callback periodically to see whether it returns true? That would allow some basic checks like "is this particular DOM element visible yet" or "how many entries are in this list" without needing to modify the production code in the page.
I don't really care about all of the calls, but I'd like to say "here's a regex pattern describing a uri. once any connection with uri pattern closes (within 10 seconds), you can render."
Yeah, this isn't great for websockets/long polling or other persistent methods, but it's fine for simple ajax-y calls.
It is, although it's not the best UX at the moment. In the details about any page that you're tracking, the past snapshot runs will be listed on the left side. You can go to any point in time, view it, and compare it to the previous time something actually changed.
We actually use Browserstack for some of our devices.
Spotbot monitors your site over time, so it'll alert you when things change. With Browserstack, you have to manually run your site in all the devices you care about each and every time you want to check something. Or you can write a Selenium test, but that's a lot more painful than just entering a URL and letting us take care of it for you!
If you're willing to do a little work on your side, you can craft a URL that will guarantee that the user ends up in a particular bucket.
One of the features that we're thinking about is doing visual diffing on A/B test variations. Then you can page through your variations and quickly see how they're different from the control, or if they're broken on any browsers.
To do that we would need a deeper integration, either with the user or with the A/B service. I think it'd be pretty cool though!
