
Launch HN: Reflect (YC S20) – No-code test automation for web apps - tmcneal
We&#x27;re Fitz and Todd, co-founders of Reflect (<a href="https:&#x2F;&#x2F;reflect.run" rel="nofollow">https:&#x2F;&#x2F;reflect.run</a>) - we’re excited to share our no-code tool for automated web testing.<p>We worked together for 5+ years at a tech start-up that deployed multiple times a week. After every deployment, a bunch of our developers would manually smoke test the application by running through all of the critical user experiences. This manual testing was expensive in terms of our time. To speed up the tests’ run time, we dedicated developer resources to writing and managing Selenium scripts. That was expensive at “compile time” due to authoring and maintenance.  At a high-level, we believe that the problem with automated end-to-end testing comes down to two things: tests are too hard to create, and they take too much time to maintain. These are the two issues we are trying to solve with Reflect.<p>Reflect lets you create end-to-end tests just by using your web application, and then executes that test definition whenever you want: on a schedule, via API trigger or simply on-demand. It emails you whenever the test fails and provides a video and the browser logs of the execution.<p>One knee-jerk reaction we’re well aware of: record-and-playback testing tools, where the user creates a test automatically by interacting with their web application, have traditionally not worked very well. We’re taking a new approach by loading the site-under-test inside of a VM in the cloud rather than rely on a locally installed browser extension. This eliminates a class of recording errors due to existing cookies, proxies or other extensions introducing state that the test executor is not aware of, and unifies the test creation environment with the test execution environment.  By completely controlling the test environment we can also expose a better UX for certain actions.  For example, to do visual testing you just click-and-drag over the element(s) you want to test.  For recording file uploads we intercept the upload request in the VM, prompt you to upload a file from your local file system, and then store that file in the cloud and inject it into the running test.  If you want to add additional steps to an existing test, we’ll put you back into the recording experience and fast-forward you to that point in the test, where again all you need to do is use your site and we’ll add those actions to your existing test.   Controlling the environment also allows us to reduce the problem space by blocking actions which you typically wouldn’t want to test, but which are hard to replicate and thus could lead to failed recordings (e.g. changing browser dimensions mid-recording). As an added bonus, our approach requires no installation whatsoever!<p>We capture nearly every browser action from hovers to file uploads, and drag-and-drops to iframes, while building a repeatable, machine-executable test definition. We support variables for dynamic inputs, and test composition so your test suite is DRY. The API provides flexible integration with your CI&#x2F;CD out of the box and supports creating tests in prod and running them in staging on the fly.  You don’t need to use a separate test grid, as all Reflect tests run on our own infrastructure. Parallel execution of your tests is a two click config change and we don’t charge you extra for it.<p>Some technical details that folks might find interesting:<p>- For every action you take we’ll generate multiple selectors targeting the element you interacted with.  We wrote a custom algorithm that generates a diverse set of selectors (so that if you delete a class in the future your tests won’t break), and ranks them by specificity (i.e. [data-test-id] &gt; img[alt=”foo”] &gt; #bar &gt; .baz).<p>- To detect certain actions we have to track DOM mutations across async boundaries.  So for example we can detect if a hover ended up mutating an element you clicked on and thus should be captured as a test step, even if the hover occurred within a requestAnimationFrame, XHR&#x2F;fetch callback, setTimeout&#x2F;setInterval, etc.<p>- We detect and ignore auto-genned classes from libraries like Styled Components.  We use a heuristic to do this so it’s not perfect, but this approach allows us to generate higher quality selectors than if we didn’t ignore them.<p>- One feature in beta that we’re really excited about: For React apps we have the ability to target React component names as if they were DOM elements (e.g. if you click on a button you might get a selector like “&lt;NotificationPopupMenu&gt; button”).  We think this is the best solution for the auto-genned classes problem described in the bullet above, as selectors containing component names should be very stable.<p>We tried to make Reflect as easy as possible to get started with - we have a free tier (no credit card required) and there’s nothing to install. Our paid plans start at $99 and you pay primarily for test execution time.  Thanks and we look forward to your feedback!
======
cmehdy
Interesting tool! And you've definition given a lot of thought to many of the
problems encountered while attempting to do UI testing automation.

I've written my fair share of extensive selenium stuff (and appium, but..
let's forget about those painful memories) and one thing that I found fairly
"easy' to add to the suites I had written was accessibility testing (using
Deque's Axe[0] tools). It literally took a few lines of code and a flag at
runtime to enable the accessibility report on my Jenkins/TestNG/Selenium
combo. However WCAG is constantly changing and it's hard to keep up, and even
Deque is not always up to date AFAIK. Do you have plans to have accessibility
testing with your tool ? (even a subset of WCAG's rules)

Another thing I've noticed is the jump in pricing from Free to 100 USD/month
which goes from 30mins to 6hours. This might be steep for a team attempting to
test out the validity of the tool against the competition - perhaps offering a
first-month discounted trial or something like that would be appealing.

I also haven't really seen if it is possible to enable concurrency (for
example, is testing on three platforms at the same time for 10 minutes on that
free tier possible? I would imagine so if one is doing the lifting with CI
like Jenkins - but perhaps you have your own solutions). Tangentially related
but you say your tests integrate with various CI solutions, does this mean one
can extract the test results in a way that allows further processing and
integration into other tools? (I'm thinking of the XMLs coming out of TestNG
there).

Lastly, I don't know if the time used is counted from the moment you start
your VMs or browser starts, or the page is loaded, or the first step is done
or something else. Clarifying that might help with a team's estimates (I had
internal tests where the struggle was entirely on having the virtual
environment/device/browser ready and the test was then a breeze, so the
significant metric was the boot-up time).

[0] [https://github.com/dequelabs/axe-core](https://github.com/dequelabs/axe-
core)

~~~
fitzn
Thank you for the thoughtful comment! Let me know if I miss any points:

1) We capture all attributes for elements and although we don't yet let you
fail a test if the attributes are incorrect, that's where we're headed. So,
you could imagine enabling/disabling accessibility validation at the account
or per-test level. This is great feedback.

2) Re: pricing - thanks for this idea! Pricing is still evolving, will keep in
mind.

3) We support parallel execution at no extra charge; it's only test execution
time. So if you want to spin up 10 tests at once, that's fine. (We may need to
scale up our infra at the outset, obviously.) There is no hard cap on the
number of concurrent tests.

4) Our API ([https://reflect.run/docs/developer-
api/documentation/](https://reflect.run/docs/developer-api/documentation/)) is
in the early stages, but includes endpoints for executing tests and fetching
results for tests in JSON.

5) Runtime is the time the browser begins navigating to the page. You are not
charged for time spent spinning up infrastructure or saving the test run. And
just to be clear, you are not charged for any time spent recording tests.

Thanks again!

~~~
cmehdy
Thanks for answering those questions.

I think it's all great news, and for customers it's reassuring to know that
running time is really just about the test itself (which I noticed is in the
FAQ, so sorry for wasting your time there). This makes me think about a couple
more things: can clients select zones from which to run the tests? It's
probably not a huge focus if your current target customers are in North
America, but can become useful if a European company considers the tool and
all loading times become noticeable. Or just if a company expects clients from
anywhere and wants to test for that (which is basically the hardest bit for a
small team, as it starts to involve having some devops experience to spin up
VMs on the cloud in different zones and all that jazz). Another tangentially
related setting would be the locale for tests execution and scenarios (I'm
thinking of date formats, monetary symbols, etc.. with expectations being
different if you're in the US or in the UK for example)

The lack of cross-browser testing is something I can see scaring away some
teams, especially with mobile being restricted to profiles of older iPhones
only. IIRC over 70-80% of iPhone users are on the latest major iOS version.
But it all ties in to other issues that I'm sure you're already dealing with
(iPhone users using Safari, something like 20% of desktop users using
Firefox/Safari, and the odd clients here and there who would have very
specific annoying requirement like "I must be able to test on Internet
Explorer - yes I know it's deprecated but my startup is B2B and I'm the small
fish there").

Your docs might be short but they are already good and readable, especially
for a target that might include a mix of pure devs and automation
QAs/SDET/etc.

All the best!

------
shay_ker
> One feature in beta that we’re really excited about: For React apps we have
> the ability to target React component names as if they were DOM elements
> (e.g. if you click on a button you might get a selector like
> “<NotificationPopupMenu> button”). We think this is the best solution for
> the auto-genned classes problem described in the bullet above, as selectors
> containing component names should be very stable.

I remember looking into this exact idea. Theoretically, if you're able to
capture the React state, and you're working with a "pure" React app, you
should be able to auto-gen readable tests from human interaction. And, if
you're capturing the state in a granular enough fashion, you should be able to
"time-travel", but for non-technical users.

IMO the biggest use case for E2E tests are for critical things like auth &
checkout. If you're able to auto-gen, maybe you can get even deeper than that.

Congrats on the launch, it looks cool!

~~~
inglor
We have a product that does this (generates E2E tests from user interactions)
- it's very tricky. If you just want to cover the auth & checkout flows it's
great but if you need to model the more complex and nuanced interactions
record + playback is really a great way to go.

~~~
shay_ker
You should be able to hack React to do what I'm suggesting (playbacks), though
last time I looked at the source code I got a bit confused and gave up.

It's historically been such a battle to deal with E2E tests and changing code,
but if this playback idea works well then the interop should be relatively
seamless. It might require an over-engineered React app, though, which is
likely the biggest issue.

------
yeldarb
We are using Reflect.run at Roboflow and it really is very slick. We have it
testing our core flow on a schedule so we know if any of our 3rd party
services or dependencies go down and/or if we introduce a regression.

Our app's core workflow involves signing in, uploading images and annotation
files, parsing those files, using HTML5 canvas to render previews and
thumbnails, uploading files to GCS, and kicking off several async functions
and ajax requests.

That Reflect.run was able to handle all of those complex operations makes me
pretty confident it can effectively test any web app.

~~~
LeonidBugaev
I would love to learn more about your setup. Does Reflect has built-in
Roboflow integration, or it ismth custom?

~~~
yeldarb
Nope, we were able to use their UI to create our tests just like any other
user would -- everything we needed was supported out of the box (minus one
minor workaround I had to add to my code because I had some janky, non-
deterministic IDs on some dynamically added hidden fields).

------
karussell
Really nice tool, thanks! I'm more a backend developer so I might have some
stupid questions: What is your competition and what are you doing better?

Update: reading through this thread and a bit web search resulted in the
following list:

[https://ghostinspector.com](https://ghostinspector.com)
[https://www.testim.io](https://www.testim.io)
[https://www.katalon.com](https://www.katalon.com)
[https://www.cypress.io](https://www.cypress.io)
[https://github.com/dequelabs/axe-core](https://github.com/dequelabs/axe-core)
[https://www.mabl.com](https://www.mabl.com)
[https://preflight.com](https://preflight.com)

> We’re taking a new approach by loading the site-under-test inside of a VM in
> the cloud rather than rely on a locally installed browser extension.

But how would I record and test things locally? I.e. I would need a public
setup right at the beginning.

~~~
fitzn
Hi! It's technically possible to test your localhost though it's a bit of work
with ngrok. What we're seeing more and more of with our customers is they
stand up ephemeral environments for each PR, or each merge, and then use
Reflect's hostname and URL overrides at run time to target their code changes
with the tests they already recorded against prod or staging. We've worked
with Release W20 ([https://releaseapp.io/](https://releaseapp.io/)) in the
past to demonstrate running Reflect tests against each PR. I know it's not
exactly what you're looking for, but similar in spirit, I think.

~~~
karussell
Never heard of ngrok, thanks for this pointer. Sounds like a magic tool :)

Will look into releaseapp.io

------
amw-zero
I've largely moved away from UI testing. The two main reasons are:

1) Flakiness / non-determinism 2) The constant change of the UI

Both of these are absolute killers to productivity on a large team. Note, I'm
not against UI testing in theory. I think if you could, you would have full
end to end tests for every single edge case so that you could be sure that the
actual product presented to users work. But in practice, end to end testing an
asynchronous distributed system (which a simple client-server application
still is) is full of non-determinism.

Re the constant changing of the UI. This is just also true in my experience.
I've worked on a navigation / global UI redesign at every company I've ever
worked at. It happens like once every 3-5 years. Within the redesigns, it's
still extremely common to subtly change the UI for UX reasons all the time.
When this happens, be prepared to spend half of the time updating all of your
tests.

~~~
inglor
To be fair, I work for a company in this space (Testim.io) and we have had
success stories with very large companies doing UI tests with the service
(like Microsoft).

I think the hardest aspects of testing are fast authoring and stability
(maintenance) - AI tools can help with that and learning data between test
runs can create very stable tests.

So as someone in this space - it's a very exciting space and I am very
optimistic for this startup (Reflect)

~~~
amw-zero
Yea I don't want to discourage work in the space. Actually, quite the
opposite. I'm just saying where the bar is for me when I'm considering handing
over money. I'm not going to pay money for something that ends up actually
slowing me down.

------
loh
Congrats on launching Reflect. Looks solid!

I soft launched something eerily similar ~6 months ago and got zero feedback
(probably because no one was able to actually try it since Google took forever
to approve the extension).

TestFront.io:
[https://news.ycombinator.com/item?id=22130590](https://news.ycombinator.com/item?id=22130590)

Ran out of money though and had to pursue other things so I put it on hold.
Maybe I'll resume work on TestFront at some point and you'll have some
competition. ;)

------
mayank
> We’re taking a new approach by loading the site-under-test inside of a VM in
> the cloud rather than rely on a locally installed browser extension.

So while you do get the benefit of clean, repeatable tests, this requirement
also puts you out of reach for most enterprise-y applications, where your pre-
production environments (i.e., what you actually want to run your tests
against) exist in a tightly locked down network environment that you can't
reach from the outside. In stricter environments, even a reverse port
forwarding setup to allow you to reach inside the LAN would be out of the
question.

I think this is a problem you're going to run into pretty soon if you want to
do enterprise sales, and one that is not trivial to solve without an open-
source flavor of the test runner (e.g. like Selenium).

~~~
fitzn
Thank you for this thought-provoking perspective. I agree we'll need to figure
out a way to provide an on-prem solution in some enterprise cases. We'll think
on this internally. Thanks, again!

------
rococode
This is super cool! I'm at a startup and we've finally started building up our
testing infrastructure lately, this seems like a great time-saver. Just tried
it out and it worked great. Two questions:

1) Is it possible to set the screen resolution? I see device profiles but
don't see if it's possible to manually create profiles. The default Desktop
res is lower than what we normally target.

2) How exactly does the visual testing work? I tried dragging over a div and
it got the text, but parts of the text were formatted differently with spans.
So does it just validate the text, or does it actually compare the visuals?

~~~
fitzn
Thanks for giving us a spin!

1) We don't support custom resolutions yet, but you're spot on that the way we
would do this is by allowing you to create device profiles in your account.
The Desktop, Tablet and Mobile profiles are all that we support today.

2) Visual Validations will capture _both_ a screenshot of the element and the
visible text of the element. In order for the visual validation to pass during
execution, the text must match and the element's screenshot must be within
your account configurable % difference. Default image difference is 0.5%
pixels. We do a pixel by pixel comparison of the images and show you a delta
when the images differ.

------
catchmeifyoucan
One thing I like about this is that it supports drag and drop support. That's
an item that I haven't seen to be super straight-forward with other suites
like Cypress and is very-much a user initiated action.

------
chris_st
For selecting things, I've found that allowing a "data-test='user-name-input'"
type attribute is useful for a lot of cases, when doing Gauge/Taiko tests.

It might make sense to allow/recognize these to save trying to find things
that way, rather than via CSS-type selectors that may change.

~~~
tmcneal
Totally! If you have data-test* attributes set up, we will use those first
when generating selectors for each action. The full list of data-test
attributes we use is listed here: [https://reflect.run/docs/recording-
tests/creating-resilient-...](https://reflect.run/docs/recording-
tests/creating-resilient-tests/#selectors). If we don't find data-test*
attributes we'll also look for other attributes that tend to have a good
degree of specificity, like alt, rel, schema.org, and aria-* attributes.

~~~
inglor
I recomment considering learning between test runs and I encourage you to
train a relatively simple model for selection on top of http-archive and
tagged data.

"off the shelf" machine learning makes it pretty easy to create very robust
selectors. I gave a talk about it in GDG Israel and was supposed to speak
about it in HalfStack that got delayed cancelled because of COVID19 - but the
principle is pretty simple.

It's amazing how much RoI you can get from relatively simple models of
reinforcement learning. Here are some ideas:
[https://docs.google.com/presentation/d/1OViIwDJJw1kjVJH5Z2N5...](https://docs.google.com/presentation/d/1OViIwDJJw1kjVJH5Z2N5chz5Usw3kOf41hCiwZXu0ng/edit?usp=sharing)

Good luck :]

------
jaequery
How does this compare with qawolf.com which uses Playwright, open-source and
free?

------
p17
I want to use Reflect to test my game with many concurrent, simulated players.
For context, the game needs to properly handle 50 players at once.

The pricing (6 hours of testing for $99) means that if I want to do a 1 minute
test with 50 concurrent players, I can only test 6 times a month. A big
benefit of testing is to ensure we ship reliable software, and we plan to ship
much more often than 6 times a month.

Is there a way you could price by unique tests instead of hours tested?

~~~
fitzn
Interesting use case. We haven't encountered this or thought about this
before. It would be a change for us to price like this, but happy to discuss
further if you want to shoot us an email!

------
idreyn
I maintain a web app that would really benefit from an E2E suite, but we don't
have the developer capacity to write one right now, so this looks like it hits
a potential sweet spot for me. To use Reflect, I think we would need to
understand the plan for what happens if the SaaS goes under -- ideally, we'd
be left holding our test definitions and some open-source version of the test
runner that we can instantiate on our own VM.

~~~
jelling
Testcafe’s test recorder would let you create tests relatively easy by
browsing and you can tweak them if needed. Ime they have great support and I
like that I can buy the software rather than having another SaaS subscription.
Ymmv.

------
futhey
This looks amazing. First test went incredibly well, easy to set up. Enjoying
the VM approach vs. extension.

Will monitor this to see how our tests perform over time.

------
mritchie712
How do you handle logging in to the app that needs to be tested? We use Google
OAuth.

~~~
tmcneal
We fully support e-mail based logins, but OAuth can be challenging.

Github OAuth for example will issue a 2FA email-based challenge non-
deterministically. We handle that by detecting the challenge and filling out
the challenge code based on the contents of the email sent by Github. This
requires a one-time setup where you add an email address we control to your
Github user so that we can read and parse it.

For Google OAuth we can execute all the steps but the two issues there are (1)
we run everything in a single window and some web apps don't like that because
they assume the oauth flow will happen in a new window, and (2) sometimes
Google prompts you to answer a security question and we don't yet support
marking test steps as optional.

What our customers have been doing instead is setting up a mechanism to auto-
log in the test user using a magic link. Basically sending in a one-time-use
auth code to a URL in their app that then authenticate the user. I think some
platforms (Firebase?) have built-in support for this.

I'm certainly happy to brainstorm what could work best for you though if you'd
like (my email: todd at reflect.run)

~~~
ss3000
I asked this elsewhere in the thread to a different user, but it seems
relevant here as well:

Curious how your service would handle a app that mandates non-email based MFA
like SMS or TOTP.

Additionally, what about testing a onboarding flow that might require some
form of manual approval?

Thanks!

~~~
tmcneal
We have the ability to handle MFA logins that use email as the second factor.
The way it works is you configure your user account to have an email address
that we control (e.g. foo@account-randomchars.email.reflect.run). When the
multi-factor challenge comes in, we receive the email, parse it for the
challenge code, and fill it in live in your test. We don't have SMS support at
the moment but we could take a similar approach there if SMS is used as a
second factor. There's a one-time setup here where we set up our system to
parse your challenge email.

No support for TOTP unfortunately, but if the SMS or email-based challenge
support would work for you feel free to reach out and I can talk to the
specifics of the one-time setup - todd at reflect dot run

------
start123
The registration is buggy - The form is misaligned - at least on Firefox, and
why does Google sign up say "continue to amazoncognito.com"?

~~~
tmcneal
Thanks for reporting! We'll get the misalignment of the form fixed up for
Firefox. The Google sign-in says 'continue to amazoncognito.com' because we're
using AWS Cognito as our auth provider, but it looks like there's a way to
configure it to be "continue to reflect.run" so we'll look into that!

------
tribeca18
Makes me think of [http://waldo.io](http://waldo.io), but for web apps! Really
excited about the new wave of no-code test automation tools - definitely helps
semi-technical team members take on more responsibility on the testing side
vs. just writing specs.

~~~
fitzn
Yes! 100% our feeling as well.

------
mchusma
How does this compare to recent YC alum Preflight?

~~~
fitzn
Hey, co-founder Fitz here! Great question. We're both tackling the same
problem. Our key differentiator from Preflight (and testim, and mabl and ghost
inspector) is that we spin up a new VM for every browser session rather than
rely on a local browser extension like all of those competitors. The comment
above highlights the trade-offs of this approach but happy to discuss further
(fitz at).

------
mmckelvy
Looks great. How much easier is this than say, writing E2E tests with Cypress?

~~~
inglor
I wrote an article about Cypress. I was a big fan (gave panels about cypress
in conferences) but am disillusioned. For a very specific use case Cypress is
great and arguably the best choice though:
[https://www.testim.io/blog/puppeteer-selenium-playwright-
cyp...](https://www.testim.io/blog/puppeteer-selenium-playwright-cypress-how-
to-choose/)

Even if you use something like Cypress you probably need something like
Reflect (or Testim where I work but don't represent - for that matter) - you
would just end up writing the framework in-house.

~~~
ss3000
Had very similar experience with Cypress here as well. The command queuing API
that abstracted away async operations sounded awesome in theory and I totally
drank the coolaid.

But once I started to write tests with it, I realized the model was optimized
for simple tests where you can afford to act only as the end-user, whereas for
the more complex flows we're interested in testing, you need to act both as
the end-user as well as, say, someone in the operations team doing a bunch of
manual approval steps, or say, the email/sms system that delivering some
secret out-of-band (through a staging backdoor API), interleaved between the
end-user actions, and not being able to precisely coordinate those async
interactions between these distinct actors made those tests extremely awkward
to write compared to something using playwright/puppeteer, where you just
coordinate promises like you usually do.

That's before we get into the limitations around single domain per test which
meant we couldn't effectively test our login flow that uses a separate domain
for login for OIDC.

------
UweSchmidt
The true test for such a tool are usually the edge cases; in my opinion the
web is simply too finnicky, and all those well-meaning custom algorithms I've
tried fell way short.

I would recommend Ranorex, which combines a comfortable record-replay
functionality (which creates real C# code) and a code-first approach and
everything in-between. A powerful "spy" assists in creating the most
sophisticated locators and turns them into first-class Ranorex objects; a
shortcut jumps from code to the repository and back; duplicate elements are
detected on the fly.

------
IAmNotAFix
This assumes the URL is publicly available, or is there an on-premise offer?

~~~
tmcneal
We don't offer an on-premise version, but we do have customers testing
environments not accessible to the public internet. They're doing it by allow-
listing a static IP and we configure all traffic to come from that IP for that
account. There's other options if a static IP allow-list doesn't work - we've
specced an approach for a prospect where we would set up an agent inside their
firewall that we use to access their internal environments. This is an
approach used by other tools to access secured environments - we haven't done
it ourselves yet though.

------
chrisweekly
Wow, this looks really useful and well-thought out. Also, kudos for the
compelling summary writeup / pitch here. Bookmarked, will def be giving this a
try.

~~~
fitzn
Thanks! Feel to get in touch if you hit any snags, info@, todd@ and fitz@

------
YPCrumble
Similar product I’ve had a great experience with is
[https://ghostinspector.com](https://ghostinspector.com)

~~~
ohadron
I use it too and I really like it.

However the premise of having a test editor in a VM where I can time travel to
a specific point in time and add steps from there could save me a bunch of
time. Also the multiple selectors per element and the ability to target React
component names sound really cool.

Happy for the competition :)

------
o_____________o
Took a look, nice job!

Would be nice:

1\. Clearer setting for notification email

2\. The ability to target an area for an element change without knowing what
the elements will be. Example: filter for recent items without foreknowledge
of which items will appear.

3\. Ability to traverse up DOM (to select parent/s) based on a selector that
was too specific. Encountering this quite a bit.

~~~
fitzn
Thank you for this feedback! I opened an issue for each :)

------
jameslk
How does this compare to other existing "no code" SaaS regression testing
tools such as screenster.io?

------
foreigner
Looks neato. Are tests versioned? I'd like to be able to see a textual diff to
changes in tests over time.

~~~
tmcneal
Thanks! Yep tests are versioned - if you click on the 'History' button when
viewing a test you'll see all the executions within your retention period. We
don't show a diff view of historical changes, however we do show a diff view
when a text change causes your test to fail. So say if your 'Log In' button
changed to 'Sign In', we would show you a diff of that text change and give
you an option to make 'Sign In' the new baseline used when running future
tests.

~~~
foreigner
That's not the kind of versioning I intended. What I'm interested in is if I
make some changes to the test setup in your UI, is there a way to see what I
changed, and when those changes happened in the past?

~~~
tmcneal
Ah - sorry for misunderstanding. No we don't have a view like that, though I
can see that being pretty handy. This is great feedback - thank you!

------
Trufa
Hey! Product looks great! My main question would be, for a rapidly evolving
product would it generate a lot of false positives?

I really enjoy the idea of business/marketing people collaborating with tests.
Congrats.

I also really like your business model so will give it a try.

~~~
tmcneal
False positive failures are a really common issue with existing E2E testing
tools, so we try to do a number of things to prevent them in Reflect tests:

\- We generate multiple selectors for each element you interact with. So if in
the future you change a class or id, we'll fallback to other selectors when
running your test. We also only choose a selector if (1) it matches a single
element in the DOM, and (2) that element is interactable (e.g. not completely
hidden), and (3) if it has text associated with it then we'll only choose it
if the text captured at recording time matches the text at run time. This
helps prevent us from selecting the wrong element when running a test.

\- For React apps, we use heuristics to ignore classes generated by CSS-in-JS
classes like Styled Components and EmotionJS. We also have the ability to
target elements based on React component name (it requires a one-time setup on
your side to enable this)

\- For Visual Testing failures (e.g. you've screenshotted an element and now
that element has a different UI) we have a simple workflow to 'Accept Changes'
and mark the new UI as the new baseline for tests going forward.

Certainly more to do here but this is one of the key problems we're looking to
tackle.

~~~
Trufa
Thank you, there are two things that are either a bug, or hard to figure out.

1) I can't delete steps

2) I can't hover.

Great product guys, keep it up!

~~~
fitzn
Thanks for this feedback!

1) You can delete steps on a recorded test but not during the recording. This
is to ensure that we have an exact copy of the initial recording. Thereafter,
you can click on the test step and click "Delete" on the middle pane.

2) These should be captured out of the box. Can you email me your test URL so
I can investigate? fitz at Thank you!

------
ggregoire
Looks good. A small suggestion for the website: perhaps I missed it, but I
watched both videos "record a test" and "view test results" on the front page
and I didn't see Reflect detects an actual regression.

~~~
fitzn
Good point. The test failure case is something we should show. Thank you!

------
catchmeifyoucan
Are there any plans to support electron apps. I have a react-based Electron
App, and also would love to test a few things that are relevant to the app
loading and getting data from disk. Just another use case to consider.

~~~
fitzn
Hey, no plans to support electron apps right now. If your app has a web-based
portal (or started as web based), then you could do test that, of course.

------
d0m
Congrats on the launch, it looks awesome. I wished I had that a few years ago
and will definitely give it a try

------
ck_one
I assume it's not possible to clean up after a test since you directly
interact with the web app, right?

~~~
fitzn
Current customers typically either run a "clean up" Reflect test that deletes
account state that was created in previous tests, or they have a periodic
internal job/logic that auto-deletes all test accounts. The next day's Reflect
tests can sign-up fresh, for example, and use the new accounts for tests.

------
cbenincasa
This is awesome, congrats guys!

------
gfodor
How well would this work with a hybrid DOM/WebGL application?

~~~
tmcneal
If the elements you interact with are DOM elements and non-DOM elements within
a canvas then we should be able to detect and replicate those actions. The
biggest remaining issue might be performance - our VMs don't have a GPU
attached so depending on the application it may be slow because WebGL is not
running with hardware acceleration.

------
browsergap
I also had the idea to run browsers in the cloud and intercept the interaction
stream to record and replay interactions, also for the reasons you say,
because it reduces a large class of bugs. But not for test automation, rather
for web scraping. I applied to YC with this around 9 times. Great to see you
convince them this is a good thing!

Are you using the Chrome DevTools protocol? That's what I built on top of. You
can see my open-source code for this here:
[https://github.com/dosyago/BrowserGap](https://github.com/dosyago/BrowserGap)

And a demo running right now of the interactive cloud browsers (without
automation, it's WIP):

[https://start.cloudbrowser.xyz](https://start.cloudbrowser.xyz) [0]

[0]: Currently on Show
[https://news.ycombinator.com/item?id=23904243](https://news.ycombinator.com/item?id=23904243)

Fitz and Todd, wanna connect?

~~~
picodguyo
"...rather for web scraping. I applied to YC with this around 9 times"

There actually was a YC company that did exactly this several years ago.
[http://www.kimonolabs.com/](http://www.kimonolabs.com/)

~~~
browsergap
Sort of but not exactly, because IIRC they used a downloadable client. I
wanted something that could run anywhere easily, hence the browser client. I
used KL a few times, and followed their story, and found it interesting they
moved to Tokyo. At the time I considered Import.IO, KL, Connotate, Mozenda and
DiffBot to be the main competitors. My main idea as a point of difference was
to be able to share the structure you've mapped on web apps, and share the
action sequences as things others can use, so everyone gets a network effect
benefit from the work of everyone else.

Thanks for posting. I don't think I knew they were YC, and did not know they
were acquired by Palantir. That's interesting, and good news. Means the market
is growing.

------
inglor
Hey congrats, I work for Testim ( [https://testim.io](https://testim.io) )
which I assume is somewhat competition?

Excited to see more players in this space - good luck! Most of the market is
still doing manual QA and that has to change.

