

Ask HN: Rate my startup, Go Test It (cross-browser testing) - martinkl

Hello HN,<p>Just recently, at Future of Web Apps in London, we launched our startup, Go Test It. Our goal is to make automated functional testing of web applications incredibly simple and efficient.<p>We have an infrastructure hosting various different browsers, and we have a fantastic test recorder which can pick up mouse clicks and keyboard events to create test scripts. Test scripts can be converted to Ruby or Python, and run directly on our infrastructure.<p>It would be awesome to get your feedback on what you think and how you would use Go Test It in your own projects.<p>Please sign up for free at:
http://go-test.it/hackernews
(use "hackernews" as invite code)<p>Thanks :)
======
spokey
Nice. And that's coming from someone who has been thinking about an app like
this for quite some time.

One minor thing: On the "Welcome to Go Test It!" page (/welcome), you have "I
have existing Selenium scripts that I want to upload." and "I do not yet have
any test scripts." I bristled at that slightly. I _do_ have a bunch of test
scripts (that I worked pretty hard to create), but they are not in a ruby,
python or selenese form. Maybe something in the gray text for the second
option that acknowledges something like "or I don't have scripts in a Selenium
format" would create less of a false dichotomy and help me feel better about
myself than selecting "I do not have test scripts" does. Maybe I'm overly
sensitive.

One bigger thing: I like the options for the "New Test Script"
(projects/*/scripts/new) page. I'll probably go back and install the Firefox
plugin when I have more time, but I wanted to see how it works quickly, so I
backed up and went to the "Write a new script in a text editor" option. Now
I'm presented with a blank text editor, with little or no clue as to what I'm
supposed to do there. At the very least you should have links to samples or
documentation for the scripting language, but you could do better than that.
You know my app's URL at this point. You could pre-populate this script (for
new users) with a basic "Is it up?" script, maybe checking the title and some
H1 text on the page, just to give me a working example to modify. I think for
new users you should minimize the number of steps to a running test. As it
stands I either need to install a firefox plugin (a pretty big hurdle, imo,
although it is right to offer that) or (I'm guessing) go off and research how
to write Selenium scripts and then come back and create one on your site. As a
random user looking to solve my functional testing problem, I'm not that
motivated to do either. Once I'm off writing Selenium code I'm likely to
forget about GoTestIt. I'd suggest that you either auto-populate a basic
script for new users, so that I can click "Run Test" right away and see what
happens or find a way to record basic scripts directly in the browser without
installing a plugin (or both).

(Also, personally I find the "Feedback" tab on the left to be a little
distracting, but I'm on a netbook so my monitor is smaller than most. Of
course, it looks a little funny when glued to the scroll bar too. I wish
UserVoice would let you attach that to the top or bottom of the screen.)

~~~
martinkl
Thanks, those are some very good points. We've tried to make it easy to get
started, but clearly we can do better still.

In what language are your test scripts currently?

~~~
spokey
I have a bit of history with writing automated functional tests and testing
frameworks. I've used a large number of tools, most frequently HttpUnit,
custom scripts built on top of Apache's HttpClient, a Jakarta Commons
subproject called Latka that sort of died on the vine (although I think it may
have replaced Apache's watchdog servlet engine compatiblity test for a time),
custom Ruby scripts based on Hpricot, and others.

My current project is Drupal based so I'm using a php unit testing framework
called SimpleTest. I've found it pleasantly easy to work with for basic HTTP
and HTML level testing but it doesn't have any JavaScript or browser-driving
capablities, so I'm probably in your target market (i.e. I have unmet needs).

------
agmiklas
A pay-as-you-go pricing option might be useful. The monthly pricing model
works well if you plan to run the tests every day/week, but I'm guessing that
a lot of people will want to run the cross-browser tests on every new
deployment, but perhaps not between deploys.

~~~
pospischil
I strongly agree with this. Also, the prices seem quite steep to me -- though
they may not be. Instead it may be that I'm not seeing how many tests can be
completed with the allotted actions at each price level. The "what can I do
with 15000 actions" is a step in the right direction, but I think it would be
easier to comprehend if you said something like "a typical application of x
size can be thoroughly tested y times a month w/ z actions".

------
oneplusone
This seems really really expensive.

Say it takes me 10 actions to fully test one feature and all its cases. Spread
that across FireFox 3 + 3.5, Safari, and IE 6 + 7 + 8, that would take a total
of 60 actions for one feature. With a web app that has 20 features testing it
once a week I would have to pay $326. What if a test fails, do I have to pay
again?

I don't see how anybody can use this with such a hefty price tag.

~~~
apinstein
I have seen a bunch of these and many are priced at this level, which is
around US $0.15 per test. It astounds me as it seems ridiculously high.

Most of my automated tests run in about 5 seconds or less. With EC2 prices at
about $0.10 per _hour_ , I can't imagine ever using a service with such a high
premium. Not only because the price is nominally expensive, but also because
it's a commodity service that's being priced at a HUGE premium.

For me, I'd consider the service if I could offload my grid testing to the
cloud for around $20 / month.

That way I could run a bunch of smoke tests after every push to make sure
nothing broke and a larger set before commits. But that can get to be a lot of
tests especially when you re-run for failures.

------
jeroen
Looks nice and useful, but scheduling is a bit unclear to me:

I have 1 testscript, 7 browsers, which makes 7 tests. If I schedule to run
every "day", will those 7 tests run every day, or 1 a day? I expect the first,
but "Our scheduler will try to start one of the tests above with the start
times separated by the time you specify." seems to imply the second.

(oh, and check this out: <http://vldtr.com/?key=go-test.it> )

------
pplante
I love the service. Its a great idea, and the scriptability of it makes
integration into an existing pipeline extremely easy.

All praise aside I cannot say I personally would use it much beyond
experimenting with it. My project uses google maps too extensively and ends up
failing out too much. Thats more my problem than yours as scripting events for
GMaps would be overly complicated and useful to only a few.

Thanks for the demo!

~~~
martinkl
Google Maps is actually one of the sites which we use internally for testing
Go Test It -- many features of it can be automated quite nicely. Particularly
with such complex JavaScript, you don't really have a choice but to test in
every browser you want to support!

------
adamt
I like it.

Sign-up nice and concept good.

Couple of comments/bugs: 1) When you first record something and click 'run' it
brings up a confusing dialog box with a textbox. I figured out eventually this
is a save dialog box, but it isn't obvious

2) Appreciate the need to queue things up, but I didn't find the queue status
thing very intuitive.

3) I used news.bbc.co.uk as my test site, and all the IE ones died opening the
front page.

~~~
martinkl
Thanks for your comments -- I know we need more work on the user experience.
We'll also look into those IE failures.

------
seajosh
I did something similar a couple of years ago called Blue Violin. It was a web
app that recorded web events, saved them, then ran the test script. The source
code is available at <http://code.google.com/p/blueviolin/> \- you might find
some code of use to you. If you have any questions, you can find me at
@JoshWatts or josh.watts at gmail.com.

------
jusob
Support of Selenium scripts is great. I was afraid only Ruby/Python were
supported from your description.

How can I delete or rename existing scripts?

it would be great to have more browser available: Safari on Windows & Mac,
Opera 8 to 10, Chrome, etc., along wit hthe option to disable javascript, Jva,
Flash, etc.

------
BlueSkies
I like the concept and the sign-up was easy.

I noted that you can't run a test until you first save it, even though there
is a Run button. You may wish to make this piece a little smoother.

I got a little confused by the queueing though. Was I supposed to wait when it
said queueing? I did, but nothing happened for a while. Then I selected the
browser type again and clicked the button to run the test. This time I had my
results.

My problem was that my application requires authentication - in particular
Twitter OAuth. Do you have any plans to address authentication in general. I
didn't know how to work around it, so I couldn't evaluate the playback.

I tried to leave a comment, but when I submitted it, the message came back
"Not Found".

\- Scott

------
hwijaya
Clickable link: <http://go-test.it/hackernews>

------
bluebird
I am in the market for such an application and my budget is up to $150 per
month but I find the 'action based' pricing model a huge turn-off. If I am
paying $50 upward, I want to be able to automate tests freely and run them
nightly without ever thinking how many 'actions' I have left.

The UI is decent, it lacks support for pages that require authentication from
what I can see, and I couldn't find any features for running tests regularly
(e.g. nightly).

I find it hard to imagine that anyone would sign-up and pay that much money to
be limited by an 'actions' meter and for an application that is not mature by
any means.

If you adjust the pricing and address some features it could be a good
application though.

------
bensummers
The signup process is rather effective -- gets you to running your first test
nice and quickly.

~~~
ErrantX
I agree the sign up process is very slick - I like it a lot.

------
avk
How does this differ from Selenium or what the guys at Sauce Labs are doing?

~~~
rams
The Sauce labs guys assume you have already recorded the script. With go-
test.it you have a firefox extension that does the recording and uploading of
scripts. Jason Huggins doesn't have a good opinion about recorders/IDEs.
Selenium is very developer focussed (As JH himself said at one of his
talks),but most testers don't code. Go-test.it seems to be based on Selenium
and focussed on the average tester. I know for a fact that there are large
companies that are willing to pay for this kind of product.

Update #1: Saucelabs records the video of tests which go-test.it currently
doesn't seem to do.

------
charlesmarshall
Now that is a really useful application! well done guys, if the price is right
(bit high at the moment for the individual / small business)

------
keeptrying
I just ran through bunch of tests for my application. This is very nicely
done.

------
ynniv
How does this compare to BrowserCam? [ <http://browsercam.com> ]

(disclosure: I work next to the BrowserCam team)

~~~
martinkl
Go Test It is much more focussed on the browser automation and functional
testing; BrowserCam is more about checking that the design looks right.

~~~
ynniv
BrowserCam does screenshots, selenium based testing, and raw VNC support, but
lacks the site design and product focus. They are part of a larger company and
don't have the same amount of control over external appearances.

I didn't mean to imply that Go Test It is inferior to BrowserCam, I was
actually curious what others throught. There may be a Mac / PC analogy here -
both have almost the same features and so the primary differentiators are
price, design focus, and product integration. If you aren't a Gomez customer,
that last one isn't going to matter.

UPDATE: I'm getting my product names wrong. BrowserCam became a Gomez product
called Reality View, and only provides screen shots. Selenium testing is done
in a related product called Reality Check.

------
aik
Very cool. As other people have have said, a pay as you go payment plan would
be pretty cool. I'd use it.

------
swombat
Can you make it work with Flex apps? If so, you've got a client. If not, I'll
have to pass, unfortunately.

~~~
martinkl
Sadly, we cannot yet easily interact with Flash or Flex apps. It's something
we want to do, but not right now unfortunately.

------
dawie
I would use it. I feel that your pricing is too expensive. I would pay $10
maybe $15 as a small company.

------
jmtame
what a bittersweet solution ;) "oh cool! look at all these bugs--hey wait..."

------
etherealG
really awesome service. very very impressive. price point seems a bit high for
another small company to be using, but corporates would jump all over it. hope
it takes off, well done.

------
prabodh
Does Go TestIt solves any problems which selenium doesnot solve...?

~~~
martinkl
It is actually based on Selenium internally. The main advantages are:

* Go Test It provides a hosted infrastructure, so you don't have to spend lots of time configuring and maintaining various VMs with different browser versions;

* Go Test It has a lovely test recorder which is much nicer to use and works better than Selenium IDE;

* Many little tweaks and improvements, like automatic screenshots etc.

------
gommm
Very good idea, I could see myself using this...

