
Ask HN: Solo app devs, how do you do user testing? - rhardih
Enlisting the help of friends and family can only get you so far, so how do you gather quality feedback pre&#x2F;post-launch, without shelling out for a professional agency to do the testing?
======
jordansmithnz
Solo dev for an app with hundreds of thousands of users. These are the things
I do that ensure smooth releases:

• run a beta test group. These users consist mostly of users that have emailed
in with feature suggestions or bugs. I ask them if they’d like to join the
beta (e.g. they can check their bug is fixed). I frequently get great feedback
from a few of these users.

• fiverr.com. Shell out $100 spread across a range of testers, and give them
all slightly different briefings. Some feedback is terrible but there’s some
gold there that’s really worth far more than $100.

• learn how to test your own product. It’s hard to do well, but you can learn.
It’s a valuable skill that can really set you apart in a regular engineering
job too.

• used phased releases. Pause the release at 1%, give it a few days for any
minor issues to roll in and fix these before releasing to 100%.

~~~
feelix
Also a "Solo dev for an app with hundreds of thousands of users"

I just test everything myself. On one OS (Mac). The worst part is having to
run VMWare with older OS versions sometimes, but not usually.

I find that bugs are almost never OS specific, and are rather just logic
errors.

I find that having other people test my code adds little value, as I can
literally just test it myself in the OS that I write it in most the time.

I know that this goes against all advice that you read, and all the "common
knowledge" that is out there... but I've been doing this for 15 years and have
got over 1 million users in total probably, and I almost never get support
emails regarding bug reports. So, in my experience, I really find that
everything I have heard is wrong. I realize that other people may have other
experiences but found this point of reference interesting.

~~~
j45
Experience building software can lower the testing requirements.

I find the degree of testing needed can be relative to the difficulty of the
problem, the unnecessary complexity of the solution, and the average skill of
the developers working on all 3.

Clever (simple) architecture will often beat clever programming in the long
run.

~~~
newsbinator
> Experience building software can lower the testing requirements.

This is a good point. The longer I've been at this, the less often, "this is
likely to work" is a thought that I let go unchallenged.

~~~
retbull
"This is totally going to work" is only said when I am completely sure that it
is going to fail catastrophically. If anything works I feel mild distrust with
my code from that point forward.

~~~
pier25
I think it's only fair, as a developer, to live in a state of constant fear
and paranoia.

(joking)

------
zer00eyz
It doesn't matter how big you are, the same cheep method works for informal
but fairly accurate quick and dirty testing.

Beer.

You think I'm kidding but if you go to a bar, there are lots of people who
will test whatever you put in front of them for a free drink. If you want to
keep your costs down even further then go during happy hour.

Is that "user testing" no... not in the sense we think of it today. Is it
testing of the usability of your product: yes it is. Im guessing that most
apps issues that are more likely the latter and not the former. Basic things
that you think are clear that really aren't (your too close to the problem and
the solution). If you explain, point and guide through the same aspect of your
feature 10 times in 10 tests people might not get it! These are the problems
you might want to solve, dumb it down, change it up, make it better.

Does this work? Yes, as long as you can moderate. If you can you be neutral
and friendly and avoid "traps" that will alter your results. Saying things
like "MY app" or reacting personally will alter your results, and learning
when to "guide them" through a frustrating point and when to let them puzzle
through it is a bit of an art (this too offers clues if your guiding people at
the same place). For me personally a drink or two helps ME in the process...

I have also discovered that this process works very well with pairs of people.
Two friends already have a rapport and will happily talk to each other to try
to get through something problematic. One can encourage or discourage the
other and their dialog can be enlightening. Letting them play together with
your app and drink free drinks can give you a LOT more insight than the next
three individuals testing. Anything more than two people becomes problematic
so look to avoid large groups.

~~~
daleco
Guerilla testing is a great starting point. Do you usualy give one or multiple
scenarios to complete? Be aware that you are not controlling the environment
and the users may be distracted by outside factors. Which may affect some of
your metrics (time to perform tasks, success rate...)

~~~
zer00eyz
> Do you usualy give one or multiple scenarios to complete?

I am OP, and it depends.

Im old enough that I have done this with paper prototypes, and it is
shockingly effective.

Now most of the code I write is compartmentalized enough that with a bit of
effort I can come up with a one off digital artifact. Just rip out the chunk
of your app that you want to test and present ONLY that. Spend a day building
ugly shims early enough in the process and you might make some dramatic
changes!

> Be aware that you are not controlling the environment and the users may be
> distracted by outside factors. Which may affect some of your metrics (time
> to perform tasks, success rate...)

Even with those things going on your going to be shocked at not only what
issues come up but how frequently. That thing you did that you thought was
simple and sane, no one gets it. Do more than half of your test subjects
comment on the same thing (good, bad, both) then you have a problem.

The key isn't controlling the key is listening...

I also want to state that a slighty drunk, distracted bar patron, getting
through your process without quitting, telling you your product is shit, or
giving up is a positive sign that in the real world people who are more
motivated than "beer" are going to have a fairly positive and expedient
experience.

------
eps
1\. Stick the mailing list sign-up form on a landing page and do a bit of
promoting in relevant places. Specifically say it's for beta announcements,
preview release testing and picking everyone's brain. This should net you some
users to bootstrap the process with, and they _will_ be responsive.

2\. Release the beta, mark it as such and ask people for feedback after they
used the product for a bit. Listen to the feedback, act on what's relevant,
keep improving the beta. Give kudos where they are due.

Have public forum, actively participate in all threads. Many people look at
how active the support forum is before trying the goods or leaving any
feedback. So the more active it is, the better. Also, moderate it aggressively
to trim junk and trivialities.

3\. Once out of the beta, don't forget to _genereously_ reward everyone who
chipped in. Ideally with a special gift, like a specially tagged lifetime
premium account, a handful of most expensive licenses, etc. The worst thing to
do is to offer a discount on a production version - this comes across as a
cheap and greedy move.

------
jermaustin1
In the past (when I was still a solo developer), after being 90% confident the
code won't completely break the environment or do something screwy with user
data, sticking the code in production and funneling 50% of the requests to it
while watching NewRelic.

There are no better user testers than your users, and the risk is only
moderate. I could always pull the server down and restore the previous version
while trying to fix whatever I broke in the deployment.

But if you aren't doing SaaS, user testing as a solo developer is not really
feasible. The best user testing I can think of that doesn't "cost" is a
Beta/RC Program for your users to volunteer to get potentially broke software
for a discount or just early feature releases. For this to work though, you
have to have enough users.

~~~
rhardih
I agree, it's easier to do for web apps since you don't have to contend with
"app store reviews", that can potentially leave a lasting blemish on your
product. So far I've done it as you describe as well and it's the "getting
enough users" part that's a pickle. If you get X users on to sign up for a
beta, only a fraction of those will actually ever install the app and give it
a spin and again, only a fraction of those will give you any real feedback to
work from. Finding real testers without breaking the budget, of your (maybe
zero-profit) app is the challenge.

~~~
jermaustin1
I've never trusted a user to provide feedback. They will complain if something
if it's terribly broken, but not if it's still usable.

That's why I have always used a 3rd party system to collect actual usage
analytics combined with server logs.

------
roryisok
I have an app in the windows store with a few thousand users. When anyone
emails me with feedback or an issue, I usually ask if they want to try the
beta version (after I've resolved their issue of course). I've gotten about
15-20 part time testers this way, they've been a mixed bag. Some provide pages
if detailed feedback, some barely anything.

When I have a decent, bug free mvp of the next version I'm going to fork out
for usertesting.com

~~~
paul7986
Forget usertesting.com use Mechanical Turk(get testers to download screen
record plug in for chrome to record/send you their tests..charge $10 or less a
test). Way cheaper and you get more screen recorded tests for less.

------
WA
Post-launch:

What's your goal? Do you want to test whether your update works correctly on
many devices? Do you want to get ideas for new features? How many users do you
have?

I have about 10,000 active users. I don't do much testing. I test myself on an
Android device and an iPhone. Since my app is built with Ionic and based on
Cordova, it works anyways and real crashes are not to expected.

Feedback is easy: I just wait until people write me an email. It's funny:
Sometimes, you don't hear about a feature request or some issue for months.
Then 2-3 people write you within a day. Then nothing again.

Back when I started, I implemented a lot of features. Now I basically reject
almost every feature request, unless it comes up extremely often in emails. I
sometimes take notes, but most of the time, I type a quick reply, thanking for
the suggestion, archive the email and only try to get a feeling for what
people want.

Obviously, this has its limits, but since I know my core users and what they
want, it works for me.

If there's a serious bug, people will let you know soon enough.

~~~
rhardih
Post-launch is mainly vetting that new features makes sense to users and
catching UX faux pas, instability or even crashes, etc. before opening up to a
wider audience, and possibly scores of bad reviews.

I was thinking about just opening up on the app store as well, and just sit
back and wait for reviews or emails, but that seems risky, since potentially
bad reviews are forever. For a paid app at least, that can have a pretty
serious impact on possible uptake. Having at least some level on confidence,
that the app is usable and stable before general availability, is the safe
play I think. To that affect, closed testing seems the only viable option.

------
nickjj
I wouldn't ever pay for an agency to do user testing, and it has nothing to do
with money.

Your ideal tester is someone who cares enough about your product to use it.

I also wouldn't even start developing the product unless I had a decent sized
list of people who really want it.

In which case, testing it is easy. Just ask those people to test drive your
app during development. If your app solves a real problem they will happily do
it for free since it solves their problem, but you could also give them perks
like giving them 2 months of your service for free when it launches.

Your users will naturally give you feedback if things are broken, so that
handles post-launch real world testing. Of course you'll want to supplement
that with heaps of automated tests too.

~~~
daleco
An agency or proper usability testing should provide you insights and ways to
correct the usability issues based on the results. What you describe may work
to get feedbacks and new feature requests. But you won't know why a user is
not using a feature, where they are struggling, what they don't understand or
see.

~~~
nickjj
> An agency or proper usability testing should provide you insights and ways
> to correct the usability issues based on the results. What you describe may
> work to get feedbacks and new feature requests. But you won't know why a
> user is not using a feature, where they are struggling, what they don't
> understand or see.

If you use heatmaps and other analytical tracking tools you can see how users
interact with your app.

That's all you really need to see why a user isn't using a feature, struggling
with the UI or happen to not see something.

An agency would just have X number of paid employees systematically going
through your UI and giving you their opinions on what they think is best, but
that might not align with what your real users are doing.

------
sarabande
I've used UserTesting.com for previous tests. If your product affects a
lowest-common-denominator user or you are testing mainly UX/interaction, I'd
recommend it. At that time I paid around $20 per usability test (I bought a
packet of multiple sessions). Testers recorded using my product for 5 to 10
minutes and it was great watching them stumble over what I had designed. They
were also trained to be over-communicative and think aloud, which really
helped. In total I did four or five of them before I had my first release
(since it was still somewhat expensive, I tried to ration my tests by fixing
all previous UX bugs before issuing another one). It's been a long time, but
if I recall correctly, a few times the testers weren't tech-competent enough
to actually do the test, but when I asked UserTesting for a credit or refund
to re-do the test because the tester was "bad", they provided one without
hassle.

You need to be make it as easy as possible to run your user test, because a
tester will give up on roadblocks that take more than 60 seconds to solve.

I was developing a language learning product, and so needed testers who were
interested in language X. In the end, the people I found using the filter
[Interested in X language, native English] weren't actually interested in
language X. Rather, they came across as people who had accepted the filter to
make an extra buck.

However, it was good enough for a random internet trial. If I wanted better
results I had to do it in person (which I also recommend).

------
jakobegger
If I need to get feedback on a feature, I build a prototype, and then show it
to one or two people (friends, family, customers that sent feature requests).
I tell them roughly what they should do, and watch if they can figure it out.
Then I change things as necessary, and show it to the next person. Usually a
handful of people are enough to find the the major issues.

\- don't "burn" beta testers. show a prototype to one or two people, and don't
show it to more people before you fixed the issues that have come up.
otherwise you just get duplicate feedback

\- ask people personally. Emails to a group of people are ignored by 99% of
recipients

\- feel free to ignore advise from testers. everyone always wants to give
advise, but that is irrelevant. the important part is: did the tester manage
to complete the task?

\- do not ignore people struggling with your app. the natural reaction to
people failing to use your app is to think they are stupid. they aren't. most
of the time your ui is stupid and you need to fix it. it doesn't matter how
brilliant you think the ui is, if testers can't use it, change it.

\- sometimes the answer is that what you built doesn't work and you just
wasted a lot of time and you need to throw it away and start over. do not
ignore this possibility

------
paul7986
For web apps I’ve always used Mechanical Turk and paid each user $10 vs. just
a dollar or less. I made each tester download a screen record plug in for
chrome, hit record and then go through all the steps in the test. Once
finished they emailed me the recording.

The above is what you get from usertesting.com and other sites but pay up the
nose($50 a test) vs. $10 or less dollars per test.

If it’s a downloadable app Just spin up a web UI/UX and have the turkers
follow same steps but have them change the browser to say iPhone 6/7/8 view.

~~~
netzone
I don't understand this, why would you want people to test something that's
already predefined? You could automate that. You want real users that will use
the app for real things. Blind testing from a list seems kind of pointless.

~~~
daleco
The goal is to detect usability issues. Your features should have user stories
associated to them, come up with few scenarios around these (e.g. book a
flight, cancel your flight from the landing page...). Then test your scenarios
with multiple users/personas. This will allow you to detect UX/usabilities
issues, and prioritize update based on how critical the issue is.

~~~
paul7986
Exactly and with multiple recordings of each user story you will be able to
see if your design is idiot proof or not.

------
acutesoftware
The most important way to do this is to actually be a user of your product,
e.g. "Eating your own dog food".

This doesn't mean logging in, trying all the features and playing with it for
an hour - you have to _really_ use it with your data on a daily basis over a
long period of time. You'll come across a lot of bugs and start logging
feature requests yourself.

Once _you_ can use it without issues [and you'll need to be careful to split
the wishlist features you come up with, against the actual bugs which need to
be fixed], then you can get others to try it.

Best way to get user testing from others?

\- email your users and ask them if they have had any issues, or want any new
features.

\- ask people to give your product a quick test - for example,
[https://www.lifepim.com](https://www.lifepim.com) is my new webapp to easily
manage your personal information [shamless plug/warning - notes and tasks work
well, contacts needs work, and calendar is not pretty]

\- pay particular attention to requests from users who pay for your product.
Everyone wants software to work in a certain way, but when someone _pays_ , it
can be a good indicator to listen to what they want. If they say something
isn't working, make it a priority to investigate and address this.

~~~
aggie
Just keep in mind with dogfooding that there is a difference between figuring
out how things work and working efficiently/effectively once you have figured
out how things work. Dogfooding is really only good for 'testing' the latter,
because you already know how everything works better than any normal user
would.

~~~
acutesoftware
The best part I've found with dogfooding is that even though you know exactly
how it all works (and yes, you can miss some issues because of this) ,
actually using the product daily can make you realise - this isn't working
well or that feature is kind of a pain in the neck.

------
wilperkins
I usually hang out in the kitchen at our shared office space and ask people.
If that's not enough, then I'll start visiting friends' offices for team
breakfast or post-work beers and do the same.

------
otterpro
There are user-testing companies that does this for small fee:

    
    
       * http://www.trymyui.com/
       * https://www.usertesting.com/
       * https://Whatusersdo.com/
       * https://Userlytics.com/
    

Disclaimer: I've also worked on couple of those companies as a tester when I
needed money and I was out of job, but thankfully I'm currently back working
as developer again. On some of them, the tester also needed to record
themselves using webcam.

------
drrob
I'd say it depends on the sort of application you're developing.

In the next couple of weeks I'm due to release a boxing game (shameless plug:
leatherthegame.com), and other than myself my only tester has been my cousin.

Having worked on the project for over 3 years I'm aware that there are areas
where I "can't see the wood for the trees", but I am confident in my own
knowledge, and only wanted an external viewpoint to see how a noob would see
things from afresh (both in terms of not having used the app before and also
him being only a lite-casual boxing fan, so he wasn't up-to-speed with some
terminology).

I know full well that once its released the proper boxing fans will get in
touch, and I'll change things post-release from that feedback, so my strategy
is very much a post-launch end user feedback one. In an ideal world I would
have had more friends who could help me test it, but there's only so many
opinions I can factor in pre-launch, and managing their experiences and
feedback would have been a mini-project to manage purely by itself. When
already time-constrained with launch schedules, app store listings and last
minute bug hunts and device optimisation I couldn't spare the time to manage a
phalanx of testers.

This has been a passion project/scratch-my-own-itch project, which is why I
feel more secure not having many outside opinions. If you're building
something more strategic, to fill a target niche in a specific market say,
then my approach will DEFINITELY NOT WORK. In this case you'd be best off
firing out a message on LinkedIn or something like that, asking for would-be
best testers.

------
euonotn34ntuent
I used to be a solo dev working in a niche field where I made plug-ins for
some creative apps. I found that one thing that really helped was being active
in the forums where users were, and (if allowed by the forum) posting that you
have a product and you'd like to get some people to test it. I'd then require
them to fill out a simple form that gathered some information about how they
work.

The form served 2 purposes:

1) It gave me the ability to balance different types of users. Hobbyists vs.
professionals. People in this specialty vs. that specialty. etc.

2) It weeded out people who just wanted something free and weren't interested
in giving feedback. Seriously, a simple form requiring a name, an email, and
the answer to 3-5 questions helped get rid of the freeloaders. But don't make
it too onerous or you won't end up with any testers.

I'd then usually give a free license for the released product to anyone who
provided useful feedback.

------
jqbx_jason
Using your product daily is essential not only to help steer the ship on what
to build but also to cue you in on what's buggy / inefficient. I have the good
fortune of having a live chat interface (product is:
[https://www.jqbx.fm](https://www.jqbx.fm)) so I get pretty quick feedback if
something that's been pushed is causing anyone any issues. However to keep
bugs to a minimum I keep most of the logic client side and have a dev
environment that I can run on production- I usually test a version for a
couple days (see: eat your own dogfood) before releasing anything. Also
forcing yourself to review your own code is a good skill and has helped catch
some issues right before pulling the trigger.

------
superasn
I generally find sellers on fiverr. They are willing to record screen and
comment for about 30 mins for $5. Not a bad deal and mostly good insightful
feedback.

------
no_gravity
One thing I have found to be very useful is to ask my existing userbase of
other projects if they want to test something new and give me feedback.

In fact this worked so well that I am currently building a new project around
this. A site that lets startups gather feedback from people who are interested
in trying out new sites and giving feedback.

It is currently in private beta with about 20 startups. If you want an invite,
feel free to send me an email.

------
fundamental
Try to make a simple tutorial video of some sort. Even if it isn't useful in a
practical sense it forces yourself (or someone else) to walk through the UI
stating their intentions and trying to execute them. Some of the best feedback
I've gotten was through watching videos like this for an application that I
work on as you can see where the stumbling points are throughout the process.

------
buf
I push the code live and see what the numbers say.

I know this isn't the answer most people want to hear, but no one tells the
truth like live users.

------
creo
I give it to my friends and ask them to do specific things. Most of the time
they will point me to something i didn't know.

~~~
rhardih
That's currently where I'm at myself. Do you feel that, that approach scales
fine and you get a broad enough idea of where problems are for the general
user and where you need to improve/change things?

~~~
ddebernardy
It doesn't scale, in that you ideally want to test your app's UI/UX with first
time users who haven't gotten used to your app's quirks yet. But it allows to
catch a lot of stuff early on at the cost of buying them a coffee or a beer
(if that).

------
lettergram
I do a free beta, then various paid betas. The goal is also to find a price
point. A user can either leave feedback in my betas and get that month free,
or they can pay. The idea that all they have to do is give feedback for a free
service usually works well...

Eventually, when it's easier to pay, some users will start leaving money
instead if feedback. At some point I'll raise prices a bit and stop the
feedback for service system.

I've done this for:

[https://projectpiglet.com/](https://projectpiglet.com/) and
[https://easy-a.net/](https://easy-a.net/)

And a few others, it's always fairly successful - _assuming you have a decent
landing page_ and you can advertise the site in your target market for cheap /
free.

------
gotrythis
I didn't read all the comments, so don't know if anyone else suggested this.

A few times now, I have successfully got my first few hundred users by
offering a lifetime discount and positioning it as a limited access beta
testing program.

Got lots of feedback, and most users kept their accounts for a very long time,
even if they weren't using the software, just do they wouldn't lose their
discount.

------
mesozoic
Integrated tests are key. Not a solo dev but my company won't give me any QA
resources so the only way forward is integration tests.

~~~
segmondy
On projects I do solo dev, I do integrated tests because after a while the
project becomes big enough that trying to cover all paths takes too damn long.
My current project got to be about 70% testing/30% code and I was miserable. I
stopped all coding and moved to e2e tests and now I'm back to 95% coding / 5%
coding and quite happy. Most companies surprisingly don't care about full
integration tests, they talk about it but are quite happy to have fingers on
the keyboard and mouse pecking and clicking away.

~~~
mesozoic
Yeah pretty much the same here. It's funny when you start listening to actions
and incentives instead of words. You realize its all doublespeak.

------
troycarlson
One tool I use is Mouseflow, primarily for session playback. It's super
helpful to see how users are interacting with the app, qualitatively, and has
helped me find bugs and UX issues that never made it into the error logs.

[https://mouseflow.com](https://mouseflow.com)

------
segmondy
Automated testing.

I start out with manual tests.

If I have a bug at any part of the code I add a unit test.

When it gets too big and starts taking too long then I start writing e2e
tests.

~~~
gitgud
The menu system being "unclear" is not a bug you can find with automated
tests.

Its true that unit-tests ensure quality software. But this quality is a waste
if real user's can't figure it out.

~~~
segmondy
True, I'm not talking about UI/UX test, but tests that test that your
application is "correct" and does the right thing. For such tests, you need to
start out a beta or just wait for user feedback or also be great in product
design and thinking like the average user. Adding help popups, and good
documentations.

------
Radeo
Probably it was already said, but quite crucial in the process would be proper
analytics. To see where the user's flow is concentrated or stops at all.

------
sudouser
external agency, 1-2 weekly tests, specially critical in the first stages of
the product.

good price, cheap compared to a dev’s salary, or compared to my time, paying
less than $500 per month.

do 5-second tests, usability, card sorting, path testing, all ux and ui
testing for greater chance of success

------
bitwize
Test it myself, and occasionally get a friend to bang on it.

------
upbeatlinux
In production (sarcasm).

------
coreyjwaldin
Checkout helio.app

