Hacker News new | comments | show | ask | jobs | submit login
Why are there not more freelance testers out there? (testninjas.com)
56 points by rosiesherry 1693 days ago | hide | past | web | 58 comments | favorite

From my personal experience, I wouldn't use a freelance tester because they won't understand the system well enough to provide value for money.

The most valuable testing is that which exposes your lack of understanding of what the system should do. These also usually require the most extensive rewrites, so you definitely want them as early as possible. Unfortunately, they also require the most extensive understanding of the system as well as an ability to detail their thinking at a level that can be understood by the development team(s).

That's why I don't see the value of freelance testers, and I'd much rather have somebody employed on a more permanent basis, either part-time or full-time.

As true as your comment is I would like to point out that there is a value in itself to have 'fresh eyes' on QA. An experienced and talented tester unfamiliar with a system/product/service can actually work as both QA/UI and UX resource if he or she has the right background.

TLDR; Sometimes you can gain valuable input from a fresh pair of eyes.

The question then is how do you assess value for money before you use the service. I work as a freelance tester and I often see that customers see the value after working with me and then return for repeat business (as someone else has mentioned - you build a reputation).

As an example, an error in an iPad app was dismissed as a known connection issue by the dev team because there were no server side logs so the request wasn't hitting the server. This was the dev team's first iOS app. My specialist knowledge of mobile testing helped identify quickly that the app was connecting to the servers but that this one request wasn't and provide console logs showing where the communication error was occurring.

This customer were to launch in public the following week and this would have killed their reputation due to the critical nature of the function that was failing. This was my first day testing with them after they had put in a year of development effort and dev testing (they brought me in specifically because they felt they were getting "tunnel vision"). This was one bug masking behind another which I spotted due to 3 years experience in testing mobile apps. I think that provides value for money without being employed permanently, however I appreciate that companies/teams don't see that value until it's their reputation that has been protected by the investment.

That's the same argument for not hiring freelance developers or project managers. And IMHO it leads to lazy coupling of software systems and services, a system that has to be built and tested by different or changing groups has to have much much better scope and decoupling.

In short, if you have small (micro) services, they are easy to build and easier to test.

I disagree. With good testing discipline, you can bring in freelance developers. Developers have the luck of not needing 100% understanding, as this is grown through the writing of code.

Testing, on the other hand, is a simulation of how your system responds to user input, and how your users respond to the system's outputs.

If your tester can't simulate a user, then you can make your services as micro as you like - it'll make no difference when you get the phone call saying the system doesn't work as expected.

If new people picking up the app have a problem simulating users, then how do you assume your users will go about picking up the up to actually use it?

Read my comment below. It's not about users understanding the program per se, it's about testers having the domain knowledge to perform meaningful tests.

Without that domain knowledge, the testers may find implementation bugs, but they won't expose the issues caused by the developer's lack of understanding. The latter is almost always more valuable to find and more difficult to fix.

Oh, ok. I guess writing automatic tests is one way to nudge developers into understanding the domain better. And getting a person on the team, even freelance, whose job it is to push for tests, might help.

Going off another tangent, getting not a generic tester, but a domain expert who's also a tester, might help.

There's a big difference between blackbox testing and whitebox testing. For blackbox testing, having someone who doesn't now the inner workings of the system to find workarounds can many times be an advantage.

Of course you should do both kinds for security critical projects.

I'm not arguing for black/white box testing, I'm arguing that for testing to have real value, your tester needs to behave in a similar manner to a real user. That way you find the gap between your own understanding and your users' understanding of the system.

Take a concrete example: a piece of medical equipment prints off reports. If your tester has no medical knowledge, he doesn't have a frame of reference to know if the numbers are correct or not. He's also more inclined to use it exactly how you specify.

A real doctor would bring his own experience of similar equipment and try to use it in ways you didn't think of. In addition, he will be able to gauge whether the reports "look right".

Very good example! And there is an absolutely huge correlation between good testing and domain experience irrespective of freelance/permanent. I've seen entire teams of permanent testers miss critical bugs because they're busy stressing the system and trying edge cases but don't actually use it like a user because they don't know their users.

I think there's much more to it than what the post says. It seems the author is looking at the freelance testing industry (is 'industry' the right word here?) through rose colored glasses.

Freelance web designers and SEOs are everywhere because you don't need any experience and often times don't even require any actual skill to do the job. All you need to know is how to repeat the whole "Social media marketing your business reach customers user experience blah blah nonsense" spiel and people say "Yeah! I want to miraculously reach the top of Google for everything and make a billion online like Zuck" and hire some dude for $300 who read "HTML and CSS for Dummies" last week at a Barnes and Noble. I know this because I used to be that asshole (not the business owner, the freelancer).

Other kinds of web/tech freelancing have very simple-to-understand values. In the minds of a lot of people who hire freelancers, Web design = Branding or Marketing = Profits. Simple. Not exactly correct, but that's the line being fed to everyone these days. Same with SEO. SEO = Rankings = $1B profit.

Testing is different though. I have to say however that if a company even needs to test software to begin with then they're probably playing at a higher level than the companies hiring freelance designer/devs and SEOs. Still, the value of a tester is still nebulous. You don't hear freelance testers out there drilling "Testing will make your customers love to use your product, increase efficiency on yours and the customer's end, and save you a ton of money" or something equally easy to market. To most businesses testing means "we played with the app on the staging server and it worked fine for us. time to push it live". It just comes off as an extra expense that isn't worth the investment.

And why is that? Because freelance testers, unlike other freelancers, have failed to create a consistent, marketable appeal to business owners (i.e. tell them why it's going to make them tons of cash) the way other freelancers have. And it's probably because those who do testing are a little more thoughtful and less about pizazz and flash than designer/developers and the nature of testing itself is kind of boring unless you're really into it yourself. Design is fun because everyone wants to be an artist. SEO is fun because people are given the idea they're in control of which keywords they'll pop up for and such. Testing? Totally boring. Some guy's gonna come in here and find bugs in our software? We can do that ourselves or just make sure to mark all the complaint emails as TODOs in Gmail.

I think you make a great point about perceived value of testing vs SEO/design. However, with the increased proliferation of downloading software through app stores, which favors apps with better reviews/ratings, the perceived value of testing should increase. For example, take a look at apps with mixed or low ratings on the AppStore and most of them will contain complaints about bugs in the app.

(random point + small shameless plug: I founded http://www.testelf.com as flat priced testing service for iOS apps)

you're very rarely going to 'make' someone money - you're going to save them money. A well-tested app doesn't necessarily out-sell the competition if the competition gets a better distribution deal through 'good old boys' networking. Demonstrably bad software often becomes dominant in specific niches, and people put up with the bugs, because "it's what everyone uses".

Your well-tested software still has a huge marketing battle and possibly integration issues to replace incumbents, regardless of how many bugs they have. You end up needing to create a negative campaign about the headaches and hassles of using XYZ, and that you don't have those issues (because it's well-tested, but you generally can't promote that to most people, as it's 'too technical').

At first, I thought you were preaching to the converted, here. Whether or not everyone here uses dedicated testers, they probably all appreciate the value of a well tested system. In theory, that is in fact the case, but reflecting on my experience this is not usually the case in practice. Testing is dropped by the wayside when the rubber meets the road - particularly in any early to mid stage ventures.

IMHO there is a strong cultural component at play. Testers are, quite frankly, second class citizens in the land of software. This is due in part to the ambiguity of the term itself, as it can mean everything from a button-clicker to an engineer (with perception often defaulting to the former).

As a result, testing becomes unsexy. Few developers with options choose to do it full time, and it becomes an afterthought. Even dev-centric approaches like TDD get brushed off as "crutches" for "junior" engineers. This all helps propagate a fairly downward-tending cycle of testers and testing lacking voice and status within the organization.

There is one niche area of testing where it is more common... security testing.

Testing in this space is often more a function of the technology stack than the industry of the client. This allows security test companies to build up great expertise without knowing anything about the client's specific app or industry.

I wouldn't even consider penetration/security testing to be in the same field. My experience in the traditional QA areas is that in order of likelihood to freelance are performance testers, automation/development testers, and finally manual testers.

I don't know if uTest is still around, but they attempted to fill a similar niche. If you are in a testing lead position, then you know that an enormous amount of contracting is arranged via recruiters to fill this need also.

I think the reasons it's really hard for freelance testing communities to get traction are

1. real or perceived lack of return - you have a 2-3 month engagement and realistically the first month is training and acclimatization (hence wasted). better off hiring a FT resource.

2. ample market supply - there are plenty of people looking for FT roles. I'm a testing manager in a secondary technology city in the US, and it's no trouble to get resumes.

3. limited scope where they're useful - I would consider freelancing or hiring a freelancer that specialized in a particular tool (Selenium, Robot, Cucumber, QTP, etc) if building out a testing framework or conducting major refactoring but otherwise consider testing an iterative process.

4. This is a bit of a bootstrapping problem, but it would need a (much) better version of Angie's List. Because of the reasons I listed before, this would be a high risk decision and most people would be very reluctant to go with an unknown quantity. There would have to be a substantial number of verified ratings and reviews. [EDIT: this is also why so much contract testing is arranged via recruiters - the recruiter wants repeat business. A bad referral can poison a relationship, so you have a strong element of trust]

Also I feel like smaller shops that do test would rather write their own automated tests (we are all developers after all) until they get enough traction to afford a fulltime tester

Dont want to do UTest marketing and I have reservations about them but they are still around and Forbes made them their 8th most promising company in USA so there is some need they are filling - http://blog.utest.com/forbes-names-utest-8th-most-promising-...

You make some good points which I'd agree with

I signed up with uTest a while ago and have never done anything else with them. I'm generally underwhelmed. Seems much more geared for crowdsourcing a testing effort, which - with all due respect to innovative business models - seems less than optimal for doing anything other than low level manual testing.

If I ever went independent (or engaged a freelancer), I'd expect an LLC/s-corp with a professional online presence, reference sheets, sample work, and slighly better than LegalZoom-grade contract templates. Consider it my smoke test of potential partners :)

We've needed a crowd-sourced human testing platform for software products for some time.

It would need granularity: as a developer I should be able to purchase N minutes of time from N people, potentially in such and such a demographic.

It would need an API: I should be able to build human testing into my deployment infrastructure.

It would need accountability: at minimum I want webcam footage of the tester using my product.

There are complications, but I see no reason why almost all of these couldn't be overcome by sufficiently smart automation.

Someone go build this!

We actually do this at MobileWorks (mobileworks.com). There's an API that lets you get N minutes from N people filtered by demographic. Some folks use our Premier product for on-demand testing set up by email -- they send test requests to premier@mobileworks.com.

If you're looking for this more as a formal end-to-end service, you can take a look at our buddies at Rainforest (https://www.rainforestqa.com/). You can also look at services like uTest, though they're real pricey.

Thanks for that - Rainforest look very interesting (what I can see of it). "QA as a service" sums it up nicely.

If https://www.usertesting.com had an API, I think it would handle the majority of the points you listed.

UserTesting may already have one, however my brief two minutes of poking around didn't reveal that they do.

Some friends of mine have actually made a business out of this over at http://testloft.com

From what I understand the phone won't stop ringing so clearly there is strong demand for something like this out there.

There's less demand for freelance testers because having a one is a nice to have rather than a must have.

Clients rarely understand the value of testers, they appear to simply be an optional additional cost. Freelance developers have little incentive to recommend them since it inevitably leads a smaller slice of client budget for them.

When an organisation gets to the stage at which it understands the value of, and develops a need for, a tester they probably have enough work and cash to occupy one full-time.

How about 'Disaster Management/Save Face' freelancers?

To cater to the companies that thought testing was an additional expense and didn't go for it.

They could probably charge more with a title like that too. :)

Wouldn't be great if freelance developers saw the benefit of a freelance tester that they worked well with and the two of them marketed themselves as a joint enterprise for even better customer value.

Yep. As a web development shop, finding a client that understands or can be convinced of the value of any testing is rare.

I'd say the psychology and politics of testing are different from other things, really quite strange.

It most white collar work, there is strong pressure to tell people what they want to hear. Well, it makes me very happy to get a clean bill of health from a tester, but if I was expecting that to happen, I wouldn't need a tester.

Testing takes a kind of special discipline that the average office worker doesn't have. Whenever I've been an organization that has tried to press ordinary people into a tester role, it's always been highly ineffective.

That said, there isn't one formula to make a good tester. Some testers I know are very systematic, they belong to quality control associations, others are expert in project management, requirements, etc. On the other hand, one of the best testers I worked with had bombed out of every other job we gave him because he was always able to screw up anything we gave him. As a tester, he was great, because if he couldn't break it, nobody else could.

All of these things, I think, point to a need to a longer-term relationship with a tester than you're going to get in a freelance situation.

This is advertising for a service that CHARGES people to sign up and test websites. The blog post serves no purpose other than to funnel traffic to their domain.

Testers are mostly a labor commodity. Seems like the body shop contracting agencies and big offshore firms (Wipro, TCS, etc.) have this market cornered. Any potential client who really values having a dedicated testing team will build up the domain knowledge in house rather than re-educate freelancers over and over, or require the body shop to build that knowledge base and do that training for them when bringing on a new tester.

And how often do you really need just one tester? What are you going to do, bring in 10 different freelancers?

Freelancers need to wear more hats than just "tester".

TL;DR - we (testers) need to tackle these problems head on ourselves rather than wait for people to "get it".

It seems to me, a lot of folks here wouldn't be adverse to using a freelance tester in general, but are extremely adverse to using bad freelance testing (and have the scars to prove it!).

I'm a contract tester and recently decided to make the transition to freelance. From my experience (and confirmed by the comments here) the main barrier is one of perception - how can testers / add value to my organisation? And how do I know this tester is as good as they claim?

There are numerous ways someone like me can help an organisation:

Help organisations find the right permanent employees; Mentor said permanent employees by sharing knowledge; provide testing experience to a known or unknown business domain; supplement the clients test lab with your own equipment; evangelise testing throughout the company, demonstrate what good testing can do, what it looks like

I had to stop myself there! And pretty much all of those examples are from recent (within 12 months) history. Luckily for me I've been able build a decent enough reputation that there is potential for repeat business.

Two more key problems: first, the kind of testers that people here need may live in a bubble where everyone knows the value of testing and as such may not realise the rest of us exist - my colleagues and I need to start dispensing some red pills :). Second, the testers that are trying to be better appear to be outnumbered by the ones that aren't (snake oil merchants, testing zombies, call them what you will) so again we need to tackle that head on too.

Nice discussion. Thanks for the illuminating comments folks.

> I think freelance software testers should be easy to find. But they are not. I think there should be loads of good freelance testers out there. But there are not. I think it should be easier to find freelance testers. But it is not.

There's a reason -- software developers have figured out that they're better off just releasing the product and hearing from their customers. It's called "beta testing" -- no special panel of testers, just users eager to try a new product, willing to put up with imperfect operation for the privilege.

And not all beta tests are identified as beta tests -- many of them appear to be a product release, but the real purpose is to wring out the bugs. The classic example is Fedora Linux, which is one very long beta test (and one that most of its users understand).

As to Web sites and online games, developers use A/B testing to accomplish the same ends, without the need to locate people whose special role is to test.

I've gotten good results through uTest (for bugs) and extremely good feedback from usertesting.com. Neither is cheap though.

Testing is, in the large, not respected.

When you have a really good, competent, creative tester, the contribution can be invaluable.

But... many organizations tend to rank testers somewhere around the janitorial staff.

(By the way, get incompetent janitorial service, and watch what happens to your work environment... And, personally, I've found many of them to be plenty competent, caring, and helpful -- better people than many of the suits.)

Then, there is the other side of testing... It does exist. People who couldn't hack (aka "handle") any more, and who really are more or less button pushers. They are the ones who provide "QA signoff" as opposed to real troubleshooting.

It can be difficult to argue your value, when much of your field is a morass on two axes: Lack of respect, and lack of competence.

What do you mean when you say tester? If you mean "someone to run manual test cases," it is probably because those sorts of jobs are pure tedium... They can be outsourced to websites too.

If you mean a software engineer that designs software to validate that other complex software systems function (often dealing w nontrivial engineering efforts and undecidable problems) , they probably don't freelance because we denigrate them by calling them "testers" and they'll get hired fulltime by companies that understand the value of their work.

I would love to be a freelance tester. I have no idea how I would start though.

I have a lot of experience testing desktop apps for my day job, but I'd love to start something on the side.

Anyone here have ideas on where to start?

I'm bias, but helping people get their freelance career off the ground is what we are trying/doing at Test Ninjas.

The time it takes to walk someone through all the use cases that need to be tested is not trivial. The reason for having a dedicated tester is so that knowledge of the system accumulates. I can't imagine how having to re-introduce your system to a new tester for every upcoming deployment could possibly be a realistic approach. In order for this to make sense I'd have to build a relationship with a freelance tester so that the same accumulation of knowledge could take place.

In some circumstances "accumulation of knowledge" in a tester is exactly what you're trying to avoid. Fresh eyes can be very valuable. Amongst other reasons, this is why we don't give developers sole responsibility for testing their own work.

That's an interesting point but I think it highlights that there are different types of testing - even above the obvious UX vs QA distinction.

I find that really good QA testing often needs someone quite technical and even someone who has some insight into the way the code works. You have to actively think about ways the system could break. It's probably closer to the mentality of a penetration test than it is to usability testing.

This is a great idea! I was actually just recently looking for something like this, to help with testing on an app.

I think my only concern would be security with those who are testing your app, because you would most likely need them to test use cases by giving them special accounts and such. Is there some sort of checks and balances for people who are testers on the site? Can you make the testers sign a NDA before they test (if the project happen to still be in BETA or something)?

Thanks for the positive feedback :) The testers work independently and often sign NDAs at the moment I am working closely with the testers and clients as a match making kind of service.

I actually run a small QA-as-a-service company (for 3+ years) and I should say that we strive to distance ourselves from "freelance" word as much as possible.

As most other cheap labors in a remote freelance mode — it is discredited. I think it would be just hard to build a personal brand strong enough to compete with tons of people, who can look like a diamond on a pre-sale phase and then just disappear.

How has that worked out for you? Do you find it difficult to find clients? Much client turnover? Are you doing mostly manual or automated testing?

Finding customers is always a tough task. We started from μ-ISVs with simple Windows applications to probe the market and see whether people are ready to buy tests and QA as a service and then slowly expanded the offered services list.

We have great customer retention (excluding some specific one-time projects, like a penetration test), I hope due to our dedication :)

I would say manual tests are most popular, and then load tests, then automation. Often customers start from a tiny manual test and then use our services more and more, in other QA areas, being happy to receive all of those from one window in a unified manner.

One of the reason is that people are scared to freelance for a month full time, and the next month not getting any job the next month, In SA we dont have lots of companies that are looking for freelancer.I believe its always good for a company to get new tester that can produce quality assurance into their systems.

Testing tends to be an intermittent need. Most apps that are to be tested tend to be off of the public Internet. I think that most orgs are unwilling to take the time to provide testers access they need (VPN access, etc) given the ephemeral nature of the relationship.

I recently found out about a service that pretty much offers this. It's called Pay4Bugs: https://www.pay4bugs.com/. Has anyone tried it? Is it any good?

There's a bunch of those types of services which we are deliberately avoiding the approach of. It's really hard to know the quality of the testers behind it all, in addition to that they can raise bugs and give you an idea of what has been tested, but you doesn't give you an understanding of what hasn't been tested.

Wouldn't it be awfully easy for an outsourced tester to say "yep, tested foo, no bugs found" without having tested it thoroughly? You can't outsource auditing that outsourced work was done properly!

Yup - just exactly as easy as it is for a developer to say "no bugs in my code" and leave the job well before a critical ID used in a database reaches the limit of the "INT" definition that was applied instead of "LONGINT". In both situations you want to build a level of trust (and contracts/NDAs/liability insurances) between freelancer and customer ;)

I originally though you wanted freelancers for creating unit/integration/behavioral tests, which presents the problem that not all codebases are testable in current state.

I would absolutely hire a freelance tester.

I wouldn't hire a freelance tester, because it's so important, we all do testing in our business. I expect every freelance developer I hire to test everything thoroughly (and I don't just mean unit tests). They're either full-service, or I don't hire them.

I've never heard a single coworker or fellow entrepreneur say, "Gee, I really need to hire a freelance 'tester'." (Or other words that would be more likely, such as 'Q&A specialist'.) That's something people either don't do at all / don't care about, or do in-house because they know it's so important.

So, aside from arguments about skills, I don't think there's any demand.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact