
Ask HN: A coding test or working for free? - alphanumeric0
I recently interviewed for a company called Rxdata.net here in New York.<p>The task was a data ETL problem transforming natural language text into specific parts of data. I opted to use pyparse to create a working recursive descent parser to pick out the relevant bits of text.<p>At this point I was wary of completing the parser for all cases found in the database table, so I decided to build it only for a few base cases which I threw in to a unit test.  I reasoned I could still demonstrate my skills without giving them much free work.<p>The solution worked well, and I submitted it to Joe, the CTO, for review.  A few hours later Joe contacted me complaining that the code crapped out after trying to run the code against their database table.  He then offered me a github branch to continue work on the problem.  I sent them the nicest e-mail I could muster explaining that I felt the code adequately demonstrated my skills and that the error is due to the parser not handling all cases of the problem text.<p>Joe&#x27;s responded with &quot;I am only trying to evaluate your work, not on just how the code looks, which is clean and well organized, but that it works correctly on the data as well.  I asked advice since the program crashed on the 5th package, which didn&#x27;t give me enough data to verify, and I wasn&#x27;t sure if it was just a system-related issue or something quickly fixed.&quot;<p>Lastly I responded with &quot;Sorry for any confusion, I was just giving you a sample of my work. The system error you mentioned is a result of the code being a sample. The cases that do work are found in the test.&quot;<p>Naturally they haven&#x27;t gotten back to me, even though my code more than adequately met the challenge.  Perhaps Joe the CTO did not understand that the other cases would be trivial to implement.  In that case, perhaps I&#x27;ve dodged a bullet.  A CTO should know better.<p>I&#x27;m wondering if anyone else has had similar experiences.  How did you handle it?  How did the interviewers respond?
======
mrcold
It's the future. Crowdsourcing through job candidates.

I had a similar experience with a company in Berlin, Germany. They didn't care
about the code or my skills. Only if the submitted solution worked against
their existing tests.

I think it's just a sleazy way of getting things built for free. Basically,
write only a test for the functionality you want. And give the actual task to
job candidates. If someone succeeds, you have both the algorithm and its test
for pennies. It's based on P = NP. Testing the result is easy. Designing a
solution is the hard part.

I now refuse both technical interviews and take at home tests. Limits my
choices a lot. But the ones that remain are usually top shelf.

~~~
mapleoin
Wait, you refuse technical interviews and you say the rest are top shelf? I'm
curious to know what your ideal interview looks like. Or do you mean you just
get hired through networking?

~~~
trcollinson
I've had a very similar experience to this commenter. I wouldn't say that I
reject all technical interviews and take home tests, but I certainly reject a
lot of them and find that I have little trouble finding really high quality
companies that will speak with me. As you say, a lot of it comes through broad
and deep networking. I know people, I have a proven track record in business
and technology, and people are comfortable working with me.

An ideal interview is one where I get to learn as much about who I will be
working with and what I will be doing as they will be learning about me.
Generally these turn into more discussion based interviews where we talk about
the issues the company is having and how I might be able to best solve them.
They still get a good idea about how I think and work by discussing the
problem back and forth. I also get to see how they think and work.

Frankly, I white board problems very well. But when companies ask me to do so
in an interview I want to see them do it as well. I want to see their ability
to code as much as they want to see mine. Same goes for a take home test. If
we are going to work together and you have to see code of mine, I very much
want to see yours to make sure it is up to my standards.

------
davimack
After having been asked this several times, I've begun to make it clear that I
will only engage in these tests if I'm compensated for the time. It's a
delicate thing to do, really, telling someone that, no, you're not going to
work for free, just on the hope that you might get a job.

But it doesn't only happen in this industry: my niece is a graphic designer &
has been asked to provide samples of her work on projects ... to the
interviewer's specification. She's done it, because she really needed the
work, but felt terribly taken advantage of, and I'd agree: it's predatory.

I have the luxury of telling people that, no, I won't take their 1) IQ test,
2) Personality Test, or 3) their Coding Challenges. Not everybody does,
certainly.

The appropriate thing to do is to either state up front that you'd need to be
compensated for your time, or acknowledge that you're giving away work as a
way to possibly get in the door & just write the best solution you can write -
not leaving it incomplete, necessarily, either, as that is a sort-of halfway
position that makes you look bad.

~~~
kaybe
Hm.. how about some kind of three-way cooperation, where you get non-profits
and charities to give specifications for work they need done, which can then
be used for interview samples for actual industry jobs.

The candidate can demonstrate their work and be assured that neither does the
company extort free labour from them nor is the work they do for the bin,
whereas the companies can see how they behave in real-world environments.

~~~
giaour
That sounds amazing. I would love to contribute to such a project (or be
interviewed using one).

(Actually, would you mind if I stole this idea?)

~~~
kaybe
Feel free! (let me know if you ever build something, maybe) :)

~~~
Ellahn
Did anybody start on this or plan to soon? If not, I'll start it and make it
open source. It's a killer idea. :D

~~~
giaour
Not yet; I'm travelling at the moment but should have time this weekend. My
email is jonathan AT jeskew DOT net if you'd be interested in collaborating.

------
cldellow
That smells a bit fishy. I think you've dodged a bullet.

I have to be a bit cautious here, because our company does something similar.

We have a little programming challenge that we ask candidates to submit. We
don't use those submissions in production. We find it gives people with non-
traditional backgrounds a chance to show their chops, and gives us some meat
to discuss during an interview.

The challenge probably takes 4-8 hours, though. Part of me does think it's a
little asymmetric for us as employers to be demanding this of candidates.
We've also done week-long work tests, where we pay the person market rate to
work with us on real-life code.

Anecdotally, people view the paid week-long trial as demeaning, but the
(unpaid) challenge as exciting and fun. We live in a bizarre world.

~~~
estefan
Obviously anyone really good won't actually do the tests because they'll be
offered a job by somewhere that doesn't do them before they get around to
yours though, sooo...

Personally I find the single best question is "what's your github username?".
People who are on it and active are... well, actually I've only ever had one
candidate who's actually been on github and she seemed OK. The majority of
people I've seen haven't been much good (and have barely heard of github).

I agree about dodging a bullet. I think the OP did exactly the right thing.

~~~
hdra
I'm curious, how does companies usually evaluate a github profile? and how
important is that evaluation to the decision?

I ask this because I find many of the software engineers I follow on Twitter
who work at major tech companies (github, twitter, google, etc) have a pretty
sparse github profile.

~~~
estefan
I started asking it really so I could see examples of a candidate's code. Also
seeing the projects they've contributed to would show that, for example, they
genuinely were interested/engaged with whatever niche they were being
interviewing for.

------
overgard
Honest question: is tricking coders into doing work for free a thing that
actually happens? I'm not saying it doesn't, I've just never heard of it.

From their end, if that was a thing they regularly did, it would be such a
hassle. After all, you'd have to find a new person to do that work every time,
find a way to split up the work into very small chunks, and who knows if the
result from a person off the street would be usable?

My guess would be that he was trying to evaluate how you respond to bug fixes
and criticism. (Still sounds like a weird interview strategy though)

~~~
jcwilde
Most companies that do this (unintentionally, I believe) are not benefiting
from the actual work product, but from the consulting regarding their business
problems.

How much do you think a consultant would charge to come in for a few hours and
discuss their needs, not to mention put some working code together for their
problem?

------
moogatronic
Something similar happened to me, but I was never provided with the
"programming test".

My interview for a freelance gig was wrapping up, and the COO asked if I would
be willing to "fix a few bugs in the system" to see if I were a "good fit" for
the project. I asked point blank if they were asking me to do work for free.
He responded that no, he was not, and that the task would take no more than 30
minutes to an hour. I said that I would be happy to take a programming test,
or provide some other proof of competency, but that it's pretty non-standard
to ask me to fix bugs in a system for free, especially for a limited hour
freelance gig.

Up until that point, we were discussing project time lines, handing over of
logins, and my number of hours of availability. I got an email the next day
saying that they were going to go another direction. Likely, trying to fund
some other suckers to fix their bugs for free.

------
feedjoelpie
I haven't run into this on the interviewee end, but my company has applicants
work on real projects, and here's how we deal with it:

Pay them.

~~~
maratd
This is a simple solution to the problem. Paying someone for an hour or two is
absolutely nothing for a healthy company.

So if the company wants to have you write code to prove your mettle, have them
pay you for the time.

~~~
xsace
Except you cannot do that in every country. You know, Labour right.

~~~
hndl
I'm not sure I understand. Care to elaborate? Pointers?

~~~
juandopazo
In my country the potential employee would have to be registered as an
independent contractor before being able to charge for the provided service.
And then if he wanted to become a full time employee he'd have to unregister
as a contractor.

~~~
xsace
Yes. In some countries, laws are strongly in favor of employee protection (for
a good reason).

That's the case here. One employer could severely abuse paying candidates for
some quick job without any contract being even mentioned between the 2.

------
gtk40
I was recently applying for work after graduation. One entry level position
and a small company (with a vague job posting) wanted me to complete a pretty
extensive problem which would involve access to their code base and take "5 to
10 hours." I also would need to use a Mac and Objective C, even though these
were not on my resume or the job posting. (I work in a .NET shop and had
mentioned using Xamarin for Android development, and they were mainly doing
iOS and OS X work, but extending into being cross platform).

I asked if they had a problem I could do that would be on more familiar
tooling, noting that even if I could borrow an OS X machine, it would take me
a while to get familiar enough to complete the task they sent me. I also
expressed concern at the length. I ended up declining to participate and they
moved on.

(Note: I have never used OS X besides briefly using someone else's computer.
Everything I have used has been Windows and Linux, with a couple of weekends
of trying some BSD-based distros thrown in.

------
kstenerud
Did he even ask you to describe how you would likely tackle the problem while
you were in their office? Things like design ideas, architectures,
technologies, pros and cons, etc?

If all he did was ask you to "do this" at their office without picking your
brains as to your thought process, then regardless of the legitimacy of the
offer, I would think twice about working there.

The fact that he's asking for a solution that works on all of the data rather
than a sample of it is also highly suspect. Either he is not being frank with
you, or he doesn't have a firm grasp of scope (most likely the latter). Either
way, he's probably not the sort of person you'd want to work for.

~~~
alphanumeric0
I went through a couple of cases with him where a regular expression wouldn't
work, how I might test my solutions, pros and cons of my proposed solutions,
how it could scale in the future, etc.

That conversation was there. After we spoke he left me to work.

------
bjourne
Why don't you post your sample code on github? Maybe the CTO just wanted free
work from you, but maybe he just wanted a candidate that was able to complete
the assignment. He could have thought that you cherry-picked the easy cases
and were unable to solve the tougher ones.

Coding tests unfortunately aren't just about demonstrating skill. Then people
like me could just link to their github QED. It's also about completing
assignments and doing stuff because you are being instructed to.

~~~
jschwartzi
Maybe the CTO should post his database on Github. If he wants the test code to
function under certain conditions, he needs to publish those exact conditions.
Not specifications, but the actual data that the test will be run against.

If he's not comfortable doing that because it's restricted intellectual
property, I might wonder what the real purpose of the test was.

------
creyes123
Any programming test that takes over an hour to complete is highly suspect. If
it took less than that to complete the code (you did not specify), then IMHO I
can understand why they were expecting a more complete answer.

Regardless, my hunch is that your instincts are right about this one.

~~~
bjourne
I disagree. I think coding tests are a great way to judge candidates. But only
if they are given proper weight. Nothing worse than submitting and acing a big
coding challenge and then being told they don't think you're a good culture
fit. Some companies employ such hiring practices and I think it is very
unethical.

~~~
fsk
That's why I've soured on coding tests. After lots of employers didn't give me
an interview after a coding test that a KNOW I aced, that's poisoned the well
for me.

Now if someone gives me a pre-interview coding test, I mentally translate it
to the employer saying "I don't respect your time."

~~~
PopeOfNope
Same here. I've been through the same song and dance. If something happens
over and over and it results in my not getting an interview, much less the
job, I'm not going to repeat it. Especially if it's time consuming.

If the company asks for a coding test, I politely decline, but I offer to
provide them with a code sample instead. Very rarely has that not resulted in
a phone interview and I didn't have to spend hours of my time on it[0].

[0]: per company. Write once, give to many scales better than coding tests.

------
dmuth
If they're running your code against actual production data, that crosses the
line into questionable territory.

I've been given programming tests before, but they were always for
hypothetical problems that were open-ended. That way, each candidate solved
the problem in a different way which was when discussed during the interview
process. In your case, it sounds like they wanted you to solve a specific real
world problem they are dealing with.

------
jcwilde
Am I to understand from your explanation that they asked you to solve a real
business problem with real business data as an unpaid "coding test"?

If so, I'd say they were clearly in the (legal) wrong even before asking for
the refined version.

~~~
bweitzman
I had an interview at a company once which was basically pair programming with
a few different engineers for a full day on real code written for real
clients. The best part was the NDA they asked me to sign 6 months later.

------
GabrielF00
It's probably not a good idea to post the name and title of the person you
spoke with when it's a small company.

~~~
mkagenius
True. OP could have asked for advice without naming names.

------
kasey_junk
Let me argue in favor of coding problems. Unpaid coding problems have a couple
of very appealing properties:

1) They are available to all applicants (not all applicants have github
profiles, or can show previous work samples, or are available for short term
contracting work).

2) They do not cause problems with typical IP agreements or extra curricular
work restrictions.

3) They can be standardized so all candidates receive the same process.

4) They are a closer proxy to the "real work" of development jobs than many
other alternatives (and there is lots of research that indicates this leads to
better results).

Those properties are very important and cannot be found in hardly any other
style of hiring filter. For this reason, its my belief that coding problems
are central to a good hiring pipeline. I believe this so much that I'm willing
to replace nearly every other part of the hiring pipeline with just the
programming challenge (though that is a hard sell with management often).

The problem with coding problems is that it is very hard to come up with good
ones. They should take less than 4 hours to solve (everyone severely
underestimates how long their challenges are), they should be indicative of
the kind of work you will you do, and they should be, if not fun, engaging. So
normally when I encounter bad programming problems I chalk it up to that
instead of malice.

It is everyones decision to not do programming challenges of course, and I
completely understand people who will not do them. But to me the standardized
hiring pipeline is the most important part and I'm willing to lose a few good
candidates to get it. For this reason, I view a programming challenge as a
good sign for a potential employer, and the lack of one to be a serious red
flag.

If someone is using a coding challenge to get free work, that is at the least
unethical and likely against the law depending on your jurisdiction.

------
herval
I interviewed at a rather fast growing startup in SF (hear about it here on hn
pretty often).

The interview was simply 'sit on that corner and fix this bug'. I said I don't
work for free, so they said 'oh sure, we'll pay for that!'. Finished the bug
fix, expecting a follow-up review or anything, so their CTO responded with
'oh, that was too fast. Hm, let me get you another bug.'. At that point I
realised their git log must have at least a hundred different names on it.

I politely refused and said if there wasn't no interview, I'd just leave. They
gave me a check and that was that.

One week later, the CEO emails me saying they'd love to hire me, but I'd have
to quit my job first and they'd evaluate my case after 2 months with them (I
was an h1b at the time).

'No thanks.'

------
rmc
I wonder about the copyright of such a thing. If you aren't an empployee and
don't sign over rights, are they even allowed use it?

------
cdnsteve
This is like asking a carpenter to prove their skills by building you a deck,
in a day, for free. Good luck. Any work performed should be paid, period.

------
brudgers
Diplomatically for Joe's sake: your standards of professionalism do not align
with Joe's vision of company culture. Apparently, someone has mistaken cost-
savings for a core company product line.

Good luck.

------
webtards
I have done at home prescreener tasks, as long as they were sub one hour,
clearly academic, and patently not related to the business. These are not
paid. I have also done on-site dev tasks and tests which were squarely related
to the core business, and have invoiced my day rate each time, which was paid.
If a company asks for more than an hour, for tasks related to their business,
and don't want to pay, run, don't walk away, and better still, spread the word
so people become aware and avoid them in the future.

------
theaccordance
Shady code test or not, I don't agree with the manner in which you handled the
situation. You agreed to do the test, submitted it partially completed, and
then replied that it should be sufficient (knowing full well you didn't meet
the requirements). I doubt the CTO didn't understand that the other cases
would be trivial to implement; he simply sees a half-finished submission and a
candidate who didn't correct the mistake when given the chance. Had this been
a college exam and not a code test, would you expect a professor to award you
a passing grade based on the entirety of the test, or the questions you chose
to answer?

One thing to keep in mind is that while you may view the code test as bad, it
might not having malicious intent; some managers simply lack experience in
putting together tests and fall back on old specs. There's two things I do
during interviews prior to writing a single line of code:

1\. Every code test I have taken has been brought up during a discussion. I
use this opportunity to ask questions about the test. If you're interviewing
with a recruiter who's not familiar with the test, they'll forward your
questions to the manager who's evaluating the test for clarification. The
manager's response will come either prior to accepting the assignment, or with
the assignment attached to it.

2\. Immediately upon receiving the code test, I review the spec in its
entirety. If I have concerns with the spec, I raise those right away, as to
not give the impression that I'm incapable of completing the spec if I bring
up these concerns at a later time.

------
fsk
I remember the guy who asked me, as a coding test, to write a program that
connected to Interactive Brokers and executed VWAP trades.

He got very angry when I offered to do it as a paid consulting project. If I
did it for free, he wouldn't need to hire me at all!

------
sidarok
It has to be a throw-away scenario and a throw-away code. Anything else is
wildly unethical.

~~~
giaour
I think this is why problems that feature singly-linked lists are so popular:
you can tell from the start that the problem is limited in scope and has
nothing to do with any system currently in production.

~~~
sidarok
Indeed.

On the other hand the interviewing company should avoid such situations too
because of the information secrecy - unless they are mandating an NDA with
each and every candidate.

------
roneesh
It sounds like a miscommunication between you two, nothing more than that.

I would view this as a blessing though. For me, interviews are a chance to
evaluate how well we can communicate. If I can't be relaxed, make jokes and
just feel free to talk, then I won't feel that way working for you during the
next year, and that to me, is a no go.

I'd say indirectly you dodged a bullet, sounds like you two might be a high
risk for constant miscommunication.

------
agounaris
I have seen a lot of such cases as well, always the best is not to accept it
in the first place and offer giving a test on an open source project for
example.

------
lexi-mono
You working on a commercial project means that they can implement your
solution and get money from it, so it'd be fair that you'd be compensated for
your work.

I'd ask to be paid for the amount of work that I'd be doing on this project or
the option to get my skills assessed on some other, non-commercial test
project that they wouldn't benefit from.

------
edoceo
When I interview candidate I make them fix a problem in our real system. First
by just talking it out (interview, unpaid) then I have them build it. Or what
they can build in the time allotted. I pay for this time. Its basically hiring
through short interview then real work on 1099style. Interviews you should do
for free but fixing my real problems I pay for. I can decide who to hire with
less time invested and know more about my new hire. #winning. Built a web-
service/ATS to help me manage the while things

------
steven2012
Don't bother continuing the interview. An employer should be more considerate
of the interviewee's time. Imagine you worked for this person?

