

Please review my new programmer screening app - kranner
http://codeboff.in

======
frobozz
Here is a bunch of feedback:

There doesn't appear to be any opportunity for TDD (or any kind of testing or
iteration), even for something simple like checking syntax (I submitted one
with a missing ';'). As a result, it does not come close enough to a real-
world programming task.

Once I had looked around, and failed to find a "run" button, I decided that
I'd try just pressing submit, and I got to the "we'll email you" screen. I'm
sure I'm not alone. This would account for the very low pass rates.

In the real world, this would be like having all your programmers working with
a plain text editor and no compiler, and having them commit untested code, in
feature-sized chunks, to trunk for the CI server to build and test.

Obviously, under the right circumstances, I could do the work on my own
computer, and paste the code into your form, but that sort of defeats the
purpose of having it online.

Following the doctrine that programs are written for humans first, computers
second, a human still needs to read the code to work out whether or not it was
well written. I don't know if you make this available to screeners.

You could try applying some other metrics (e.g. max/mean function length), to
give more than a pass-fail.

~~~
kranner
Thanks for the feedback!

\- You're right, I think a lot of people expect to be able to check their code
first. When I was planning this, I figured if I were the candidate, I'd use my
local IDE to compile and test the code first. So this was a bad assumption
which I need to fix.

\- Screeners do have access to the code a candidate submits. It goes in a mail
to them along with the pass/fail rating and is also accessible in their
accounts.

\- Pass/fail is definitely on the crude side, but I was going for an 80%
solution first.

~~~
frobozz
I forgot the positive part of my feedback - It does look good, and you've
certainly hit the 80% solution. nice work.

~~~
kranner
Thank you.

------
chrislloyd
I really like the idea of this site, but it needs some polish.

1\. As everybody has said, more languages.

2\. One of the aims is to have the candidate as calm as possible. Seeing that
90% of people can't write a C program that reverses a linked list makes me
nervous about the strictness of the test.

3\. Rather than giving just a subset of the environment how about you make
each test have to produce a binary. Then tell candidates that the test is for
`gcc -o reverse reverce.c && ./reverse 1 2 3 4` to output "4 3 2 1". I presume
you are running them in a sandboxed VM? That will help people run and debug
the program on their own machines.

~~~
kranner
Thanks!

\- With the 'open' tests, encouraging experimentation was the intent. It
definitely needs an interactive syntax check as others have suggested.

\- Each test does produce a binary, which runs in a sandbox (or rather a few
nested sandboxing mechanisms). Your suggestion is interesting but I don't
think I fully understand it. For one it might restrict us to solutions
expressible as strings, and the formatting requirements might make the task
stricter.

~~~
chrislloyd
I guess I didn't explain myself well enough. Give candidates the entire file
and tell them how it will be run. Then it's trivial for them to develop on
their local machine.

I'm positive most of the failures you'll get are syntax related. Let people
check their own! If you want to check their syntax for them, that is going to
be a lot of work and make it harder to add new languages. Include the unit
tests with what you give the candidate so you are not just limited to string
output if you want, but you could give just them the formatter too. Keep up
the good work though, it was actually quite fun doing a couple of the tests :)

~~~
kranner
Thanks for the encouragement, it's very heartening!

------
thejash
This is a good idea. It needs more features and polish, but I see no good
reason why someone would NOT use this for hiring. I think the especially
important points will be:

1\. Support for all reasonable languages.

2\. LOTS of different tests--the more unique, the less likely the solutions
can be copied and pasted from either someplace online, or a friend who applied
and took the same test.

It would also be awesome if you told the candidate "program in this window
because the employer will be able to watch you code". Then show the employer a
fast-forward view of the programming process. Does this person guess and check
their way to a solution? Do they plan out the whole algorithm first? What kind
of mistakes did they make?

If you had all that stuff, I think this would do a reasonable job of at least
filtering for raw programming ability.

~~~
tomjen3
>This is a good idea. It needs more features and polish, but I see no good
reason why someone would NOT use this for hiring

Other than, as Joel pointed out a long time ago - the best people wouldn't put
up with it because they can always get a job elsewhere.

>It would also be awesome if you told the candidate "program in this window
because the employer will be able to watch you code". Then show the employer a
fast-forward view of the programming process. Does this person guess and check
their way to a solution? Do they plan out the whole algorithm first? What kind
of mistakes did they make?

Absolutely not - I program in a real editor, and I won't be judged by some
wannabe textbox.

~~~
thejash
> the best people wouldn't put up with it because they can always get a job
> elsewhere.

How are you supposed to know that someone is one of the "best people"?
Presumably by word of mouth and recommendations. I'm not saying that all
applicants should be forced to use this system--if you know that someone is
great, no need to test them. This is designed to filter out all the REST of
the applicants, the ones who may be good, but don't have many references
(straight out of college, etc)

> Absolutely not - I program in a real editor, and I won't be judged by some
> wannabe textbox.

Fair enough (I enjoy using a decent editor too), but: 1\. These are not
supposed to be hundred line solutions. You can live with a crappy editor for
30 minutes. 2\. Maybe applicants could submit a video instead if they wanted.
3\. It's not strictly necessary, it's just another nice data point.

~~~
tomjen3
About your second point, sure I can program is a textbox but I am going to
make mistakes that I would not normally do and I don't want a potential
employer to see that, because he will subconsciously think that it is bad even
though he should know (and properly does) that that sort of things happen.

And at a job interview I am not going to let something like that count against
me. Same reason I am going to go there in appropriate clothing, even though it
shouldn't make any difference it does.

------
kikibobo69
I like this idea a lot. We routinely screen people with a really simple
programming task to just make sure they aren't completely lying (which happens
more than I can believe).

1\. I took a test, implemented reverseWords (which is a test we often give
candidates), and was rather shocked to discover that in your estimation, I
failed. Yet, pasting the code into vim, and the unit test you say I failed ...
I didn't fail. The fact you failed me when I should have passed, isn't great.
:) More info on the failure would be helpful.

2\. If I were using this on a real candidate, I would like to know how long it
took them to do the test.

I think it's a great idea; I'd consider using it for pre-screening once the
kinks are worked out.

~~~
kranner
Could you paste your code here or email it to me at karan@codeboff.in? I'd
love to find out what happened.

In my experience code that runs on OS X often fails on linux, because gcc on
OS X seems to initialize newly malloc'd memory to zero, when gcc on linux
doesn't (and it shouldn't, according to malloc's man page).

Thanks!

~~~
calloc
Shouldn't? malloc doesn't make any guarantees that the memory is initialised
or not. In the Linux malloc man page it specifically says it is not cleared,
on Mac OS X it doesn't specifically mention that it is initialised. On FreeBSD
it says it is initialised

Yes, programmers should make sure that memory is indeed cleared, preferably by
using calloc or memset.

I just ran a test on my FreeBSD laptop, allocating 256 MB of ram (on a 64 MB
physical ram laptop), and then walking through it, and with just malloc I got
all 0's. I then had the program write random out to those 256 MB, free it, re-
run the program, still no garbage values. Same on Mac OS X. I don't currently
have access to a Linux machine so it is not possible for me to run the test
there.

So apparently that way may run on Mac OS X and FreeBSD could fail on Linux.

------
tomjen3
As an applicant, what do I get out of using your service? Why shouldn't I just
skip ahead to the next job posting, if some would be employer demanded that I
used your service _?

_ if you are Google this does not apply, but otherwise the first problem you
have is to answer "why would I ever work for that company", since most large
companies are the same anyway.

~~~
kranner
Personally, from the other side, I've seen literally hundreds of people apply
to jobs with only one or two open positions.

Most of those candidates turn out to be just noise, but the few that are not
don't always get a fair chance because screeners/interviewers are just too
swamped. A tool like this can take a large chunk of that load off, and any
candidate who invests time in taking this kind of test will know that they are
ahead of the 'unwashed masses' without having left the comfort of home.

------
frankie41
The service looks nice and the idea is definitely a good one. One questions -
you guys look very similar to <http://codility.com> . AFAIK they have been
around for a while (see discussion here
<http://news.ycombinator.com/item?id=1039140>), have more programming tasks,
support TDD and more languages. It's unclear to me whether you just want to
compete in the same space or are you anyhow different from them?

~~~
kranner
Thanks! Essentially we'd be in the same space. The ways this will differ is:

1) cheaper, one-time pricing, 2) opening up over-played questions to free,
public use, 3) less complexity for test-assigners (at least I'd guess so, I
haven't really used Codility to assign tests to candidates in real life).

Give it time, it's just getting warmed up. Personally I think this space will
(and should) see a lot more competition. Single founder here.

~~~
kranner
Whoops: I meant pay-per-use pricing, not one-time. So it's for companies with
a one-time hiring need and, for instance, individuals looking to screen
outsourcing partners.

------
bobbywilson0
1\. I think that it would be more interesting instead of just testing if a
applicant can do a specific task, let them try and fix it if they failed.
After all, it isn't such a problem in real life if someone doesn't get
something right.

2\. What about having them refactor some messy code?

3\. Exercises where they write tests for untested code.

~~~
kranner
Thanks!

1\. I am going to add an interactive syntax check feature as a lot of people
here have recommended. (As a side-benefit, I can skip asking for an email and
present the results immediately after the test).

2\. Funny enough, this was my original vision for the project, but according
to feedback on a blog article I wrote to test the hypothesis, people preferred
to write their own code from scratch rather than fix the nearly-working test
problem code.

Please see: \- [http://blog.codeboff.in/2010/07/03/will-the-real-
programmers...](http://blog.codeboff.in/2010/07/03/will-the-real-programmers-
please-stand-up-automated-edition/) \-
[http://blog.codeboff.in/2010/07/19/bug-fixing-test-
problems-...](http://blog.codeboff.in/2010/07/19/bug-fixing-test-problems-are-
out/)

3\. Might not be important enough to take time up in a test. As I understand,
everyone hates long tests.

------
rodh
It would be fantastic if the candidate could just click a "run" button at any
time, and (provided with some input string), see the output.

Of course this would make the pass rate a lot higher... but, assuming you
tracked each iteration, would give the recruiter a chance to look at how the
candidate goes about identifying their mistakes.

~~~
kranner
I will add the interactive syntax feature seeing that everyone wants it.

Now about tracking iterations, from a candidate's POV, I'd be worried about
how my work was coming across, at a stage when I might not be ready to show it
yet. I think it's respectful of people's privacy to let them show only the
final code that they want to show, and not track their every move on the test.
If someone did it to me, I'd be unhappy.

~~~
rodh
You could just take the code snapshot when they choose to run it. Warn them of
this, so they only end up showing the recruiter code they think will either
work or is suitable for iterative development.

~~~
kranner
Thanks for the suggestion! This goes in my notes.

