

Tell HN: TechTestNow, an app to help you hire the best coders - sandaru1

We are launching the beta test phrase of our startup, TechTestNow. TechTestNow allows you to run online coding tests to select the best programmers to interview. You can setup a test and give the link or invite them by email. Then the candidates are asked to take a test<p>http://beta.techtestnow.com/<p>At the moment, we support 2 languages Java and Ruby. We are adding more questions and languages gradually. And as soon as we have more data, we are going to analyze and fine tune the ranking algorithms.<p>Here are two example tests,<p>Java : http://beta.techtestnow.com/en/user/testLogin/secret/TTN4c2479e455a032.22415626<p>Ruby : http://beta.techtestnow.com/en/user/testLogin/secret/TTN4c247a1e3fc618.66097557<p>To view the system as the recruiter, login as "sandaru1/hnhackers". (please don't delete/deactive above tests)<p>Any feedback would be appreciated.
======
kls
I do like the idea of compiler driven tests over Q&A, but you still fall in
the same trap of only getting answers for a very contrived case. There are
more dynamics to a good developer than just a few hypothetical situations.

For example someone could be great at Hibernate but not really have some a lot
of work with pure JPA. Just because they are not familiar with the JPA does
not mean that they have not mastered ORM concepts and would be able to do JPA
development as good as the rest of us in a week or so.

Which brings me to my next point. I would throw time completely out. With the
situation I just framed a senior level developer is going to do what all of us
do. Fire up the browser look at the JPA api and implement it. This is what
makes a good developer. Someone who can gather code together and create a
solution.

On almost any given project there are going to be new API's that developers
will have to familiarize themselves with. The ones that can are far more
valuable developers, because they disseminate that information back to the
rest of the team.

Your test does not account for that and by making it time based you are
disadvantaging the very ones that posses that valuable skill. If the answer is
right, it's right.

As well you should consider offering the service to recruiter and hiring line
managers, not as a competitive test, but rather as a filtering tool.
Competitions are regarded in academia, but in the corporate world people just
want to know if you can do the job.

~~~
nps
What you say is quite true and resonates with my personal experiences.

I myself do not possess an encyclopedic knowledge, nor will you see my fingers
flying over the keyboard.

On the other hand, what I have noticed is that my understanding of the basics
of coding, i.e. how to mix and match loops, arrays, conditionals, etc has
vastly improved over time.

I frequently forget advanced or language specific topics - and I don't really
care, since I can easily re-read about them online and refresh myself.

I have worked as an examiner for computer science university students and also
as an interviewer for new hires for companies. What I have noticed is that the
best candidates are similar - they have a rock solid grasp of the fundamentals
of the language, and just twist and turn that knowledge to acquire the results
they need.

You yourself have mentioned this - a good developer with an understanding of
ORM can easily adapt to whichever framework he is required to use.

This mastery of the basics is what we are trying to assess with TechTestNow.
You might notice that none of the questions require in-depth knowledge of the
APIs (although that knowledge might help you solve some of them faster, there
is always a solution that uses only basic constructs).

The main reason the tests are timed is because, all said and done, a good
programmer does need to have a certain coding speed !

Also, in real life, a lot of time is spent not on figuring out how to solve a
problem, but rather on figuring out what the problem is in the first place -
what do we really need to do ? (and can it be done at all ?). Although the
problems given in TechTestNow might seem contrived, they do have the advantage
of being unambiguous, so that the programmer can just concentrate on writing a
solution.

------
ismarc
Your method of grading the tests fails. I couldn't determine if you were
counting number of times compiled against the person, or just looking for
something VERY specific in the final result, but both methods failed to
properly categorize technically right solutions, even if they didn't meet what
was entered as "correct". Even worse, there's no interface to determine what
solution you were even looking for, requiring the test requester to either
trust your (unique) judgements on correctness of answers or review the
playback of every entry, removing the major benefits of the site. The fact
that every person gets random questions also doesn't help compare capabilities
of different testers. And the final thing missing, the ability to see the
requirements + final code submitted without going through the playback.

Note: This isn't a "wah, I did it all right and it said I was wrong!", I
specifically used correct, valid answers that were unlikely to match the model
you were looking for (empty for loop body for decrementing a variable to a
certain value, etc.).

~~~
nps
I went through your solutions and I must disagree. Your solutions did not
generate the correct output for the questions.

For example, I'll take the following question :

'Given the string variable 's' and the integer 'n', return a new string
consisting of n repetitions of s.

For example, if s = "abc" and n = 3, you should return "abcabcabc".'

The code you have written is as follows :

String duplicate(String s, int n) {

    
    
      String result = s;
      // First assignment already done
    
      while (n > 0) {
         result += s;
         n--;
      }
    
      return result;

}

If we analyze the following code (assuming that s = "abc" and n = 3),
initially, the variable 'result' is assigned the value "abc".

The loop then runs 3 times - when n = 3, when n = 2 and when n = 1. Which
effectively means that "abc" is concatenated 3 more times giving the string
"abcabcabcabc", which is incorrect.

I do agree that the solution is _almost_ right. Note that you are allowed to
compile your code and run it against our testdata (and even your own test
cases) in order to establish it's correctness - which means that you did have
the opportunity to find out that there was a bug in your code and fix it.

But in computing, how often is it acceptable to get things 'almost' right ?
Would you be happy if Google returned search results which were almost right ?

I know that if a human conducted the interview, he would have helped guide you
through the test - in fact, we are almost done adding the capability for
recruiters to monitor test sessions in real time and converse with candidates.

I can also provide a case by case analysis of your other solutions if you
wish.

~~~
ismarc
question 2 of 6 for duck: Write code to trap these exceptions and return 1 if
a FineException is thrown and return 0 if a CoarseException is thrown.

    
    
      int exceptionTrap(int a) {
        try {
          exceptionThrower(a);
        } catch (FineException fe) {
          return 1;
        } catch (CoarseException ce) {
          return 0;
        }
        return -1;
      }
    

You're saying the above is invalid? Tests out perfectly fine on my
system...are you trying to say there shouldn't be the last return because you
say "You may assume that an exception is always thrown"? If a function returns
a type of int, all codepaths then need to return a type of int, programming
101.

Others indeed do have some errors as I quickly typed them up (off by 1,
forgetting that java array length isn't the same as C++ std::vector.size).
Number 3 could definitely use a better example, such as a = 12 (so the result
would be {12, 11, 10}.

I admit, I checked number 2 closely, saw a horribly invalid failure reported
and only glanced over the rest. Independent of the other failures, you cannot
have incorrectly graded questions, and the rest of the points still stand, the
requester has nothing to compare against, leaving the judgement up to your
pass/fail (which is clearly not 100%) and the random assignment of questions
to different test takers results in no real comparison ability between skills.

~~~
nps
You're right about question 2, my apologies. There indeed was a bug in the
test data for that question (which we have fixed now). However, the rest of
the questions given in the test work properly.

The reason different test takers get (somewhat) different question sets is
because our algorithm alters the difficulty of the questions based on the test
taker's performance. I.e. if you did the first few questions well, you'd start
getting progressively tougher questions. On the other hand, if you do badly,
you'll be given successively easier questions.

------
sandaru1
Clickable Links

<http://beta.techtestnow.com/>

Java :
[http://beta.techtestnow.com/en/user/testLogin/secret/TTN4c24...](http://beta.techtestnow.com/en/user/testLogin/secret/TTN4c2479e455a032.22415626)

Ruby :
[http://beta.techtestnow.com/en/user/testLogin/secret/TTN4c24...](http://beta.techtestnow.com/en/user/testLogin/secret/TTN4c247a1e3fc618.66097557)

------
jawn
Wow the recorded sessions feature is really cool. Props to you guys, I hope
you do well.

------
zaph0d
Where are the questions? The UI is not very intuitive. Good luck anyway!

~~~
nps
I think that the confusion is because you attempted to take the test using the
login credentials for the recruiter.

Recruiters cannot take tests - only candidates can. Note that you can easily
create a new account at the 'take test' page (please use one of your email
addresses).

~~~
zaph0d
Recruiters should be able to see the questions and answers.

------
AmberShah
What is the difference between this and <http://codility.com/>?

~~~
sandaru1
TTN has quite a few more features. The key ones would be :

1\. Viewing a playback of how the candidate coded the question.

Even a bad candidate might be able to 'prettify' his final code (or even
craftily cut and paste code while solving the problem).

However, it is almost impossible to do so in a playback of the entire coding
process.

It is like the difference in complexity between creating a fake photo vs
creating a fake video !

2\. Adaptive testing

Depending on the performance of the candidate, the questions presented will
increase or decrease in difficulty - allowing the system to accurately assess
the candidate's skill level (that is the advantage of using several small
questions, as opposed to a few large questions).

3\. The ability to embed the testing widget both on your website or even on a
job posting on a job site - that way candidates never even have to leave your
website.

~~~
AmberShah
Number 1 is actually a downside, to me. I don't really care whether a coder
starts with ugly code (don't we all?) but I care what it looks like when he
hits "submit". Also, do the rules specifically disallow writing the code in my
IDE and then copy/pasting it in? I think that would be a silly rule. What
about a good developer who writes a test first, and of course, codes against
that test.

However, number 2 and 3 are nice.

~~~
nps
Both of the points you make are quite valid.

As you said, it is the final code which is more important. But don't you think
that it is also important to view the process by which the programmer arrived
at the final code ?

An experienced coder will quickly reach a solution. Although he might make
several mistakes on the way, he can usually quickly detect and correct them.

Someone less experienced would take longer, make more mistakes and probably
take several attempts to correct those mistakes before getting it right.

Even though the final code might be more or less the same, the playback would
help you discern who has better coding skills.

Of course, this would mainly be of use when attempting to choose between two
closely matched coders.

Regarding copying and pasting - at the moment the rules don't explicitly ban
this. The problem with allowing it is that the likelihood that the coder might
attempt to cheat (and get away with it) is vastly increased (and cutting and
pasting makes the playback feature worthless !).

That is why we went to the trouble of enabling syntax highlighting,
compilation and testing within the widget, so that it would be as close to an
IDE as possible.

BTW - regarding testing - aside from the 2 default test cases, it is also
possible to enter your own testdata in the widget.

------
Prisen
The coding window assumed I have an american keyboard, which made it sort of
hard to program.

~~~
nps
Thanks for your input. Can you give me more details ? What were the problems
you encountered ?

~~~
Prisen
Special symbols, like []() etc. are located on different buttons on an
american keyboard compared to e.g. a swedish one. So when I push shift+8 I
usually get "(", but here I get "*".

------
seeker_ak
grate work guys

