
Codility - Automated tests of programming skills - grk
http://codility.com/
======
defdac
Very good idea, but I don't program that way. I use a competent IDE to tell me
syntax errors and obvious programming mistakes while I write. If I can't debug
my algorithm with trial & error I write a test first and modify my algorithm
until it complies with the test. To think that a programmer will always come
up with the exact and most optimal solution to any given problem is a bit
naïve. In my 10 years as a programmer I have never seen one wonder-kid that is
able to do that. The real wonder kid assets in a team also tend to be those
that are more creative than syntax-anal compared to those that can write
syntax-error-free-code with pen and pencil and remember large parts of a
phonebook...

~~~
marcin
Well, that's one more reason you should like Codility - you can run test
compilations (the VERIFY button) which will show you syntax errors. And in
bottom left corner you can apply your test data sets. In other words - you
work like in your every day environment. Check it out and let me know what you
think.

~~~
defdac
I did, and now I see comments stating that you should be able to run tests on
your code while developing it. That's awesome and what I was missing. Next
step intellisense and on-the-fly syntax checking. I hope no programmers click
the "compile"-button nowadays to check if their code is correct...

------
JoachimSchipper
This is quite nice, even if the exercise isn't all that difficult.

It would be nice if the boundary conditions were better specified (people seem
to have been bitten by overflowing integers). This could be inferred if you
told people that you're using a 32-bit machine (the sum of elements is at most
SIZE_MAX * INT_MAX; if int and size_t are 32-bit, SIZE_MAX * INT_MAX <
INT64_MAX.)

I was also somewhat surprised to see that my code compiles without #include
<stdint.h>. I'm not sure this is correct (it could be solved by putting the
code written by the candidate _before_ the test code).

Finally, I assume you have proper sandboxing set up?

~~~
jakacki
We deliberately underspecify the compiler version, we think in such situation
a programmer should attempt to write code as portable as possible.

#include --- good catch! We are not very strict about missing includes, but
this is definitely not something we want to penalize.

Sandboxing -- absolutely! :-)

------
cf2000
I think this is cute but I would think hard about the kinds of questions you
develop for this. Coming from the perspective of someone working for a big
company with a ton of software people that is not a tech company, I could use
something like this in general. But, the question you asked is the kind of
thing I ask CS undergrads (almost exactly, in fact) because it allows for
clever solutions without actually requiring someone to be a particularly good
programmer. I can work through it with them and if the basic solution is easy,
prompt them to give me something more clever, so I know how they think, which
is good when you're looking at an inexperienced developer. However, for an
experienced developer that I want to hire to churn out Java code, the
difference to me between a correct but simple O(n^2) solution vs a clever O(n)
is not terribly meaningful for determining whether they can actually do what I
want them to do. I want something that requires people to write more than a
handful of lines and demonstrates an ability to keep track of a few more
variables correctly, and honestly I don't care all that much about their
algorithmic complexity (although I might care a great deal about their space
usage, usage of external resources, etc). So, if this is the audience you are
looking to attract, consider thinking about this as a screening tool for
people outside of the startup world that actually need "knows how to really do
shit in language X" more than "knows how to write clever algorithm,"

~~~
jakacki
I don't think Codility checks whether one knows how to write clever algorithm.
Look at tasks from ACM Collegiate Programming Competitions, they require
clever algorithms. Codility deliberately focuses on very fundamental tasks.
Also it does not pretend to be a final sign-off, it merely tells you which
candidates to reject due to lack of rudimentary programming skills. It's not
that surgeons make injections that often in their daily practice, yet I would
not trust a surgeon who doesn't know how to make an injection.

~~~
charger
I like this comment. A typical surgeon earns a bachelors degree, then attends
medical school, before finally spending three to six years in a residency
program (in the United States). This is before becoming a practicing surgeon.
Unfortunately, there is nothing similar (in the United States) that is related
to software engineering -- though many here would like to see something
similar to an engineering license for technology professionals; there is
nothing currently on the horizon (that I know of). In some ways, I think that
Codility can help with this state of affairs. This is particularly interesting
when we consider comparing metrics for education and experience; and perhaps
'real-world' success. The Codility statistical comparison, taking these
factors into account, would be very interesting indeed. And could possibly
result in data that furthers the state of the art in general.

~~~
ununoctio
{In Pascal Language+Assembly} {$ASMMODE INTEL} function equi (A : Array of
longint; n : longint ) : longint; var c:Longint; label noOverflow1; label
noOverflow2; label ciclo; label fine; label Over; label tot; Begin Asm DEC n
JS over XOR ECX, ECX {Somma1} XOR EDI, EDI {Somma2} XOR EAX, EAX MOV c, EDI
MOV ESI, n tot: MOV EDX, A MOV EDX, [EDX+ESI*4] PUSH EDX ADD ECX, EDX JNO
nooverflow1 ADD c, ECX nooverflow1: DEC ESI JNS tot; SUB ECX, c SUB EDI, c
ciclo: POP EDX SUB ECX, EDX CMP ECX, EDI JE fine ADD EDI, EDX JNO nooverflow2
DEC EDI nooverflow2: CMP EAX, n JA over INC EAX JMP ciclo over: MOV EAX, -1
fine: end; End;

~~~
junkeater
lol I got 101 on the test.

------
Blasa
This is interesting. It might be worth noting somewhere how many bits is an
Int on your test rigs.

I got 94 using C in 12min (didn't check for overflows of large numbers, silly
haskell making me used to Integers).

~~~
jakacki
Generally we don't want to penalize people too badly for assuming particular
(but realistic) type ranges, nevertheless we think that perfect solution
should be as portable as possible across different implementations of the
language (e.g. one should not assume that C's int is 32 bits wide, since
implementation with 16-bit int can still be Standard compliant)

------
patio11
That is an amazing piece of software. I hope you make a mint.

I'd give people access to their language's output facility for playing around
to the solution, but that is a minor nitpick. (And I suppose I could open an
irb instance on my machine for exploration and then give you code when I'm
good and ready.)

My Ruby solution, in case we've got anybody who is struggling:
<http://pastie.org/771438> This would be my first cut if you asked me to
whiteboard this.

~~~
moconnor
Pasties are a good idea, but results pages are more interesting. Here's my sad
effort after 8 mins in python:
<http://codility.com/demo/results/?id=demoXFXD64-GCZ>

The site's very slick. My only feedback is the histogram on the results page
could be more prominent - it's cool, useful information but I ended up
inspecting the html elements to work out what it was.

Also the results page seemed to keep refreshing for me while waiting for
results (Safari 3). Surely that should've been AJAX'd?

~~~
lincolnq
regarding your python code: do you know about 'enumerate'? It returns an
iterable of (index, value) pairs of a sequence. :)

~~~
almost
enumerate is very useful, here's mine:

    
    
        def equi ( A ):
            right = sum(A)
            left = 0
            for i,x in enumerate(A):
                right -= x
                if right == left:
                    return i
                left += x
            return -1
    

What's interesting is that all the versions I've seen so far are pretty much
identical apart from very minor things. This would seem to me to be obviously
the correct way to do it but I'm interested if it's just _a_ way. Did anyone
solve this in a different way of equal or greater elegance?

~~~
anc2020
Mine kept one cumulative list adding left to right and another summing right
to left. This is conventionally slower, but conceivably faster if you do it in
parallel, leaving the final loop to be something like:

    
    
        for i in xrange(len(A)):
            if right[i] == left[i]:
                return i
    

The final loop here is slightly tighter, so as long as your parallelism didn't
add big overheads, this will be faster.

Original: <http://pastebin.com/m6e742f56> Parallelised:
<http://pastebin.com/m32a60b1>

------
samstokes
I like how the performance evaluation seems to be fairly realistic - I used
Ruby and my algorithm could have been more efficient (O( _n_ ) space, and O(
_n_ ) time but with more traversals than necessary), but I scored 100 because
I met all the _stated_ constraints.
(<http://codility.com/demo/results/?id=demo8XKE77-H8G> if anyone wants to
refer to and/or mock my code.) It seems to me "ability to solve problems _well
enough_ under time pressure" is exactly what you want a simple hands-off test
like this to screen for.

In face-to-face interviews you'd probably follow up by adding constraints:
"your algorithm uses O( _n_ ) space in addition to the input, can you do it in
constant space?". It would be cool if Codility supported that kind of workflow
- following up the submitted solution with refinements on the original
question. (Relatedly - do you support the workflow where you get a second try
if you didn't get 100, maybe because the evalution revealed a constraint that
wasn't clearly stated in the problem outline?)

~~~
jakacki
With paid account it is possible to reopen a task after one solution has been
submitted, so the recruiter can "give another chance" to the candidate
(although we generally suggest not to), which supports the workflow you
describe.

------
Erwin
Is the code editor they're using something publicly available? I'm looking
around for a light in-browser code editor with programmer-oriented features
(e.g. syntax highlighting, this also seems to auto-indent Python code to some
degree).

~~~
ideamonk
Even I noticed that I had seen something ditto somewhere... Hmm so Codility's
editor is copy of the one SPOJ uses, so which one do they both use? Can I too
get it :)

<http://twitpic.com/x6wkq> and even the code matches line by line I guess -
<http://twitpic.com/x6wnp>

~~~
jakacki
Codility uses open-source component EditArea.

> Hmm so Codility's editor is copy of the one SPOJ uses

Please don't flame. Codility has not copied it from anybody, its EditArea-
based interface is in place since 2008.

~~~
ideamonk
I beg to edit the word "copy", but I can't do it now.

Hmm so Codility' and SPOJ use a similar editor, so which one do they both use?
Can I too get it :)

Thanks for pointing out. /me looks into EditArea

------
marcin
Thanks for the positive feedback. Do not hesitate to ask us any questions.

best Marcin Codility

~~~
jkkramer
Nicely done overall.

You might want to lock down functions like glob() and readfile() in PHP --
they effectively give the coder read access to your file system.

~~~
jakacki
Good point, thanks! But no worries, this is not our real filesystem :-)

------
Estragon
It gave me a test concerning finding the "equilibrium index" of a sequence. It
said to assume that the sequence is very long. (The question is repeated here:
[http://forums.sun.com/thread.jspa?threadID=676596&start=...](http://forums.sun.com/thread.jspa?threadID=676596&start=0))
This problem seems underspecified, because it doesn't state whether you can
iterate repeatedly over the sequence in question. If you can, it's easy. If
you can't I'm not sure how to do it.

~~~
bsaunder
_iterate repeatedly over the sequence in question_

Does that ever seem like a good idea? The description suggests it may be a
long list. That should give you a hint that you don't need to do this to solve
the problem.

~~~
Estragon
I'm pretty sure you have to iterate over it at least twice. I guess the
specification is implicit in that.

~~~
anc2020
The best case possible is to iterate the list once - you sum the list from
left to right holding cumulative answers and if the final sum is zero, that
index is a valid answer.

The worst case of the best algorithm is to iterate twice - firstly as above,
and then again in the opposite direction, comparing values from the first
iteration.

------
psranga
Really nice site. Well done. The example problem that showed up to me
(equilibrium point detection) was not trivial, but didn't require advanced
stuff which a halfway decent programmer wouldn't use day in and day out.

I also like the way you tell me what you scored on and what my result was.

I wonder if you should relax the condition that you can submit only once.

I gather TopCoder offers something similar. What's your strategy to compete
with them?

PS: got 94 using C++ in 16 min b/c I didn't check overflow of large numbers.
Good thing you check for that.

~~~
marcin
Right now TopCoder is focused on coding competitions. We are focused solely on
recruitment - those are pretty separate markets at the moment.

------
ThinkWriteMute
You know, now that I think about it...The description of the test was pretty
incoherent. If I don't know what my test data is, and if the problem
description is incoherent, I can't really make it work.

It's also taking a metric fuckton of time to evaluate my solution. Which
apparently fails. For reasons I can't debug because I can't see the test data.

Any company that relies on code snippets, incomplete problem statements and
undefined test data to judge coders will get what they deserve.

~~~
andrewcooke
maybe there's more than one task, but the task i took appeared to be
completely specified (the one where you find a balance point - i may have
remembered the name wrong - for an array, so that the sum of elements to
either side is equal).

it wasn't obvious to me what to do on first read, but when i paid a little
more attention, everything was there (including edge cases, as far as i could
see). in particular, they gave a formal numerical expression that was
unambiguous (and which didn't seem to contradict the text).

~~~
marcin
Yep, it's good to read the specs in detail ;)

~~~
loup-vaillant
The spec _was_ clear, but the fact that the test dataset used for the test was
the very one given as an example wasn't. plus, it was a bit frustrating not
being able to fiddle with the test dataset. I would have done it locally with
my compiler, but the necessary boilerplate would have lost me time. So, I
didn't test enough, and made a silly error which went undetected because of
the trailing zero. I still got 75%, but that looks quite miraculous.

EDIT: It appears I _can_ customize my test dataset, and I didn't see it. Silly
me.

~~~
jakacki
Yes, it is even possible to add your own data sets, Codility will tell you
what you code returns for them. I guess we should make it more apparent in the
interface.

------
grk
5 minute version of the demo problem in Ruby - 100%:

<http://codility.com/demo/results/?id=demoMPC2W9-JDG>

------
tsiki
As a student this is very interesting since it could also be used to measure
self progress. There's a myriad of ways to test your abilities in most
subjects, but if I want to find out if I'm any good at programming or to
measure my progress, there are very few easy ways to do that. Are you going to
sell this to just business's?

~~~
tudorachim
Have you tried topcoder? It's the same as codility, but with cooler/harder
problems.

~~~
tsiki
Thanks for the tip, topcoder looks pretty interesting in general, too.

~~~
eru
Also Project Euler might be worth a look. It focusses on solving math-related
problems with progams.

------
stonemetal
For C++ it would be nice to know what compiler is being checked against. At
least in the demo question I would use a 64 bit int to hold the sum if the
arrays are really "large" but MSVC names the type __int64 vs G++ using long
long and other such minor details. Same for the other languages aka python 2.x
or 3.x etc.

~~~
jakacki
We want to avoid sticking to particular compilers/interpreters. An expert in
MSVC should not be penalized only because we use GCC. Also we design the tasks
so that they are compiler-agnostic (actually they are even language-agnostic).
It is sometimes a bit tricky due to non-standardized type sizes (e.g. C, C++,
PHP), but generally the tasks can be solved using very limited and portable
subset of a language.

------
anc2020
Got my 94% C solution up to 100% by using long long ints for intermediate
sums.

<http://pastebin.com/m6e742f56>

Edit: And won't this be a problem if interviewees can access the internet? The
interviewers will have to make sure they can check the browsing history of the
computer being used!

~~~
inukun
Why? After all, you are going to use the internet while on the job too. Just
making the problems more difficult should fix that problem.

~~~
anc2020
I don't mean the interviewers should shut off internet access entirely, but
what I mean is that if they do allow access to the internet, they'll need to
check the interviewee hasn't just googled for the solution to their codibility
test (and they'll also have to make sure the interviewee can't delete their
browsing history too).

~~~
marcin
Don't worry, we have some nasty surprises for the cheaters ;)

------
kbob
Codility people, if you're reading, here's a bug report.

The test window's layout is fubar if you use a small window size. I was using
800x600 and the language selection buttons were hidden behind the source code
text box. Something (couldn't tell what) overlapped the first two lines of the
instructions.

Firefox 3.5.7/Ubuntu Linux 4.10, FWIW.

~~~
kkowalcz
thx! we will look into that! Chris

------
alextgordon
I'm not sure the timer is a good idea. For one, I found the seconds counting
down _very_ distracting. It also encourages quick and inefficient solutions
that are easy to hack out, over more efficient algorithms that may take longer
to code.

Maybe you should time the programs, instead of the programmers.

~~~
marcin
We are timing the programs as well - with the scalability assessment.
Recruiter can change the timing anyway, but we feel that a bit of pressure is
good. You don't always have comfy conditions on the job after all.

~~~
loup-vaillant
Right, but maybe not down to the second. I personally didn't mind, but
counting seconds causes a distracting flickering. Maybe you should consider
activating it on the last minute only ?

~~~
jakacki
Hm, would something more smooth (analog?) be better?

------
anc2020
Might need some way of making explicit which libraries are available?

eg. if the programmer goes off and writes a multithreaded solution, they might
only realise too late that pthread isn't included. Admittedly they should have
checked first using verify, but you never know with interview pressure.

~~~
jakacki
Unless it is stated explicitly, we do not provide anything extra but the
vanilla programming language and standard libraries. It is more clear for some
languages (e.g. Java, C++) than other (e.g. PHP, JavaScript), but we try to
stick to the reasonable canonical set of libraries.

------
theycallmemorty
Seems like a cool idea. If they were smart they'd not only target companies
looking for people but people looking for jobs. e.g. Adding a "Allow the
codility community to take this test" check box. Then you have the a whole
world full of people taking your test.

~~~
kkowalcz
Actually it's there - if you click on "Take a free demo test"
(<http://codility.com/demo/take-sample-test/>) on the Codility page there is a
free test and you will be given the option to tweet your result at the end.

------
ThinkWriteMute
Wow, beautiful design.

That said: Jesus I must still suck, or maybe Math simply isn't my strong
suite.

~~~
olliesaunders
Yeah, that demo problem would probably take me at least several hours to
solve. I'm bad with sequences and I have a volatile short term memory which
means I make mistakes and am slow to process numbers and logic. Interestingly
these flaws cause me abhor any over complex logic or code where you have to
remember more than 4 or 5 things at once to use properly, which makes my code
clean. So maybe I'm not a good programmer (I can't write the complex stuff
very easily) but the code I write is usually good. Try testing for that
Codility!

BTW 10-20% of the population is like me.

------
Kliment
Very nice product. Much-needed quantification in tech recruiting. Maybe
consider marketing it to the stackoverflow people. They have been running a
job board with much fanfare since recently. (94 points for me in 7 mins, yay)

------
j_baker
Great. So now we're going to be hearing about TDH zealots (Test Driven
Hiring).

~~~
kbob
I've been a TDH zealot, as you put it, since 1986. It's amazing how hard it is
to evaluate coding ability in an interview without it.

I readily concede that coding ability isn't the only thing a dev needs, but it
is absolutely required for most jobs.

~~~
j_baker
For the record, I was kidding. :-)

------
melito
Just a heads up:

I took a demo quiz and it worked fine. Great idea for a product.

However I think I'm too competent for the test because I was able to run
something I probably shouldn't have been able too.

~~~
jakacki
Yep, we've seen it... :-) Thanks for pointing this out!

------
shib71
This would be kind of cool simply as a brainteaser site with problems taken
from Project Euler.

------
scotth
This is great!

I've already sent it around at the company I work for. I'm hoping we can put
it to use.

------
andrewcooke
also thought this was nice after trying it out.

one thing i missed, though, was the ability to write and run my own tests.
perhaps a "scratchpad" that evaluates code and returns a result would be a
nice addition?

~~~
marcin
Bottom left corner. It's obviously not visible enough :)

~~~
TomasSedovic
I just can't see it. Is it in the demo test?

My bottom left corner is empty. Except when I click "verify" it shows message
whether the tests passed or failed.

But I don't see the tests themselves, nor did I notice a way of writing my
own. I've tried it in Firefox, IE 8 and latest dev Chrome.

So, is it possible to write my own tests when I'm _taking_ the test, or only
when I'm creating a new one?

~~~
marcin
Sorry, some bug on our end - it should be there. We are fixing it right now.
In the meantim - if you sign up for a trial account
[<http://codility.com/accounts/codility-register/>] it works fine there (a
blue "add a test case" button in the bottom left).

~~~
andrewcooke
thanks. another smaller point i just remembered - i was a bit surprised to see
that the evaluator could see my submissions to the "validate" function. given
that you do say they won't influence the score i assumed that they would be
discarded.

i can see there are several reasons why this is useful to the evaluator, and i
can also see that you're getting close to saturating the user with
instructions, but in a perfect world it would be good to have "full
disclosure" (i guess you do, in that anyone can run the example....)

~~~
marcin
Good point, thanks for the feedback.

------
rpledge
Very cool product. Well done

