So if people are continuously creating unmaintainable messes and technical debt we should just throw up our hands and say "oh well, we shouldn't judge"? Assessing capabilities is required for improvement. If someone is creating poor quality work we can either replace them or train them to do better. But if we can't assess them then we won't know when to do either one.
I am coming to the conclusion now that actually talking to a programmer is the best way to judge.
I have had various interviews in the last couple of years. One I got offered the job without so much as any technical test, based on my CV. I almost felt a bit cheated there, as I wanted to prove I was good. I interviewed for a startup with non technical founders. They had pulled a Python test from the internet which seemed 90% about lambdas in Python (I can use them, but in most cases list comprehensions provide a more readable syntax).
I actually like being able to explain my thought process in an interview. Even if I don't get it right, then they may see that I am "almost there". Likewise if I don't know something, I can tell them.
I have a fair idea about my colleges work, not from their code, but by the way they talk about it, and seeing the tools they use. (The "best" in my office of 4 chooses Java for most of his stuff for some reason - not what I would expect from someone good. The worst code was written by someone who ranted about how I should be choosing Python over Perl.)
