I used to have a math puzzler that I'd ask. Almost nobody got it right. One guy got close, and we ultimately hired him. He asked for the full answer, and I gave it to him. He told me I was wrong. I pointed out that I had asked this question at least 50 times, and knew it in and out, and that he should reconsider. Then he sent me a spreadsheet proving that I was wrong. Whoops.
It's a question that checks intuition on probability and the law of large #s. I don't want to give too much more since I do pull it out on the very rare instance where someone is very bold about their stats knowledge.
To your credit, you are among the very few who would admit this kind of error and you likely took it gracefully.
But how many times did you ask this question without actually verifying that YOU could answer it? And what does that mean about your process for generating and grading interview questions? And why did it have to be the interviewee who told you?
Maybe interviewees should be more confident, but maybe interviewers should be a little less self-confident. This is a huge cultural problem for the industry, but it's right in our blind spot.
And we have to ask why there are less women in tech since it became dominated by erroneously self-confident, high-testosterone personalities who think that hiring people like themselves is a meritocracy?
Without getting into too many specifics (on rare occasions I pull it out again) it was a question of statistics and law of large numbers intuition. I was checking for how directionally correct people would be on the subject, with 3 parts of the question to see how deep they knew it. More about intuition than actual computation. There was a subtle error on how aggressively to apply the law of large #s.
How many times had I asked it? 50? I had a couple folks answer it like I thought it should be answered. He was the only who nailed it enough to catch the error too.
The reality is interview performance is very different than work performance. Interview questions test interviewing ability more than anything else.
Over time I've realized that the only reliable predictor of success is seeing people's work results first hand when they don't think they are interviewing. The danger is this too breeds issues of familiarity and lack of diversity, because one only hires from their narrow circle of the world.
This makes it very tough to make college hires. The only things I've seen that worked are "High enough" GPA (with "High enough" being higher on non-technical majors) and some sign that the student had to significantly multi-task (lots of extra curriculars, research, or working through school). Most 21 year olds don't have interview training, so it's a very bad way to pick them.
I was in the position of that candidate before, and the interviewer refused to (do the analog of) letting me set up the spreadsheet, then vetoed me from the rest of the interviews.