I was talking purely about those abstract interview questions where the full problem can be summarized in a few lines and the expected solution is ~50 lines max. I've just tried with Claude 3.7 and it's produced a seemingly plausible solution at the "word finder" problem (2D grid of letters - given a word, return if it's present in the grid as a series of adjacent cells in any direction).
I agree that they're far from taking our jobs, which is my point - those interview questions are absolutely not representative of what we really do and optimize for the wrong outcome.
Fair enough, but my bet is that even trivially recontextualizing an interview question is enough to wreck an LLM's performance.
If you change the "find a word in a grid of letters" question to a "find a Collatz sequence in a grid of numbers" question, does it still work? As an interviewer, I would expect a qualified candidate to spend maybe an extra 5-10 minutes asking clarifying questions and understanding the difference between the two.
I was talking purely about those abstract interview questions where the full problem can be summarized in a few lines and the expected solution is ~50 lines max. I've just tried with Claude 3.7 and it's produced a seemingly plausible solution at the "word finder" problem (2D grid of letters - given a word, return if it's present in the grid as a series of adjacent cells in any direction).
I agree that they're far from taking our jobs, which is my point - those interview questions are absolutely not representative of what we really do and optimize for the wrong outcome.