In the beginning a had no idea what should I do, so I found it quite complicated, but then after a bit of thinking this was quite easy .
Well yes in order to make it more animated I think this should have a different approach.
I don't do much JS (as you can likely tell) but found the task approachable. I think that the low barrier to entry is a good thing for an interview question.
My approach is described in the comments. I started by writing it out and then went from there.
So. I took the exercise in a different direction. I thought of game of shells and my mind got stuck on the card flip matching game, so that's what I went with. I spent most of my time worrying about how much time had gone by, which wasn't helpful, but I think it came together.
I think the idea of this is great, and I may use a deviation of it for future pre-hire peer reviews.