I've had people not know why I'm reaching out; I've also gotten references selected by the candidate that did not have good things to say. eg "X is difficult to work with."
And most people aren't good impromptu liars. So pushing a bit with a reference on what did you work on together? Why is this person fantastic? Would you hire this person? can get you far. And if the reference has left their shared employment, the classic: why haven't you hired this person?
True. In spite of my earlier comment, someone earlier in their career who didn't work out--especially at a smaller company--may have trouble providing a reference. It is also the case that I've seen people get fired who just had no clue that they were obviously unsuited for their job and were totally blindsided.
You’d be surprised by how many people don’t think that through. There are always a surprising number of people apparently just going through the motions.