First, it was usually the assumed knowledge that got them. They'd mess up at calculus because of algebra; or get algebra wrong because they couldn't add. When you don't know the basics, you get the wrong answer even when you do all the new stuff right. The whole thing starts to seem futile, like climbing a mountain of sand.
Second, just as above. Most of my students had no idea that math, like weightlifting, is supposed to hurt a bit. They thought that heavy, stretching sensation you get when you learn new concepts meant they were stupid, that they couldn't do math. They didn't realize that every feels that, if only briefly. If you're in the bottom third of the distribution, and a third of people are, you never get to the other side of that feeling before the class moves on.
The answer isn't magic teachers. It's for kids to learn that learning is possible. You do that with practice, and feedback, until they get it right.
It is all about relating things and explaining in a way that can be easily referenced in your mind.
By the way...several of these people who could not pass the TABE tests at the time went on to get their GED's because of that one little step with fractions.