You're right, I think this was probably a major factor in why he was voted in (I wasn't alive at the time so this would just be a guess from my side). I think the big problem with Carter is that he was what you could call a "peace-time president" in a period where America was basically still very much at war. What America needed was something closer to a Churchill, and they essentially got it in the form of Reagan later.
I think Carter would've been assessed very differently if he'd been president in the 90s or early 2000s.
Or maybe he was. After Nixon/Ford people were tired of corrupt career positions. Carter was folksy, one of us.