I don't think this is true. For example, as a math teacher, I couldn't do a very good job predicting how easy or difficult students would find particular problems. But I could easily predict which problems would be easier and which would be more difficult. I could do that even though I personally understood all the problems.
I'll attribute difficulty to the energy required to resolve a system. For example, pulling weight. The complexity of the action is the same. But the difficulty depends on the weights to pull.