Programmers have the difficulty of the unforgiving reality of having to make things _work_ - pass tests, compile, perform. This makes their job challenging, but it also makes it rewarding in that you know when you've gotten it right, when that piece of work is done.
Project managers have a task much more like herding cats: a poorly (or non) defined problem, moving goalposts, and a lot of "least worst compromise" kind of decisions. They have to deliver business results while keeping engineers and execs happy, in many cases with much of the responsibility for a projects success and little of the authority (i.e. they can't hire or fire engineers because the engineers don't actually report to them.)
I'm not saying one is actually harder than the other - that's very situation-specific. Just that the "difficult" parts of the project manager's job are less obvious to a programmer. (And, I'd note, often the _exact_ types of discomfort that most programmers would love to avoid dealing with themselves.)
(A good project manager is a shit umbrella, a bad one is a shit funnel. If you're in a shitstorm, learn to love your umbrellas.)