To give you an example. Try writing code in Java these days. With Eclipse very soon(And trust me you will feel this in days) you will realize that you are hardly doing anything that is intellectually heavy lifting. Unless of course you are doing something that is totally new to your area of work. In that case its totally different. Which is the same case with assembly workers too(A totally new product unknown to you will be difficult to assemble).
But imagine doing applications which have nothing more than a DB interaction layer and some business logic. You can do this sort of a job for first 6 months of your programming career, and then you can have your muscle memory built to work with nearly auto completion method out there. Soon you will see that you will be doing nothing other than assembling already well known blocks of code without putting any sort of hard intellectual effort.
Rinse-Repeat this with a few domains/programming languages/architectures. Very soon you will find that you are doing nothing more assembling reusable blocks of logical pieces.
The only difference between hard labor and something like programming is the physical tiredness. This I completely agree with you that physical labor is more demanding to your body.
But other wise there is no reason for us to believe that the nature of our work is any different than anybody else.