I think your last point holds true, but it's really not an equivalent amount of stress you're putting on your body in an office. You can always go outside more after work and remain physically active to counteract the damage you're doing by sitting in an office all day, but if you're in a physically intensive trade, you can't avoid the damage you're doing by remaining less active after work.
Also you're going to have less energy to do the physical activity you want to do.
I paved roads for a summer at 18 and even though I was a very young energetic kid, I was barely getting anything out of my evening workouts or soccer games with friends after 10 hours outside shoveling asphalt.
