I absolutely believe this is true.
One thing that influence this is that the second (third, fourth) time that you are doing it, you are better at it and know exactly what to do. So, you get into shape faster.
"Epidemiological studies in human ageing cohorts also suggest that low birth weight and gestational malnutrition are strongly associated with reduced skeletal muscle size, strength and gait speed in older age"
More or less known among older folks.