Don't you still run into labor force participation rate issues due partly to wealth accumulation when shrinking the family size?
For example, if you receive your parents inheritance at age 50, you might be inclined to retire early. Even if you have to split it with one sibling, that could still be enough for early retirement depending on the specifics of the assets.
Even between relatively extreme examples of "every family has 2 children" and "every family has 3 children", the difference in inheritance isn't that big. So I wouldn't expect much more early retirement than if there was mild population growth.
For example, if you receive your parents inheritance at age 50, you might be inclined to retire early. Even if you have to split it with one sibling, that could still be enough for early retirement depending on the specifics of the assets.