We also have immigration into the US. I read somewhere that growth in the population vis-a-vis immigration (and the higher birth rate of recent immigrants) propped up US growth, and explains both why the US didn't suffer as badly as Europe post-recession, and also why our demographics will look better down the road (better ratio of workers to dependents).
Japan has very little immigration. Europe and America are obviously grappling with these issues politically, but it's pretty clear from the economic numbers that immigration is important for growth if you don't have stable population growth. (Japan and parts of Europe are in decline, as would the US be if not for immigration, I believe.)
And at some point the growth stops, but the policy of expanding generally doesn't.
Reality is, Europe and America are going to have further population declines a la Japan. Shrinking population = shrinking growth.