Working hard is making sure you don't toss your winning ticket away. More people, working to keep their families out of poverty, changes the odds of the lottery.
But that doesn't change that it is a profound fortune to be born into such a family.
I agree with your points, I have seen people use similar justifications for why they have not improved their lives. Which is unfortunate but I think it's the wrong battle.
Many more times I have seen people from middle class or better backgrounds talk about how they got where they were because of hard work or drive or whatever. And then use that as justification for why some social program should be cut, like food stamps. The way I look at it, the "might as well not try" argument harms a few people who have the least influence in society. The "I worked hard and made it so we don't need social programs" argument is made by some of the most influential people and the programs they cut effect millions.
How could I make sure my offspring have a better life than I did? If my only option is "luck", then why bother trying?