You got the "luck" part right. The rest, the "other half" you spoke of first, is solid gold bullshit.
We can talk about hard work and smarts all you want. You can even bring up outliers—cases where people have pulled themselves up from hardship and succeeded anyway. But they are not statistically supported. The truth is that the better upbringing you've had, the more opportunities you have had to improve yourself and your abilities, and the larger support network you have, the more likely it is you will succeed.
It really is that simple. It is situational. Purge your mind of the bullshit, because it is incorrect flawed logic.