Take, for instance, many countries in Africa. What prevents these countries from not rising in the poverty ladder is not the lazy, dumb people of these countries.
First, they can't sell products to other countries because the majority of countries are land-locked, meaning they have no way to ship their goods out at a reasonable price. They don't have any ports in their country and the neighboring countries with ports are hostile or unsafe.
Second, wealth is not a stable commodity in these countries due to coups and constant political restructuring and military action. Even if you were a very hard ambitious worker, there is nothing you can do about external forces (read: rebels invading your home/shop and stealing all the wealth you have built up).
Third, little of Africa is actually arable, and Africa can't even become an agrarian society because there isn't really the resources for them to do so. And the resources that are valuable to the rest of the world that could generate money for African countries - like, oil and diamonds, become hotly contested and a source of conflict and instability, furthering reducing the value of storing wealth.
Take some time to learn about the world if you have a strong opinion on this matter.
If you compare any rich country with any poor country, you can make a similar list of things the poor country is lacking that make it poor. Those things are all forms of wealth. Whether it is natural or man-made doesn't matter.