Compared to most Silicon Valley founders he was, his adoptive parents were blue collar with no college education. His dad made most of his money as a repo man.
Part of the agreement his biological mother made with the Jobs family to allow them to adopt was that Steve had to go to college.
"Poor" may be stretching things a bit.