There might have been many times more people living entirely isolated populations which would have left no descendants. Which is not unlikely given the extremely low population density.
Even if all modern humans were confined to Africa if there were only 10s of thousands of them distributed across a significant part of the continent there is no way they would've had actual direct ties to each other.