We are looking at very long time periods, the cave La Cotte de St. Brelade had continuous Neanderthal occupation for 250 000 years, the span of recorded history is roughly 5000 years.
So called modern humans and neanderthals shared living space for roughly 2000 years, they were genetically close enough to have viable offspring. Like a donkey and a horse produce a mule that is infertile.
With human neanderthal offspring it is likely the male offspring was infertile but the female was fertile
Most remarkable for me is how long the neanderthal was a stable group and even now they contribute quite a large amount to the human phenotype.
I wouldn't call anything on the likelihood of fertility without genomic analysis. These branches of humanity were similar enough that the offspring could be viable, in fact, sometimes you can find people with more neanderthal residual genomic admixture nowadays too.
