There were many invasions into Britain. Romans, Vikings, Anglo-Saxons, Normans. And before that Corded Ware as well, that fully replaced dna of local population. And some Brits also went away and settled in Bretagne. So, yeah the picture is not as simple as that.
The interpretation that you have has more to do with English identity, which is strugling to accept that they once were invaders that certainly were not assimilating into Cornish, Welsh and Cumbrians, while even today dialect differences are present where these various people existed. That and in modern times descendants of common folks assuming identity that was belonging to aristocrate class devoid of any guilt for past and has nothing to do with them.