Nigeria is all the way on the other side of the African content while the Arabian peninsula is right across the very narrow Red Sea. So it would make less sense for Ethiopians to speak a language that’s related to languages spoken thousands of kms away in West Africa than for them to speak a language that’s related to the languages spoken on the other side of the Red Sea - which is exactly what they do.
Edit: also, if you look at pictures of Ethiopians and Eritreans and of Yemenites you’ll see they’re way more similar in appearance than Ethiopians and, say, Nigerians.
Yeah, my wife is Ethiopian and she was shocked to see in TV news coverage how "Ethiopian" a lot of the Houthi rebels in Yemen looked, but this is actually not surprising once you know the histories of the Horn of Africa and Red Sea at a greater detail.
Edit: also, if you look at pictures of Ethiopians and Eritreans and of Yemenites you’ll see they’re way more similar in appearance than Ethiopians and, say, Nigerians.