This goes for a lot of services – why do YouTube recommendations take up the majority of its homepage? Why are Instagram account suggestions pinned to its sidebar? Why are there news article recommendations below the article that you just read? Suggestions are meant to help users discover content. It's like how there were curated link directories before search engines existed, to help users discover new websites on the internet. This is the same idea, but limited to a specific platform so that you spend time on only their share of the internet.
As an extreme example, what if all forms of suggestion/recommendation/curation features on all platforms were gone, and only a search bar exists? Then your engagement is limited to the scope of your own thoughts, and you would just leave when you have nothing in mind that you want to search (and thus see fewer ads, generate less revenue, etc). I think there is a balance to be achieved, but companies certainly error on the side of more opportunities to drive engagement.
Anyway, your best bet is probably to cook up a browser extension/script to hide what you don't want to see. Maybe it even exists already. Though of course, the DOM probably changes all the time.
I actually thought about youtube as a contrasting example. Youtube has their recommendations on their homepage— that makes sense to me. I haven't chosen any content yet, so they want to help me.
But in this case on Twitter I've already chosen a specific profile to view, and now they're obscuring that with recommendations.
But then where do you go once you're done viewing that one specific profile? You probably close the tab because you finished catching up on their tweets, and/or go to some other webpage that they linked to in one of their tweets.
The suggestions implore you to continue browsing on Twitter, instead of leaving Twitter altogether. The cynical take is that the purpose is to drive engagement and revenue, while the UX design take is that this helps users find more conversations that they're interested in, that they wouldn't otherwise find ("you follow Foo, people who follow Foo also follow Bar, so you might like Bar").
Well if it were designed as an earnest feature, it would go on the side right? I can't imagine any UX pro saying 'let's plop this right in the middle of their requested content'. So I see it as pretty cynical. Any time software says 'I know what the user wants more than they do' I tend to think that's a mistake. But I think the consequences of this sort of thing plays out over a long period of a time. Maybe past when some executive leaves for a new job.
As an extreme example, what if all forms of suggestion/recommendation/curation features on all platforms were gone, and only a search bar exists? Then your engagement is limited to the scope of your own thoughts, and you would just leave when you have nothing in mind that you want to search (and thus see fewer ads, generate less revenue, etc). I think there is a balance to be achieved, but companies certainly error on the side of more opportunities to drive engagement.
Anyway, your best bet is probably to cook up a browser extension/script to hide what you don't want to see. Maybe it even exists already. Though of course, the DOM probably changes all the time.