Good point. Here is why I don't see that as a problem.
1. Things are democratic overall. You tend to follow people that share things that you like. You see what they post AND what they "upvote". Whenever someone votes on something, it also shares to their followers. Combined with some weighting on how popular a post is, and how recently it has been touched by someone else, this effectively means that you get a list of posts relevant to your interests, sorted by popularity. Any one person leaving would have negligible impact on your feed's content.
You get a hive mind or you get regression to the mean, by which I mean you either aggressively filter for people with specific qualities, which is apparently a hive mind scenario, or you end up with the same kind of commentators you find on every other let-everyone-in platform.
Instead of subreddits, you follow users and tags. When you upvote posts, you share them with whoever follows you.
It's sort of like a community web of trust. It works really well.