Then please spend some of your time making it better. Your contribution will be welcome.
With fewer people helping, any open source project will suffer from being horrible and unpolished.
> whether they're intentionally held down, or just neglected and without as much attention.
There is definitely no intentional holding down that I'm aware of. It is clear that the flavors do receive less attention, but that's just down to popularity. Those who care about them more than the Unity route should contribute!
> People who have seen their productivity drop because they've been forced to use Unity
Nobody has been forced to use Unity. Xubuntu has existed for a long time, and more recently so has Lubuntu.
If anything has been forced, it is the dropping of GNOME 2. But that has happened upstream (including in Debian) and Ubuntu cannot reasonably be held responsible (even though this seems to be an unfortunately common misunderstanding). AIUI, difficulties with GNOME upstream is one of the reasons that Unity was created.