That's the entire point of a permissive license. You grant other parties the right to do as they wish, given a very lax set of requirements (usually just attribution). Would you feel offended if someone only distributed a binary build with their changes, or even charge for that? That's literally what the permissive license is about.
What may not be immediately obvious, is that it still favors the downstream developer to contribute their changes back; the network effect scales exponentially with the number of participants. Making the changes copyleft complicates the matter, as their copyright status may become less clear as further downstream contributors chip in.
There's always some trade-off, but this is exactly what the MIT license permits. If it bothers you, you'd probably find yourself preferring copyleft anyway.
What may not be immediately obvious, is that it still favors the downstream developer to contribute their changes back; the network effect scales exponentially with the number of participants. Making the changes copyleft complicates the matter, as their copyright status may become less clear as further downstream contributors chip in.
There's always some trade-off, but this is exactly what the MIT license permits. If it bothers you, you'd probably find yourself preferring copyleft anyway.