(a) more recent arrivals don't have as much of whatever quality distinguished the original members
Try to make it obvious who distinguished members are, so people can emulate them. Colour-coded usernames based on age/reputation, starred 'high quality' comments, reputable upvoters (quora-style).
(b) the large size of the group makes people behave worse, because there is more anonymity in a larger group.
Try to reduce anonymity. Tiny little grey names are fine for 100 users, but after a point they all look the same. We're visual animals - maybe we need recognizable avatars, symbolic or otherwise, so we recognize the people we're replying to. Even allowing users to choose a single unicode symbol in addition to their username might be enough.