I completely agree; it's very oversimplified. Utility is not a scalar, and to the extent averagism is correct, the "average" is not a literal arithmetic mean.
As just one counterexample, I think there is some moral or at least aesthetic value in a breadth of life experiences existing. If the aforementioned one person or one million people were the only people in the universe, the million people would have a much wider breadth of experience, leading to a more interesting society. On the other hand, if we assume a total population of billions, I'd say the significance of this factor dwindles to near nil.
Also, I believe utility maximization is only one of many moral precepts in the first place.
As just one counterexample, I think there is some moral or at least aesthetic value in a breadth of life experiences existing. If the aforementioned one person or one million people were the only people in the universe, the million people would have a much wider breadth of experience, leading to a more interesting society. On the other hand, if we assume a total population of billions, I'd say the significance of this factor dwindles to near nil.
Also, I believe utility maximization is only one of many moral precepts in the first place.