The reason why we say "arithmetic mean" when we want to be precise is because "average" can mean any centralizing statistic. The great Carlin no doubt realized the joke would not go off as well with the word "median", and the way he told it wasn't wrong.
