The reason why we say "arithmetic mean" when we want to be precise is because "average" can mean any centralizing statistic. The great Carlin no doubt realized the joke would not go off as well with the word "median", and the way he told it wasn't wrong.
Think of how stupid the average person is, and realize half of them are stupider than that.