While Mozart's family was not of the aristocracy, they were of the bourgeoisie, which at the time indeed was a very small part of society – the great majority of Austrians in the 18th century were peasants. I think it is fair to consider his upbringing quite a privileged one.
