The difference between an icon and a letter is small and unclear. & is a symbol but was considered a letter as an example. Chinese characters are words etc.
Good point. Letters.. punctuation.. symbol .. the lines are blurry. If I may I'd say that & is a symbol that represent a grammar connective. Which is a generic abstraction and won't cause explosion like having symbols for every word out there.