Both predate computers as shorthand spellings of words and phrases. @ for example dates to 1500s Italy. # is often used to denote a number is an ordinal number, or as a substitution for the word pounds (weight, not currency).
As you see, these are symbols that can be replaced with a textual description of what they mean - being something like "macros" for text. So as i wrote, we are already using symbols that aren't text and we do enter some of them via dedicated keys on the keyboard.
The main issue here is to see these symbols for what they are: an arbitrary set of symbols that we limit ourselves to because they are printed on the keycaps of the keyboards we have.
But we don't have to only use symbols found on keycaps, we don't have to use keyboards that essentially inherit the same layout and most of the keys from centuries old typewriters, we don't have to limit ourselves to a handful of letters and symbols chosen decades ago for devices that were already primitive in 1980.
The by far main reason we do all that isn't because the current system is better but because of what is essentially software peer pressure.
No, we dont because non-keyboard symbols are irritating to type. Look at the history of APL and other symbol based languages.
In any case, the computer doesn't care. It is either a character or something similar stored as binary representing part of an instruction to be compiled into other binary.
I have seen a few proposals for creating languages which are exclusively stored as a generic AST, sufficiently abstract that it can be edited in any variety of "languages" in a capable editor, then stored back into the AST. I havent seen a good proposal for making that actually workable, however. It sounds a bit like storing code as the output from LLVM then expecting an editor to be able to open it as C# or JavaScript or Lisp.
Their wikipedia pages give several modern day usages, including those I listed and the sibling comment (@ meaning at the price of, # as shorthand for pounds, etc)
