Well, no, they're minor elements everywhere. You don't need to be able to capture every phonemic distinction in a language to get a near-perfect transcription, as witnessed by the fact that people understand foreign accents without difficulty. The much larger problem in understanding foreign speech is the odd word choices and lack of grammaticality, but those problems don't arise when you're transcribing native speech.
For some comparisons, think about the fact that Semitic languages are traditionally written without bothering to indicate the vowels, or that while modern English has a phonemic distinction between voiced and unvoiced fricatives, this has a very uneven correspondence to the same distinction as it exists in the writing system. In the case of the interdental fricatives, the writing system does not even contemplate a distinction. And there's nothing particularly problematic about this; if you delete all the voicing information from a stretch of English speech, it stays about as intelligible as it was before. (A voicing difference in stops is not even audible to English speakers. It's audible in fricatives, but no one is going to be confused.)
> For some comparisons, think about the fact that Semitic languages are traditionally written without bothering to indicate the vowels, or that while modern English has a phonemic distinction between voiced and unvoiced fricatives, this has a very uneven correspondence to the same distinction as it exists in the writing system.
And there's a very uneven correspondence between vowels as they exist in speech, and as they exist in the English writing system. Thought dissent mannequin swipe them or bite roar a lie.
You're right that usually, in English, you can understand a sentence with aspiration information stripped out. But just because it's not (usually) significant in English, that doesn't mean that's universal across all languages! Wikipedia has a short lists of languages where aspiration makes a difference. https://en.wikipedia.org/wiki/Aspirated_consonant#Phonemic
> In many languages, such as Armenian, Korean, Lakota, Thai, Indo-Aryan languages, Dravidian languages, Icelandic, Faroese, Ancient Greek, and the varieties of Chinese, tenuis and aspirated consonants are phonemic. Unaspirated consonants like [p˭ s˭] and aspirated consonants like [pʰ ʰp sʰ] are separate phonemes, and words are distinguished by whether they have one or the other.