Supposedly, our left and right ears aren't even equal at birth...each having different pathways to the brain. Then, of course, unequal damage over time. Makes me wonder how much tediously matched drivers matter.
Our brains are pretty good at compensating for the failings of our bodies. For general recreational listening matched pairs are not all that useful since the majority of music is mixed with the knowledge that 99% of people are not going to listen to it on something with an accurate stereo image or even flat frequency response.
I had a sad realization after finally buying the very nice 6XX with a small tube amp - that while it sounds incredible, it also makes my hearing loss/tinnitus more prominent. Probably since I was focusing much more on the intricacies of the sound. Ended up selling them because it just wasn’t an enjoyable experience
We are already asymetrical, adding (inconsistent) asymetry on top of it is not going to improve on it. That's why matching is... something we can do to improve audio. At least it's doable.