> While linguists specify a matrix of vowel sounds, a lot of people use vowel sounds that land in between them
Definitely, but this is equally true of mat, met, and mate.
As for marry/merry/Mary, there are speakers with a partial merger. Usually they maintain a distinction between "marry" and the other two. That does not discount the fact that there are English speakers who feel (rightly) that the strength of the difference between these three in their own dialect is equal to the strength of the difference between mat, met, and mate in most dialects.
When it comes to mergers more broadly, many speakers who grow up with distinctions that some of their neighbors don't make will wind up falling into the nebulous middle ground you're describing. There's a cognitive burden placed on someone who consistently distinguishes sounds that others in their community do not, because the speaker who makes the distinction will regularly misinterpret what they hear out of the mouths of others. ("Wait, did they just say 'Mary'? Oh, no, it must have been 'marry.'") This is true even when there can be no confusion over homophones: "What does 'fahl' mean? Oh, they must have meant 'fall.'" This is one mechanism that makes it quite easy for a vowel merger to spread. Speakers are conditioned not to pay too much attention to phonological differences that are not a part of the grammar of others in their social circle.
Definitely, but this is equally true of mat, met, and mate.
As for marry/merry/Mary, there are speakers with a partial merger. Usually they maintain a distinction between "marry" and the other two. That does not discount the fact that there are English speakers who feel (rightly) that the strength of the difference between these three in their own dialect is equal to the strength of the difference between mat, met, and mate in most dialects.
When it comes to mergers more broadly, many speakers who grow up with distinctions that some of their neighbors don't make will wind up falling into the nebulous middle ground you're describing. There's a cognitive burden placed on someone who consistently distinguishes sounds that others in their community do not, because the speaker who makes the distinction will regularly misinterpret what they hear out of the mouths of others. ("Wait, did they just say 'Mary'? Oh, no, it must have been 'marry.'") This is true even when there can be no confusion over homophones: "What does 'fahl' mean? Oh, they must have meant 'fall.'" This is one mechanism that makes it quite easy for a vowel merger to spread. Speakers are conditioned not to pay too much attention to phonological differences that are not a part of the grammar of others in their social circle.