I'm not sure I always clearly distinguish between the three of them. My "merry" is most certainly not normally the same vowel as "met" (it is a hybrid that vowel sound and some other one). Neither is my "Mary" reliably the same sound as "mate". I think my "marry" normally comes closest to the vowel sound of "mat", but I'm pretty sure it generally is not all the way there. (Although it can be, especially if I am trying to enunciate clearly). Over all I'm pretty sure I have at least two and possibly three disguisable vowels (with Merry and Mary probably the merged ones if it is only two) among pronunciation of all three. If I have three vowel sounds they they are allowed to vary enough that there is overlap between the pronunciations I will produce.
That is the awkward thing with vowel sounds. While linguists specify a matrix of vowel sounds, a lot of people use vowel sounds that land in between them, and the sounds are not just exactly the same every time, but have some level of variation between them.
And of course, thing probably differ with word stress, as vowel sounds often do in English. Not sure how they vary though.
> While linguists specify a matrix of vowel sounds, a lot of people use vowel sounds that land in between them
Definitely, but this is equally true of mat, met, and mate.
As for marry/merry/Mary, there are speakers with a partial merger. Usually they maintain a distinction between "marry" and the other two. That does not discount the fact that there are English speakers who feel (rightly) that the strength of the difference between these three in their own dialect is equal to the strength of the difference between mat, met, and mate in most dialects.
When it comes to mergers more broadly, many speakers who grow up with distinctions that some of their neighbors don't make will wind up falling into the nebulous middle ground you're describing. There's a cognitive burden placed on someone who consistently distinguishes sounds that others in their community do not, because the speaker who makes the distinction will regularly misinterpret what they hear out of the mouths of others. ("Wait, did they just say 'Mary'? Oh, no, it must have been 'marry.'") This is true even when there can be no confusion over homophones: "What does 'fahl' mean? Oh, they must have meant 'fall.'" This is one mechanism that makes it quite easy for a vowel merger to spread. Speakers are conditioned not to pay too much attention to phonological differences that are not a part of the grammar of others in their social circle.
That is the awkward thing with vowel sounds. While linguists specify a matrix of vowel sounds, a lot of people use vowel sounds that land in between them, and the sounds are not just exactly the same every time, but have some level of variation between them.
And of course, thing probably differ with word stress, as vowel sounds often do in English. Not sure how they vary though.