The downside? As user tastyface said in the comments here, "As a life-long loner, travel made my social life a whole lot worse. I barely talked to anyone during my travels..."
It hasn't made my social life worse per se, as there were ups and downs even when I was living in one place in the US, but it's amazing how long one can go being invisible (the effect is probably made worse by already knowing how to speak the local language). In actuality, I would say the invisible factor has more to do with modern city-living than being a DN. If one just sticks to the minimum, like answering "do you want paper or plastic?", "would you like something to eat with your drink?", etc etc, then it's totally possible to be invisible, so to speak. After a few days of that, maximum one week, I'm itching to get in contact with my friends or do something social.
A slightly more concrete downside is making lasting friends, especially if you tend to go for expat circles. Everyone eventually moves on, despite the time and energy placed into making those friendships. As for local friends, no matter what city or country I'm in, I hear the same thing: it's hard to make real friendships with the locals. As a remote worker, I usually make a small number of real friends (expats who, as mentioned, eventually move on) and a healthy sized number of 'have a beer' friends (both expats and locals).