1. First, most people who follow the "only follow your passion" advice WILL fail, so not only will they have lost their "girlfriends, wives, relatives", etc., but they'll feel the dejection of it not being successful.
2. More importantly, and this is really my main point, but I think most people who follow this advice and succeed will find it is a pyrrhic victory. When you've reached what you thought was they pinnacle of your success, but you look around and you see you've burned most of your bridges to get there, it turns out to be a very lonely place.
I think the vast majority of people will find that if they give appropriate attention to their self care and their relationships, it will make achieving their goals easier, not the other way around.