You certainly don't need to hate people to not want to go out to dinner with them. Not after 8+ hours of work. And often when there are team/company dinners, they are often an attempt to make up for lots of overtime and/or having to travel.
If after all of that, you would rather actually see your family for a change, or catch up with friends, or get some of your chores done that've been piling up due to work, or relax with some hobbies - then somehow you're painted as the bad guy who hates their coworkers and isn't a team player just because you don't want to spend 24/7 with them?
If after all of that, you would rather actually see your family for a change, or catch up with friends, or get some of your chores done that've been piling up due to work, or relax with some hobbies - then somehow you're painted as the bad guy who hates their coworkers and isn't a team player just because you don't want to spend 24/7 with them?