Sometimes it refers to statements like "commit early, commit often" or "move fast and break things"; that's fine. Sometimes it refers to office interaction, such as whether you tolerate interruptions and social interaction: an introvert in an office full of extroverts or an extrovert in an office full of introverts; that's fine too, though at some point you'll scale large enough that you can't necessarily keep being that selective.
On the other hand, sometimes it means "are you going to play foosball with us and drink with us", which is wrong. And sometimes it means "do you have exactly the same personality/likes/dislikes/ethnicity/gender/etc as us", which is wrong and illegal.
In general, "cultural fit" is a dangerously vague excuse without a concrete reason. Say what you really mean, and if you can't bring yourself to say it, perhaps you shouldn't be using it as a hiring criteria.