I feel like focusing on the asymmetry of it is a large part of the problem. If you're not walking into the interview prepared to disqualify these people as potential employers, that is doing a disservice to yourself.
But the interview process is such a time suck (not to mention that you have to nick out on your regular job which you can only do so many times for a half a day before they get wise) that by the time you are walking into an office for an interview, you've already made a non-trivial commitment. You can only repeat it a handful of times (assuming you have another job or are otherwise professionally engaged) before it becomes a real burden.
Now, if you are unemployed and you have a substantial financial cushion or can pick up enough 'anytime/anyplace' freelance work, sure, interview with 5 companies a week, take your time and be picky. But how many job seekers are in such an ideal position?