We don't know how much of this stuff exists in the interstellar space. It's possible that a probability to hit something there is much less than ISS hitting something big enough to destroy it, even with all radars and maneuvers. So I wouldn't consider it as an argument against interstellar travel, not yet.
I'd say it's one of many potential problems. We've already observed a couple interstellar comets, right? So we actually already know that there are objects out there. Certainly it wouldn't hurt to send some laser accelerated probes out there and see what we find, but human travel seems like pure sci fi.
