Malicious relays have the option to give you bogus content or real content. The block-level hashes of Bittorrent prevent anything bad getting through. So just avoid relays which are not giving you good blocks.
At another level, if relaying as honest, but curious (e.g. spying) you need to randomly select several relays and use them all in a long relay path. Pioneered by Tor team, results in excessive bandwidth usage.
But what about the issue of filtering out malicious relays? How do you achieve that without a centralized directory server?