You think incorrectly and having that assumption, you're exposing yourself to more risk. To assume Linux USB drivers, file systems and file managers don't contain vulnerabilities is to assume wrong.
Sure, there is also the potential threat of someone breaking into my home/office and installing secret videocameras to record my keyboard and get my truecrypt password (or more easily, opening the "safe" and figuring out what string of characters in a certain notebook is it). However, I don't worry about such things, and neither would I worry connecting a random usb to my practically disposable netbook.
