Yeah, I'm afraid you're definitely wrong on this. I've used Modafinil extensively at several different doses. Even at a quarter pill (50mg) you will struggle to feel even remotely tired if you take it after 9am (assuming you go to bed at a normal-ish time).
You should probably be saying "I" instead of "you". Anecdotally, I've never had trouble going to sleep even a couple hours after having modafinil. If anything, it's a more peaceful transition to sleep too. Modafinil makes me feel relaxed and refreshed. Like going to bed after taking a shower, or being in a hot tub or something.