If you're successful you will say "Wow. I almost listened to him" like you dodged a bullet. if you fail, you'll have a much better chance of success next attempt.
I actually don't think that - I think he's generally a decent guy who cares about his employees, but cares about his company more. He's even said something to that effect: part of his advice to me was to make sure I can setup a corporate structure that'll survive the departure of founders or key employees, because things do go wrong, and the company is his dream. Perhaps there's a lesson in there: don't let your identity get so wrapped-up in your company that it becomes only your company, one where other people don't feel like they have a chance to make a difference in it. Come to think of it, I've heard similar things from other (failed) entrepreneurs.
You're right though: I guess I just need to go with my gut. Eh, I'm young, I've got plenty of time for failures before success.