Newton 1: to accelerate in space, you have to (continually) eject mass in the opposite direction.
Tsiolkovsky: In space, you have to carry the mass you eject with you, so you need to eject more mass to accelerate the mass you're going to eject later. So you need to carry that too. And don't forget the mass you'll eject slowing down at your destination!
Newton again: Perhaps you could cut down the mass you need to eject by accelerating it to really high speeds.
Newton 2: The power required to accelerate the ejected mass goes up with the square of its final velocity. But your acceleration only goes up linearly with exhaust velocity. Tough.
Special Relativity: Ahem, actually sub-linearly. And there's a hard limit, which is pretty low.
Thermodynamics: your mass ejection system has to generate power to accelerate the ejected mass. Some of that power will be lost to the rest of your spacecraft because nothing can be 100% efficient. Your spacecraft will melt if you try to go too fast.
Interjection by Cosmology: hey, don't forget everything is a really long way apart (Special Relativity is really limiting that way), and there's invisible nasty stuff in between the places you want to go.
Wannabe interstellar travelers: Hmm. We'll just build big telescopes.