If we ever achieve some form of reasonable, in-system spaceflight, then we are really going to have to start worrying about every crazy person with a spaceship bumping a smallish space rock into an impact orbit with earth. Or even just unfortunate accidents. But it wouldn't take much of an asteroid to burn down through the atmosphere and create a Tunguska-level impact.
The most recent Expanse novel is an interesting take on how apocalyptic such an event could be http://amzn.to/2aD2AFl
Sure, but intrasystem travel will only be economical if we can mine and get fuel, energy, and mass from non-Earth sources.
If we get to that point, scaling up just consists of waiting a few decades of exponential growth to kick in. It might not even involve humans, but just letting automated robots continue to build more robots.
If the technology is there, moving mountains instead of ships is just a small jump.
The most recent Expanse novel is an interesting take on how apocalyptic such an event could be http://amzn.to/2aD2AFl