"Premature optimization" has earned a negative reputation because it has a tendency to inflate dev schedules unnecessarily.
So I tend to just crank something, anything out just to have something. Once you can see what you have, it's often a lot easier to modify than come up with in the first place.
OTOH, I like to think that everything I build is a foundation for the next thing to be built upon it. I am constantly getting bitten in the ass by some grossly underperforming building block. If a prototype runs poorly a dozen times, it's a concern. If the same code runs poorly a million times, it's a disaster.
It's a constant trade-off. Get something running vs. build solid building blocks. Make a mistake one way and never release. Make a mistake the other way and have a time bomb to clean up. Sounds like OP has a lot of the same issues.