When you're small and starting out is definitely not the time to be mucking around on a learning curve.
Learning a new tech in a startup is doing a lot of 'busy' work that is only beneficial to you as you're learning something, it doesn't benefit the business, it slows it down. You're also more likely to make fundamental mistakes in your implementation as you don't know the tech.
And switching when you're running is not as hard as you'd think as you already have the domain knowledge of how the solution actually needs to works.
It's all a balancing act, if the new tech is a fundamental selling point (for example your program's 10x faster than incumbents) I can understand it. If it's to deal with future scalability problems, well, that'll be a good problem to deal with later.