.. so, you measured the performence (sluggish), saw the need for improvement and improved it (snappy). It is not premature optimization. It would be premature optimization if it happend without mesaurement and without need.
I agree with your examples above. If you choose the right data strcuture/algorithms/patterns without sacrafising readability or development speed. By all means. But don't spend hours to improve something which dosn't need improvement.
Beyond that, measure before you optimize, as such interventions will require larger amount of effort, so it makes sense to do them in the order of highest-impact first.
(Also note that "performance", while usually synonymous to "execution speed", is really about overall resource management. It's worth keeping memory in mind too, in particular, and power usage if your application could be used on portable devices. Which is really most webapps nowadays.)