I sometimes sigh at poorly performing apps. I've been sighing a lot at the feedly app lately. But that doesn't stop me using it, because the functionality is compelling enough.
Putting effort into performance is probably something you'll want to do at some stage if your app gets big enough. But a lot of the time you have more important things to do right now. Users will put up with an awful lot if you've got something they really want; get that part down first.
But if you do not take care the "more important things" will have to be refactored or even rewritten allo over because you build them on a poorly performing base.
Finding the bottlenecks should always be a priority. Fixing them can be done at a later point.
Putting effort into performance is probably something you'll want to do at some stage if your app gets big enough. But a lot of the time you have more important things to do right now. Users will put up with an awful lot if you've got something they really want; get that part down first.