When I get half a million users I’ll start worrying about it. Most startups fold way beyond this point. Saving dev time and the pleasure and ease of working with Ruby more than make up for the speed difference.
It will scale well beyond that point if ut is properly organized into services.
Check out some of the comments above about "service oriented monolith".
That being said, know your limits. If you are doing a lot of image processing for instance it may be worth investing in an external service in a language that is a better fit for that problem.
