> What kind of speedup is available for big Rails applications? If 90% of the time in an application is spent in database calls, then there’s little opportunity for improvement via JIT technologies.
This is written as speculation of course, but it matches some long-time "conventional wisdom", that most Rails apps spend most of their time on I/O waiting for something external.
That may have been true once, but i don't believe it's true anymore.
My Rails apps now, when things like "n+1 database query problems" are properly eliminated, all spend more of their time on CPU than I/O, 60-80%. (I think most of that CPU time is ActionView rendering. Unsure about CPU parts of ActiveRecord, that aren't waiting on DB, but converting db results to in-memory AR objects etc).
When this comes up and I ask around on social media who else has actually looked at their app and can confirm or deny, almost all who respond agree. I would be curious to see here too.
Definitely some more systematic rather than anecdotal investigation is called for. But I think it's time to stop repeating that it's likely that a Rails app is I/O-bound rather than CPU-bound, my experience leads me to think the reverse is currently likely.
[*edit* after I wrote this comment, I noticed multiple other comments in this thread saying basically the same thing. OK, so. Can we put this rumor to rest?]
> I think it's time to stop repeating that it's likely that a Rails app is I/O-bound rather than CPU-bound, my experience leads me to think the reverse is currently likely.
I'd like to offer an anecdotal counterpoint. I worked at Shopify for several years. The big problems were always due to I/O. Usually that was in the form of long-running network calls. You're calling the FedEx API for shipping rates in the cart. All of a sudden FedEx starts taking 5 seconds instead of 1 second, and all of your worker capacity evaporates. That kind of thing.
Or, if databases are more your speed, we faced another problem with the GraphQL API needing to rattle off a bunch of requests to the DB, without being able to do so in parallel. So the more different slices of data you needed to fetch, the longer the single GraphQL call would take.
We spent significant amounts of time finding ways to work around Rail's inability to do these sorts of things in parallel (or at least, the inability to do it without lots and lots of Spell-o-Tape). Writing proxy services in other languages that could parallelize an incoming request and fan it out to the actual Ruby boxes, etc.
I like Ruby but I feel like it's pretty unequipped to deal with the problem of long-running I/O and worker saturation. There's work happening there but it will be a long time before you have the same ease-of-use as languages with better concurrency support out of the box.
You could argue that these sorts of long-running calls are not the norm, but in my experience it's way more common for a web app to need to perform some call to an external service at some point during its lifetime than not, and that's not a use case that Ruby/Rails are built to elegantly handle.
Thanks for the anecdotal counterpoint, that is what I was looking for!
I think we're talking about different things though.
I think OP was suggesting/questioning whether a typical Rails apps may spend 90% of it's overall time in I/O.
I am suggesting the reverse, that typical Rails apps (so long as they don't have bugs where they fetch more than they need) will spend spend 60-80% of their time in CPU rather than I/O.
I am curious if this is true of shoppify's app; my guess is it still would be (especially cause I think shopify will not have very many bugs where it's fetching things inefficiently or that it doesn't need to fetch, shopify will be higher-quality than a random rails app).
You are suggesting, I think, that at shopify atypical, problem requests were always due to I/O.
I think all of this can be simultaneously true.
I think a side issue to what I'm talking about, but:
> We spent significant amounts of time finding ways to work around Rail's inability to do these sorts of things in parallel
Rails 7 actually recently introduced a feature to do multiple ActiveRecord fetches concurrently... perhaps motivated by/developed by Shopify? https://pawelurbanek.com/rails-load-async
I haven't used that one yet. In the past when I've needed to do things like external API requests in parallel, I have had success just using futures/promises from Ruby::Concurrent, without even telling Rails about it. I've had to do this only rarely, but I haven't run into too much trouble. But one thing you can run into if your threads try to store things in ActiveRecord themselves, is running out of database connections with Rails "long connection checkout" model, involving explicit handling of pool checkouts once you do your own threads. Sequel (vs ActiveRecord) has a much more connection-efficient way of pooling database connections, where they are transparently checked out of the pool, under the hood without you usually needing to manage it explicitly, for only as long as it takes to execute a single statement. When doing heavily concurrent things with AR, I have often wished it used Sequel's concurrency/connection-pooling model.
With Ruby::Concurrent, I found similar weird-isms until I wrapped in `Rails.application.executor`. Rails has its quirks and purists can be put off by all the "extra work" that needs to happen for things to play nicely together.
To that I say: Rails gives me superpowers and I need to abuse them appropriately.
Totally -- I've been trying to do multi-threaded concurrency like this since before Rails actually had `Rails.application.executor` -- it was possible, there were some gotchas. It is very helpful to now have API available at that level of abstraction. It can still get weird sometimes. There are still some gotchas. Usually about ActiveRecord.
With what we know now from seeing how it all works out, if we were able to do it over again greenfield, I think it's pretty clear that sequel's concurrency/pooling model is preferable to ActiveRecord's. Although I'm sure it would have it's own downsides I'd run into if I used it more and at scale.
You're right, I think we're using IO/CPU bound in different senses. You're talking about where an average request spends most of its time; I was thinking more in terms of Shopify's scaling bottlenecks. Even though individual requests probably spent a lot of their time in CPU, we didn't really have to worry about CPU as a bottleneck for the larger application. Whereas IO was a different matter, and we had a lot of difficulties ensuring that we had enough capacity for spikes in the IO-heavy stuff. So the overarching Shopify application was "IO bound" in that sense.
I think it also depends on architecture and what the specific endpoint does. Let's say it's a microservice architecture with lots of network calls (to fetch data from different services or even outside API calls) ...how will CPU be 80% of the time? Doesn't sound reasonable.
But if there's not much calls at all and it's rendering one huge view then yeah maybe it's mostly CPU...
> Usually that was in the form of long-running network calls.
That's why doing a network call to another service in the middle of a request is pretty much banned from the monolith. Anything that doesn't have a strict SLO is done from background jobs that can take longer, be retried etc.
Now you mention FedEx so I presume you were working on the shipping service, which by essence is kind of an API bridge so it's probably why it was deemed acceptable there, but that's far cry from what a typical Rails app look like, except maybe in companies that do a lot of micro-services and are forced to do inline requests.
> that's not a use case that Ruby/Rails are built to elegantly handle.
I'd argue the contrary, the elegant way to handle this is to perform these calls from background jobs. Look at shipit [0] for instance, it's syncing with the GitHub API constantly but doesn't do a single API call from inside a request cycle, every single API call is handled asynchronously from background jobs.
I was actually referring to background job workers in my comment. Queuing is still queuing, whether its request queuing due to a shortage of web workers or job queuing due to a shortage of job workers.
Yes, there are more levers one can pull for job workers, and it's probably easier to horizontally scale those workers than web workers for various reasons. But regardless of which workers are performing the long-running I/O, there's still a hard bottleneck imposed by the number of available workers. They're still going to inefficiently block while waiting for the I/O to complete. The bottleneck hasn't been truly eliminated; it's just been punted somewhere else in the application architecture where it can be better mitigated.
Background jobs may be the most elegant solution for handling long-running I/O in a typical Rails app, but that's still less elegant than simply performing those requests inline and not having to worry about all the additional moving parts that the jobs entail.
> that's still less elegant than simply performing those requests inline and not having to worry about all the additional moving parts that the jobs entail.
I strongly disagree here. Going through a persisted queue gives you lot of tools to manage that queuing.
If you were to just spawn a goroutine or some similar async construct you lose persistence, lots of control on retries, resiliency by isolation etc. When you have "in process jobs" re-deploying the service become a nightmare as it becomes extremely muddy how long a request can legitimately take.
Whereas if you properly defer these slow IOs to a queue, and only allow fast transactional request, you can then have a very strict request timeout which is really key for reliability of the service.
Those are all fair points. My only counterargument is that for a certain class of requests, the simplicity of not needing to worry about a separate background jobs queue outweighs the benefits that the job queue provides. There's some fuzzy line where those benefits become worth it. And you're probably going to cross that line earlier with a Rails app than with an evented one. There are lots of cases in Rails where problems are solved via background jobs that would most likely just stay in the parent web request in an IO-friendlier environment.
Rails or not, in any platform/framework, if you do an API request inline in a request that could take max time N, then the total request time could take max time N+m, so if you don't want requests to take >N, you don't do requests inline.
What am I missing, how does Rails make this especially bad?
Or is it that in another platform/framework, it's easier to allow requests to take long N+m to return if you want? True that would be easier in, say, an evented environment (like most/all JS back-ends), but... you still don't want your user-facing requests taking 5 or 10 seconds to return a response to the user do you? In what non-Rails circumstances would you do long-running I/O inline in a web request/response?
> Or is it that in another platform/framework, it's easier to allow requests to take long N+m to return if you want? True that would be easier in, say, an evented environment (like most/all JS back-ends), but... you still don't want your user-facing requests taking 5 or 10 seconds to return a response to the user do you? In what non-Rails circumstances would you do long-running I/O inline in a web request/response?
Yeah, that's what I'm getting at. It's true that even in evented backends there's a line beyond which it's probably better to put the long-running stuff in a background queue, but it's a higher bar than in Rails. I've run pretty high-throughput Node and Go apps that had to do a lot of 1-5s requests to external hosts (p95's probably up to 10s) and they didn't really have any issues. In my opinion, it wouldn't have been worth it to add a separate background queue; the frontline servers were able to handle that load just fine without the additional indirection.
byroot makes good points in a sibling comment about retries and more explicit queue control being advantages of a job queue pattern regardless of whether you're evented or not. I just think that those advantages have a higher "worth it" bar to clear in an evented runtime in order to justify their overhead (vs Rails).
I'm curious if you could share more on the GraphQL part of the situation you describe here, and if being architected on your more traditional "RESTful" (however you want to define that, but usually Rails defaults get you a wonderful implementation of "RESTful") API convention would have made any difference in the performance issues you encountered at Shopify.
We didn't have the same problem with the old REST API, because the requests were naturally sliced by whatever resource type people were requesting. Whereas the GraphQL API allowed clients to request big slices of disparate data all in the same request. If we had (for whatever reason) exposed a REST API that fetched lots of different slices of data and combined them like GraphQL does, those requests would have faced the same problem.
So it was really due to the nature of GraphQL that the problem materialized, rather than being a GraphQL vs REST issue per se.
The whole I/O is the bottleneck thing is the same as saying “memory is cheap”.
It’s true in a way, but that doesn’t mean we should all ignore it. It’s usually not a big problem, until it is.
I love Ruby, but I love crystal in terms of performance, except for compilation times.
My preference at this point is rail's in api mode with a separate javascript fronted. I haven't had had to scale enough to really see it, but I often see rendering discussed as CPU heavy and I do wonder how much it helps that all my rendering is the OJ gem.
If you go this route you end up having a drastically more complicated front end, and usually without the speed benefit. Unless you really need a SPA I would stay away.
> This is written as speculation of course, but it matches some long-time "conventional wisdom", that most Rails apps spend most of their time on I/O waiting for something external.
> Can we put this rumor to rest?
You have only given one example though. What rumor is there to put to rest? I think the statement is still accurate: "most Rails apps spend most of their time on I/O waiting for something external"
To your specific concerns though - I have also see ActionView and ActiveRecord casting take up non-trivial amounts of time (I recall a situation in a past job where the mysql driver being notoriously bad with allocations of enums - probably our own doing however).
For Rails specifically, I can't say enough good things about AppSignal. Being Rails specific gives it really great performance analytics right out of the box (it hooks into Rails standard instrumentation), and it handles error reporting for you as a bonus.
Of course mileage varies. I think it is fair to say that it is still the case that for many rails apps most of the request/response time is spent waiting on I/O. The DB I/O performance can be greatly improved by minimizing number of queries, indexing, query optimizations, ETC. DB I/O performance bottleneck is not a problem unique to rails, but maybe felt by rails devs more because the abstraction ActiveRecord provides doesn't make it obvious when you're misusing the DB.
Views are definitely one of the places that can get slow. Each partial is an IO read and they happen sequentially. Looping and rendering a partial is a common mistake that kills performance.
For example using the data from the db (model instance) as cache key is quite effective solution for being able to deliver most view/fragments from cache.
The downside is that some care must be taken with keys, which parts are cached (eg: logged in pages).
This is written as speculation of course, but it matches some long-time "conventional wisdom", that most Rails apps spend most of their time on I/O waiting for something external.
That may have been true once, but i don't believe it's true anymore.
My Rails apps now, when things like "n+1 database query problems" are properly eliminated, all spend more of their time on CPU than I/O, 60-80%. (I think most of that CPU time is ActionView rendering. Unsure about CPU parts of ActiveRecord, that aren't waiting on DB, but converting db results to in-memory AR objects etc).
When this comes up and I ask around on social media who else has actually looked at their app and can confirm or deny, almost all who respond agree. I would be curious to see here too.
Definitely some more systematic rather than anecdotal investigation is called for. But I think it's time to stop repeating that it's likely that a Rails app is I/O-bound rather than CPU-bound, my experience leads me to think the reverse is currently likely.
[*edit* after I wrote this comment, I noticed multiple other comments in this thread saying basically the same thing. OK, so. Can we put this rumor to rest?]