
The Only Reason My App Worked Was Due to a Slow Database - jv22222
http://justinvincent.com/page/1860/the-only-reason-my-app-worked-was-due-to-a-slow-database
======
fleitz
Shouldn't the child pool simply be reduced and the queue increased which would
have the same effect as the slow MySQL database?

Requests should spend their time queued rather than letting the operating
system thrash between processes, queues are stupid anyway in the context of
webserving, they should use stacks as the best case scenarios are largely the
same, and the worst case scenario for a stack are much better than a queue.
(Yes, stacks are unfair, but it's better that a few people are served than no
one in the event that the server gets bogged down.)

The issue is that the child pool now has too many children and the children
spend their time thrashing, before it was acceptable because the children were
mostly doing nothing. This sounds much more like an apache configuration issue
than anything else. I wouldn't point the blame at 'fast' MySQL, but easy to
misconfigure apache architecture.

Or simply frontend apache with nginx, and limit the number of backend
connections. If you're using a fork/join apache which it sounds like are you,
you'll get much better performance from this anyway. Setup this way apache can
dump the output directly to nginx which can then spend time spoonfeeding it to
the client.

------
spolsky
Press the TURBO button on the front of your PC.

<http://en.wikipedia.org/wiki/Turbo_button>

~~~
waivej
Oh my! I remember the Turbo button. Good times back then.

------
emehrkay
I submitted this a few days ago, but he could use a very simple Object Pool to
handle this. Set up a pool with 3 or so Ajax request objects and make requests
to that pool.

Here is an example where I setup four MooTools Fx.Morph objects. Then I set up
400 divs that whose width will be modified using the next available Fx.Morph
object in the queue. It is pretty simple, powerful stuff. Your objects must be
reusable though so I dont think jQuery's built-in ajax/animations would work
(I use this in "production" with MooTools' Request objects)

<http://jsfiddle.net/XkTPJ/11/>

you simply get an instance of Pool, add your actions, by calling pool.act(fn).
In your callback you'll receive one of the objects in the pool when it is
ready. You use it, and when youre done, put it by by calling this.add(obj)

    
    
        pool = new Pool();
        pool.add(new ObjectInstance());
        pool.act(function(obj){
             //use obj and put it back
            this.add(obj);
        });

------
mattlong
Since you got rid of the MySQL bottleneck, it seems your server architecture
now simply isn't able to handle higher concurrency. If you suddenly got 5-10x
the number of users, you'd have the same problems as before adding your AJAX
queueing, right? That's, of course, assuming nothing else "breaks" first due
to the increased traffic.

------
pharrington
a legitimate concern from someone who's hellbanned:

No offense intended here... but it sounds like you could easily become the
victim of a DoS attack. All that's needed is for someone to create their own
front end and you'll be back to normal. Have you investigated some sort of
solution on the server side, and not the client side?

~~~
tedivm
It isn't nearly that simple- they'd have to create the custom front end and
get _everyone_ to use it. I doubt one person constantly hitting the site would
have any issue, it's the combination of all of his users doing it that caused
the problem.

~~~
dmishe
> it's the combination of all of his users doing it that caused the problem.

well that's the point of ddos attack

~~~
Ineffable
Yeah, but if you want to do a DDOS attack you can just use one of the already
configured tools out there that will bombard the server with as many requests
as possible. A few chained ajax requests are never going to replicate that
sort of load.

------
jbert
I guess your next step should be to profile the apache reqs?

Unless you know that you're doing CPU-heavy work in your apache procs (and
it's unavoidable or already optimised), it seems that you could get back the
responsiveness of your site?

It seems surprising that you're CPU-limited on the web head.

------
chrisacky
I've had a vaguely similar occurance once that took forever to track down.

When I would persist a particular model back to mysql, I would fire off a job
in Gearman, that would then grab the saved object and asyncronously send it
off to lots of other sources.

After moving servers it would always send the previous state of the object.
Having previously worked flawlessly.

Eventually I trackked it down to the fact that my Gearman job was being kicked
off before the actual write to MySQL, so then when I did the select in my job
process, it would grab the previous snapshot.

Pretty easy fix once I figured that out.

------
Moto7451
I once had a similar issue with a Silverlight program I wrote. In my case I
ended up adding worker threads to handle data processing with an event to
signal when it was done processing and followed up that tweak by consolidating
my SOAP requests as much as I could (ironically making it even faster helped
fix it). Timing issues can be rather tricky sometimes but the upshot is that
if your going to have a problem, running too quickly isn't a bad one to have
to deal with.

------
josegonzalez
When I load a dynamic page with content that isn't ready, I push the missing
content requests it into an array. When the page is done loading, it polls the
server with the data it needs. Server replies with data and client keeps
polling every second or so for more stuff until it's done.

Update: I will never be without work.

~~~
bri3d
I second this type of approach - when I developed an iOS app that needed to
display the results of multiple long-running API requests made on the server-
side, I had the app make a request and then wait for data to be pushed to the
device on a persistent "back-channel" connection the app kept open. This way
data could be displayed as it was available, but the app didn't need to poll
or open multiple TCP connections (expensive on EDGE, which was common at the
time).

------
tbrownaw
So... what, the client had a polling loop with no delay other than the time to
actually make the ajax request?

~~~
jv22222
The client makes a lot of ajax requests since it's so ajax heavy and there is
no throttling on the concurrency. When 5-10 requests hit the server from the
same client at the same time it is a "little" bit like a dos attack.

~~~
jv22222
@jonknee - It's true there could be less. That's a future optimization. Here's
the type of requests that are happening...

ajax/get-accounts

ajax/get-streams

ajax/get-other-settings

ajax/store-client-view

ajax/get-tweets

ajax/sync-relationships

~~~
mapgrep
Any idea why that's pegging the CPU? My naive analysis would be that once the
database has sent/stored the data, the server is just doing some light
filtering/serialization and sending it off to the client. But obviously
there's more going on if these requests (en masse) can get you to 100 load.

Shot in the dark: Is there some kind of sorting going on?

------
iamabanana
Why are you not using any caching for these read requests?

~~~
jv22222
Nothing to cache. Each response is unique. Well 99.9% anyways.

------
BerislavLopac
Which only proves my old adage that Web wasn't designed for high-speed
Internet. We need something new.

------
AznHisoka
I didn't really understand why from the article, but that's pretty cool
nonetheless!

~~~
jemka
Essentially, mySQL was the bottleneck. He fixed that, then Apache became the
bottleneck. He just didn't realize mySQL was his bottleneck (or that fixing
that would increase the load on the webserver) and therefore "blames" the
functioning of the app on mySQL's slowness. Now, whether or not there are
changes to Apache he could make to avoid having to queue ajax requests is not
something I can speculate on.

