
Rough Estimates of the Dollar Cost (of Memory) of Scaling Web Platforms - Part I - davidw
http://journal.dedasys.com/2010/01/12/rough-estimates-of-the-dollar-cost-of-scaling-web-platforms-part-i
======
profquail
I would've liked to have seen Apache/Mono (mod_mono) in there as well. I've
been trying to make sure my ASP.NET sites are cross-platform just in case I
want to move them over to a Linux server sometime, but I haven't had a chance
to do any load testing on them yet to see how it compares to Windows Server or
any of the other Linux app platforms.

~~~
davidw
If you can get me an easy-to-follow setup/config, I'd be happy to include it.

------
pquerna
This is just wrong.

The Virtual Memory Size (VSZ as he calls it in the article) is not what is
important.

Measuring actual memory preasure and costs on modern Linux is pretty non-
trivial, just look at this wiki page about Spam Assassin:
<http://wiki.apache.org/spamassassin/TopSharedMemoryBug>

He needs to measure the real size of a multi-process daemon like apache by
using a script like the ps_mem.py:
<http://www.pixelbeat.org/scripts/ps_mem.py>

That can measure the real memory use of each daemon, and while I don't think
the overall picture will change much, measuring VSZ is just inaccurate.

~~~
davidw
> while I don't think the overall picture will change much, measuring VSZ is
> just inaccurate.

I agree... which means that your "it's just wrong" comment is inaccurate too,
because the general relationship of the various systems to one another, in
terms of memory usage, is not likely to change much: it's obvious that one way
or another, Rails doing complex dynamic stuff with templates is going to use
more memory than nginx serving static files, with other things falling in
between.

Thank you, in any case, for the pointer to ps_mem.py. I am using a similar
script, but it doesn't look at smaps, so I'll improve the one I have or use
theirs.

------
megamark16
I'd like to see how a fairly straight forward webapp running on different web
app frameworks compare. Lets say you write a simple polling app (similar to
the Django tutorial example) using Django, Rails, ASP.NET/Mono, and Node.js
:-) Write the app to use whatever features you can within the framework but
with matching output, perhaps sans caching, and run them all on similar
architecture where possible (apache/passenger maybe?) then see how those load
tests compare. If nobody else does this than I might, I'd be interested to
know what that looks like.

~~~
skorgu
Yeah, I think this would be a great idea. I'd like to see a set of traffic
generators that exercise actual functionality not just ab/pageload tests too,
that works more of the stack.

It should be reasonably easy to automate a deploy/test run in EC2 too, then
you could do a big batch of comparisons with configuration changes, different
db backends, tuning parameters, etc.

Sort of like the programming language shootout but for webapps.

------
aaronblohowiak
Apples:Oranges. Why not compare Rack or Sinatra to PHP, which would be a much
closer comparison. Or, perhaps try CakePHP or one of the 1000 other php
frameworks out there. Similarly, should be including web.py instead of Django.
Comparing Rails to Jetty is pointless! Compare Rails to Spring/Hibernate.

~~~
skorgu
There are a couple of different fruits being compared (and maybe making this
more plain would avoid this kind of critique):

Apache, Nginx, the static servers are baseline connection costs.

PHP, Jetty/JSP, Mochiweb templated are one tick up, baseline connection plus
the template cost. I'd like to see Tornado included here personally.

Rails and Django are full stacks. It would be nice to see Cake or Symfony
included here and I personally would like to see Compojure as well but that's
much more niche than the others.

Edit: If nobody beats me to it I'll do the date/time template in Tornado and
Symfony this evening.

~~~
davidw
> Edit: If nobody beats me to it I'll do the date/time template in Tornado and
> Symfony this evening.

Thanks, I'm looking forward to seeing it!

~~~
skorgu
Edit: Rather than fighting with escaping code in HN, here are scripts to setup
symfony and tornado examples: <http://gist.github.com/275785>

The symfony one requires some more loving to the apache setup which is well
documented ([http://www.symfony-project.org/getting-
started/1_4/en/05-Web...](http://www.symfony-project.org/getting-
started/1_4/en/05-Web-Server-Configuration)). The script suggests the
frontend_dev.php to make sure things are set up right but for actual
performance benchmarking you'll have to tweak your apache.

~~~
davidw
Ok, I'll have a look. By the way, you have no contact information or anything
in your profile, so I have no way of getting in touch with you.

------
davidw
For everyone saying: "I'd like to see ...", give me some instructions on how
to set it up on Ubuntu, and I'll see about adding them. That is part of the
reason why it's a "Part I".

~~~
houseabsolute
I'd like to see non-3D graphs. ;)

~~~
davidw
Oops, fixed the one that was.

------
sadiq
Jetty's initial heap size can be tuned.

Setting the initial heap to something small (say 16mb or 32mb), while having a
decent max heap, would change those graphs a fair bit.

~~~
davidw
Cool, thanks. I'll collect some of this advice and then update the article.
Part of what made doing this difficult is that some things I know reasonably
well, and others not so well, so I had to trust some of the out of box
settings.

