
Infinitely Scalable Framework with Amazon Web Services? - mattjaynes
http://blog.nanobeepers.com/2007/04/07/infinitely-scalable-framework-with-aws/
======
mattculbreth
This is pretty interesting. I'm not sure I understand what you mean though
about separating databases "by user". Could you elaborate on that? How would
my application's database change if I were a user of this service?

~~~
mattjaynes
Sure. Basically each user would have their own database file generated on
account creation. Then in your app, instead of doing something like:

connectDB("/path/to/db")

you would do:

connectDB("/path/to/$user/db")

Also, if you needed to query all of your users data in aggregate, you would
use the 'ATTACH' capability of SQLite to essentially create a master database.
Note that you will need to design your scheme appropriately (thoughtful
table/column naming, etc) for this to work well. See:
<http://www.sqlite.org/lang_attach.html>

------
blader
From what I've read of evaluations of S3, the latency of S3 requests seems
significant. Significant enough that using S3 as a backend database doesn't
seem feasible. Is this true or am I mistaken?

~~~
blader
I forgot to mention that I think this is fantastic idea. I'm just not sure if
it will be able to replace a local mysql cluster.

~~~
mattjaynes
Thanks, AWS is amazing and has been super fun to play with. This solution will
of course only work for a certain subset of apps. Apps that require more
database complexity will not be a good fit for this framework. My apps are
typically designed so simply that they could just use a flat-file storage
system instead of a db. SQLite is awesome because it provides the flat-file
storage, but in a format that allows you to query it like a normal database.
The simplicity of that model is amazing and hard for me to resist. I've used
mysql clusters in most of the projects while working for companies, but I
often found that the added complexity did little for us. My last company hired
two database guys with fat salaries just to manage it. My goal with this
framework is to really reduce the cost of maintenance, backups, etc. S3 is
distributed and so really doesn't need much in the way of backups - but
backing up this system is a simple as tar gziping the directory of flat-file
databases.

This was interesting from pg's Viaweb FAQ:

<http://www.paulgraham.com/vwfaq.html>

"What database did you use?

We didn't use one. We just stored everything in files. The Unix file system is
pretty good at not losing your data, especially if you put the files on a
Netapp.

It is a common mistake to think of Web-based apps as interfaces to databases.
Desktop apps aren't just interfaces to databases; why should Web-based apps be
any different? The hard part is not where you store the data, but what the
software does.

While we were doing Viaweb, we took a good deal of heat from pseudo-technical
people like VCs and industry analysts for not using a database-- and for using
cheap Intel boxes running FreeBSD as servers. But when we were getting bought
by Yahoo, we found that they also just stored everything in files-- and all
their servers were also cheap Intel boxes running FreeBSD.

(During the Bubble, Oracle used to run ads saying that Yahoo ran on Oracle
software. I found this hard to believe, so I asked around. It turned out the
Yahoo accounting department used Oracle.)"

------
BitGeek
The terms1 of S3DFS don't work for me so I suggest people look at MogileFS. I
just learned about it recently, so I can't give a review... but it looks
interesting.

<http://www.danga.com/mogilefs/>

1 Will pay good money for software, their prices are ok, just don't like the
feeling I get that they will have my company by the balls.

------
mattjaynes
Added this in the hopes of getting some good feedback from the community on
this framework I'm developing.

The post is derived from this comment:
<http://news.ycombinator.com/comments?id=9991>

------
enomaly
You may want to take a look at <http://www.enomalism.com,> it provides
migration to and from Amazon EC2 as well as the ability to create your own
geo-targeted cloud.

