
Ask HN: Back end architecture design for site that aggregates data across APIs - alcaras
Hi HN!<p>I run a little statistics website for World of Warcraft: subcreation.net (source at https:&#x2F;&#x2F;github.com&#x2F;alcaras&#x2F;mplus.subcreation.net)<p>I&#x27;m working on updating this site and am wondering how I should re-architect my backend to to minimize costs.<p>Current architecture is:
- Some script queries APIs -- writes data to DB. Ideally rate limited &#x2F; queues so I am a good citizen an obey api limits. (I am using App Engine Task Queues but these are of course deprecated and not available for Python 3 ... and Cloud Tasks are more work to set up).
- Then some script that, when data gathering is complete, generates HTML &#x2F; JSON files and writes it to somewhere the web server can read it.
- Users visit and view HTML pages.<p>Current set-up:
Google App Engine runs tasks to query APIs, write to Datastore, and then output HTML to Cloud Storage. Strangely,  egress costs are the highest expense -- on order of $20&#x2F;month for something like 300 GB outbound (~30k MAU), which seems egregiously high(!) to me.<p>Not sure as to NoSQL vs SQL -- I seem to be dealing with reasonably structured data but originally opted for DataStore over CloudSQL since DataStore was cheaper.<p>Wondering how to do this more cost effectively.<p>1. App Engine + Cloud Firestore + Cloud Storage<p>This is effectively updating the current architecture, but moving to Python3 and Cloud Firestore. Could write static json to Cloud Storage to enable fancy filtering as well.<p>2. Shared VPS + Python + MongoD &#x2F; Sqlite<p>Not crazy about dealing with server admin, but this seems cheaper, since VPSes seem to have much cheaper egress.<p>3. AWS? Azure? Other options?<p>Appreciate any thoughts &#x2F; ideas :) I am currently leaning toward option #1, but am wondering if exploring #2 would be better from a cost-basis, since my main user facing interaction is static HTML pages (or, if I make the site more interactive) HTML pages with AJAX to static json files.
======
jimsmart
Re option #2, we have found Linode's smallest offering, Nanode, to be
remarkably useful for a bunch of tasks that sound similar to yours (and indeed
some bigger tasks). Only $5/month, which includes 1TB of transfer, and easy
enough to switch to bigger plans if needed.

