
How do i choose architecture for my scraping web app? - tnsaturday
I have written a bunch of simple Python scripts that parse and scrape information on the web and store it in JSON files, so that I can access that data and work with it. Now I want to build a web application on top of my project. Basic use case would be:<p><pre><code>    User inputs his search request in html form.
    User gets response without reloading the page.
</code></pre>
Seems fairly simple, however a couple of questions arises when it comes to choosing a technology, or technology stack:<p>1. SQL vs JSON.<p>I store my data in a JSON files now which seems to work quite nice: I have an array with 700-800 objects of about 5-6 key-value pairs of unicode strings and urls:<p>My old laptop search through that data blazingly fast, but what about the web? What will happen, when multiple users will try to access the same file at the same time? The question is, how much slower is reading and searching in JSON file as opposed to SQL database?<p>2. Python in PHP way.<p>I use Python as a general-purpose language, I write backend in PHP. This time not only do I want to write parsing&#x2F;crawling part of this app in Python, but I want to create and serve web pages using Python.<p>However, it finds out that Python gets quite complex when it comes to developing a web app. You have to use a full-fledged web framework such as Django which seems as quite an overkill to me, especially in my case when I do not need to worry about storing user&#x27;s data, no sign up or sign in is required, no email checking. So, second question is, can I form and serve valid html documents with Python using inline tags as with Apache + PHP stack? like the following:<p>If not, can it be done significantly easier than in Django using all that MVC (also they call it MTV and I haven&#x27;t figured out why yet) stuff?<p>3. AJAX<p>Third part comes right about serving that content in search result container without reloading page. How do i access local JSON file with jQuery or should I query my server with a XHR?
======
gusmd
I've recently been in a similar situation. After some research, I decided to
go with Flask. It is so easy to get started, you will drop that _python is
complex for a web app_ opinion very quickly.

Now, over the last couple of weeks, I've migrated my app to Quart [0] due to
asyncio support. It is pretty much a drop-in replacement of Flask, and has
great support for websockets, which I use extensively. You could use that (or
the also supported server-side events), in conjunction with JS, to update your
webpage.

[0] [https://gitlab.com/pgjones/quart](https://gitlab.com/pgjones/quart)

------
is_true
What kind of data? That's key IMO.

If Django is too much for your needs maybe you could start with Flask.

