Why did you decide to go this way? Is it because the user-facing parts of the app (the bits where Flask helps) are only a small part?
I mean, if I'm developing a CRUD app with small amounts of supporting non-interactive code, I can't see a reason not to use Django (and put the other stuff in manage.py commands).
On the other hand, if I were to write something like an analytics system, which had a web UI for users to inspect and play with results, I can see that I'd feel constrained by Django when writing the non-web parts.
As someone who loved getting started with Flask, but then switched to Django after getting stuck without a good structure, I'd really like to understand whether the choice of Django vs. Flask should be made based on individual preferences (comfort of an imposed structure vs. freedom without it) or based on the type of project (mainly web, or mainly non-web)?
I love SQLAlchemy. Maybe things have changed (I'm a jaded Django 0.96 refugee), but as soon as you take away the Django ORM, you lose the admin and then all your advantages just sort of fall away. But really, that's my personal preference with a lot of bias from being there in the early Django days. I think the Django Rest Framework is absolutely awesome, and it almost pulled me back to Django but it just took my first morning of screwing around with some config to remind me of why I prefer the simple frameworks.
And actually, I could already feel with feature creep there. I wanted to have a remote filestore, so there's a Django plugin for doing it. But a) why is my framework in charge of how I store the files for my application (that means I need Django loaded in ipython!) and b) it didn't work quite how I needed and I could see storm clouds brewing.
In terms of general structure, if you know you're building something bigger, I'd suggest at least splitting the web bit and everything else as soon as possible. For me, leaving the web app to handling routing / validation / marshalling / sessions / templating etc feels like the right abstraction.
I look back on any apps I've built / worked on that a) started off as a CRUD system and b) are still going and most of them are a mess. Even if they're clean, the core business logic is mixed up with the standard framework structure. If you really work at it you can separate things out in the code, but my feeling is that by starting with an app that doesn't know about the web, you'll make it much easier to discover where that abstraction lies.
TLDR; use Django if it works for you. Whatever you choose, try to keep the framework out of your application.
A simple (and I think common) situation may illustrate what I took away from your comments:
- You create a CRUD app in Django with two types of users: staff (who use the admin interface to modify anything) and regular (can create/modify their own data, but only read other people's data)
- The staff make a mess via the admin interface, so you decide to add some type of undo thing. You look around for an existing addon that does model versioning, but people still spend time reverting changes back.
- You decide you need some business logic (changes going through some review/approval process) and are scratching your head about how to do that in Django.
I can see then why "I look back on any apps I've built / worked on that a) started off as a CRUD system and b) are still going and most of them are a mess."
In regards to your last question, here are my subjective ways of thinking about this. Firstly, I don't think there are many pre-defined or general rules with software development. I've been lucky to work on a pretty diverse set of projects in my career, and I can tell you success or failure almost always boils down to fundamentals not tools - how well do you understand the space you're working in (for web development - how well do you understand databases, caching, concurrency, scaling strategies) and how well do you understand the abstraction tools provided by your language? If your decision making is grounded in these fundamentals, then the tools you choose are not going to make that big an impact. From this point, tool choice is really about economizing the work you'll have to do and managing risk. If you're gonna be building a bunch of CRUD stuff and non-technical users need to view and modify data - then Django and its admin system makes a ton of sense. If, on the other hand, you're developing JSON APIs that you'll be deploying into high-volume conditions that you'll have to react quickly to, you don't want to pin your ability to scale and move quickly on fundamental decisions made in a large underlying library with many unknowns, in this scenario Flask makes a lot of sense because it's not overly prescriptive and plays nicely with almost everything. For this particular project none of us were very familiar with Django, so there were far too many unknowns for us to be comfortable staking a lot on it.
Vanilla is the holy grail. You mention manage.py commands but why not if __name__ == '__main__': do the thing. Or the multitude of tools like invoke (https://github.com/pyinvoke/invoke) and click (http://click.pocoo.org/5/). I digress...
Django is great for a monolith. Stack Django REST Framework on top of it and you're even bigger and nastier. I would never use it again because the amount of time spent bending it to my will often exceeds the time I'd spend building my own system. Fair warning, I have been using Django in production since version 0.96 and have built many companies on it. After ten years of building web apps, using every MVC under the sun, and working with every size/shape team under the sun you really start to appreciate simple things that compose well together. They're easier to reason about. They're easier to replace. They're easier to pull out and turn into a standalone library or shared service. Are you hitting the wall with one tiny part of your app? Replace it with something written in an entirely different language and communicate via JSON, or over a message bus. Not everything can be mapped to a WSGI request/response.
I am so glad aidos has the top comment on this post because his/her style is what we should all be aiming for from the get-go.
I'd definitely suggest reading up on the SOLID –https://en.wikipedia.org/wiki/SOLID_(object-oriented_design) – principles. Furthermore, the adapter concept is often explained with scary Java code samples, but the core of it is wonderful. Essentially you should consider your API as a vanilla system in your language of choice and you build adapters which then allow for external services to communicate. One adapter could be HTTP (via Flask perhaps) for a front-end website. One adapter could be HTTP but one that only speaks JSON, which you might serve under api.yourdomain.com. Another adapter might be via a queue system which allows for arbitrary commands from an external source to get written to a durable queue, so that they can be replayed later.
I took some melatonin so at this point I feel like I am just babbling and about to fall asleep.
tl;dr build things without frameworks, use your language well, be vanilla, compose tiny things to build big things.
I fall mostly into the second camp. For non-trivial functionality I will (in order of preference) 1. Find a good existing framework or library 2. Refactor something I've already written to be reusable 3. Develop something specifically for the task at hand.
Even when I plump for 3 I'll be thinking about how it could be made reusable - I just postpone actually putting any effort in that direction on the grounds of YAGNI.