What helped me a lot getting away from being limited by ORM capabilites while on the job:
- Get the raw SQL of some slow or memory intensive queries the ORM produces and try to optimize them by hand. Try different approaches to get the same result, measure and understand them by using 'explain' and visualizers like http://tatiyants.com/pev/
This works great together with libraries like sqlalchemy where the ORM is optional and build upon a abstraction of SQL that you can use directly. This way you can use the ORM for the 80% where it works just fine and hand write the rest in Python without having to deal with raw, fairly inflexible SQL in application code.
- Try moving workload from the application to the database. In the past I often ended up doing ORM queries to get a large numbers of objects and then further process and even join them in Python. In Most of these cases doing it in the database is way more efficient and lets you get away with a slow language, synchronous requests running on small servers for a surprisingly long time.
- Do business intelligence type queries for reports and monitoring. Through doing this I discovered a lot of database features that I didn't encountered commonly in web application development but that nonetheless came in handy several times for it. Also since you often need to combine data in ways that it wasn't necessarily originally designed for, you really need to start thinking about how your data is structured and how to get it in and out efficiently.
- Don't immediately dismiss relational databases for tasks where they might not be the infamous "best tool for the job". Chances are, that the relational database you already have in place is good enough for your use case and that it will safe you headaches of setting up, understanding, synchronizing and maintaining a entirely different db system. E.g. Vanilla Postgres for timeseries data worked just fine for us for years before moving to a more specialized solution with TimescaleDB. Also used it with success for non-relational data, simple graphs, key/value stores and for queues.
- Get the raw SQL of some slow or memory intensive queries the ORM produces and try to optimize them by hand. Try different approaches to get the same result, measure and understand them by using 'explain' and visualizers like http://tatiyants.com/pev/
This works great together with libraries like sqlalchemy where the ORM is optional and build upon a abstraction of SQL that you can use directly. This way you can use the ORM for the 80% where it works just fine and hand write the rest in Python without having to deal with raw, fairly inflexible SQL in application code.
- Try moving workload from the application to the database. In the past I often ended up doing ORM queries to get a large numbers of objects and then further process and even join them in Python. In Most of these cases doing it in the database is way more efficient and lets you get away with a slow language, synchronous requests running on small servers for a surprisingly long time.
- Do business intelligence type queries for reports and monitoring. Through doing this I discovered a lot of database features that I didn't encountered commonly in web application development but that nonetheless came in handy several times for it. Also since you often need to combine data in ways that it wasn't necessarily originally designed for, you really need to start thinking about how your data is structured and how to get it in and out efficiently.
- Don't immediately dismiss relational databases for tasks where they might not be the infamous "best tool for the job". Chances are, that the relational database you already have in place is good enough for your use case and that it will safe you headaches of setting up, understanding, synchronizing and maintaining a entirely different db system. E.g. Vanilla Postgres for timeseries data worked just fine for us for years before moving to a more specialized solution with TimescaleDB. Also used it with success for non-relational data, simple graphs, key/value stores and for queues.