
Dynamic Django Models with Model Models - jonatron
https://www.protoapi.net/blog/2/
======
pjungwir
There's this application I've been writing and re-writing my whole career: I
call it "some users design forms, other users fill out forms." It is hard to
design the db as anything other than EAV. (You can push more or less of it
into JSON columns, but there are tradeoffs.)

I have often wondered about pushing the customizability down a level though,
like what the author does here, where each form creates a separate database
table. It's too crazy for me, but it's tempting. And I have actually done it
before where users' forms automatically generate individual tables in a
separate flattened reporting database. It works really well for giving
enterprise customers SQL access to an OLAP view of their data, even if I
wouldn't use it for my source-of-truth OLTP structure. (Some day I'm going to
write a blog post about that.)

Even just for derived reporting tables, you quickly discover all kinds of
limits in your database: reserved words, max columns per table, max columns
per query, number of foreign keys pointing at a central table, etc. I haven't
hit anything fatal though (with Postgres).

Anyway, I'm excited to see that someone actually pulled this off, complete
with Django models and migrations, even if it's just for fun.

~~~
TylerE
Postgres JSONB columns could give you a lot of that without actually creating
seperate tables - for instance indexes over embedded keys.

~~~
ppeetteerr
This. The best of both worlds + you get querying inside of the JSON.

------
rheide
This is the kind of thing that I love to see as a blogpost, and I admire the
ingenuity of the author, but I would hate to ever see this in a production
codebase.

~~~
shezi
Unfortunately, most enterprise applications will have something like this,
because they'll need to be customisable for individual companies. I've worked
with a CA product years ago that had written their own SQL engine for doing
exactly that. I would have much preferred the Django solution.

~~~
paulie_a
And unfortunately it will be the responsibility of the next Dev to sigh, and
delete it.

~~~
shezi
Realistically, they'll sigh, curse and build more of the same...

------
ppeetteerr
"Warning: You probably don't want to do this because it's unsupported and very
hacky."

The opening line should have stopped here. Dynamic data structures in untyped
languages are risky, hard to debug, and prone to breaking One can argue that
document stores (e.g. CouchDB and MangoDB) have been around for a while but
their strength (and that of JSONField), is the ability to store hierarchy, not
arbitrary structure (which they also do). A dynamic model like the one in the
article serves as a database-stored deserializer, which is more of the domain
of the business logic, rather than data.

Having said all this, the article is dope :o

~~~
t-richards
>Dynamic data structures in untyped languages are risky

What exactly do you mean by "untyped languages"? Python and every other high
level language like it definitely has types. Maybe you meant "dynamically
typed"?

~~~
dangerbird2
It's a snarl word for dynamically typed languages, to conflate strong, dynamic
type systems like Python's with true untyped languages like Assembly, Forth,
and BCPL.

------
orf
An alternative to this is to use a Form. You can have a model that produces
the form configuration (relations for each field in the form for example) and
produce this Form on request.

Then, when the form is saved, take the resulting dictionary and save it to a
JSONField in a 'result' model. If you want to re-display the form with the
results, just create the form from the configuration model and add the data
from the result model.

I've had great results with this technique, and you can model any field except
a foreign key. Django forms provides the validation, and the backing storage
is dynamic so you can change fields easily. It seems less fragile than
creating models like this article proposes.

~~~
shezi
That's about what the author says right at the top of the article.

However, this is only more flexible in the front end. If you want to do
anything with your data in the back end, a JSONField will help you less than
fully structured data. While you can do some queries on JSON fields, most of
the richer queries will have to be done in python, slowing processing down.

If you're looking for runtime-changeable, structured data that you can run
complex queries on, ModelModels sound like a good alternative. Most enterprise
applications are in this area.

------
coleifer
Another way of describing this is:

* Django has facilities for mapping Python classes to database tables

* Author uses metaprogramming to construct classes to match table definitions decided at runtime

* Author uses built-in schema migration tools to synchronize backend

There's nothing about this that _wows_ me, although it does demonstrate
knowledge of Django internals and Python metaprogramming. It doesn't impress
me because this is the bread-and-butter of what Django does.

~~~
jonatron
That pretty much sums it up.

------
imperio59
One big downside with this approach no one is talking about is that your model
is no longer in sync with the rest of your code via commits.

This is generally what causes headaches later on, when you accidentally delete
a field in production or forget to add a new one to mimic your dev
environment.

But we had something like that where I last worked and it was okay. There was
also no concept of dev db which was a different topic altogether...

~~~
candiodari
Given that the whole point is to allow the customer to have their own schemas
that is not a problem, but exactly the intention of the schema.

------
TeeWEE
This is basically what MetaModelling is about. A model of your model. Boom
[https://en.wikipedia.org/wiki/Metamodeling](https://en.wikipedia.org/wiki/Metamodeling)

Its used a lot in defining DLS's. Or Programming languages in general. Also
editors often need to work one level higher in the abstraction chain.
Abstractions around sourcecode.

Lisp is the master is this.

------
danpalmer
I work on a large Django codebase, and while this is a cool blog post, it's
quite important once you reach a certain size that these things aren't
actually done in practice.

If you wanted to _fundamentally_ build your product around this concept that's
fine, but Django probably isn't the right framework, however no feature is
worth the pain of this.

For what it's worth, Django models and doing good relational design solve a
huge number of problems. We've got ~380 Django apps and ~350 models and
haven't yet needed dynamic models.

~~~
aldoushuxley001
How is it possible to have 380 Django apps in a single codebase? That sounds
insane. Is that not absurdly complex to manage? I can't even imagine.

~~~
danpalmer
It actually works pretty nicely. Most of our apps are small and focused, and
we nest apps within each other heavily.

It's fairly common for example to see an app introduce some models, Python
APIs to work with them, and business logic, and then to have a sub-app for the
web API and bits relating to that, and a sub-app for the admin interfaces (we
don't use the Django admin).

Example:

\- orders

    
    
      - refunds  
    
      - shipping  
    
        - royal_mail  
    
        - click_and_collect  
    
      - fraud  
    
        - fraud_blacklists  
    
      - returns
    

This means that working on our Click and Collect integration, one only needs
to hold in their head the current app, and maybe some APIs provided by the
apps above it. It's relatively uncommon to import across apps on different
branches of the tree, and when we do that's typically only to top-level apps
containing the most core models, or for very explicit public APIs provided to
the whole site (which might be re-exported from an __init__.py up the tree
anyway).

I find it a pretty nice codebase to work on. We're hiring too
[https://www.thread.com/jobs/software-
engineer](https://www.thread.com/jobs/software-engineer)

~~~
toyg
_> We encourage applications from people who aren’t cis white men, who are
currently over-represented on the engineering team at Thread._

...

------
Walkman
This is extremely clever! Something like this needs a deep understanding of
Django and Python internals. The bullet points are written like they put it
together under 10 minutes or so :D I love it!

~~~
thecatspaw
To me it looks like a walkthrough how to set it up, rather than a guide.

If I googled "Dynamic Django models", _this_ is what I would like to see.

------
Fradow
My first instinct was to scream "just use a CMS", but there are obvious
advantages to this technique, so here are my thoughts to make it less hacky if
someone ever intends to run that in production.

\- the most obvious one I don't see mentionned is that you should use a
separate database than your main one for all the generated tables. Pretty sure
it would save on operational headaches.

\- it would probably be useful to generate all the Python code that could be
used for making migrations, if only for debugging purposes

\- understand early on which Fields you want to use with which options. I
wonder if ForeignKeyField would be possible. That would be a great feature,
but probably bring headaches.

\- obviously, try to submit a PR to Django for the migrations loader hack,
maintaining Monkey-patchs is troublesome

All in all, it's a pretty cool hack, but even with those modifications, I'd
still be very wary to use that in production instead of battle-tested
alternatives, unless you have a use-case that really warrants having full-
blown SQL schema for your dynamic data.

Edit: on second thought, I can think of a use case where using that in a CMS
would have saved me troubles (at least different tables for different Models).
When you have a production site receiving Model A data (for example, your
clients orders) AND you want to edit Model B data (for example, your site
dynamic pages), you can't do that with EAV all in a single table, but with
using dynamic models, you could just copy the Model B data, or even go with
more complicated scripts.

------
adpirz
A real-world answer to the problem presented here seems to be Django CMS:
[https://www.django-cms.org/en/](https://www.django-cms.org/en/)

~~~
dfsegoat
It's not really realistic to bring this into a large, pre-existing application
(REST API) from our experience - which is admittedly limited.

Re-building the app around django cms would be the way to go - but being able
to do this from scratch in an existing application is pretty nice.

------
harel
I've done something similar as an experiment a few years back but I've used
Meta Classes which define themselves, to support a schema-less model backed by
MongoDB. It felt like a more flexible and correct approach at the time. In the
end after my experiment ended, I've scrapped it decided to wait for PostgreSql
9.4 to be released with the BJSON field support. Then JSONField was my new
friend.

------
egeozcan
Great hack! A sane person would just use something that suits much better for
this use-case though. Processwire[1] comes to mind, for example.

[1]: [https://processwire.com/](https://processwire.com/)

------
fleetfox
This seems to solve all my EAV use cases.

~~~
collyw
It is essentially EAV isn't it? Well integrated into Django so it works with
the admin and migrations.

~~~
spapas82
No it's not EAV. EAV is an sql (anti) pattern where you have a table with
varchar fields such as name, type and value. So you'll have tuples like (id,
int, 99), (name, string, sera) or (enable, boolean, true). This makes it very
difficult to do aggregates and you can't do any consistency checks from the
db.

What the article proposes is much better than this: A way to create models (ie
tables in the database) dynamically. To make it more clear if you are not
familiar with django, it'd be the same as if the CREATE TABLE sql was
generated dynamically depending on the user selections.

So in this case each model will be in its own table and the fields will have
proper types. You may even be able to have referential consistency using a
ForeignKey field!

~~~
babayega2
> You may even be able to have referential consistency using a ForeignKey
> field

Since you say "you may" ... , do you happen to see any problem that may arise
with this implementation ?

~~~
spapas82
Well, the original article doesn't actually mention ForeignKeys that's why I
written may (it only has TextField and IntegerField). Of course nothing stops
you from extending them; you'll need to of course add the model where you'll
want to create an FK to.

So the MakeField model will have an optional field to denote the class to
which you want to create the FK; probably an FK to ContentType would suffice
for this field
([https://docs.djangoproject.com/en/2.1/ref/contrib/contenttyp...](https://docs.djangoproject.com/en/2.1/ref/contrib/contenttypes/#the-
contenttype-model)).

I know I've written field and FK too many times, I hope it makes sense; if it
doesn't tell me and I'll provide a more thorough example.

------
tomtomtom25
On a slightly different topic, I was wondering as to what is the best approach
to implement(or use a library) a progress bar in file upload (large files)
forms. I've been looking around for quite a while but no success.

~~~
jonatron
It's suspiciously like you're asking me to shamelessly self promote
[https://github.com/jonatron/django-admin-resumable-
js/](https://github.com/jonatron/django-admin-resumable-js/)

