
Django 3.1 - gingerjoos
https://www.djangoproject.com/weblog/2020/aug/04/django-31-released/
======
grenoire
JSONField has been an absolute godsend in combination with Django's ORM. I had
been using it with Postgres and will likely keep our backend the same, but I
cannot recommend it enough. You will have to write some validation and schema
code on top if you want your data to have similar (but weaker) guarantees to
the usual typed fields; the benefits from the flexibility you get are
immeasurable though.

~~~
nojs
Can you give some examples where a relational schema isn’t suitable?

~~~
zmmmmm
Sparse data.

For example, if there are 1000 possible attributes, only 5% are populated for
any given row. If you have 1000 columns you are going to have a ridiculously
wide and mostly empty table.

~~~
fabian2k
In Postgres, NULL values take one bit of storage as far as I understand. So a
table has to be very, very wide before this becomes a benefit. Of course if
you have attributes that you don't control but are e.g. user-specified, JSONB
column are a good choice.

------
sammorrowdrums
I'm surprised the discussion has derailed into whether or not to use JSON
fields... Certainly you don't have to, but they've been in Postgres contrib
for a long time.

Async views have dropped, and that is genuinely exciting. It's taken a lot of
work, and there's still a ways to go before Django is truly async native, but
this is one of the big steps, and being able to do some concurrent IO on a few
routes could be a huge saving for some projects that call out.

Otherwise a lot of stuff has to be done in event queues, to avoid blocking the
main thread, and sometimes that means a bad UX when users take actions and
aren't offered the guarantee that they are complete - in times where that
might be the best option, were you not risking blocking thread.

~~~
btown
As an FYI: Using a gevent monkey patch has been a way to get async Django for
years. Overhead is inefficient in CPU cycles and you need to stay away from
doing CPU bound things like Numpy manipulations, but for an app server that’s
bound by external API call latency, it practically gives infinite concurrency
compared to a thread-per-request model. And no need to worry about event
queues. You can feel free to synchronously call requests.get with a 10 sec
timeout and still serve prod traffic from a small handful of threads. And most
days we don’t even worry about async from a coding perspective. Anyone can
email me with questions.

~~~
mgraczyk
I'd add that crucially, DB ORM operations just work with gevent. It will be a
while before async database operations are supported natively by Django. For
me that is a complete blocker.

~~~
georgewalker355
> DB ORM operations just work with gevent

My impression was that django+gevent requires some care for the db part. The
psycopg2 docs state, for instance

"Psycopg connections are not green thread safe and can’t be used concurrently
by different green threads." [1,2].

In addition, gevent cannot monkey patch psycopg2 code because it is C and not
python. This is handled by calling psycopg2.extensions.set_wait_callback()
[1,3,4]

I just now realize that you're probably referring to making the ORM itself
async capable which is on the roadmap
[https://code.djangoproject.com/wiki/AsyncProject](https://code.djangoproject.com/wiki/AsyncProject)
in which case I totally agree.

[1] [https://www.psycopg.org/docs/advanced.html#support-for-
corou...](https://www.psycopg.org/docs/advanced.html#support-for-coroutine-
libraries) [2] [https://stackoverflow.com/questions/12650048/how-can-i-
pool-...](https://stackoverflow.com/questions/12650048/how-can-i-pool-
connections-using-psycopg-and-gevent) [3]
[https://github.com/gevent/gevent/blob/master/examples/psycop...](https://github.com/gevent/gevent/blob/master/examples/psycopg2_pool.py)
[4]
[https://github.com/psycopg/psycogreen](https://github.com/psycopg/psycogreen)

~~~
btown
In practice this is just a one-time setup. We use
[https://github.com/jneight/django-db-
geventpool](https://github.com/jneight/django-db-geventpool) which uses
psycogreen under the hood. It isn't perfect (we're currently tracking down
some rare cases where connections aren't properly returning to the pool,
though we believe this to be error on our side) but it's more than sufficient.

While making the ORM async capable would be great, I'm not sure it will ever
make sense to migrate towards sprinkling "async" explicitly across our
codebase. We'll see, of course.

------
jungletime
As someone that has an active app still written in Django 1.6. The larger my
project and more complicate the more I wanted to ditch the model/view/template
separation. And attach methods to the model so that it can be called from
everywhere, returning html code in a string directly.

Other times, I wished Django had something analogous to a component, where
everything is just encapsulated in a single file. I don't want to separate
javascript/html/css view/template. I want a single file I need to update and
an easy way to call to render.

The template system is also difficult to use, if you get complicated
scenarios.

I needed to show event details in a modal if that was the user preference. But
the page could also be embedded. This lead to me having to render differently
depending on the device, whether it was embedded and if it was a popup or not.
This lead to an explosion of possibilities, a total 2x2x2 = 8 different views
for the same data in the same template.

The most practical way was with if / then statements but that still lead to me
repeating html code. And being difficult to reason about and test.

I also got into situation where the template wasn't parsed and updated.
Probably because of too many includes or inheritance in the template. For
example, I wanted to write a custom css file that would use google fonts that
the user selected in the UI. The only way I found to work was to bypass the
template and write a function in the view that would render_to_string the css
file.

~~~
mixmastamyk
> I don't want to separate javascript/html/css view/template. I want a single
> file

This was always possible.

~~~
canada_dry
>> I want a single file > This was always possible.

Any good (code/github) examples of this??

~~~
mixmastamyk

        <body>
            <style>...</style>
            <script>...</script>
            …
        </body>

------
scrollaway
I've been using 3.1 alpha/beta for a while and it's got some pretty nice
improvements. "JSONField for all databases" is a huge deal IMO, as it
simplifies a lot of fairly common use cases.

------
Jonnax
Cool, they've implemented async views.

ASGI has kinda passed me by. For some small project's I use Gunicorn.

What's the setup for ASGI that's popular?

~~~
iaml
They support Daphne/uvicorn
[https://docs.djangoproject.com/en/3.1/howto/deployment/asgi/](https://docs.djangoproject.com/en/3.1/howto/deployment/asgi/)

I think in python world uvicorn is more popular choice, might be wrong tho

------
d33
Pardon the rant, but I feel that the advantages don't outweigh the fact that
with each release some of my stuff gets broken and I need to adjust. Django
puts DeprecationWarnings basically everywhere and it's a hell to maintain
projects that had been alive for a few years. God forbid you do anything with
the interfaces they expose. The problem is only getting worse when you
consider your dependencies, which in many cases don't keep up with the release
cycle of Django. It's a mess.

~~~
WD-42
I'm sorry, but this is just programming. You have two choices: 1. Never
upgrade and keep everything stable but go through hell when you inevitably
must update something or 2: keep up with the latest and go through some minor
pain with every release.

It's a lot like taking care of a house or car.

The world does not stand still, much less programming frameworks.

~~~
zelphirkalt
Hmmm, with well chosen, thought through abstractions, which only presume the
minimum they need to presume about their usage, the frequent changes you
describe are not a given.

The question is, what it is, what frequently needs to change and why no
appropriate abstraction has been found to reach stability.

~~~
WD-42
We're talking about a huge web framework here. Your comment might apply to
small isolated pieces of code. But Django is built on top of thousands if not
millions of abstractions. It's absolutely unreasonable to think that it won't
change.

------
konschubert
Slightly OT:

How does HN feel about the recent craze to make web python asynchronous?

To me, the performance gains are dubious in many cases and the complexity
overhead of handling cooperative multitasking just seems like a step back for
a language like python.

~~~
whouweling
Well, if you have external API calls in your Django app and you are running
sync (which I would absolutely advice, with running async it is really easy to
get an unpredictable performance which is sometimes hard to track down) having
the ability to run some views async is really crucial.

Otherwise your application might me humming along smoothly at some point and
coming to a sudden complete standstill or performance plummets when a random
external API endpoint starts to time out. Yes I have been bitten by this :-)

To fix this while running sync I have dedicated separate application processes
for the views that do external calls, but this makes the routing complex.
Alternatively you can juggle timeouts on the external API calls but this is
hard to get right and you need to constantly keep track if calls are not timed
out just because the external endpoint is a bit slower at some point.

So I think this solves a very real-world challenge.

~~~
viraptor
> plummets when a random external API endpoint starts to time out

You should add something like
[https://pypi.org/project/circuitbreaker/](https://pypi.org/project/circuitbreaker/)

Continuously failing external requests should not make each one of your
responses slow.

~~~
whouweling
Interesting, will certainly try it out, thanks!

> Continuously failing external requests should not make each one of your
> responses slow.

It is not really a matter of the responses becoming slow, the problem is that
if you are running sync with i.e 6 application server processes and you have
just 6 hits on an endpoint in your app that is hung up on an external API call
your application stops processing requests altogether.

~~~
WD-42
Exactly this. I see the whole django async stuff being far more relevant for
applications with lots of traffic or high request rates where you are already
running on beefy infrastructure with a ton of workers and any small
improvement in performance translates into huge real world cost savings. Your
standard blog, no so much.

------
leetrout
I hope they prioritize some support for ROLLUP and friends.

I’d never used it before and it was fantastic but I had to drop down to raw
sql to do it. SQLAlchemy has had support for well beyond a year.

I’ve used Django since 2008 and I love it with all its warts but I’ve really
grown to prefer SQLAlchemy.

~~~
jsmeaton
Adding new SQL support to the Django ORM is tricky. There isn’t a nice low
level abstraction for generating and composing SQL. It’s mostly a bunch of
lists containing different kinds of data (strings, Col, Expression) and they
don’t compose well.

On top of that, you’d need to come up with a decent frontend syntax that
aligned with the existing methods.

I think Django made a mistake when first defining the language of aggregates
by overloading the values() method and not using group(). To support rollup,
values() would need to support it but only when used after an annotate. Not
nice.

I often think about what it’d take to use alchemy for the sql building beneath
the Django frontend. That would open up so many more possibilities and
features.

~~~
leetrout
Yea I started to bring up the existing ticket for grouping on the dev list and
ask what people thought about it.

I also ended up writing a very tiny transformer function and using that
directly because core only has a couple supported casts and I needed Postgres
timestamp types so I could extract and rollup on the year / month / day. That
gave me some insight in to some of the patterns in use in “lower level” Django
differ from the expressiveness / composability of SQLAlchemy.

~~~
true_religion
SQLAlchemy has a great architecture, but it comes at the expense of being tied
to the use case of accessing a relational database.

Django models are more abstract and anemic in querying, but it means you can
use the same API and write access layers for non SQL databases. At work, we
have an in memory database, and Elastic Search all queried in an identical way
to the Postgres models.

~~~
jsmeaton
I’d be super keen to see your ElasticSearch model implementation if you’re
allowed to share it?

For what it’s worth, I dislike the unit of work model that SQLA uses, but the
layered architecture is enviable.

------
kumarvvr
OT Question : When choosing a stack for creating sites, would you choose
Django over ASP.NET Core or PHP?

If so, why?

~~~
rs999gti
Does the backend language of a CMS matter?

We are a .NET shop, we'd love to use Django, Drupal, etc. But does that also
mean we need a dedicated Python resource to support these CMS'?

Or could we use these CMS' out of the box?

Or would we just be better off with a .NET CMS like Umbraco or Orchid?

~~~
kumarvvr
Once you have an app deployed, it takes a life of its own. Absolutely, the
whole stack matters a lot.

~~~
rs999gti
So the backend language of the CMS does matter, because we'll eventually have
to modify it to fit business needs?

------
jungletime
Something I wish that Django did was user defined functions in the template.
It has for loops, which is good but it forces you to write the html in top to
bottom procedural manner.

It would be far better to be able to define a function that you can call for
bits of html code that might repeat in the same template.

Since I stopped using Django at 1.6 does the new version let you define
functions in the templates?

~~~
__alexs
It has blocks for this
[https://docs.djangoproject.com/en/3.0/ref/templates/builtins...](https://docs.djangoproject.com/en/3.0/ref/templates/builtins/#std:templatetag-
extends)

~~~
jungletime
That isn't as nice to having a function inside the same file you can call
over, IMHO.

Also lets say you define a block for rendering a button with a different
color. Then use 20 different buttons on one page.

Does that mean that the block code would need to be loaded from a file 20
times and parsed each time. Seems like huge performance hit to do it that way.

Is there a technical reason why you can't have a function in the same file.

Am I misunderstanding how it works?

------
jtdev
Does the ORM still write horrendous cursor based loops where simple joins
would suffice?

~~~
stefano
By default yes, but you can configure queries fetching behavior using
select_related/prefetch_related to avoid the N+1 queries problem.

~~~
collyw
Sadly not so many people know how to use those. From the Django projects that
I have inherited at least.

~~~
selectnull
Django ORM mainly hides the SQL from the developer (which is kind of the idea
of any ORM). But if a developer does not understand the underlying SQL
concepts, they will soon write performance wise horrific code. But, if you
remove the ORM from that equation, I don't see how that same developer doesn't
make the same mistakes. So in the end, I don't think that Django ORM (in its
core) can help much in this area.

There are of course tools that can help with that (django-debugger and
others).

~~~
yebyen
This is a hot-take from Aaron Patterson, Rails and Ruby core team dev, in his
keynote this year where he addressed a very similar idea on a perhaps related
query generation topic...

[https://youtu.be/9JEEabL90AA?t=1360](https://youtu.be/9JEEabL90AA?t=1360)

To give you the tl;dr (he goes through profiling and a great deal of data to
help show what a core dev needs to do in order to help us solve this one
specific case, and...)

Aaron comes to the conclusion that there is a "performance pro-tip" which many
Rails devs have learned, similar to what y'all are discussing in this thread
for Django, and in this one specific case he outlines in some detail, Aaron
considers that there is a bug where the programmer needs to know this "pro-
tip."

As there was really no reasonable way for the Rails engine to interpret this
instruction from the programmer as to mean "do it the slow way" when there's a
better way to do the same thing with no drawbacks, and it would have been
possible for the engine to safely do the smarter thing (it was IIRC to
precompile some better-shaped SQL, bringing back the same result set in fewer
queries, (making up the most runtime performance by spending less time in the
SQL compiler overall, in the worst performing pathological case, all this is
shown with data)...) and that having this behavior here is actually much
better for a novice programmer's behalf, in order to make that better
performance happen without requiring a developer to manually build such hints
into the application code, at abstraction layers where they really shouldn't
have to be thinking about those things anyway.

~~~
doteka
While a good point, I would say the problem is caused by ORMs existing.
Somehow somewhere we decided that a person who thinks SQL is too difficult
should be using a database.

~~~
Daishiman
Nobody who builds an ORM thinks that.

An ORM saves experienced SQL users from having to write boilerplate SQL and
hack on their own garbage ORM, which is what any large project ends up doing,
attempting to compose queries and filters in vain.

I have never ever heard of ORMs as an argument to avoid learning SQL, and
AFAIK no author of well-known ORMs holds that opinion.

~~~
jtdev
"boilerplate SQL" is an oxymoron...

Literally every pro-ORM dev that I've ever worked with was so incredibly weak
with basic SQL fundamentals that I must conclude that they preferred ORMs
simply due to a reluctance to learn SQL.

~~~
yebyen
There is a strong argument for conceptual compression, the idea that every
component has an optimum abstraction layer to understand it at, and with that,
less-optimum layers for understanding. An ORM that does its job well allows
you to take the complexity and pack it away for another day. We can unpack it
later when we need to understand those details, but most of the time we'd
rather be focused on the details of a domain-specific problem that we're
trying to solve, with intricacies that are specific to the domain, that
ideally need not become compounded against problems introduced separately by
(for example) a persistence layer such as a database, as that makes it harder
for the domain experts by introducing another layer of complexity that they
must grok in order to build or spec an appropriate solution.

------
waihtis
I'm closer to a hobbyist than a professional dev, but the async views seem
like a big functionality. Having done some Django apps, getting a synced up
view for some changing variable was always a bit painful.

~~~
StavrosK
I'm not sure you're talking about the same thing, this is about asyncio/green
threads.

~~~
waihtis
re-read the documentation but I'm not following. This example:

    
    
      async def current_datetime(request):
          now = datetime.datetime.now()
          html = '<html><body>It is now %s.</body></html>' % now
          return HttpResponse(html)
    
    

seems like an example that I was thinking of.

~~~
StavrosK
I don't know what you're thinking of, unfortunately, but that piece of code
just returns the current date when you visit the page.

~~~
waihtis
After taking a uninterrupted reading slot to understand the async views,
you're right, and I had the concept understood wrong.

~~~
StavrosK
Yeah, I think you were thinking some sort of auto-refresh.

~~~
waihtis
That was exactly what I was _hoping_ this to be.

~~~
arusahni
Perhaps django-reactor would be of interest to you? [1] It's targeting
LiveView-like functionality for Django. I haven't used it (so I can't vouch
for the claim).

1: [https://pypi.org/project/django-reactor/](https://pypi.org/project/django-
reactor/)

------
FreeHugs
What would be a typical use case for "Asynchronous views"?

~~~
woile
async is cool for I/O bound operations. You don't have to wait the
request/response to finish in order to start processing another request.

Talking with a db, or doing http requests are IO operations, so instead of
blocking the process, django can now start processing another one. When the IO
operation is done, it continues where it stopped.

~~~
FreeHugs
Why would Django not be able to process other requests while one process is
waiting for IO?

I don't know much about Python, but in a typical PHP/Apache setup, Apache
simply starts as many processes as needed to answer all requests.

~~~
Ralfp
Apache or FCGI?

In python world you have few Django processes spawned by WSGI/ASGI app
container, and those processes spawn python threads to handle requests.
Because python uses GIL, no more than single thread is executed at single time
per process. At specific times GIL stops running bytecode for one thread and
moves to next one (eg when you do IO). This means other threads keep running,
but in theory you can run out of threads.

In async you use tasks instead of threads for http requests, so single process
can handle multiple requests at same time, only spawning threads at limited
number of situations.

------
ram_rar
I wish, there is something equivalent like this in golang world. Django has
been one of the best well maintained open source projects.

------
samirsd
can the async functionality replace celery?

~~~
heliodor
No. Celery can execute tasks after your view returns. Async can't do that.

~~~
bredren
Would you please elaborate? If all of a task is encapsulated in a django async
view, would this not be the same thing?

~~~
heliodor
To run async code, you use 'await' in python. That right there tells you that
you have to wait for it to finish. You want to receive the result. The view
will wait for a result before returning the http response.

Celery and all task queues are fire-and-forget. You tell it to do something
for which you don't want the result, so you can continue executing your code
right away. The task queue can even take an hour to process, for example. Or
you can schedule it to run later.

~~~
samirsd
thanks

