
Crossbar.io: Real-Time Distributed RPC in Python (Interview) - sametmax
https://www.podcastinit.com/episode-101-crossbar-io-with-tobias-oberstein-and-alexander-goedde/
======
sametmax
I've been following crossbar for 2 years now, wrote some doc and tutorials for
the team, but only this week have I finally used it in production. Made a real
time dashboard for a big transport facility, coupled with vuejs and django.

In short, it's a fantastic piece of software: RPC and Pub/Sub can't be easier
and cleaner. The fact it also packs a process manager, a static file server
and a WSGI servers makes it amazing.

But we really need a framework around it because right now it's a lot of
manual work.

It would probably take a year or so to make something worth it and not "yet-
another-microframework", but I've been studying the beast for some time and
using it would allow to create a Python framework loaded out of the box with:

\- full async web (and non web) microservices

\- live settings, including routing

\- task queues, with PUB/SUB talking directly to the browser

\- task runner, including for building static assets

\- advanced cache busting strategies

\- hot reloading to help you in dev, or to deploy in prod

\- load balancing and fault tolerance strategies

On top of the regular features.

Additional stuff we could add, not related to crossbar:

\- configuration framework (there is literally no good general solution for
configuration management out there. The less terrible being pyramid's and
zope's)

\- life cycle and error handling helpers (this is somewhere async framework
fails big time) with clean logging, event model and debug tooling.

\- wrappers offer a micro-framework like API that you can unfold into a full-
scale framework when your needs grow. Aend not just one or the other.

\- clear story for DB. Qraphql is something to explore but it's a hard topic,
with bad asyncio support for ORM and nosqlite db.

\- 2 pass template rendering. Meta template generating templates, then
generated templated used on the fly as views. This allow many tricks like
pregenerating translated page, URL routing, static assets name based on hash,
etc.

\- administration web console, not for the database, but for the project
itself.

We really need something to compete with new gen frameworks in Python. Right
now there is nothing trying to be on par with meteorjs and the likes.

The trap is that Django is good enough right now, so investing the time and
resource into dev something THIS new is hard to justify. And I read the code
base, it's not easy to groke (async never is anyway, but here you have to
manually deal with the event loop) so finding a team would be hard, and doing
it alone even harder.

~~~
jarpineh
Django integration sounds interesting and it's something we have been mulling
about. Could you point any specific examples or code? How did you deal with
RPC and Pub/Sub with Django? Solutions that come to mind are adjacent view
functions that Crossbar calls when messages arrive (meaning users wont touch
those directly) or perhaps going directly to Django with scripts (Django
management commands)?

About your wish list:

I understand you'd build these features on top of, or relying on crossbar.io?
This sounds like something regular Django wouldn't have a place in. I'd very
much like a system with features these features. Django's structure and
direction doesn't seem to allow things like this. Class based configuration
for various things, use of startup time settings everywhere. Django has a very
wide and stable base for regular web stuff, but when you need things like JS
web apps, streaming, reactivity, asynchronic distributed operations it gets
complicated pretty fast. After more than 10 years with Django I'm still unsure
on how to tackle things like this properly, without hacks and loads of
boilerplate.

~~~
sametmax
> Django integration sounds interesting and it's something we have been
> mulling about. Could you point any specific examples or code? How did you
> deal with RPC and Pub/Sub with Django? Solutions that come to mind are
> adjacent view functions that Crossbar calls when messages arrive (meaning
> users wont touch those directly) or perhaps going directly to Django with
> scripts (Django management commands)?

There are a lot of things to get Django integration because Django is a lot of
things, but I can see 3 points worth mentioning quickly:

\- auth. Crossbar has it's own version of auth, which means you need to create
a WAMP auth service and call django.auth.contrib's authenticate() function in
it to get authentification in it. It also means you need to create some manual
hook for logging out.

\- pub/sub and rpc: crossbar has an HTTP bridge, and it's probably the best
way to get Django in the loop for all events. However, you must remember than
you may have many Django processes in production, and only one will receive
each pub.

\- orm. Django's ORM is blocking, meaning you have to run all calls to it into
loop.run_in_executor. It's very annoying and there is no way around it.

> I understand you'd build these features on top of, or relying on
> crossbar.io?

Yes. Although the webpart would be handled by something else, but connected
with an integrated wrapper.

> This sounds like something regular Django wouldn't have a place in.

Indeed. Django being blocking, you can't really make it the center piece of
such a work. You could, however, provide adapters so you can integrate with
existing Django project to ease transitions of existing code bases.

> I'd very much like a system with features these features.

Maybe I should start a kickstarter or something like, gather a team and work
on it. I gave it a try in short sprints, but a few hours does not cut it.
After a while I ended up opening several tickets on the official asyncio
repository because as soon as you work that close to the event loop, you run
into edge cases you must take in consideration.

> Django's structure and direction doesn't seem to allow things like this.
> Class based configuration for various things, use of startup time settings
> everywhere. Django has a very wide and stable base for regular web stuff,
> but when you need things like JS web apps, streaming, reactivity,
> asynchronic distributed operations it gets complicated pretty fast. After
> more than 10 years with Django I'm still unsure on how to tackle things like
> this properly, without hacks and loads of boilerplate.

We can't blame Django for that though. It has been a useful design for a long
time, plus the design is still holding quite well today. It's just starting to
show limitions because we compare it to much more recent techs. Still, I
agree.

AppConfig and Django channels are hacks written as a workaround of the
framework design.

But let's not kid ourself, the amount of work that went into Django is
enormous, and it's still the framework making me the most productive to this
day. Any alternative would have a hard time just catching up with what it can
do before even starting to offer killer features that Django doesn't have.

Some other issues:

\- async is hard. A good async framework should aim at making common task as
easy as possible, including debugging. This is really hard, and few
developpers spend time on that.

\- async is hard (again). A good async framework should aim at providing a
solid even management framework, with a clear lifecycle and robusts hooks.
This is really hard, and few developpers spend time on that.

\- configuration is un unsolved problem in Python. There is no such thing as a
"standard configuration framework" and we all end up parsing config files,
reinventing live settings in database, or dealing with redis/etcd. But
configuration is a really, really hard problem : you have serialisation,
access rights, value sharing, data validation, hierarchy, handling of
different data source and several layers of conf depending of where in the
life cycle you are in an app. This is a project on it's own.

\- crossbar code base is not made to be integrated. It's a design decision
from Tavendo's team: provide a piece of infrastructure (like nginx, redis,
etc) and not a library or a framework. So you would probably have to make some
heavy mocking and monkey patching to bend it to a programmable API. Or, god
save us, making a fork.

~~~
jarpineh
Thanks for the tips about Django with Crossbar. It doesn't sound entirely
smooth sailing ;) Be that as it may, we got some IoT things starting and it
might fit the bill.

I'd certainly back such a kickstarter ;) I took a look at asyncio just a few
days ago and got dizzy fast. As you said, async _is_ hard. I just wish it
could be more explicit without having all that boilerplate.

These days doing much more functional programming I'm thinking that
configuration should be data instead of global state we got now.

I think that Django is still great and keeps getting better. I'm not about to
replace it. I just hope it wasn't so hard to use functionality and data in
Django projects by less conventional means (such as when async ops, long
running processing or keeping track of changes is required).

