
Subinterpreter Support for Python (2018) - tosh
https://lwn.net/Articles/754162/
======
davidw
This is from 2018, and like someone notes in the comments, this is similar to
the model Tcl has used for quite some time.

Handling interpreters:
[https://www.tcl.tk/man/tcl8.5/tutorial/Tcl43.html](https://www.tcl.tk/man/tcl8.5/tutorial/Tcl43.html)

And Tcl, threads, interpreters and message passing:
[https://www.activestate.com/blog/threads-done-right-
tcl/](https://www.activestate.com/blog/threads-done-right-tcl/)

------
marmaduke
> The PEP provides for a shared-nothing concurrency model.

Isn't multiprocessing already a shared nothing model?

~~~
pdonis
Yes, but it requires multiple Python processes, which is often overkill for
the concurrency needs of an application.

------
solotronics
It seems to me that in python3 using: [ asyncio, event loop,
ProcessPoolExecutor, or ThreadPoolExecutor ] satisfies this problem. If I am
misunderstood please correct me!

[https://pymotw.com/3/asyncio/executors.html](https://pymotw.com/3/asyncio/executors.html)

~~~
pdonis
The only one of these that solves the GIL problem (that only one thread in a
given Python process can be running Python bytecode at a time) is
ProcessPoolExecutor, or more generally forking separate Python processes for
each computation you want to do that uses Python bytecode. The others all
still are limited by the GIL.

What subinterpreter support will do is basically to allow multiple GILs inside
a single Python process, each in its own thread, so multiple threads in the
same process can now run Python bytecode.

~~~
duckerude
Notably, subinterpreters might be able to share data without copying it.
ProcessPoolExecutor can only share data by pickling it.

~~~
pdonis
_> subinterpreters might be able to share data without copying it_

In the current proposal as described in the PEP, they wouldn't.
Subinterpreters will only be able to communicate through channels, and
arbitrary objects can't even be sent through channels; they would have to be
pickled or marshaled, similar to sharing data between processes.

~~~
duckerude
If I'm reading the PEP correctly, it's possible to send memoryview buffers
through channels without copying the underlying data.

The few other types supported by channels (None, bytes, str, int) are all
immutable, so I would guess that those are also shared without copying.

~~~
pdonis
Yes, these particular types of data can be shared (as you say, sharing
immutable objects obviously doesn't require copying since they're immutable);
but I took "shared data" to mean being able to share arbitrary objects, since
that's the sense of "sharing" that is relevant when we're talking about
various forms of concurrency. If all I can share between concurrent
computations is those limited data types, then I have to pickle/marshal
everything else, which is effectively the same thing I have to do with
multiple processes.

~~~
duckerude
I see. I was paraphrasing from the article originally, and didn't mean "share
data" in a very general way.

Sharing memoryviews lets you share e.g. numpy arrays, so it should be powerful
enough for some applications.

