There's a big difference between "the GIL is net neutral or positive in terms of performance" and "removing the GIL from Python after 30 years of having it baked in is pretty hard to do cleanly." What you're observing here appears to be the latter.
If lacking a GIL actually incurred a 40% performance penalty in general and not as a consequence of design decisions in Python's internals, we'd expect to see the fastest languages all sporting GILs, and that is really not the case.
It’s unclear what you mean by design decisions of python’s internals.
Certainly python could have instead been a compiled language but then it’d be fundamentally different language. Are you implying that all of Pythons features could be preserved without a GIL and no performance loss if say a full rewrite was possible?
I don't think it's possible without breaking changes.
But if we're willing to accept some changes to the C API, it's at least possible to make the GIL less global.
It really should be obvious that it's possible to have multiple Python interpreters in the same process, where multiple independent threads run Python code in parallel, with occasional calls into a shared data structure implemented as a multi-threaded C extension module.
Currently the GIL makes this impossible: you either can have the multi-threaded C extension module in a single process (but then the Python threads will be limited by Amdahl's law unless you spend >99.9% of CPU time inside the extension module), or you can use multiprocessing which lets the Python code run in parallel, but than the C extension module will have to figure out how to use shared memory.
This is a massive, massive problem for us. We're already invested almost a year of developer time into changing C++ libraries to tell all those pesky std::strings to allocate their data in a shared memory segment. Even then, we've only managed this for some of our shared data structured so far, so this only allows us to parallelize around half of our Python stuff. So in effect, the GIL is causing massive extra complexity for us, and still prevents us from using more than ~2 CPU cores.
We would have been massively better off if 15 years ago we picked Javascript instead of Python. JS is also "single-threaded" like Python, but the JS interpreter lock is much less of a problem because it's not global.
Now Python currently has a "sub-interpreter" effort underway that refactors the interpreter to avoid all that global state and make it per-interpreter. But I somehow doubt that this approach to remove the GIL will be accepted, because again, it will slow down single-threaded execution: e.g. obtaining the `PyObject*` of `None` is currently simply taking the adress of the `_Py_None` global variable. Python will need to split this into one `None` object per interpreter, because (like any Python object) None has a reference count that is mutated all the time. But existing extension modules are accessing the `Py_None` macro -- they don't provide any context which interpreter's `None` they want! This can be solved with some TLS lookups ("what interpreter is the current thread working with"), but I suspect that will be enough of a slowdown that the whole approach will be rejected (since "no performance loss for single-threaded code" seems to be a requirement).
I think Python will get rid of the GIL eventually, but it will take a Python 4.0 with a bunch of breaking changes for C extensions to do so without significant performance loss.
If lacking a GIL actually incurred a 40% performance penalty in general and not as a consequence of design decisions in Python's internals, we'd expect to see the fastest languages all sporting GILs, and that is really not the case.