
Python multiprocessing is different under Linux and Windows - gthank
http://rhodesmill.org/brandon/2010/python-multiprocessing-linux-windows/
======
jnoller
I said this in a reply to another comment, but it's worth repeating. Yes,
brandon is right, there is a difference, albeit an intentional one. I agree
with him that having an option to force this behavior to match across
platforms is desirable, however, I do not currently have the time to implement
it myself.

So, in short; I agree, patch welcome or we wait until I or someone else has
the time to do it.

------
blub
He should stop trying to test for Windows behaviour on Linux and use a VM.
Having the multiprocessing module expose an option for turning off forking
would be useless, as Windows is not UNIX and there are many other subtle or
not so subtle differences.

~~~
neilc
If multiprocessing exposed such an option, it would effectively be harder to
write non-portable code, which seems like a good thing (to do optionally). You
could then switch on "no, do a sensible fork(2)" purely as a performance
optimization. Seems like a pretty obviously good thing to add.

------
rbanffy
Someone should call Microsoft and report a bug on how Windows handles child
processes. This seems serious ;-)

Can anyone check if this problem also exists in the Cygwin version of Python?
It's closer to the Unix source and should behave in a more unix-ish fashion. I
am glad I no longer can ;-)

I always install Cygwin when I work from a Windows box. It gives me a varnish
layer of civility on what I feel is a mostly barbaric environment.

That said, the POSIX subsystem NT had should be able to handle proper process
forking. I am not sure how easy (or not) would it be to run Python on it, but
that should also work.

~~~
alextingle
NT's POSIX subsystem never supported fork(). They, um, overlooked that bit...

~~~
daeken
Actually, that's incorrect. Both the old-school POSIX subsystem and the
Interix (Services For Unix) subsystem support a fork system call with copy-on-
write semantics. In fact, the NT kernel itself can fork processes trivially,
but this isn't exposed in the Win32 subsystem, like most of the fun stuff on
NT.

~~~
rbanffy
I always found it weird having a bunch of Windows kernel operations that are
not exposed to Windows apps.

How would you call those?

~~~
daeken
Ntdll.dll contains the userspace side of the NT API. Specifically, Zw* and Nt*
contain most of the APIs that map directly into the kernel. Then there's the
Rtl* stuff which provides a lot of fun functions for manipulating processes,
and the Ldr* stuff for loading binaries; Ldr is a lot of fun to hook into, if
you want to manipulate binaries with a high degree of transparency.

Edit: One thing to note, though, is that it's easy to subvert the Win32
subsystem by using these APIs directly. I.e. if you want to fork a Win32
process, you're going to need to not only fork it the standard NT way, but
then tell CSRSS what you're actually doing. It's easy to run into a situation
where the Win32 subsystem won't play nicely with you, because it just doesn't
have enough information.

------
demosthe
Shared resources across multiple concurrent processes is not a good way to
ensure a full night's sleep. I would follow the documentation's advice and
pass arguments to the child process and not try and depend on the state of
global objects.

~~~
mkramlich
Agreed. I think the #1 rule of multi-platform support is to NOT support
multiple platforms. Been there, done that, you won't like the tee-shirt.

But if you "must" do it, let the implementation/agent for each platform be as
different as they need to be, with no shared mutable memory, and just pass
messages.

