On: Chris Lattner on garbage collection vs. Automatic Reference Counting (2017)

[jnwatson]

The done vs destroyed dichotomy is a cross-cutting concern for programs that need explicit control over resource lifetimes.

RAII is actually an antipattern in Python if you actually care about resources being released.

[shakna → jnwatson]

The surprise is less about resource usage and more about the scope lifetime. 'with' introduces a new binding that outlives its body. It looks, at first glance, to be a temporary variable. But it isn't.

[chombier → shakna]

I always wondered why does this variable leak outside the `with` scope, because this is really the one place where it should not.

IIRC python3 fixed scope leaks in list comprehensions, does anyone know why they didn't fix this one as well? Or does it have actual uses in practice?

[shakna → chombier]

I don't know the _motivation_ for why it leaks, but it leaks because the method calls of a context manager are __enter__ and __exit__, but not __del__. [0]

The motivation is especially opaque, because though CPython does run __exit__ immediately, it isn't guaranteed to do so by the standard. So __exit__ will happen at some point, like __del__ with the GC, but it won't actually call __del__, which makes GC'ing the value more complicated as __del__ can't safely run until __exit__ does.

I _think_ the motivation is something to do with __exit__ being allowed to throw an exception that can be caught, whereas __del__ isn't allowed to do that, but I don't find it that convincing. And I've never seen an example where the introduced variable is ever referenced afterwards in Python's documentation (including in the PEP itself).

[0] https://peps.python.org/pep-0343/

[chombier → shakna]

Thinking about it, my guess is that properly scoping the variable does not really prevent the user from leaking the value anyway:

    with context() as foo: 
        leak = foo

This is unlike list comprehensions, where it is syntactically impossible to bind the comprehension variable to a new name accessible outside the comprehension.

So maybe the reason is that it is not really worth the trouble.

[ptx → shakna]

Everything is always* scoped to the entire function in Python. Perhaps that's not always ideal, but at least the rule is simple.

* Well, not variables in generator expressions. There might be a few other exceptions.

[pphysch → shakna]

Consider the common pattern:

    with open(path) as file:
        result = func(file.read())

    another_func(result)

If `with` used a separate scope then it would be a lot less ergonomic. How might we bring data outside that scope and be confident that `with` isn't forcibly cleaning things up? Would be a lot more verbose.

[shakna → pphysch]

I think you've missed what I've meant by a new binding. That the body executes is entirely intuitive.

However:

    with open(path) as file:
        ...

    if file.mode == 'r':
        print("Huh? file is still a thing?")

[xscott → shakna]

Python doesn't have block scope, but most people don't seem to notice:

    def foo():
        if False:
             x = 123
        print("x is visible out here:", x)

That will generate an error because `x` is "referenced before assignment", but the variable exists for the entire function. This can bite you in subtle ways:

    x = "is global out here"
    def foo():
         print("all good:", x)

    def bar():
         print("not good:", x)
         if False:
              x = "now x is local, even before this declaration"