
Why Rust closures are somewhat hard - amelius
https://stevedonovan.github.io/rustifications/2018/08/18/rust-closures-are-hard.html
======
Animats
There's a lot of confusion about this.

A lambda is just a function without a name. (This feature tends to come with
special syntax, although it doesn't have to.)

A nested function is a function defined inside another function which can
access the variables of the enclosing function. (A nested function can be a
lambda, but it doesn't have to be. Some languages have named nested functions.
A lambda doesn't have to be a nested function; it doesn't have to pull in any
variables from an outer scope.)

A closure is a nested function which can outlive the outer function, keeping
its data alive. (A closure can be named, the usual case in Python and
Javascript, or anonymous. So a closure need not be a lambda.)

Closures are easy to implement in garbage collected languages, but hard in
explicitly allocated ones, because extending the lifetime of the imported data
gets complicated.

So the options are:

\- Lambda without any external data access -- typical use, comparison function
for a sort.

\- Lambda with external data access, but not outliving its enclosing function
- typical use, iteration expression

\- Named function with no external data access. Typical use, a local function
in languages that don't do local functions well, such as C.

\- Named function with external data access, not outliving its enclosing
function. Typical use, internal function within a function to avoid passing
extra parameters.

\- Named function with external data access, outliving its enclosing function.
A true closure, but not a lambda. Typical use, saving state for a callback in
Javascript by passing the function to something that will save it and invoke
it later. An object, really. This was how LISP did objects.

\- Lambda function with external data access, outliving its enclosing
function. A true closure. Same uses as above, but in different languages.

Most languages offer some subset of these six options.

~~~
jarjoura
I think the OP makes it clear that this nuance is actually quite tricky in a
native systems language like Rust.

A good example in C++ would be:

\- A function (non capturing/lambda) is created with

    
    
      auto f = [] { ... };
    

\- A function (capturing/closure) is created with

    
    
      auto s = "...";
      auto f = [&] { s; ... };
    

In source code both of those look similar and you can even define them with
the same type: std::function<void()>

However, the resulting assembly for both couldn't be more different. The
Lambda case is as raw as any normal C function type, which the closure case
creates a C++ class that closes over the outer method's state. If you're not a
seasoned C++ engineer, this nuance will be lost on you.

It's fair to say that everyone should understand Lambda Calculus rules, but
that makes the language less accessible to new-comers. In the case for non-
systems languages, you can blur the lines with ease, but something Rust and
C++ just cannot afford to do. That makes understanding the nuance important to
be effective.

~~~
vlovich123
I’m pretty sure that the generated assembly for both your examples written as-
is is the same. The optimizer should see through the lambda sugar unless the
example gets weird or you type-erase via std::function.

I think your broader point still holds but perhaps could do with a clearer
example

------
t3hz0r
Note that the workaround to force `change_x` out of scope in the section
'Implications of “Closures are Structs”' is no longer necessary with non-
lexical lifetimes. So Rust closures are (somewhat) less hard now :)

------
cobbzilla
The fact that you can even do stuff like this in Rust is amazing. It’s
simultaneously: relatively clean code, very efficient and type-safe. I like
it.

~~~
capableweb
Not sure I agree with "relatively clean code" when one of the examples show
"fn compose <T>(f1: impl Fn(T)->T, f2: impl Fn(T)->T) -> impl Fn(T)->T {"
which is just a mix-match of keywords and other things, with tons of syntax
embedded in just one line.

But as always, depends on where you come from. I mostly deal with lisp
languages nowadays, so guessing it's just my view that the line quoted above
seems complex enough to not be interested one bit in Rust.

~~~
lachlan-sneff
That line can be simplified to:

    
    
      fn compose<T, F: Fn(T)->T>(f1: F, f2: F) -> F {

~~~
tomjakubowski
Lambdas have unique types, and you can't use a generic parameter in a return
type (which is impl Trait's raison d'etre), so I think it would have to be
this:

    
    
        fn compose<T, F: Fn(T) -> T, G: Fn(T) -> T>(f: F, g: G) -> impl Fn(T) - > T
    

or:

    
    
        fn compose<T, F, G>(f: F, g: G) -> impl Fn(T) -> T
          where F: Fn(T) -> T, G: Fn(T) -> T

~~~
Diggsey
Maybe you just miscommunicated, but you absolutely _can_ use generic
parameters in a return type:

[https://doc.rust-
lang.org/nightly/std/iter/trait.Iterator.ht...](https://doc.rust-
lang.org/nightly/std/iter/trait.Iterator.html#method.collect)

... and `impl Trait` exists to be able to have return types which are
difficult, verbose, or impossible to name.

~~~
YatoRust
Yes, I think he meant that it would be a bad idea to lock all of the type
parameters to the same type, because all closures have different types. So you
wouldn't be able to do something like this

    
    
        compose(|x| x, |x| x)
    

because the closures have the same type.

~~~
lmm
Because the closures _don 't_ have the same type, if I'm following you
correctly.

~~~
mathw
Because the parameters demand the same type (and for the return value), but
the supplied parameters are not the same type because the only way to get that
is to pass in exactly the same instance of the closure.

It's a fun little thing, and a good example of why impl Trait in argument
position is a really nice addition to Rust, even though based on what we were
originally excited about (impl Trait in return position for returning
Iterators and other such things), the argument position form didn't seem so
important.

------
chrismorgan
> With Rust 1.26, there’s a simpler but completely equivalent notation:
    
    
      fn new_invoke(f: impl Fn(f64)->f64, x: f64) -> f64 {
          f(x)
      }
    

That actually _isn’t_ completely equivalent. With the former example,
`invoke::<_>(x)` works, but with impl, `new_invoke::<_>(x)` _doesn’t_ work.
This is a deliberate aspect of the design of impl in argument position, and
part of the reason why it can be better to avoid it in libraries.

In the case of functions, this difference probably doesn’t matter, because you
normally can’t name the type of a function anyway, and are _extremely_
unlikely to wish to; but in other cases being able to type the turbofish can
matter for ergonomics.

As an arbitrary example, take std::convert::Into::into: the type parameter is
on the _trait_ rather than the _method_ , so you can’t do `x.into::<T>()`, but
if you need to constrain the type you must do so otherwise, e.g. `let y: T =
x.into();` or `<_ as Into<T>>::into(x)`. (In that case in particular, you’d
write `T::from(x)` instead, but there won’t always be such an ergonomic
replacement.)

------
zelly
Everything that's hard in Rust could be solved by GC.

With Rust, programmers have to spend most of their mental energy worrying
about management of memory, which has largely been automated already.

I always go back to this quote from Andrei Alexandrescu (creator of D):

    
    
      A disharmonic personality. Reading any amount of Rust code evokes the
      joke "friends don't let friends skip leg day" and the comic imagery of
      men with hulky torsos resting on skinny legs. Rust puts safe, precise
      memory management front and center of everything. Unfortunately,
      that's seldom the problem domain, which means a large fraction of the
      thinking and coding are dedicated to essentially a clerical job (which
      GC languages actually automate out of sight). Safe, deterministic
      memory reclamation is a hard problem, but is not the only problem or
      even the most important problem in a program. Therefore Rust ends up
      expending a disproportionately large language design real estate on
      this one matter. It will be interesting to see how Rust starts bulking
      up other aspects of the language; the only solution is to grow the
      language, but then the question remains whether abstraction can help
      the pesky necessity to deal with resources at all levels.
    

RAII and borrow checking is a crutch. You just limited the set of programs you
can write to those that can be written in the block-lifetime-scoped manner,
which is smaller than the set of good programs (see: pretty much any graph-
heavy data structures). The limitations of this programming model show up
everywhere, like in the way closures have to be implemented.

There will be more innovation in GC that will make manual memory management
even more useless. In a lot of cases the JVM does a better job of freeing
memory than a programmer. I don't want to spend my time programming worrying
about the same thing (memory) that K&R did in the 70s. I don't want to bet
against innovation and technology.

~~~
imtringued
Gargabe collection is a much bigger crutch. Once you use it your entire
program is contaminated by the performance characteristics of a single
improperly written function. You cannot mix real time code with non real time
code. If you do your real time code will be delayed by the non real time code.
The unpredictability of the GC has become a property of your entire
application.

~~~
zelly
Yes, but let's be real, 95% of programmers do not build real-time systems.

Especially not the type of programmers Rust is appealing to. They seem to be
more excited by WebAssembly than the stuff that actual low latency people
worry about, like tuning obscure FPGAs and vectorizing math code. These are
industries like audio processing, SpaceX, industrial equipment manufacturers,
self-driving cars, high frequency trading. Definitely not code targeting
x86_64 cloud VMs or iPhones.

In hard real-time environments they do not even do memory management. They
just statically allocate the whole heap and never malloc or free while the
program is running. So even if Rust is better at memory management, it won't
make them switch from C because it wasn't an issue.

------
rafaelvasco
I feel most things in Rust are harder to do, at least at first sight. A
compromise for the enhanced safeness I guess. Could give it a try again
sometime.

~~~
taberiand
It seems to me that it's not that Rust is hard, it's that all programming is
hard but Rust exposes the complexity so we can make a more informed decision
about the trade offs of safety and performance etc, while other languages tend
to hide the complexity (null reference, race conditions, etc).

~~~
rafaelvasco
Yeah. Makes sense. Rust just explicitly exposes certain features so that the
programmer has more control.

~~~
stefs
something i've read times and times again: programmers of other languages who
learned rust tend to discover potentially problematic code in their earlier
work.

~~~
hoseja
That's just normal for learning to be a better programmer.

------
hechang1997
This article does a great job explaining the subtleties about rust closures.

A subtle point often missed is that if you want the anonymous structure of
your closure itself to own a value, so that you can have a closure with a
`'static` lifetime (which lives forever) that still could be called multiple
times, you must use move keyword. If you just move the values into the closure
manually, your closure will be a `FnOnce` since the compiler will mistakenly
believe that you want to move that value each time the closure gets invoked,
which can only happen once if the moved value doesn't implement `Copy`. In
constrast, if you use the `move` keyword, compiler will correctly move the
value into the anonymous structure of the closure itself during its
construction, and the closure will be a `FuMut` and/or `Fn` as long as you
don't consume the said value in the closure.

~~~
hechang1997
[https://www.snoyman.com/blog/2018/11/rust-crash-
course-05-ru...](https://www.snoyman.com/blog/2018/11/rust-crash-
course-05-rule-of-three)

------
jimbob45
I don't understand the intuition of closures and they turn me off to languages
immediately. They feel like a hack from someone who didn't want to store a
copy of a parent-scope variable within a function. The idea that I can touch
variables that have gone out of scope (and that have ostensibly been GC'd)
makes me feel that it is impossible to reason about variable lifetimes when
dealing with closures. Is there some perspective I'm missing here? Is it
literally just the language invisibly adding the parent-scope variables and
their values to the top of my function?

~~~
chubot
I don't have a problem with the intuition but I'm with you on programming
style.

It's sort of a contrary opinion but I prefer to be explicit about state, and
closures are sort of silently bundling up state for you behind the scenes.
It's not very apparent from the syntax.

I think languages should have a lighter-weight syntax for classes and that
would subsume many use cases for closures.

\-----

While I've never programmed in PHP, it appears PHP actually has a pretty nice
solution for this problem! You declare the variables captured using 'use':

[https://stackoverflow.com/questions/1065188/in-php-what-
is-a...](https://stackoverflow.com/questions/1065188/in-php-what-is-a-closure-
and-why-does-it-use-the-use-identifier)

    
    
        $callback =
            function ($quantity, $product) use ($tax, &$total)
    

In other languages the capture of 'tax' and 'total' are implicit.

~~~
smilekzs
C++ lambda has explicit capture list as well.

------
lmm
It feels like the part that's missing is the ability to abstract over these
different types of function, polymorphically. At least, that seems to be where
things fall down when we try to talk about e.g. implementing a Functor trait
in Rust.

As a concrete example, the `compose` method given should clearly be the same
for `Fn`, `FnOnce`, and `FnMut`. Can we write it once and reuse it for `Fn`,
`FnOnce` and `FnMut`? If not, why not?

~~~
kccqzy
Because Rust doesn't have higher-kinded types. Well it does but it doesn't
allow you to define or implement traits for them.

Also, what happens when you compose one function with FnMut and another with
just Fn? The answer is clearly FnMut. These three traits are related by a
subtyping relationship. I don't know enough Rust to answer the question of
whether such a hypothetical compose function can return the right one from
this hierarchy.

------
Koshkin
Well, looks like everything in Rust is somewhat hard.

