Just holding an invalid reference is UB, even if you don't do anything with it (the example still works as the reference gets invalidated by safe code).
A reference is never really "held": from a language-semantics standpoint, it only exists when it is actually used. In this example, copying, reborrowing, or accessing the reference would be UB, but simply letting it fall out of scope would not be UB (modulo the UCG issue oconnor mentioned; but I personally doubt that this status quo will change). You can try this yourself with Tools > Miri on the Playground (https://play.rust-lang.org/?version=stable&mode=debug&editio...). The distinction is far more relevant for unsafe code than for safe code.
