That is not an issue with the safety of the Vector, it is an issue with the safety of 'int' and raw pointers. If the Vector grows, that pointer points to freed memory.
But yes, in my implementation I have a safe version of int called 'i32' which overrides the & operator and doesn't allow it to return raw pointers.
It absolutely is a concern about vector. Iterator invalidation is a property of a type and its interfaces. One could design a vector implementation that doesn't invalidate pointers and doesn't provide this footgun to users.
There are significant costs to this safety, of course, just like adding bounds checks.
But yes, in my implementation I have a safe version of int called 'i32' which overrides the & operator and doesn't allow it to return raw pointers.