I'm not making a one-way argument. If the underlying object actually is a Derived, you can freely cast between pointer-to-Base and pointer-to-Derived. That is what "and vice versa" means.
But if the object isn't actually a Derived, you can't cast to pointer-to-Derived:
Derived derived;
Derived *pDerived = &derived;
// This is legal because it's equivalent to:
// Base *pb = &derived.base;
//
// ie. there actually is a Base object there that the
// pointer is pointing to.
Base *pBase = (Base*)pDerived;
// This is legal because pBase points to the initial member
// of a Derived. So, suitably converted, it points at the
// Derived.
//
// The key point is that there actually is a Derived object
// there that we are pointing at.
pDerived = (Derived*)pBase;
Base base;
// This is illegal, because this base object is not actually
// a part of a larger Derived object, it's just a Base.
// So we have a pDerived that doesn't actually point at a
// Derived object -- this is illegal.
pDerived = (Derived*)&base;
// Imagine if the above were actually legal -- this would
// reference unallocated memory!
pDerived->some_derived_member = 5;
To my mind, 'illegal' means that the compiler will complain. In this case, I don't even see weird, scary UB; this is just a case of the standard being completely unable to say anything about what will happen.
After spending too much of my life chasing these bugs, here the compiler will do exactly what you told it to, which probably means making your day miserable.
But if the object isn't actually a Derived, you can't cast to pointer-to-Derived: