Accessing beyond bounds is undefined, but leaving pointer arithmetic outside of objects undefined would preclude a lot. For example, building an OS page table or DMA, or RDMA, or MMIO, etc...
No, even out-of-bounds arithmetic is undefined. 6.5.6 Additive operators paragraph 8 (cribbed from Stack Overflow [1]):
When an expression that has integer type is added to or subtracted from a pointer, the result has the type of the pointer operand. [...] If both the pointer operand and the result point to elements of the same array object, or one past the last element of the array object, the evaluation shall not produce an overflow; otherwise, the behavior is undefined.
This doesn't say anything about pointer arithmetic on pointers to raw memory though. For example, using an mmap file, there isn't any object, and there aren't any bounds.
That's correct --- the standard doesn't say anything about raw memory. Only pointers to objects defined by C are defined to work; everything else is an implementation-specific extension (and so, covered under 'undefined behaviour').
I believe that C99 added the ability to losslessly cast from a pointer to a uintptr_t and back again, but, IIRC, the compiler didn't have to support this in C89.
These cases do work if you are operating on (uintptr_t)&object instead. The C machine model is more restrictive than the von Neumann machine model supported by the likes of x86 and ARM.