RAII for memory management implies that destructors free owned memory, which can quickly cascade out of control for complex objects (e.g. you still need to think very carefully about memory management).
You are right in the sense that for high performance applications you still need to explicitly take into consideration memory allocations, allocators, indirection, into consideration. RAII helps you implement your preferred design, but it doesn't make you ignore the details.
RAII is the way.