I have a similar experience but I have to admit that pointers became reasonably trivial once our teacher told us to use pen and paper (or a digital equivalent) to schematically draw how these pointers work, much like this article does. The difficulty with pointers is, for me, keeping track of them in your head, especially when you work with arrays. Otherwise it's actually pretty straightforward!
> The difficulty with pointers is, for me, keeping track of them in your head, especially when you work with arrays.
I think this is mostly an artifact of C's type system. I wish pointers were taught in a language with similar runtime semantics as C with a more expressive type system. Drawing it out helps a ton, but the way C forces you to embed that structure into code doesn't help.