That happened to me. I have a very early memory of a trivial thing (eating a radish and not liking it) from when I about two. It happened at my family's apartment, before we moved into our first house. I later visited that apartment when I was ten, and a lot of the details match my memory(e.g. the location of my Dad's little garden relative to the stairs) EXCEPT the actual visualization of the apartment building, which got replaced with an image of one near our new house twisted to fit the memory.
