Likewise. I have only used them twice for that reason. I will also say the two times I have had terrifying hallucinations, they were induced by intensive meditation, not by drugs. The thing is, I think that you get out of these things what you put into them. The time I tried mushrooms, I could move in and out of reality as I chose. I attributed this to preparation because many of the others who tries mushrooms with me could not do this.
The time I tried LSD was different. I could not move back to reality, but I spent most of the time in meditation. Finally later I gave up on meditating only to have a series of life-altering epiphanies and hallucinations immediately following.
These experiences can happen without drugs to be sure, but there is something about taking oneself out of the normal world that seems to facilitate them.