Maybe it is because we were using Eclipse at my day job and I kept forgetting all the emacs commands, kept having the reflex to use the common key shortcuts for copy, paste undo etc. from the other applications I use. Maybe it was how slime would go crazy when doing scheme. Or maybe the fact that finding the keys for emacs sometimes proved problematic (when using cygwin outside of xwindows for example). Or the fact that aquamacs was slow to open on my mac, or that its default configuration was slow to open from the command line on my linux machine when I wanted to edit a line in a config file. How jumping around or scrolling a file was slow when you didn't know exactly what you were looking for. There were tons of small things that eventually added up and I always abandoned after a few weeks because I was losing productivity, was wasting time configuring or customizing it and the actual gains in productivity or features weren't that big compared to Eclipse, NetBeans, TextMate, XCode or other IDEs or editors.
Most of the comments raving about emacs (or at least the people I have met raving about emacs in real life) somehow seem to believe other editors are like notepad and lack search and other useful things. So I naïvely ask, is it really worth it?