I'm a (reluctant) Notes/Domino admin at a small company. We use it for email and a few small apps.
My impressions of where it went "wrong":
* It was built before a lot of de-facto standards for UIs came about. So it uses different buttons, shortcuts, styles and UI paradigms compared to other software (mainly Office). This creates friction for most users. Unfortunately they can't "fix" it without irritating the users who've got used to the traditional Lotus way.
* It got too complicated for ordinary users to develop their own apps. So instead of all the users being familiar with its power (and so understanding of its weaknesses), 99% of users just know it as a "buggy email client"
* IBM gave up on expanding its power or fixing any of its bugs, and instead just treat it as a cash cow for the companies which are still using it.