The not-implemented instruction trapping in the 68K could be easily evolved to cover less frequently used instructions to make room for more optimized implementations of the frequently used ones.
What really killed the 68000 was the move to RISC, in particular Apple's move to PowerPC. That took away any hope of future evolution (they even managed to release the 68060 after that, but that was it) and collapsed the high-end 68K business.
The Motorola Series 900 machines were interesting - I had one under my desk at work for quite a while. They had stackable units, including one that contained a SCSI 3.5" floppy drive that was way faster than regular ones. We also had a DG unit with the m88k.
What really killed the 68000 was the move to RISC, in particular Apple's move to PowerPC. That took away any hope of future evolution (they even managed to release the 68060 after that, but that was it) and collapsed the high-end 68K business.