I used to choose Eclipse over IntelliJ for similar reasons years ago. When it came to refactoring and other integrated features IntelliJ won, but Eclipse could do partial compilation on save and the run of a unit test automatically within moments, something which on IntelliJ invoked a Maven build that often took minutes.
The performance of feedback and the speed with which you can go through the basic cycle of coding and testing it works as expected is the speed at which you can develop code and the editor is a pretty critical part of that.
Interestingly enough I had kind of the opposite experience
Since I haven't done much with Java itself, the build times weren't as impactful on me.
What made a big change was how IntelliJ, despite being pure Swing GUI, was order of magnitude lower latency, from as simple things as booting the whole IDE, to operating on, well, everything.
Then I switched from Oracle to IBM J9 and I probably experienced yet another order of magnitude speedup.
The performance of feedback and the speed with which you can go through the basic cycle of coding and testing it works as expected is the speed at which you can develop code and the editor is a pretty critical part of that.