Sure it would be a significant hit. But only for relatively few pages, and I imagine most of them would either stay cached or are very cold.
Of course you could trade implementation complexity for speed as always. My main point was to have effective ECC without any additional support from motherboard and memory modules.
The important detail to make your theory plausible, and without a huge performance impact and/or jittering, is how the kernel memory management system is going to decide what exactly is those "relatively few pages" on a server that runs a plentiful of processes, many of them including critical/heavy ones such as databases? One such example comes to my mind - transparent huge pages. And it didn't turn out to be quite successful.
I imagine soft-ECC would be more plausible if it could be applied/enabled:
1. Per-process (e.g. whole application)
2. Or fine-grained per-allocation within a process/application
I think both could be implemented through a (system) memory allocator by taking the advantage of page alignment LSBs and/or (non) addressable bits of the virtual memory. Those spare bits in memory can be used to store the encoding of ECC algorithm (Hamming, Reed-Solomon, or something more primitive but less robust).
And then, depending on the ECC algo, one could substantially minimize the performance impact of encode/decode by using SIMD.
I imagined executable pages and pages explicitly marked by the application/kernel via VirtualProtect[1] or similar.
Of course an application that marks GB of data this way could play havoc with other processes, so maybe some OS limits on how many non-executable pages a process can mark may be needed.
And there certainly might be some further dragons I'm not considering.
Of course you could trade implementation complexity for speed as always. My main point was to have effective ECC without any additional support from motherboard and memory modules.