That wouldn't fix the current situation, since if you wanted to reduce somebody's karma, you just downmod their positive submissions. Almost all submissions are positive, so nevermind. People with mostly negative submissions are probably spammers anyway.
I'm against downmodding beyond zero.