The name hellban is from Something Awful, but it's a much older idea. The basic problem with banning problem users is that they'll often just register a new account and keep making bad posts. Hellbanning tries to solve this problem by hiding from the user that they're banned; they can continue to post and everything appears to work from their end, but their posts are hidden from all other users. The hope is that eventually they'll get bored with getting ignored and just move on.
A lot of people find this distasteful for obvious reasons, but it's fairly effective. The occasional good posts from hellbanned users (which are the vast minority; most are terrible, spam, or at best noise) are simply a result of that moderators are not perfect.