I've looked at doing this a couple of times by segmenting subscribers that never open or click a campaign. Instead of silently unsubscribing them I send them an email with a GIANT UNSUBSCRIBE BUTTON and explain why they fell into that segment.
The two times I've tried this I've had a number of responses asking why they made their way onto the list as they enjoy the email but block any tracking. Just a word of warning.
Removing "inactive" subs has proven to be a dud for me in many ways as well.
The first time I tried it silently and got a lot of people very annoyed they were removed. The second time I actively emailed people and they were similarly offended. I now don't bother (though if open rates were low, I'd probably just do it again).
The two times I've tried this I've had a number of responses asking why they made their way onto the list as they enjoy the email but block any tracking. Just a word of warning.