Thus, it makes no sense to ask what is or is not a "privilege that can be taken away", since this is a question with no logical or scientific meaning.
The only thing you can ask is "given a specific set of criteria we are trying to optimize for, is it better or worse for this to be a privilege that can be taken away". Which, of course, depends on the criteria.
A civil society breaks its pact with law abiding people people when it allows these people to go on living when they've not only broken the social compact, they've destroyed it by dint of murdering, raping, selling drugs, ruining the lives of others, some of whom were parents who have left a child parentless, sometimes orphans, who then become a burden on the state (albeit a worthy burden). Actions like these are grounds for state-sanctioned execution of the offender.