contrib.auth can be made to do most of these if you still want the ability to play nice with other people's code who expect you to use auth. You'll have to create and manage User objects which it sounds like you're unwilling to do, but you can't have it both ways.
I think having it there is better than not. In most circumstances it saves me a lot of time. If I come across a circumstance that I can't use it then I'm just back to where I would be anyways and will have to write a lot more code and modify 3rd party apps.
When I want to do something more original and specific where I don't need all those apps, Django just gets in the way - and that's when I turn to a more lightweight and flexible framework.
Use the best tool for the job. Sometimes Django is that tool. The problem I have is with companies and individuals who think it's the only tool.