On the other hand, I have found that most of the time the "I don't care, it works for me" people are just lazy or incompetent. Others mentioned blocking your site for screenreaders and ethical scrapers, I can add non standard (mobile) browsers to the list.
You might say you don't care about them. I say that is the same as throwing your household waste over the fence and let others fix it. It doesn't exist if you can't see it! The web as a whole becomes more inaccessible, it fortifies existing monopolies and mono culture.
It does not have to be perfect, you don't have to spend days. A little effort and awareness goes a long way.