Are you a charismatic amoral-yet-benevolent despot with a knack for turning violent tuggs into loyal enforcers? Anarchy sounds so great, but keep in mind that other, less enlightened, despot wannabes are going to come after you and yours, so keep at all times enough resources for the defense of your turf.
Not really interested? I am not surprised.
But if you bear with me, you can stretch the "violence" argument a long way.
If you have a no government at all, you have this "Mad Max" scenario. The strong take from the weak until the only way to survive is to pledge allegancy to one Warband or another.
If you have a government that only concerns itself with protection against foreight invasion, you will have the semblance of order and institutional life, but underneth the organized crime will rule. They will not limit themselves to illegal business like drugs (though they will exploit it in a monopolistic way), but will quickly develop extensive protection rackets that will be a burden to the whole economy.
Then, if you have a government that only concerns itself with foreight defense and internal defense or property rights, you will have the robber barons scenario. The rich will do as they may, and anyone that opposes their interest will face a weaponized law enforcement at their doorsteps. That, or will be victim of paramilitary groups like the pinkertons, who enjoy the unofficial protection of law enforcement.
And so it goes...
I agree it comes to a point where giving more responsibilities to government causes more problems that it solves, but to despise every and all regulatory oversight in the name of liberalism and (idealized) anarchy is an oversimplified model.
Unfortunately this approach generally fails with anything more complex than a tiny village of 1-200 people at most. There's no particularly system with which hierarchical/authoritarian structures can easily be replaced (observe the tendency of most religions to end up concentrating temporal power even where said power is doctrinally discounted) and attempts to replace fallible human agency in the hierarchy with an incorruptible technical solution typically veer into absolutism and tyranny, whether by institutional capture or simple inertia.
The basic problem (it seems to me) is that we don't have a good theory of government by which we can measure institutional health and reliability. Anarchists and libertarians are excellent at pointing out the shortcomings of government, but as you point out their response to this is basically dismantle it, which seems to be about as sensible as saying the cells in your body will enjoy greater freedom if one would only commit suicide. Governments exist for the same reason that complex multicellular life exists.