People who are going with the flow of a conference often want to stay up and hang out until 10 or 11 and meet people, and it's not difficult to organize or get local companies to organize events that go that late, like hack fests. People who decide to stay up later, well, at that point they're not going with the flow of the conference. Hopefully they know what they're doing. And if they don't, and something bad happens, of course the offender should be blamed.
I just don't think having to wing it should be a normal part of a conference experience.
Edit: I realized that I've only been to small (<500) conferences. It still seems like bigger conferences could have evening events, but they might be trickier to plan. Perhaps having a few evening events that are announced and that the organizers can vouch for would be a solution. Also I don't think it's all-or-nothing; the more well-organized activities there are that fill up the evenings, the less chance of something bad happening at a poorly-organized evening activity.