In general, it's pretty messed up that there's an exceptionalism about drug related crime (and some other kinds of crime).
Crime is crime. If they don't take custody of a house because some kind of crime X happened there, they shouldn't do it for drug related cases either. They can always arrest the person dealing the drugs and forgeit the drugs themselves.
There was a case a few years ago where the parents lost their house because their son once was caught with drugs in the house.