Addiction and excessive use are the only things about drugs, including alcohol, that tend to break apart family and community. If there's no stigma attached to occasional and social use of psychedelics or other drugs then I don't see how they would negatively impact family and community any more than occasional use of alcohol does. I agree that some people will suffer negative consequences from drug use but upon close examination of those cases it's very rarely the drug itself that is the source of the trouble. People abuse drugs in order to escape otherwise inescapable situations in their lives, if every day you wake up to misery and pain then taking a pain killer or dissociative is the easiest way to escape that and often the only way if the person has no family, community or healthcare support.