I think it is appropriate to call it such. In all the times I've taken mushrooms I would have been completely unable to respond to an emergency situation. The same with LSD. No other drug I've taken would have left me that incapable, with the exception of extreme amounts of alcohol.
I'm not saying it's not fun or worthwhile, just that hallucinogens leave you impaired in a way other drugs don't.
I would tolerate a statement like "unable to respond to an emergency situation" - although, this varies substantially with the substance, quantity, person, and nature of the emergency. And the recognition that there are plenty of non-pharmacological situations that render you equally unfit for dealing with certain kinds of emergencies - good luck dealing with a back injury while skiing off-piste.
However, throwing out a statement like "drugs make you nonfunctional" does not help the discourse.
I'm not saying it's not fun or worthwhile, just that hallucinogens leave you impaired in a way other drugs don't.