MJ has a HUGE group of people who use it medicinally. My wife can not eat or drink without it, due to chronic nausea to the point that without it, she throws up every 20 minutes. Medicine like Zofram barely help even at max dose, which carries with it other side effects.
Calling everyone who smokes stoners is the equivalent of calling everyone who drinks a glass of wine an alcoholic. I regularly enjoy MJ - and I think it would be short-sided to consider me a stoner, considering im a successful father of 2 with a great salary and a happy family.
The good news is - the "minority" you speak of is people like you... the outdated mentality is in the minority now, with the majority of the country FOR legalization - even in traditionally red states like TX.
This makes the practicalities of banning marijuana different to those of alcohol but is not necessarily a comment on the ethics of doing so.