If you're going to be pedantic at least do it well. Going by the dictionary definition of drug: "a medicine or other substance which has a physiological effect when ingested or otherwise introduced into the body" both caffeine and cannabis fit the bill.
Cannabis is a plant that contains cannabinoids that are drugs. As a chemist, I typically don't refer to unrefined plant material as a drug. Because people don't typically consume caffeine raw, most people don't consider coffee consumption "doing drugs"; so, I believe cannabis deserves the same consideration.
Language matters, and I don't consider it to be pedantic to make this point (however subtle the distinction is).
Edit: And, for reference (since you missed the definition of cannabis):
Cannabis is a genus of flowering plants that includes three different species, Cannabis sativa, Cannabis indica and Cannabis ruderalis.
