Sure, but most people don't and saying that without making sure they know what you mean is at best misleading.
Also, counting the modem as making the whole device non-free would require you to count plenty of other things as well, such as microSD cards, accelerometers, audio codecs, SIM cards, even USB-C cables, as all of those things (and more) contain non-free blobs inside. In my opinion it's not a useful stance to hold if you care about freedom - even FSF doesn't do that.
I agree with you but I think it is better that people realise this and then use the device as ‘as free as we can get without making me a hermit’ instead of just hiding it.
The point is - it is truly 100% free at the level that actually matters: the user controlled operating system. This is the point where I feel completely comfortable with calling it "100% free" without further explanation (especially when the last, and I think the only, smartphone to ever reach that level was GTA02 from 2008), just as I am completely comfortable with saying "I made this cake from scratch" without having invented a universe.
Some people (me as well) do. It is very frustrating that we have this legacy thing where any radio devices are ‘protected’ like this.