I think acting as if peanuts are actually nuts for purposes of communication is much more defensible than acting as if tomatoes are vegetables, in short you are dying on a hill that was paved over long ago.
I agree most people will conflate them, but someone who's allergic to peanuts but not tree nuts (or vice versa), i.e. the people the labels are intended for, are going to care about the difference.
