The mechanical driver part of the subwoofer might be able to be made smaller, but the surface that moves the air still needs to be large - there is a relationship to the wavelength of the sound involved that I no longer remember. This is why tweeters are small.
Tweeters are small to allow them to change direction more quickly, since they have a much smaller mass. Changing direction more quickly means being able to reproduce higher frequencies with less distortion. (If you have a very massive driver trying to reproduce high frequencies, you can try to picture the driver physically distorting when trying to move back and forth too quickly.)
Larger drivers can generally handle more power, and lower frequencies require more air movement (requiring more power) to be perceived at a comparable level to higher frequencies. An 18” woofer can move much more air than an 8” woofer. The design of subwoofers is of course much more complicated than just “use a bigger woofer” but it makes the task much easier.
Another way to think about it - 2-3” headphone drivers are capable of reproducing bass and subsonic frequencies that are perceived at a reasonable level. This is because they’re right next to your ear, so there is very little air that must be moved to reproduce those frequencies. Same with teeny tiny IEM drivers, which require even less power. Larger drivers just make it easier to design a subwoofer that can move enough air to reproduce those frequencies in a room.
