You can go a fair bit smaller, but the primary size driver is the optics. Pancake lenses, which combine refraction with reflection, allow the depth to be much smaller while having similar optical power. However they cause image artifacts and have really poor light transmission, so it takes a good amount of R&D to get them good enough. A traditional refractive system, especially for a high-fidelity device, is a better choice for now than optimizing for size. That can come later.
