If you inflate a balloon with a water droplet on it, it's possible that the balloon becomes 20x its initial size, but the water droplet stays "compact" (and does not increase) due to its surface tension.
You can imagine that the space between matter particles is expanding, but since the binding forces between them are much stronger, they come back to the same relative positions. Only at scales where gravitation/electromagnetic forces have little to no influence you will be able to see things drifting apart
