You can no more build a full-featured virtual hosting stack with email (and spam/AV scanning), web, databases, mod_fcgid or fastcgi with suexec, DNS, etc. in 64MB without a control panel than you could with a control panel. Likewise, if you just want web service, you can build one with Virtualmin in X+11 MB of RAM (where X is the amount of RAM the system and web server require).
Of course, Virtualmin (or cPanel or Plesk) are not targeted to very small deployments. They're for shared hosting, which tends to be on a larger system where a few MB for the control panel is of negligible impact.
I just want it to be clear that "control panel == memory hog" is not really accurate. It's probably all the stuff that users expect from a control panel-equipped system that makes it require more memory. ClamAV, for instance, requires more than 200MB. MySQL for a single moderately popular website can consume 300MB. Apache can grow as big as your imagination, because there are so many modules people can enable. A thoroughly stripped Apache isn't all that big, but the average Apache process is a couple hundred MB.