I'm not going to argue that everyone should use Virtualmin or any other control panel, and I'm sure if I were trying to fit my websites into a 64MB system I wouldn't be using a control panel. But, I do want to mention that Virtualmin itself can be configured to use approximately 11MB of RAM (though it is an application server that can cache libraries and modules, and can be as much as ~130MB, with about 20MB resident, in a large system with all caching features enabled). The services being managed are, by far, the biggest consumers of resources.
You can no more build a full-featured virtual hosting stack with email (and spam/AV scanning), web, databases, mod_fcgid or fastcgi with suexec, DNS, etc. in 64MB without a control panel than you could with a control panel. Likewise, if you just want web service, you can build one with Virtualmin in X+11 MB of RAM (where X is the amount of RAM the system and web server require).
Of course, Virtualmin (or cPanel or Plesk) are not targeted to very small deployments. They're for shared hosting, which tends to be on a larger system where a few MB for the control panel is of negligible impact.
I just want it to be clear that "control panel == memory hog" is not really accurate. It's probably all the stuff that users expect from a control panel-equipped system that makes it require more memory. ClamAV, for instance, requires more than 200MB. MySQL for a single moderately popular website can consume 300MB. Apache can grow as big as your imagination, because there are so many modules people can enable. A thoroughly stripped Apache isn't all that big, but the average Apache process is a couple hundred MB.
I'm running nginx, PHP-FPM, MySQL, NSD, exim, UnrealIRCD, ZNC, and thttpd for CGIs on my 128mb VPS, and it uses around 50mb RAM. It only goes up to over 64mb used when my file/database backup script is running. The only thing I don't run on the VPS is incoming email. (I use Google Apps's free version)