My grandfather did that in the 50s with my dad helping a little. You need to scale for sq footage, the average American house has increased from 1000 sq ft to something unaffordable recently. By sq foot its probably not so much of a deal.
Apparently it takes only a couple weeks for one man to build a house, but the labor involved in general contracting to get the subcontractors to do their thing exceeds the time spent swinging a hammer. He sub'd out to dig and pour the foundation, some electrical, gas, some plumbing...
For those of a certain generation it was how you proved to the employer that you could complete long complicated projects, basically the equivalent of completing your college degree (which was hard to complete during WWII)
This is still done on a regular-ish basis for large garden sheds and garages. Home Depot and others will dump a pile of wood at your location with some plans and bulk pack of nails and other hardware and a weekend or two later you'll have a garden shed.
The permitting and inspection process is more about revenue generation than safety.