When i was a kid, removing or significantly altering the muffler was the #1 mod to obtain more speed, and then of course there was the psychological effect of making noise to let everyone know you were around, which to me isn't that different to beasts growling loud to mark a wider territory. I guess that is normal for kids, not so much for grown ups.
Yes, 50cc scooters were subjectively as loud as 500cc grand prix bikes and made only 40 km/h. I hated them even when I was 14, the age when I could start driving one.