Background: I have ADHD so I have to force myself to not jump in at the deep end whenever I hear of something novel and cool.
After getting licensed, you'll probably get frustrated with the UV-5R and look into a more capable hand held unit, or a mobile radio. These start at around $150 and go up from there (plus antennas, power supplies, etc).
Can I read somewhere about which frequencies are used by ham operators, and descriptions of things like UHF and VF and narrowband and such?
I roughly split spectrum in my head as "longer waves" (kHz, AM radio, needs impractically large antennas), "HF" (thousands of miles of range, largish antennas), "VHF/UHF" (line of sight, 25 miles, walkie talkies), and "microwaves" (1GHz+, short range, tiny antennas for wifi).
Someone already mentioned the Baofeng style of cheap Chinese radio. They're good enough but they also produce pretty dirty signals that there's a chance they aren't legally compliant. That being said, everyone seems fascinated with them right now so they get away with it for the moment. I wouldn't let that stop you since if you stick with it, you'll move up soon enough and it'll become the backup/beater radio.
I'll add on that $25 RTL-SDR (get a "TXCO" one like a Nooelec Blue) is also a pretty cheap entry into listening to absolutely everything. With that, you can start making antennas, listening to satellites, running your own FlightAware ADS-B scanner, etc. To listen to HF, you can add a $75 upconverter.
Will do wonders for what you'll be able to hear(and send). I've got a similar Diamond one on my VX-8DR and I can pick up the repeaters ~20mi away. Can't hit them though, but that's what cross-band is for :).
I'm able to run a pretty resonable APRS setup w/ BF-F8HP + above and a mobilinkd for ~$120.
This is as cheap as it gets, given that you've already got internet access and (I assume, not sure if it works on mobile - theoretically it should, it's all HTML5 now) a capable PC.