I just wrapped up a graduate level course in cryptography and the Khan Academy videos and exercises went a very long way to help me grok the math needed to understand the fundamental math. Good exercises on there to explain why probability distributions can break a cryptographic scheme. Very good video explanations on Diffie-Hellman on that site.
