There are tons of problems that need to process large data, but touch each item just once (or a few times). You can go a really long way by storing them in disk (or some cloud storage like S3) and writing a script to scan through them.
I know, pretty obvious, but somehow escapes many devs.
Needless to say I wasn't recommended for the job, and it taught me a valuable lesson: if you don't first give them what they want, you can't give them what they actually need.
As soon as you're touching it more than once, sticking it in RAM upon reading makes everything much faster.