You can just write Spark SQL, set the executor memory to whatever the machine is...

threeseed on Feb 12, 2020 | parent | context | favorite | on: Does my data fit in RAM?

You can just write Spark SQL, set the executor memory to whatever the machine is and not worry about whether it's in RAM or not.

Spark will naturally use RAM first and then disk as needed.