For an AI noob like me: can you use spot instances to train models? They are about 1/3rd the price on AWS compared to on demand ones, so it'd make a significant difference.
Yes you should use them. They can be taken away from you with 2 min notice. (It doesn't happen a lot in practice though. I have been running a different instance for over a month. AWS doesn't force you if they don't have to)
If you are going to run a long training job, ensure you are creating checkpoints. Be sure to use persistent storage, EBS and ensure that you check the option that it doesn't get deleted if the instance is stopped, so your checkpoint remain in the disk and you can easily restart.
Yes you can. In Oregon you could eventually get this instance at $9. I say eventually, because of course Spot allocation is not guaranteed. ( And neither is On Demand ...but that is a story for another day)
