Hi HN,
I wrote this and wanted to share a few more words.
This post discusses a trick for training neural networks. Usually when we train a NN, we just show it what the correct answer is, but we don't say anything explicit about the wrong answers.
This trick, Complement Objective Training, is a way to give the model "feedback about its mistakes". In practice, it makes the model better and train faster.
The important thing is that it's pretty easy to implement. Many cool research ideas are a months work, but this takes 15 minutes with pytorch lighting so it's in the realm of low hanging fruit.
So they don’t even try to explain why splitting it into two steps works better? I wonder if anyone tried splitting knowledge distillation optimization like that.
The important thing is that it's pretty easy to implement. Many cool research ideas are a months work, but this takes 15 minutes with pytorch lighting so it's in the realm of low hanging fruit.