On: Creating Artificial Mechanical Turks with Pretrained Language Models

[cuteboy19]

Isn't this just "transfer learning"? Surely there has to be a better way than "momma bird pukes into baby bird's mouth" type of training

[gwern → cuteboy19]

No. Transfer usually means using the same NN model (eg. GPT-3 checkpoints being retrained on Github and then called 'Codex'), or possibly some sort of distillation/sparsifying approach. This is about auto-generating training data, maybe not even meant to be used by a neural net at all.