How do you figure? It would look like people from all over the place running API workloads on a wide range of tasks and topics (if the goal is to distill the model generally, that’s what you’d need). So many people are already using GPT-4 to synthesize fine-tuning datasets, I think it would be invisible.
However, you might wonder what the goal is. This “API distillation” is good for teaching a pretrained model how to do lots of things. But the end result is always constrained by the quality of the pretrained base model, and API outputs don’t help at all with that part.
However, you might wonder what the goal is. This “API distillation” is good for teaching a pretrained model how to do lots of things. But the end result is always constrained by the quality of the pretrained base model, and API outputs don’t help at all with that part.