That's an interesting question which we haven't studied so far. There is a good argument to be made that ML models trained on public data do not work well on private data on the interesting queries, where I consider those queries interesting that only make sense in the specific project. The core issue there is around the specialised vocabulary that you would only see in the (private) project, which the model would be unfamiliar with. This could be mitigated by using absurd amounts of data, models that can generalise more easily (character-level/BPE/subtokens/...), and finetuning on project-specific data, but it's an open question nonetheless.
It's also easy to test without access to private repos, by just splitting some repos off as a separate test set, so should hopefully be something that other people can also make progress on!
You could test on GH and MS's own private repos to get some idea of the performance. These might still not be representative but it would be an interesting data point.
It's also easy to test without access to private repos, by just splitting some repos off as a separate test set, so should hopefully be something that other people can also make progress on!