What would be some practical alternative model architectures one could use? I've seen EfficientNet be used fairly often (apart from ResNet/ImageNet etc.) in transfer learning.
The answere is unfortunately "it's complicated" and "it depends on the task". Very large EfficientNet variants are impractical anyway. For detection I've been very impressed with https://www.paperswithcode.com/method/vovnetv2. You can use "plain" convolutions when you need higher accuracy and depthwise separable convolutions when you need speed, at the expense of a relatively small accuracy drop. Generally GPU hardware prefers the kind of nets where it can "stretch its legs" so to speak. VoVNet does have more parameters, but it's _far_ easier to train to practical levels of accuracy, and it does well on NVIDIA hardware, and especially under TensorRT. Note that you can plug in VoVNet2 as a backbone into just about any detector, it doesn't have to be CenterNet. For mobile GPUs MobileNet V3 is still adequate for a lot of things.
If anyone from NVIDIA is reading this, I feel like this is a good, and under-explored area of research that'd be very tractable for you: figure out architectures custom-designed to do well on recent NVIDIA GPUs, and especially on Jetson Xavier.
One thing people don't realize is that EfficientNet/EfficientDet aren't necessarily the best choice _for their specific dataset_. In a way, a lot of these academic networks are overfit to the task of e.g. detecting objects in MSCOCO. If your dataset doesn't look like MSCOCO, there's no guarantee whatsoever that they will do well on it. Same with ImageNet for classification. ImageNet is very hard. To do well on it your net has to do something most humans won't be able to do without substantial training - recognize the various dog breeds. If your problem is simpler (which nearly all of them are), chances are you don't need as complicated a model to do well on it. Indeed, a "complicated" model is likely to actually do worse than a model that's "just complicated enough". Due to e.g. overfitting, or being more sensitive to noise in real-world data, and so on. Not to mention it will naturally limit your experiment throughput, which is one of the most important factors for getting a good model that does something practical.
I appreciate your answer and your take on the subject, I'll definitely check VoVNet2 out. I think the practice of taking whatever model is currently achieving SotA results and fine-tuning it on one's data is bringing results, but at the same time it makes it too easy to just blindly apply those models.
I think part of the problem is that it's currently too easy to reach for the big models - why train one from scratch where you can just tweak a few layers and get better results.
That's the thing, if your dataset is not similar to the dataset on which your model was pre-trained (which it often isn't), the initial weights may actually prove to be either worthless, or even detrimental. That is, they may be worse than random initialization.
