It depends on what the BERT was pre-trained on. If it was pre-trained on Google News Collection like most Transformer models are, fine-tuning on something like scholarly data won’t perform so well due to vocabulary mismatch. Besides as the author mentions training models like LightGBM are efficient and perform reasonably.
I agree with this.
In my experience, LightGBM is often good enough. The additional improvement provided by neural net models is not worth the substantial increase in development time.
