The hype for neural networks is deserved. Some major contributions to the field resulted in increases in accuracy for fields like NLP, computer vision, structured data, machine translation, style transfer, etc.
XGBoost did not change much from the "Greedy function approximation: A gradient boosting machine." paper, but uses a few tricks to be much much faster, allowing for better tuning.
I'd say Tensorflow/Keras can handle a wider variety of problems, with the same, or improved accuracy, than tree-based methods can. NN's do well on structured problems (the domain of tree-based methods), but also own computer vision and, increasingly, NLP. I agree on the pitfalls of applying neural nets vs. forests.
It is true that tree-based methods are academically a bit out of vogue: The exciting stuff is happening in the neural network space. You have more chance of getting published with deep learning (this used to be the other way around).
I agree that neural nets are state-of-the-art and do quite well on certain types of problems (NLP and vision, which are important problems). But a lot of data is structured (sales, churn, recommendations, etc), and it is so much easier to train an xgboost model than a neural net model. You need a very expensive computer or expensive cloud computing to train neural nets, and even then it is not easy. Ease of implementation is an important factor that gets overlooked in academia. And on non-NLP and non-image datasets, usually the single best Kaggle model is an xgboost model, which was probably developed in 1/10th the time it took to make a good neural net model. Xgboost has come a long way since it was first introduced, with early stopping being an example of a significant improvement.
Academia does not have to run the model in production. It has few computational constraints, and most datasets do not have a feedback loop, requiring combating drift, debugging, and retraining. Papers are often accepted when they equal or beat state-of-the-art. Not many academics have to deal with the business side of running models in prod.
All of this leads to ease of implementation being overlooked. Especially on NLP, you see a lot of overengineering with deep neural nets (where the feature engineering is hidden inside the architecture). These models are hard to implement/reuse.
But yeah: academia/theoretical machine learning creates the very tools for applied machine learning.
XGBoost did not change much from the "Greedy function approximation: A gradient boosting machine." paper, but uses a few tricks to be much much faster, allowing for better tuning.
XGBoost is popular for structured data competitions on Kaggle. Even there: The winner is often an ensemble of XGBoost and Keras. And some structured data competitions are won by neural nets alone: http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it... and https://www.kaggle.com/c/higgs-boson/discussion/10425 (Neural nets won the Higgs Boson Detection Challenge, where XGBoost was introduced)
I'd say Tensorflow/Keras can handle a wider variety of problems, with the same, or improved accuracy, than tree-based methods can. NN's do well on structured problems (the domain of tree-based methods), but also own computer vision and, increasingly, NLP. I agree on the pitfalls of applying neural nets vs. forests.
It is true that tree-based methods are academically a bit out of vogue: The exciting stuff is happening in the neural network space. You have more chance of getting published with deep learning (this used to be the other way around).