One of the most insightful comments I have seen about deep learning for NLP is a reddit r/machinelearning comment seen in this post . As the poster in that thread mentioned, much of the theory about using neural networks for NLP tasks was laid out pretty clearly in a paper by Yoshua Bengio from 2003 . At a high level, the basic learner is very similar to many of the other deep architectures for images and audio.
There is also a pretty great blog post  by Radim Rehurek (author of gensim) about how he implemented word2vec (a "deep learning"-style model for text) in Python, while also getting a performance improvement over the C version!
Any resources to learn that stuff?
I thought the introduction to neural networks from Andrew Ng's coursera course (even though it meant writing MATLAB) was quite good, and allows you to implement backprop, cost functions, etc. while still having some other helper code to make things easier. I highly recommend working through that course if you are intersted in ML in general .