
Oxford Deep NLP – An advanced course on natural language processing - melqdusy
https://github.com/oxford-cs-deepnlp-2017/lectures
======
jkbschwarz
Im taking this course at Oxford and they have been working through the
practicals 1-3 (further ones will be posted)

For anyone considering working through this outside of Oxford: I think the
practicals are the real gems here and should be doable without the practical
lab sessions that you get when attending the course. With that being said,
they use a dataset a bit closer to a real world assignment. Therefore, it
requires some patience when wrangling the data especially for the later
practicals.

However, the patience should pay off and it is rewarding once you build your
own nonsense spewing TEDbot!

~~~
axplusb
Regarding practicals: is there anything more than the overview slides ?

~~~
mcintyre1994
The org : [https://github.com/oxford-cs-
deepnlp-2017](https://github.com/oxford-cs-deepnlp-2017)

That contains repos for practicals, eg. Practical 1:
[https://github.com/oxford-cs-
deepnlp-2017/practical1](https://github.com/oxford-cs-deepnlp-2017/practical1)

------
demonshalo
here is what I don't understand about deep NLP (please keep in mind that I
just began exploring this field):

I am currently working on an algorithm that uses elementary text cues in
combination with large data-table lookups to determine things like relevant
keywords of news articles scraped from various sites. I have given my results
to hundreds of people independently to provide me with some feedback regarding
the quality. Here is the current breakdown:

80% of the cases I get perfect score.

10% of the cases I get acceptable score.

10% of the cases needs improvement.

My questions here are:

1\. if deep nlp can only provide us with the same level of
efficiency/accuracy, then why the hell would we use it?

2\. if deep nlp can provide us with more efficiency than what is stated above
then wouldn't it be safe to assume that is UNREASONABLY efficient?

3\. why are most people using deep nlp or ML in general right off the bat.
Theoretically, it would be far more interesting to construct a model where the
result of a statistical/linguistically parsing is fed to some sot of ML algo
in order to tackle that 10% of bad cases.

~~~
jventura
I've worked in the NLP research area (mostly with statistical metrics), and I
can safely say that 80% (precision) is the empirical threshold that most
metrics are able to reach quite easily. The threshold between 80% and 90%
starts to get difficult and above that you have to do some tweaking to adapt
to the specifics of your problem.

With that said, your values do seem to be in line with what I consider to be
easily reachable, so it kind-of depends on how much work you need to do with
the neural networks to extract those keywords. I'm not very knowledgeable on
how NN are applied to this field, but I'm assuming that a drawback of that
approach is that it may resemble a black-box in the sense that it may be hard
to tweak the internals.

I prefer statistical metrics because they seem more simple to derive. For
instance, you can think of things like "a relevant keyword is usually related
with (or closer to) other relevant keywords" and you can test that hypothesis
only by counting distances between words. This is what I've done in 2012 with
quite good values, you can check the paper here:
[http://www.sciencedirect.com/science/article/pii/S1877050912...](http://www.sciencedirect.com/science/article/pii/S1877050912001251)

~~~
jamra
I believe that NN provide the logical relations for you so perhaps they can be
used to train and then inspected to understand the data better.

~~~
jventura
As I said, don't know much about deep NN, but what I can remember of neural
networks from my college years, each "node" on the network only have weights
associated with the inputs, which makes it a blackbox. In other words, it is
not easy to "grab" a node from the network, check the weights associated with
the inputs and understand how it relates to a language..

------
seycombi
YOUTUBE-DL will download the lectures [https://rg3.github.io/youtube-
dl/](https://rg3.github.io/youtube-dl/)

~~~
jdemler
Thank you. I just wanted to complain about the need for the adobe flash
player.

------
hmate9
I am currently taking this course at Oxford and definitely recommend following
this.

We will be using TED talks as our dataset, to create Question Answering, text
completion, generating entire TED talks ourselves etc. Definitely very
interesting and it is being taught by leading researchers in the field!

------
roystonvassey
Began this course earlier today and I think they appear to be pulling off the
right combo of first principles foundation and tough problem sets, like cs224n
(Karpathys CNN class). Other NLP courses that I've taken so far have gone over
my head.

------
orthoganol
> The primary assessment for this course will be a take-home assignment issued
> at the end of the term. This assignment will ask questions drawing on the
> concepts and models discussed in the course, as well as from selected
> research publications.

Comes as a surprise that it's not a project, as, in my experience, all ML/ DL
courses I've seen online from US universities (Cal, Stanford, etc.) require.
Different university culture across the pond?

~~~
NPDW
Oxbridge is very much exam based assessment. With your degree depending only
on the exams taken at the end of the final year. So the take-home assessments
are there just to make sure that you understand what you're doing and can
progress to the next part of the course.

~~~
arethuza
I must say that I am rather glad the degree I did at a Scottish university was
based mainly of 4th year exams, with an element of 3rd year exams and
coursework.

The idea of a averaging _all_ of your work across your course would have been
a disaster for me as I did rather poorly in year 1, scraped through year 2 and
did spectacularly well in years 3 and 4.

~~~
tonyedgecombe
I'm not sure about Oxford but certainly for Cambridge you aren't graded
overall, instead you get a grade for each part of the tripos[1] (two parts).

[https://en.wikipedia.org/wiki/Tripos](https://en.wikipedia.org/wiki/Tripos)

~~~
NPDW
Yes but you generally treat your degree as being graded by whatever you got in
part II

------
melqdusy
Stanford's version
[https://web.stanford.edu/class/cs224n/](https://web.stanford.edu/class/cs224n/)
Note: the videos will be available later.

~~~
wlrd
Any idea when the lectures will be up?

~~~
julien_c
Last spring's session's videos (so 9 months ago) are up on Youtube:
[https://www.youtube.com/playlist?list=PLmImxx8Char9Ig0ZHSyTq...](https://www.youtube.com/playlist?list=PLmImxx8Char9Ig0ZHSyTqGsdhb9weEGam)

------
Fede_V
If you want more advanced materials, both Kyunghyun Cho and Yoav Goldberg
posted excellent notes:
[https://arxiv.org/abs/1511.07916](https://arxiv.org/abs/1511.07916) and
[https://arxiv.org/abs/1510.00726](https://arxiv.org/abs/1510.00726)

------
kalal
'Advanced' course with 'Sesame Street' introduction:
[https://ox.cloud.panopto.eu/Panopto/Pages/Viewer.aspx?id=b7d...](https://ox.cloud.panopto.eu/Panopto/Pages/Viewer.aspx?id=b7d66d78-0deb-46d5-bc14-b1852b9d95e8)
One of them should go :)

------
mailshanx
How does the course content and rigor compare to the Stanford deep learning
for NLP course? From a cursory glance at the practicals, it seems like the
Stanford version has more variety and depth of problems.

~~~
roystonvassey
If you're well-versed in DL + nuances of text-mining, then the Stanford one
might be great. But, if you're a beginner like me, you'll find it a bit hard
to catch up [1]. The Stanford assignments + Tensorflow class were very
helpful. The assignments especially really get you thinking.

[1] I took the Stanford one and went through the videos (both 2015,2016).

------
webmaven
Link should probably be to the org rather than a specific repo:
[https://github.com/oxford-cs-deepnlp-2017](https://github.com/oxford-cs-
deepnlp-2017)

------
RegW
"Prerequisites: This not meant to be an introduction to Machine Learning
course. Hopefully you've all got some knowledge to machine learning, otherwise
you may find this a bit opaque. So at least you should understand/have taken
courses in linear algebra, calculus, probability, ... we are not going to do
anything particularly challenging in those areas, but ideas from those areas
will be useful."

around 7 mins 30secs into the introduction

~~~
cr0sh
Those sound like standard pre-reqs for any ML course - it should probably be a
standard disclaimer on every course except for intros (even there, though,
you'll want the linear algebra and probability knowledge - at least the bare
basics). I can only imagine the number of people who jump into such courses
and get in over their head very quickly...

------
alfonsodev
Here [1] is an example of the videos, the player has a handy search feature
and links to video parts.

 __update: __It would be great to have a way to take your own notes, any
chrome extension that can help with that ?

[1]
[https://ox.cloud.panopto.eu/Panopto/Pages/Viewer.aspx?id=ff9...](https://ox.cloud.panopto.eu/Panopto/Pages/Viewer.aspx?id=ff91caf5-fa7c-42de-8b3d-41f4bc2365b4)

~~~
javiermares
Where could I find a list of links to the videos?

~~~
alfonsodev
On the README [https://github.com/oxford-cs-
deepnlp-2017/lectures/blob/mast...](https://github.com/oxford-cs-
deepnlp-2017/lectures/blob/master/README.md)

~~~
javiermares
Thanks!

------
option_greek
What are the practical uses of language modelling RNNs ? (apart from writing
grammar/syntax checkers)

~~~
spangry
I wonder if they could be used to improve speech recognition accuracy. So
you'd have two models running when someone utters a sentence: the first would
generate the x most likely phrases that it thinks were spoken, and the second
(the RNN) would select the highest ranked 'plausible' sentence (i.e. a
sentence it would have been able to generate itself).

I guess that's a bit indirect, but these RNNs are essentially learning the
'rules' that actual phrases conform to. It'd definitely be better than trying
to hard-code the rules (especially for a language like English!). And the
training data is very easy to get: just feed it a few thousand ebooks, the
comments section from HN etc.

~~~
bshillingford
Standard neural network-based speech recognition pipelines (i.e. RNN + CTC)
always use a language model. Unlike a seq2seq model (or any autoregressive
model, or a structured prediction output), CTC models output timesteps as
conditionally independent. Hence, everyone uses an RNN LM or n-gram LM or both
when retrieving probable sequences from a CTC model (e.g. with beam search).

------
ejanus
Is there any good introductory material? I have tried severally to understand
the theory and the spirit of DL and ML, but I have not been able to connect
the dots. Please direct my path.

~~~
imakecomments
Calculus, Linear algebra, Statistics, Probability theory. You won't get far
studying "basic" ML without an elementary understanding of those subjects.
Unless you blackbox the implementations and hack them, but you'll have trouble
understanding why things work/when to use certain techniques vs others or how
to tune for optimizations.

------
pratap103
I'm reading 'Deep Learning' right now so this is going to be really useful.
Thanks a lot!

~~~
snowcrshd
Who's the author? I'd like to learn more about deep learning techniques, so I
appreciate any suggestions on introductory materials!

~~~
alexpw
The book 'Deep Learning' generally refers to deeplearningbook.org, though
there are others by that name. Goodfellow, et al, I think of as the canonical.

------
hoju
Darn - I did an MSc at Oxford last year and they didn't offer this course then

------
mcintyre1994
This looks amazing, thankyou for sharing!

------
mrcactu5
coookie monster and the fairy keep exchanging apples <\----> bananas

how does this help me solve NLP

------
jray
Edit.

