What makes the Sentiment Treebank so novel is that the team split those nearly 11,000 sentences into more than 215,000 individual phrases and then used human workers via Amazon Mechanical Turk to classify each phrase on a scale from “very negative” to “very positive.”
Can someone here please explain whether the use of Mechanical Turk here is a cop-out from building a better computational model, or just an ordinary use of supervised learning in place of unsupervised?
This is just an ordinary use of supervised learning; they just used Mechanical Turk instead of undergrads. The point of the human classification was to generate the labeled data for model training.
It's neither, I think. It's basically a way to get training data. The novel part is splitting sentences into phrases and asking about the sentiment of each part, which provides the algorithm richer data to train off.
- certain combinations of words within phrases score all over the place
- hand those to mechanical turk for human classification
- understand where the results differ from the model
- patch the model where necessary when it breaks down.
The example they gave with the "but..." at the apex of the sentence is difficult primarily because it's ambiguous to what proceeds it. It could be positive or could be negative, especially from a programmatic standpoint.
Really fascinating stuff. Can't wait to see the code.
No, wait. So as someone who isn't remotely well-educated on machine learning, what teaching model would this be called? "Reinforcement learning" is an AI term, but it's more what this sounds like: sometimes the model is wrong, so we hand the data off to a human being who comes up with a definitive Right Answer which is then used to fix the model.
What "landongn" describes, where certain data 'score all over the place' - i.e. where the model tells you it is uncertain about some phrases, and where those phrases are subsequently manually annotated by a human, sounds like 'Active Learning'. [0]
If the model is telling us that it is uncertain about specific examples, and would like more information on examples like those, that's active learning.
That sounds different from what you describe in your post, depending on what you mean by 'sometimes the model is wrong, so we hand the data off to a human being'.
It depends on how we know the model is wrong.
If we know its wrong on a test datum, which is part of a big set of test data humans labelled without any input from the model, then its standard 'supervised learning'.
If, instead, the model is 'wrong' because it expresses uncertainty for particular test data, then, if we go and have a human classify that data it was uncertain about, and retrain the model, then we are probably doing Active Learning. In this case, the model/system is (at least partly) guiding the learning process.
Reinforcement learning is neither of these things exactly - it describes a more general framework, where the system is getting rewarded based on how well its performing.
Lets say you want to choose 1 of 5 labels for each datum.
In supervised learning, the system gets given the right label for each training example.
In a RL setup, it might be shown an example, have to guess a label, and maybe be told if it got the right guess, but if it guessed wrong, just told it was wrong - but not necessarily told what the right answer was.
There's a little fuzziness to how all these terms are used in practice.
Before signing up for an online course I'm pretty set on starting my new semester at Technion and figuring out what my Real Life We Will Count This Against You workload is.
Can someone here please explain whether the use of Mechanical Turk here is a cop-out from building a better computational model, or just an ordinary use of supervised learning in place of unsupervised?