
Is Positive Sentiment in Corporate Annual Reports Informative? - rahimnathwani
http://clsbluesky.law.columbia.edu/2019/08/26/is-positive-sentiment-in-corporate-annual-reports-informative/
======
olooney
Very odd study. The usual reason why people introduce sentiment analysis into
a model is that they don't have enough data or aren't comfortable enough with
NLP to train a classifier against their own labels. Instead they grab pre-
trained sentiment analysis model off the shelf (e.g. Vader) and now the have
the much simpler problem of determining if a binary variable (sentiment
positive/negative) is related some response variable which can be tackled with
statistics 101. This can also be seen as an attempt to leverage transfer
learning because such models are trained on a very large corpus with gold-
standard labels; this can be beneficial when your corpus is too small to be a
representative sample of the entire english language.

But in this study, they manually label and train 8,000 sentences taken from
their 10-K dataset. (A terribly small corpus, but the unsupervised word
embedding step should help with that, at least a little.) Not great, not
terrible. But they label them with _sentiment_ and only later look for an
association with the real variable of interest (abnormal return!) Why the
extra step? If you're training your own model anyway, why not label the data
with the real variable of interest from the start?

I also don't see any attempt to control for the obvious confounding variables,
such as the actual numbers in the 10-K. This leaves open the question of
whether the effect is mediated by the other, more structured information in
the filing. It's entirely possible for sentiment to have some mutual
information with the variable of interest but for its benefit to entirely
disappear once these are taken into account (a phenomenon known as mediation.)
I would prefer to see this a single end-to-end model which is fed both the
unstructured text data and the structured financial information as input
features and the actual performance change as the output. The actual
contribution of the unstructured text data could then be assessed with a tool
like LIME (Local Interpretable Model-Agnostic Explanations.)

------
mazimi1
Using abnormal returns to train a classifier, as suggested by olooney, is not
appropriate in this study for two reasons. First, there is not enough
observations. The classifier labels sentences not the entire report. Each
report contains more than 1000 sentences on average and there are 60,000
reports and 60,000 returns. Second, the purpose of the study is to see if
textual content (in this case sentiment)is related to return. Using returns to
train a classifier would assume that sentiment is related to return, which is
the question that paper tries to answer. Also, the paper does control for the
quantitative information in corporate reports. All the control variables in
the regressions are from the reports.

