
Predicting Stack Overflow Tags with Google’s Cloud AI - amrrs
https://stackoverflow.blog/2019/05/06/predicting-stack-overflow-tags-with-googles-cloud-ai/
======
yeldarb
Slightly off topic but I’ve been playing with this dataset for a while now to
learn ML. It’s remarkably humbling.

No NLP approach I’ve tried has been able to predict question score based on
content better than my baseline of “choose the mean”. (I’ve tried random
forest on bag of words, AWD-LSTM, and Google AutoML so far).

The author of this post tried score prediction as well and pivoted to tag
prediction after she couldn’t find anything that worked well:
[https://twitter.com/srobtweets/status/1125860523377979398?s=...](https://twitter.com/srobtweets/status/1125860523377979398?s=21)

It’s so crazy to me that which posts get popular might just be random. And
makes me wonder about the correlation between post content and popularity on
other social sites like HN and reddit.

~~~
AlexCoventry
It's not that it's random, it's that the signal is too abstract for such
simple methods to grasp it. Machine Learning which can predict Stack Overflow
question scores is going to have to understand what's being asked, and how
useful the responses are going to be for people interested in the question.

~~~
djakjxnanjak
It may seem obvious with the benefit of hindsight, but I would have guessed
that the better answers might be different in terms of word choice or style.

~~~
AlexCoventry
I'm not saying it was a silly thing to try, just that it's wrong to conclude
from the result that the scores are random. I.e., I was mostly responding to
"It’s so crazy to me that which posts get popular might just be random."

~~~
yeldarb
I’m not convinced they are randomly distributed but before I started playing
with it I was fairly certain they weren’t.

Now all I know is I haven’t seen evidence that they aren’t.

I could see how there could be some element of luck in whether a post gets
attention before getting buried. It depends who’s online when it’s posted and
whether it piques their interest, and many other random factors that don’t
have anything to do with the content of the post like how quickly other posts
come in afterwards to bury yours, and how any automated algorithms decide to
surface it.

------
Darkphibre
I've used simple TF/IDF with a custom stemmer as a similarity scoring tool
against the XDK documentation. It worked pretty well, I'd be curious to know
how that'd fare vs. the neural model (i.e. I suspect it'd be better in
identifying the rare, high-signal words that were excluded by the author's
400-most-common word limit).

For my side project: As we received emails from developers asking for
clarification/help with the APIs, the system would provide relevant
documentation URLs so that anyone could pick up an inquiry and brush up on the
API (and have a handy link if the the docs could be leveraged in the
response).

------
btgeekboy
Between this and the Pi calculation "achievement" it appears Google's all-in
on the grassroots marketing these days.

~~~
boulos
Disclosure: I work on Google Cloud.

Both Emma and Sara are in our Developer Relations (aka Dev Rel) organization,
like Kelsey Hightower and Felipe Hoffa.

They’re explicitly not in Sales, and their job is focused on explaining and
demonstrating stuff to Developers. They go to meetups, give invited talks,
write blog posts, and so on.

Sorry if that doesn’t come across as clear. They work for Google, are paid by
Google for that work, but aren’t measured by revenue or anything.

~~~
manigandham
That’s called marketing. Also sometimes pre-sales with solutions engineering
and onboarding.

~~~
boulos
No, we have those functions, too :). It really comes down to how people are
measured for their work, and how it's directed. Nobody told Emma or Sara to do
the things they did. Nor is anyone going to go see "how much pipeline got
generated".

DevRel is measured on how many people they reach, but even that only vaguely.
Marketing events, by contrast, are more often measured by pipeline. And you
can be certain that onboarding (Professional Services) and Sales/Solution
engineers are measured on revenue and time to revenue.

------
gridlockd
Out of curiosity, does anyone ever use tags on Stackoverflow? Does anybody use
search on Stackoverflow? Does anybody ever use in-site search on _any_
website, instead of just using Google?

~~~
inglor
I'm a "heavy" Stack Overflow user
[https://stackoverflow.com/users/1348195/benjamin-
gruenbaum](https://stackoverflow.com/users/1348195/benjamin-gruenbaum) and I
use tags all the time.

Otherwise you're just playing "fastest gunslinger in the west" trying to
answer generic stuff before anyone rather than sniping questions you genuinely
find interesting and can teach you.

~~~
IshKebab
I think there are two kinds of SO users - those that ask questions when they
have problems, and answer questions when they solve them; and those that find
questions to answer (for internet points, kudos, product support etc). Tags
mainly help the latter.

------
onurcel
If you want to experiment with question tag prediction on your laptop, you can
also play with fastText : [https://fasttext.cc/docs/en/supervised-
tutorial.html](https://fasttext.cc/docs/en/supervised-tutorial.html)

------
ACow_Adonis
Kaggle ran a facebook recruitment challenge trying to do something similar a
few years ago for those interested (I am because I participated in it):

[https://www.kaggle.com/c/facebook-recruiting-iii-keyword-
ext...](https://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction)

I'm in a waiting room right now, but is anyone interested in summarising or
commenting on the differences/ gains/ losses between the performance and
aspects of comparing the two? :)

------
lettergram
For those interested, I’ve done similar work in the past. I’ve also written a
guide along with explanations of how it works (on sentence classification):

[https://github.com/lettergram/sentence-
classification](https://github.com/lettergram/sentence-classification)

It uses Keras and goes through everything from encodings, to the way various
networks function, to hyperparameter tuning.

------
milad_nazari
Similarly, predicting issue tags on Github would be interesting.

------
aficionado
Links to dataset do not work.

~~~
turtlegrids
Not sure why you're being downvoted. I also cannot get to at least one dataset
-[https://storage.googleapis.com/cloudml-demo-
lcm/SO_ml_tags_a...](https://storage.googleapis.com/cloudml-demo-
lcm/SO_ml_tags_avocado_188k_v2.csv) which is linked to from the fourth
paragraph directly above the "What Is The Bag Of Words Model?" section
heading.

~~~
boulos
I pinged Sara. Probably just forgot to make it publicly accessible.

~~~
turtlegrids
Thanks!

