
Botrnot: an R package for detecting Twitter bots - tysonzni
https://mikewk.shinyapps.io/botornot/
======
dexen
I was surprised by the number of false positives reported here, went ahead and
tested on several Twitter accounts of my friends, both professional and
personal. 9 out of 25 tested were classified as 'bot' with probability > 0.6.
Only 11 were classified as 'humans' with probability > 0.7. And that's on 25
accounts of people I know personally.

Given the preposterous error rate, I deem there is no actual classification
logic in R, and instead it uses the (very fallible) humans to do the actual
classification via a Mechanical Turk-style API.

Wait, do I need to add "/s" to this post, or is it obvious enough?

~~~
peterhi
Same here. I am a bot with 84.7% confidence

Dread to think how they came up with this

~~~
na85
83% chance that I'm a bot, apparently. Neat idea though.

------
nl
I've done some work in this area - it's disappointing how terribly the Twitter
product has failed to evolve to take account of bot usage.

There are (of course) some useful bots, but lots of incredibly harmful bots,
they they should be treated differently to actual humans.

But Twitter can't ship product, so it's not really worth suggesting what they
should do.

In the mean time, my colleagues and I got a nice WWW18 conference paper about
a new _unsupervised_ (!) way of detecting some type of bots on Twitter. Like
most things it's completely obvious in retrospect...

~~~
tysonzni
Can you share the paper on the new method?

~~~
stingraycharles
Seconded, I would be very interested in this paper!

EDIT: Perhaps it's this one ?
[https://arxiv.org/pdf/1711.09025.pdf](https://arxiv.org/pdf/1711.09025.pdf)

~~~
nl
No it's not.

That's supervised for one thing. Also this is bot detection, not attempting to
detect fake news.

I'm not the first author and I'm not sure what the convention is for pre-
prints at WWW.

It's the kind of thing I'd love to talk about because it's so damn amazingly
effective, but it isn't really my place to say anything.

------
dsacco
I looked at the GitHub README for the project, which says

 _> Uses machine learning to classify Twitter accounts as bots or not bots.
The default model is 93.53% accurate when classifying bots and 95.32% accurate
when classifying non-bots. The fast model is 91.78% accurate when classifying
bots and 92.61% accurate when classifying non-bots.

Overall, the default model is correct 93.8% of the time.

Overall, the fast model is correct 91.9% of the time._

How is this accuracy determined? There is no information available explaining
how this determination is quantified, nor what the caveats are.

~~~
minimaxir
From the code
([https://github.com/mkearney/botrnot/blob/master/R/features.R](https://github.com/mkearney/botrnot/blob/master/R/features.R)),
the percents are # correct / # total, which makes sense.

However, the percents are from the training set; there’s no test/validation
set, which is a problem when working with bespoke text data as a feature.

~~~
tysonzni
Do we know that no cross validation was done? I couldn't figure out what the
dataset is or where it came from just by looking at the repo.

~~~
minimaxir
k-fold cross-validation was done (the cv.folds parameter to gbm), but that
doesn’t help when the model overfits like hell.

~~~
gerty
I agree this is probably what happened. From the estimation object I see
around a total of 3500 training samples with around 90 predictors. 3-fold
cross-validation was done so at each iteration only around 2150 samples were
used for training... And it seems none of the samples were used for out-of-
sample checking.

------
minimaxir
Per the README to the corresponding repo:

> The default [gradient boosted] model uses both users-level (bio, location,
> number of followers and friends, etc.) and tweets-level (number of hashtags,
> mentions, capital letters, etc. in a user's most recent 100 tweets) data to
> estimate the probability that users are bots.

Not an exact science, but shows what you can do and deploy quickly with
R/Shiny.

The author’s rtweet package is very good for making quick Twitter data
visualizations.

------
andrew-lucker
I tried putting in verified users and they were all "probably bots". By
definition is that not the only type of user publicly acknowledged as "not a
bot"?

~~~
nl
Yeah it's not very good. It marks the author's account[1] as "not a bot" when
it appears to be just reposting Instagram posts.. ie, it is a bot.

Plus it said I was a bot (85% chance anyway), and I'm pretty sure I'm not.

[1] [https://twitter.com/mkearney](https://twitter.com/mkearney) (I assume
this is his account?)

~~~
opsiprogram
It correctly classified me (I am a bot) w/ 96% accuracy.

------
peatmoss
While the quality of the model can be debated (I noted lots of false positives
too), I do note that it’s kind of cool that we’re all sitting around and
poking at an app written in an R web framework.

If you haven’t:

1\. Downloaded RStudio IDE

2\. Built a hello word Shiny App (better still for a flavor of the thing a
hello world app using the shiny dashboard package)

3\. Deployed your app to shinyapps.io

I highly encourage you to do so if for no reason than to see how streamlined
RStudio has managed to make web app deployment for people who often don’t have
much of a programming background.

I’m continually impressed with the work RStudio does, even if I’m a curmudgeon
and still write all my code in Emacs instead of their IDE. If RStudio expanded
to support Python similarly well, I imagine they could really be the place
most data scientists work.

~~~
bllguo
RStudio is by far my favorite IDE. I would write much more Python if there was
something as good for it. Anticipating more Python support, which I believe is
part of the plan

------
prateek_mir
It is classifying me as a bot with 94.6% probability. Does it give too much
emphasis on retweets ?

~~~
Pokepokalypse
False positives are usually due to high number of retweets and. . . uh. . .
check your followers. You may have a lot of bots following you or you may be
following a lot of bots.

------
derrasterpunkt
I recently watched a talk from 34c3 (chaos computer club conference) which
were held at the end of last year about Twitter bots, their existence and
their detection. The speaker couldn't find a lot of bots that were cited in
studies and that their methodology were somewhat arbitrary.

Definitely worth a watch:
[https://media.ccc.de/v/34c3-9268-social_bots_fake_news_und_f...](https://media.ccc.de/v/34c3-9268-social_bots_fake_news_und_filterblasen)
(The video is German but there should be a translated version of it on the
site)

~~~
shaki-dora
I believe there is a bit of confusion in the public discussion, mostly of the
2016 election:

While “bots” featured large in initial discussions, the details that have
emerged since, especially in the recent indictment, actually point at
software-assisted humans as the main avenue for these activities.

I believe the “bot” monicker because (a) it’s how most people would tackle the
task, perhaps ignoring the cheap labor pool this shadowy PR operation had
access to, and (b) the bot-like behavior exhibited by many of these accounts,
which was actually a result of not being native speakers, following a script,
and having no deeper knowledge of politics and culture to fall back on.

------
ehudla
The classification is based on an R package for Generalized Boosted Regression
Models[1]. Can anyone knowledgeable opine about this choice?

[]1
[https://cran.r-project.org/web/packages/gbm/](https://cran.r-project.org/web/packages/gbm/)

~~~
jupiter90000
I'd opine the choice of model matters less than the general approach used. In
this case, features are taken from the data that are more like metadata about
an account, metadata about the frequency and such of tweets, and automatically
generated text features about the tweet text which are used to train/predict
the binary classifier. Whether it's gbm, lda, logistic regression, etc. used
on this data is probably not that huge a deal (maybe small % differences in
accuracy) assuming each are used correctly.

What I'd want to also see however, would be curation of a relevant corpus of
tweet content that could be integrated in the classification process in terms
of actual tweet content (bag of words, sentence structure, & other NLP
features, images, other media) being used to determine a given account's
similarity between bots and non-bots -- instead of relying essentially on
metadata similarity in addition to crude auto-extracted text features like
simple total number of words and character counts. I can't say for sure if
it'd improve predictions but would seem key for this type of problem.

------
betolink
I got .6 probability, that's pretty high and last time I checked I could fool
the Turing test.

------
mcintyre1994
I tried a few things, it seemed to be working well but now I keep getting "An
error has occurred. Check your logs or contact the app author for
clarification."

------
plcancel
@jack, 0.693?

------
tysonzni
Developer: Michael W. Kearney

Link to github:
[https://github.com/mkearney/botrnot](https://github.com/mkearney/botrnot)

------
amelius
What use is this really if bot creators can incorporate this tool, and adjust
their tweets until they pass the test?

------
glangdale
Welp, apparently I'm probably a bot. Time to go into the bathroom and cut my
arm to verify...

------
_susanoo
It seems even @potus has a probability of .929 of being a bot. Is this fake
news?

~~~
mcintyre1994
@realdonaldtrump is only 0.008, interesting comparison! I wonder if it's
picking up something weird from the style of @potus changing from Obama to
Trump, and in particular something related to the fact Trump seems to favour
his personal account even as POTUS.

------
benliong78
Kinda wish you named it: 'Robot or not'
[https://www.theincomparable.com/robot/](https://www.theincomparable.com/robot/)

------
drefanzor
I think you need to fix your bot algorithm.

------
opsroller
This is the name of a browser plugin mysef and a few others have been working
on. Glad you jacked the name.

~~~
wpietri
Jacked? I just tried searching for "bortnot" and I can't even find your plugin
in the search results. You can't blame somebody for stealing a name that
nobody can even find.

~~~
Nition
Did you search "bortnot", or "botrnot"?

~~~
wpietri
Sorry, botrnot is what I searched for.

