
Predicting Churn: When Do Veterans Quit? - ukdm
http://www.gamasutra.com/view/feature/176747/predicting_churn_when_do_veterans_.php?print=1&utm_source=buffer&buffer_share=cb737
======
patio11
You can do a very, very similar thing for non-game SaaS services, too.
(Hubspot calls theirs CHI -- Customer Happiness Index. It attempts to predict
who is going to quit based on backtested statistical models. In practice, you
can get almost sickeningly good results with heuristics that take ~3 lines to
program. You can also then save those accounts by something as simple as "Have
the sales team email them and talk" or, in my recent experience, "Send them an
automated email reminding them that the software exists.")

~~~
archon
As a user, those emails have sometimes had a negative effect for the company
emailing me. Something they send me reminds me that the program exists, that
I'm still paying a monthly fee for it, and that I really don't use it very
much at all.

But in all fairness, I doubt anything they could have said or offered would
have changed my mind at that point. It was just a matter of when I cancelled,
not if.

~~~
rfergie
Stochastic Solutions have several papers about this problem:
<http://www.stochasticsolutions.com/papers.html>

In a subscription business who can be saved, who will leave regardless and
then the tricky group of people who will leave if they get a sales call?

~~~
thibaut_barrere
This looks really interesting - thanks for sharing!

------
wccrawford
My previous employer knew I was going to quit long before I told them.

But then, the fact that I was sending out resumes was probably a huge hint. I
started sending them when I was a little disgruntled. Eventually, they heaped
crap on me until I got serious about it and found a new job.

Not once did they attempt to do anything to stop me, and asking them to pay me
what I'm worth was met with, "Can you wait a year?" This, despite the fact
that every review I ever got was great. Not just good, great. The only
complaints I got was that I was too quick to answer questions literally. If
someone asked if something was possible, I told them. They wanted me to read
their minds and ask the question they should have asked, instead of the
question they did.

I eventually learned to do that, even, though. I have to admit it made things
smoother. But jeez was it a pain.

------
zach
The best response would be to offer a reason for people to come back after
they leave for a week or so.

The worst response would to offer a reason for people to _leave for a week or
so if they weren't going to already_.

If the company basically offers an incentive for not logging in, word will get
around very fast. Then players have an excuse for taking a break, that they
are gaming the system to earn an incentive.

Penalizing players is ineffective too because you want to welcome back your
wayward customers instead of starting to burn bridges ("I'm going to throw out
your stuff! Okay, I'm putting it in the trash right now!")

So unfortunately, I think all a company can do is have emails and community
managers get in touch. Of course, that encourages players to stop playing for
a week whenever they want to escalate a customer service issue. It's like you
can't win.

I discovered at Blizzard that guild features were enormously strategic. When
players get into an active guild, especially with people they already know, it
becomes hard to leave. The glue that keeps players put is social expectations.

------
aaronjg
That's a really interesting approach. Basically mining a couple of key
features from the data set, and plugging it into a supervised learning
algorithm. The really cool thing is the simplicity of the features used, just
daily activity and daily playtime.

The write up of the methodology is very clear, but I'd love to see some more
description of the results. Ninety-five percent accuracy is a pretty bold
claim, and I'd love to see some ROC curves to back it up!

~~~
mturmon
The high accuracy surprised me too.

To compute their accuracy, their methodology seems to require determining
whether someone is a veteran user, and having a clear quit time for them
(otherwise the user can't be used for training or testing). Maybe after you
make these determinations, the resulting population is easier to deal with.

------
praptak
This is a bit scary. Imagine your employer using machine learning on your git
commit patterns (or swearwords per comment) to check when you might quit.

~~~
yummyfajitas
I can imagine it. Why is it scary?

Would it also be scary if your manager observed your attitude/emotions through
personal interaction and deduced you might want to quit?

~~~
dagw
I'd be all for a tool like that, assuming it actually worked and it was used
in a productive way. Imagine if you're getting tired of writing RoR code all
day and pondering quitting and all of a sudden your boss comes up to you and
says "hey how would you like stop coding Rails for a while and join a team
that's about to evaluate Clojure for an upcoming project". That's just the
sort of thing that could make everybody happy.

~~~
GFischer
The problem is, at least in my experience, the ratio of people who would use
it for good vs those who would misuse it would be very bad.

Of course, it's also the employee's fault for getting into such an environment
in the first place, but sometimes you just need the job.

I use ManicTime Tracker, if my boss got hold of all the data contained in
there, he would probably consider firing me - I average more than 2 solid
hours of surfing the web on any given workday... wait, I have the exact data
:) , that's the point!

I was at my work PC for 1059 hours so far this year.

Of those, I spent 325 hours on Firefox and 95 on Chrome, that averages about
13 hours a week of web surfing (probably 10 hours procrastinating or reading
and 3 actually researching problems).

I also spend a shocking 8 hours a week reading and replying to mail (to be
fair, we don't have any bug tracking or project management, so mail becomes
both), and 5 hours a week on SQL (I do a lot of querying and reporting).

I'm less than 4 hours a week actually on a development environment, split
between VB6, .NET and Forte4GL (our ugly legacy system).

And I was hired to be a "systems analyst"... And that's only time spent at the
PC, it doesn't count time wasted on meetings and stuff.

This is a real-life example of why they say that on a large corporation, you
only do actual work 1 or 2 hours a day (as opposed to a startup where you
might do code or programming-related stuff 6 or 7 hours, I hope :) ).

It's really depressing to put it in numbers. Fortunately, I'm going to quit
next year and dedicate full-time to my startup (which is just getting started
right now :) ).

~~~
JoeAltmaier
It's called Short Timer's Syndrome. When you know you're leaving, your
dedication and work ethic plummet. You don't need a lot of statistics to
notice it; you just have to be looking.

Many managers are too busy looking up the management tree, sucking up to
superiors, working their own career to actually manage their subordinates. If
you're expecting to leave then you're in luck, they won't notice. But that's
probably the reason you're leaving too!

~~~
GFischer
I didn't know it had an official name :) . There are papers and everything
(example: <http://www.opus12.org/uploads/O12-SCI-V02-N01-P30.pdf> )

I guess surfing the web is a form of short-timing - I can't actually leave
early or come in late, since I'm heavily penalized for those, like any self-
respecting bureaucracy, my company equals time at the desk with productivity,
and heavily penalizes lateness.

------
ShabbyDoo
The article cites customer service interactions as the obvious (and obviously
effective) remediation step once a player is identified as likely to quit in a
few weeks. I'd like to know if this is really true. What if the game's
fundamental problem is that players become bored over time? What can a CSR do
to fix this? Probably not much. The analysis shows that changes in gameplay
frequency are far from a random walk and do indeed predict future declines,
but it doesn't offer much in the way of clues for how to keep customers
engaged. What if it could be shown that a breakdown of a player's in-game
social structure (I'm not a gamer, so I'm being vague) also predicts churn?
Perhaps helping the player to find some new relationships would be effective.
Is game play repetitive? Maybe decreasing frequency of novel experiences
predicts churn? This would be good feedback for game designers. Predicting the
future isn't of much valuable if it isn't actionable.

------
westi
This is really interesting.

I can think of great possibilities for using these methods to analyse the
behaviour of users in any number of online services to identify the ones that
might need some form of out reach to help them out and keep them as users
rather than loosing them due to the problems they are having.

------
pydave
There was an article with a similar goal previously posted to Gamasutra from
altdevblogaday, but it's got data and software for you to play along:
[http://www.altdevblogaday.com/2012/08/10/business-
analytics-...](http://www.altdevblogaday.com/2012/08/10/business-analytics-
with-regression/)

------
bloaf
I bet you could apply this method to other participation-based business. One
that springs immediately to mind is gym membership.

