I think a major hurdle we have to overcome with big data is separating causation vs correlation. As the data set scales, we gain ever-increasing confidence in the correlation, but an ever more complex set of causations.
Take their House of Cards example. Netflix saw a strong correlation between David Fincher, Political Thrillers, and Kevin Spacey. Fantastic. But why? What did people like about these things? Why did this 'work'?
Let's try to replicate this decision: take great directors (Wachowski siblings), a strong cast (Emille Hirsch, John Goodman, Susan Sarandon), and nearly unlimited budget ($200m) to reboot an existing, well received franchise. Should be a hit, right? Wrong - it's a complete and utter failure known as 2008's Speed Racer.
When we say we want to be data-driven we actually mean we want to be insights-driven. We want to understanding the "Why?" from the data's "What"; it's the 'Why' which lets us know how to react next. It's easy to confuse the data's specificity with insight's certainty, but they are distinctly not the same: We can pinpoint conversions down to 6 digits of significance without having a clue why it occurs.
What we really need is Big Insight, but that's a significantly harder problem, not because we don't have the technology to create a solution, but because don't even know what the right questions are.
I'm optimistic about the possibilities of a system like IBM's Watson in helping solve this, but as it stands, Big Data's utility is giving us 99.755% certainty that we have no idea what is going on.
Take their House of Cards example. Netflix saw a strong correlation between David Fincher, Political Thrillers, and Kevin Spacey. Fantastic. But why? What did people like about these things? Why did this 'work'?
Let's try to replicate this decision: take great directors (Wachowski siblings), a strong cast (Emille Hirsch, John Goodman, Susan Sarandon), and nearly unlimited budget ($200m) to reboot an existing, well received franchise. Should be a hit, right? Wrong - it's a complete and utter failure known as 2008's Speed Racer.
When we say we want to be data-driven we actually mean we want to be insights-driven. We want to understanding the "Why?" from the data's "What"; it's the 'Why' which lets us know how to react next. It's easy to confuse the data's specificity with insight's certainty, but they are distinctly not the same: We can pinpoint conversions down to 6 digits of significance without having a clue why it occurs.
What we really need is Big Insight, but that's a significantly harder problem, not because we don't have the technology to create a solution, but because don't even know what the right questions are.
I'm optimistic about the possibilities of a system like IBM's Watson in helping solve this, but as it stands, Big Data's utility is giving us 99.755% certainty that we have no idea what is going on.