
Twitter Cuts Off DataSift To Step Up Its Own Big Data Business - nailer
http://techcrunch.com/2015/04/11/twitter-cuts-off-datasift-to-step-up-its-own-b2b-big-data-analytics-business/#.acq8mm:kcu0
======
ransom1538
2008 - Facebook had a meeting with us Crowdstar describing how our current
business model (quizzes) was getting their own section in their TOS, and would
cease to exist.

2010 - After switching to a game model and doing quite well - neck and neck
with Zynga. We were walked into a Facebook meeting room and told a bizarre
story how Mark was upset about friend feed forms. They explained that we could
only share (by means of post feed forms) our games with people that already
had our games. Pleading did nothing, they ended that model. Zynaga is still
flailing around.

2012 - I changed companies and focused on mobile advertising. Another decent
path with 12 employees or so. We got a phone call _. [EDIT REMOVED REST OF
STORY]

2015 - I received another phone call from a large company (They deal in
search) _. I wont go into this one.

* Phone calls from large companies are extremely bad. That means they don't want things in writing.

So sure, have fun with working with large companies. Just remember one thing:
they hate you.

~~~
harperlee
Honest question, because I'm neiher american nor familiar with their common
law:

    
    
        Let's call the company that called us [redacted]
    

Does that really hold water, legally speaking? Because we all know what are
you implying...

~~~
raverbashing
If you think this is inappropriate you shouldn't be quoting it

~~~
harperlee
You are completely right, I didn't think that the parent would delete it (it
was a honest question, really). But I can't edit it now. Perhaps someone from
HN can?

~~~
dang
Ok, we redacted that bit.

~~~
harperlee
Thank you!

------
ColinWright
Never, ever trust another company for your business model.

Here's a previous comment:
[https://news.ycombinator.com/item?id=9262245](https://news.ycombinator.com/item?id=9262245)

That was about FaceBook, but exactly the same holds here.

Quoting:

    
    
        Certainly I'd be incorporating an exit strategy,
        and my game plan would be to leverage off FB to
        start with, to grow as fast as possible, develop
        my own community and eco-system, and then be able
        to shed FB and replace it with something else.

~~~
vdaniuk
>Never, ever trust another company for your business model.

This is just another thought-terminating cliche not unlike "if it is free, you
are a product" and I can't help but respond.

Large technological companies create ecosystems where smaller players can
coexist with "hosts" in a mutually beneficial relationship. Ecosystem
companies are not able to predict the future of the ecosystem, its evolution
for all intents and purposes is mostly unpredictable. Sometimes the
relationship works out, sometimes it doesn't.

Startups have 70-90% failure* rate, the possibility of the host cutting of
some parts of the ecosystem to integrate their food chains into the corporate
borg is factored in the business and investments strategies of the smaller
companies.

You should not trust another company for your business model, but building
your tech business in the ecosystem of a large fish is an absolutely valid and
quite possibly lucrative strategy.

*definitions vary

EDIT success/failure

~~~
gtirloni
The problem is that there is nothing preventing the "ecosystem host" from
deciding that a niche an "ecosystem player" is exploring is lucrative and
worth pursuing.

You can fly under the radar for a while but if Twitter/Facebook/Google/etc
decide your area is lucrative, they will go after it, whether that will kill
one of their "ecosystem players" or not.

There is no partnership involved, they provide a service as a huge "blob" that
is making them money and they don't care about 1 out of 1 million of their
regular users. Your small company cannot call itself a partner, you've as much
leverage as any user when it comes down to deciding to sacrifice you in the
name of something else more lucrative.

I'd say explore the benefits to bootstrap but have a strategy to depend less
and less on them or your time will come. And if you really want to tie your
business survival to one of these ecosystems, get yours and theirs
responsibility down in writing (and even that won't help much because a breach
of contract for them will mean nothing in terms of dollars, while for you
it'll be your existence).

~~~
droopyEyelids
Ah. So you mean, "If you want to be perfectly secure for all future time you
must be completely self sufficient."

Can't really argue with that. Maybe not useful advice, but it's self-evidently
true.

~~~
ColinWright
It's more like:

    
    
        If you build a great business based on something over
        which you have no control, and which is such that the
        ones who do have control actually may have an interest
        in cloning what you do and terminating your facility
        without warning, be sure to have a "Plan B".
    

Yes, sounds obvious, but _so_ many people seem to ignore it, and in some cases
actively disagree with it. You seem to be disagreeing with it - what would you
suggest as more useful advice and a more reasonable stance?

Honestly, would you build a business based on a Twitter or Facebook facility,
relying entirely on them not arbitrarily terminating your access?

What is your advice?

~~~
lifeisstillgood
Perhaps to realise these companies are walled garden abberations, and that in
an open competitive world, the Eco system would exist but the monopoly would
not.

As such encourage and build openness by default into _your_ products, as for
sure users want that (how many people for example wanted _only_ to email other
AOL users, or view only other AOL websites?

------
code_duck
I was involved in a company that heavily depended on a popular site's API. We
could clearly see they thought of the API's best value for them as filling in
features before they had time to create them, and as a testing ground for
ideas. It became harder and harder to stay ahead as our business grew, as at
any time our features could be replaced with an official version. We decided
to stop development.

Now, a few years later our largest competitor has been doing quite well with a
product along the same style... but now, without warning, they've been cut off
from API access and are facing requests for hundreds of refunds. Glad it
wasn't us.

------
frabcus
At ScraperWiki, we had our Twitter to spreadsheet tool cut off (at a much more
modest size!). I wrote up an analysis of where you can get Twitter data from
and why which is relevant:

[https://blog.scraperwiki.com/2014/08/the-story-of-getting-
tw...](https://blog.scraperwiki.com/2014/08/the-story-of-getting-twitter-data-
and-its-missing-middle/)

------
bane
> He said that before this, the pair had been discussing a renewal of the
> deal.

This should be a case study for anybody looking to get into a BD role.

A partnership is fundamentally coming from a position of weakness, and the
relationship is a way for both partners to partially fill in whatever gap they
believe they have in their strategy.

Sometimes partnerships go south, especially when the relationship is highly
asymmetric.

In this case Twitter was the stronger of the two partners and they decided not
to renew. That sucks, but that's life. Don't build your entire business around
a single partnership, because that signals to your partner that all they have
to do to get rid of you is replicate what you do and cut you off. Once you've
gone through the blood sweat and tears to prove the market, you've just given
your larger partner free market development.

DataSift's website lists a number of data sources [1], but Twitter comes up
first. More importantly, I suspect these sources are slightly massaged to look
like they have more sources coming in than are really useful for their
business model. e.g. is Wikipedia _really_ in the same space as Twitter? They
also have a bunch of minor sources but they're listed like they all rank
equally. This is fine, most businesses would do the same to make themselves
look better -- it's just advertising.

Another part of business partnerships is that you have to expose your weakness
to your partner. Twitter is going to know at some point that DataSift's
business model is heavily built around their firehose. They might even have
some sense about what volume of business all the rest of those sources makes
up to DataSift's business. Twitter _also_ , through their other partnerships,
has a better sense of the size of the Twitter Analytics market. Even if
DataSift's piece is 15% of the overall market, Twitter has other partners it
can use to get a feel for the other 85%.

Partnerships are dangerous. Know what you're getting into before you get into
them, and try to only do them with similar-sized partners or you'll find
yourself getting screwed over/screwing somebody else over.

1 -
[http://www.datasift.com/platform/datasources/](http://www.datasift.com/platform/datasources/)

~~~
jackgavigan
_> Sometimes partnerships go south, especially when the relationship is highly
asymmetric._

It's arguably not really a partnership if the relationship is highly
asymmetric.

------
DanielBMarkham
Twitter is quickly approaching the point where it has lost my goodwill and I
will actively look for something to replace it. (Apple has already passed that
point)

I understand the way big ecosystems and walled gardens work. I also understand
that every large technology company that has come down the pike in the last 30
years has wanted to conquer the known universe. That does not mean I have to
actively assist them in this activity.

~~~
tl
I'm losing faith that this is a viable approach. For example, laptops are a
domain where your options are Apple, Lenovo, Dell, or a company that whose
future in the laptop market is bleak. All of the options have done goodwill
losing things in the past.

Can't we all just "conquer the known universe" together? We need a framework
for educating these companies that open solutions have greater reach than
closed solutions even if they are less directly profitable. The people making
decisions are often successful enough this should be possible.

~~~
vijayr
I am not sure these companies don't actually already know "open solutions have
greater reach than closed solutions". They are run by extremely smart and
capable people - but they are more concerned with pleasing the shareholders
and hence short term profits than creating truly open systems (with some
exceptions here and there) and we can't blame them for it. A more effective
approach would be to create/support open systems and educate _consumers_
instead.

------
njyx
Have some fairly strong feelings about how much of a bad idea this is for
twitter in the long run - posted here: [http://www.3scale.net/2015/04/loosing-
innovation-twitters-fi...](http://www.3scale.net/2015/04/loosing-innovation-
twitters-firehose-mistake/)

I have no idea if DataSift and NTT were good partners, but not having anyone
else access the Firehose suggests they think they can cover all significant
use-cases. This almost always turns out to be wrong.

It also sends another chilling signal to say "we're a platform, but when we
feel we want to roll over part of the ecosystem we'll do it".

~~~
clay_to_n
Gnip was DataSift's biggest competitor, and Twitter acquired Gnip. So firehose
access is still totally available, but only through Twitter itself (b/c of the
Gnip acquisition).

When the big company you relies on buys your competitor, you should be very,
very scared.

~~~
bvm
i heard a (possibly apocryphal) explanation of that acqusition that had
something do with a forgotten invoice on Twitter's part.

------
jackgavigan
It's great for companies like Twitter and Facebook when startups build on
their platform. The startups innovate and experiment with new features and
business models (at their founders' and investors' expense). The lousy ideas
fail, the good ones survive and the platform owner can cherry-pick by either
replicating the successful feature/business model themselves or buying their
favourite.

Then, by cutting off others' access to the platform, the platform owner can
make their own service a monopoly and capture a significant proportion (if not
all) of the others' customers and revenues.

Phase 3: Profit!

~~~
MichaelGG
DataSift was simply reselling Twitter's data, along with whatever enhancements
and cleaned up API niceties they offered.

And getting 80% of the revenue for that (according to the article).

How does this benefit Twitter? How does this benefit startups wanting to use
Twitter? The data is still available, just not through resellers.

~~~
clay_to_n
DataSift and Gnip had access to the Twitter firehose (every single tweet
posted, live). Normal businesses / developers can't get access to that.
Twitter's "streaming" API is a tiny fraction of the firehose.

So DataSift wasn't just reselling their data - they, along with Gnip, had a
partnership that allowed them to get access to the firehose. For many social
media companies, the firehose is a necessity, so they bought contracts with
Gnip or DataSift.

Then Twitter bought Gnip, and is cutting off DataSift's access to the
firehose.

~~~
MichaelGG
Right so if we say Gnip is Twitter, then anyone can go buy data from Twitter
(even if it's a wholly owned subcompany).

DataSift didn't have much of a value add (OK, historical data, common filters
for Twitter and Reddit and others, etc.) But nothing that Twitter can't just
in-house now that the market got nice and established.

~~~
jackgavigan
Yes. Having acquired Gnip, Twitter can now offer the same services that
DataSift offers.

So, Twitter terminates their "partnership" with DataSift, and all of
DataSift's customers (i.e. the market that DataSift spent money and effort
_getting_ "nice and established", educating potential customers, helping them
figure out how to make use of the data, etc.) who want to continue benefitting
from those services will now migrate to Twitter/Gnip.

I bet Twitter/Gnip will come out with some tool to make it ultra-easy for
DataSift's customers to migrate across.

------
vram22
I was once invited to be a co-founder of a startup (by the other founder),
whose business model was to build something using the API of <insert-big-
social-network-company-here>. I declined, for the same sort of reasons
mentioned in this comment by georgemcbay -
[https://news.ycombinator.com/item?id=9364391](https://news.ycombinator.com/item?id=9364391)
\- i.e. "yes, this fact should really give pause to people who are thinking
about basing an entire company around an API, portal or other service that is
100% controlled by a large company. Any such dependency should be seen as the
huge risk factor that it is."

Have read of multiple other such cases where the rug was pulled from under
such startups that relied on a single point of failure (SPOT) -
[http://en.wikipedia.org/wiki/Single_point_of_failure](http://en.wikipedia.org/wiki/Single_point_of_failure)

I told that founder the reason why I thought his model was not a good idea,
after thinking for a while and then declining the offer.

------
steven2012
Anyone doing business with Twitter is a fool. There are few companies that
show as much disdain to their partners as Twitter does. With partners like
these, who needs enemies?

------
bsdpython
For all the great things going on in open source software, truly open source
data isn't all that common. I love playing around with third party APIs as
much as anyone but as software developers we need to be extra careful that we
are not building on sand. Just the thought of being cut off limits lots of
interesting ideas and side projects for me. In some ways I want to build a
platform just to get 100% access to some really fun data thrown off from the
platform in which to build fun things on top of.

------
ergest
_Talking to Nick Halstead, the CEO and founder of DataSift, he said that his
company was “blindsided” by Twitter’s announcement, which it made without any
warning to DataSift._

He should have totally seen it coming and been prepared. When you don't think
about the future and don't have the foresight to build competencies to take
advantage of what you discover, you end up being blindsided. This is not about
hating on Twitter or siding with them, it's pure competition.

~~~
weirdsift
That's untrue. DataSift was explicitly telling customers the contracts were up
for renewal and that they did not have anything solidified.

~~~
bvm
But it wasn't totally unexpected, the rumour mill has been running for months.

------
untog
This is terrible for Datasift, but I really hope they saw this coming. Twitter
buying Gnip should have been the writing on the wall.

~~~
frabcus
I'm sure Datasift saw it coming.

I researched this last summer. Even before I was explicitly told all the
original firehose contracts were five years long, I could infer it from
attitudes and behaviours.

I'm excited that Datasift are launching interesting new Facebook features.
They're a great company, with good technology.

I hope Twitter's "no data resale" attitude leads to them becoming irrelevant,
as it is so destructive of much innovation.

~~~
bane
Yeah, I'm sure it was significant source of talk within the company for a
while. But what can you do? It's not like there are other Twitters out there.
The value of Twitter is that there's only one. It looks and sounds like
they're trying to diversify out to other sources, but none-of-them are Twitter
and for whatever reason Tweets are like crack to market research firms.

~~~
frabcus
Well, Instagram and Twitter are converging in terms of feature set. They're
pretty well indistinguishable now.

They've not totally converged in terms of audience, but I think have
surprising overlaps. Lots of Twitter data use cases you can get from
Instagram, I bet.

~~~
untog
_Well, Instagram and Twitter are converging in terms of feature set. They 're
pretty well indistinguishable now._

I don't see how that's true at all. While Twitter now has well-developed image
capabilities it's still not the core of what people use it for. Also Instagram
has no equivalent of retweets.

------
bayonetz
It's a shame, for sure, for DataSift.

We have been working with DataSift for about a year now though and it is
pretty damn clunky. I'd say GNIP is probably even worse at the moment but
maybe not for long?

As a user, if GNIP can capitalize on being part of Twitter, I gladly welcome
the improvements this could bring. I'm not super confident though as now
Twitter/GNIP will also be a monopoly in the space with the usual disincentives
that come along with that. Also, our use case is strongly biased towards the
"historic query" features and neither service seems as interested in that as
they are in the real time streaming features.

Also, I had considered earlier developing a niche competitor solely focused on
the historics space for companies with research-oriented use cases like ours.
I suspected Twitter would be dropping this hammer though so glad I didn't
waste my time!

------
cinquemb
I wonder if using the hundreds if not thousands of twitter api credentials
checked in on public repos, one could mitigate against things like this?

Though I suspect companies not in the jurisdiction of the US or its "partners"
will probably be more likely to get away with an approach like this if caught
or public about it, while still providing value added services based on walled
gardens like facebook and twitter.

------
Animats
That's not surprising. Twitter is very controlling about their service. They
had an open API only until they got market share. Then the jaws gradually
closed on Twitter-related programs. Twitter strongly discourages client
programs other than their own. I once wrote a Twitter client with spam and ad
blocking, just to see what it would be like, but I knew Twitter would never
approve it.

------
hellbanner
"Parasites are not crushed by the strong, they feed off of them" but sometimes
the strong simply buck them off.

Apple did this with their introduction of Gamecenter, killing OpenFeint, and a
couple others (can't remember their utility).

------
wslh
I feel bad for DataSift. I always understood that their business was really
risky. More when Twitter acquired Gnip.

~~~
weirdsift
I hope some of the people got good salaries. And maybe got some of the almost
$80M raised.

DataSift was proud of the money they raised. The said it made them "more
difficult to acquire than Gnip". I was amazed when I heard that. It sounds
like they are happy they took on so much money and were moved passed by Gnip.

But it is worse, I think. For many services they require you to provide API
keys. Rate limits are a problem? No! Create more API keys and give them us.
Why are they not blocked? If any big service decides to block API key rotation
... game over? If we do it in-house, then we can use many proxies and pretend
to be separate. Maybe.

Or maybe not. Maybe because of VC connection DataSift can rotate API keys and
with impunity. Who knows?

But yes, to me it is very very risky business, but not too difficult
technology. (Maybe that is why I have not had a big hit. Too little risk.)

So does HN think it is successful to raise money like them? Do the founders
make more than $500K/year total from this venture?

------
grizzles
Their strategy to handle this business risk was to ...accept it? It seems kind
of shortsighted given twitter has ALREADY did a version of this to a bunch of
other companies/apps.

At it's core, what twitter provides is no big deal. The ability to post 140
chars. They could have been quietly funding an open twitter protocol (it will
happen eventually) but they chose to keep their heads in the sand.

------
phesse14
Wonder how you have done it, but this post was already published. A breach on
HN?

------
paulhauggis
This doesn't surprise me at all. If you are going to use the infrastructure of
an existing business to start your own, you will need to have a plan for
figuring out other sources of users and diversifying your risk. Otherwise, you
will be out of business if you ever get too big or you do something that the
larger business disapproves of.

This is a good lesson in business.

------
zkhalique
This is why we are building a new kind of social platform!

[https://qbix.com/blog/index.php/2013/04/a-new-kind-of-
platfo...](https://qbix.com/blog/index.php/2013/04/a-new-kind-of-platform/)

One where you won't be cut off if you're a developer. And one which
organizations can host themselves.

~~~
tlrobinson
Why will you succeed where App.net, tent.io, etc hasn't?

~~~
Karunamon
Both of those services are still alive - seems a little bit silly to say they
"weren't successful" unless your definition of success means having as many
users as Twitter.

~~~
tlrobinson
For general purpose social networks, yes, that's my definition of success.
They aren't very useful otherwise.

