
Why Twitter’s decision on Scraperwiki is bad for data democracy - frabcus
http://gravyanecdote.com/uncategorized/why-twitters-decision-on-scraperwiki-is-bad-for-data-democracy/
======
wslh
This issue is just an instance of a big problem: there is not such thing as a
data democracy, only a theory. It's not about Scraperwiki only, the SLA for
the free Twitter API is very restrictive (i.e: number of calls per hour).

Twitter (and others) works like a cartel, imagine that you want to pay to
increase the number of API calls per hour, you don't have a straightforward
way to do it. Twitter has a lot to learn from what Google did almost a decade
ago: make it easy to pay and work with your service instantly (AdWords).

~~~
theworst
A little off topic, but I'm curious where the HN crowd lies on the ethics of
scraping.

I'm a bit on the copyleft side -- I say, if something is made public, it's
fair game. If you want to protect your data from being scraped, you very
easily indicate that by putting it behind a login.

But in my opinion, anything google is allowed to access and put in the index,
I can access.

~~~
wslh
I think at the startup level you can "hack the system". The issue comes when
you grow.

~~~
theworst
As a practical, or ethical issue?

IE, is it just as "wrong" in your opinion to scrape sites, regardless of size
or profitability? Or are you espousing the practical argument that it's fine
because you can get away with it when flying under the radar?

------
zAy0LfpBZLC8mAC
What, a centralized service is not good for democracy? Who would have guessed!

------
metaphorm
I don't get it. Is the author under the impression that Twitter data is some
sort of essential common resource? That's ridiculous. Its private and and its
for sale by a for-profit company.

This isn't about "data democracy". Thats a weasel word the author used to try
and sound persuasive despite his weak argument. This is about the author
wanting to have cheaper access to Twitter data and whining about how much it
will cost him to get that data.

~~~
frabcus
Hiya! Francis from ScraperWiki here. There are a couple of things going on:

Firstly, it is about inconsistency between how developers are treated and how
people-who-can-use-Excel are treated.

Secondly, it is that _vetting_ of "data use" is so hard, it means low end
users just can't buy it at all (financial and red tape reasons). Even if they
are happy with the per-Tweet cost.

Speaking for ScraperWiki, we want Twitter to make more money from their data.
More details here: [https://blog.scraperwiki.com/2014/08/the-story-of-getting-
tw...](https://blog.scraperwiki.com/2014/08/the-story-of-getting-twitter-data-
and-its-missing-middle/)

~~~
metaphorm
Twitter seems to have a strategy for monetizing their data that is at odds
with your company's strategy. This is not about "data democracy", its just a
business dispute. You're making it sound like its ideological but its not. You
want something from them and they said no. That's all.

------
angersock
Or, you know, you could learn to code. That would open up a myriad of
opportunities for you, and help prevent this sort of loss of independence in
the future.

~~~
mcherm
I really don't think that will help. It is not that Twitter is insisting that
people stop using (unlimited, unmetered) scraping and move to (unlimited,
unmetered) APIs. Rather, it is that Twitter has stopped offering unlimited,
unmetered APIs and is now charging for use.

~~~
angersock
I agree that that is the real problem--that said, the article took the stance
of "Why is Twitter being so much nicer to developers than non-coders?"

------
markbnj
Any links to information about the decision? Googling the two company names
didn't turn up much. Has this been publicized?

~~~
chippy
Here is a link from ScraperWiki from the article:

[https://blog.scraperwiki.com/2014/08/the-story-of-getting-
tw...](https://blog.scraperwiki.com/2014/08/the-story-of-getting-twitter-data-
and-its-missing-middle/)

~~~
markbnj
Thanks!

------
Patrick_Devine
Couldn't Scraperwiki release a tool which allowed anyone to scrape Twitter and
have it upload the data back to Scraperwiki? Would that get around the SLA?

~~~
frabcus
Probably not.

See Twitter's response to this blog post of somebody trying to do something
similar: [http://mashe.hawksey.info/2011/03/export-twitter-
followers/](http://mashe.hawksey.info/2011/03/export-twitter-followers/)

Even if we could, the users would have to register a Twitter developer
application, making the user experience much worse than just OAuth, for no
good reason.

