
Introducing Twitter premium APIs - stablemap
https://blog.twitter.com/developer/en_us/topics/tools/2017/introducing-twitter-premium-apis.html
======
ComputerGuru
Twitter already betrayed their entire dev base all those years ago when they
effectively killed off all 3rd party clients by setting a lifetime cap on
oauth tokens per developer just so they could force users to use their clients
which inject ppa/ppc ads into the stream (and then didn’t even bother keeping
those clients up to date, pissing both the developers and their users off in
one go). Stupidest move ever on their end. All they’d have had to do is
mandate the inclusion of promoted tweets in the firehouse and they’d have kept
developers loyal to their platform while raking in the cash.

~~~
rhinoceraptor
Also, if I’m not mistaken, retweeting and hashtags were done first by third
party clients. Even the word tweet was coined in an email Blaine Cook sent
discussing twitter’s auth API for 3rd party clients.

~~~
oliyoung
you were right in that they weren't Twitter's idea, but they were community
driven.

@ and hashtags came from IRC and retweets (back when it was the RT....) just
evolved

~~~
icc97
IRC then via Chris Messina [0]

edit: although disappointingly he seems to use this fact as quite a personal
branding tool now

[0]: [https://en.wikipedia.org/wiki/Chris_Messina_(open-
source_adv...](https://en.wikipedia.org/wiki/Chris_Messina_\(open-
source_advocate\))

------
me_bx
A few years ago, the free search API could retrieve tweets since the beginning
of twitter (although twitter did not guarantee completeness of the results):
[https://web.archive.org/web/20150311065138/https://dev.twitt...](https://web.archive.org/web/20150311065138/https://dev.twitter.com/rest/public/search)

Twitter then limited this API endpoint to data from the past seven days only,
and now just introduces a paid version for only last 30 days data.

I find it fair to charge for elevated API access. On the other hand, the 30
days limit for the paid version is pretty ridiculous. Makes it more worth to
just scrape the free online advanced search...

~~~
andypiper
That's categorically not the case - the index for the search API has _always_
been around 7 days since Twitter acquired Summize (which is where the search
API came from). There was never an artificial limitation such as the one
you're describing. From the page you're linking to from the archive:

"The Search API is not complete index of all Tweets, but instead an index of
recent Tweets. At the moment that index includes between 6-9 days of Tweets."

Scraping the Twitter website is against the terms of service and may result in
permanent suspension of access.

~~~
gboudrias
> Scraping the Twitter website is against the terms of service and may result
> in permanent suspension of access.

Scraping is always against the ToS, so no surprise there. I doubt it's really
ever stopped anyone.

~~~
oculusthrift
i would imagine it stops researchers.. who usually try do things legit

~~~
iagovar
It does. We contacted them seeking academic discounts, and there wasn't. I was
in a little university in Spain. No one is going to drop 20k like that.

------
jakereps
> I wish these premium APIs were available during our first few years. As we
> grew, we quickly ran into data limitations that prevented expansion.
> Ultimately, we raised a round of funding in order to accelerate our growth
> with the enterprise APIs. With the premium APIs, we could have bootstrapped
> our business longer and scaled more efficiently.

Quoting a company that is essentially saying they had to seek funding because
of twitter limiting API access is an odd choice for this announcement.

------
zippergz
I wish it were true pay-as-you-go rather than starting you in a sandbox and
then jumping immediately to $150. Working on a small side project, $150/mo is
a lot of money in the beginning, especially if I'm doing a relatively small
number of requests. I guess I need to see what the sandbox entails, but if
it's not real live twitter data, this still feels like a big step function in
cost from free to premium. Just charge me per-API call or on the amount of
data transferred with no minimum.

~~~
andypiper
The Sandbox is real live Twitter data, free access, but a smaller number of
requests, no counts endpoint, and no enrichments.

~~~
zippergz
Ok, cool, that might serve my development purposes, then.

------
wastedhours
I don't know that the testimonial in the post makes me feel better about the
product to be honest "they couldn't provide us what we needed so we had to
give away part of our company to pay through the nose for the privilege for
that feature, so it's great they've released this now, years after it'd have
been useful. How's that feature you're waiting on working out for ya?".

I know it's the current thing to bash on Twitter, and I'm sure there are
decent use cases out there, and glad they're finding extra revenue streams
outside of ads, but the speed and level of iteration seems rather low
considering the resources they have.

 _Of course, I 'm just a username on a website and have never scaled anything
to 300m users, but like a lot of people, think Twitter could still be great...
but seemingly still has a lot of miles to go. And highlighting that in an
announcement post seems... strange_

------
tomkin
Twitter's biggest mistake was hiring thousands of people when they only needed
30. It doesn't need thousands of people. It doesn't even need 100.

Twitter is not a technological marvel. It's something Oprah promoted, and
Donald Trump bitches on.

And now, it's an API with a (social) class-based access, and without irony,
also promotes itself as a leader in conversation and exchange.

~~~
Jagat
I often see these kind of comments and find them extremely myopic. Sure you
can create a twitter clone all by yourself, but good luck scaling it to ~150k
tweets per second writes[1]. All the while ensuring they're fanned out to all
the respective followers some of whose tweets may need to be fanned out to
tens of millions of users in a matter of seconds.

You'll need 30 engineers just to build and maintain the datastores, caching
mechanisms, and service deployment infrastructure. You then have the actual
services, real time streaming platforms, streaming applications, batch infra
(hadoop), analytics pipelines, notifications, security, anti-spam,
recommendation/machine learning pipelines, ads infrastructure etc to name a
few. Can't find it now, but there should be a circa 2011 architecture of
twitter in one of the blogs somewhere. I recommend you have a look at it.

If you're comparing it with WhatsApp, note that WhatsApp messages are mostly
1->1 and it's easier to shard the servers by recipient. Also, not having to
bother persisting all the messages forever simplifies the system by a lot.

[1] [https://blog.twitter.com/engineering/en_us/a/2013/new-
tweets...](https://blog.twitter.com/engineering/en_us/a/2013/new-tweets-per-
second-record-and-how.html)

~~~
tomkin
I think you're making a few assertions that would be more honestly served if
you conceded as assumptions.

Twitter validated it's own employee count with an absurd overhead.

It has concerns of scale, performance and user experience. So does GitHub,
Shopify, Pandora, Spotify, Snapchat, LinkedIn, MailChimp, WordPress.com, etc,
and yet, manage will much less resources and overhead. Oh yea, and fucking
Netflix - which - as far as I can tell - has much more and difficult problems
to solve at a much faster pace. They employ 4,700.

As of this comment, Twitter employs 3,583 people.

What they needed to do was hire 100 talented people, allow third-party
clients, try and occasionally acknowledge the developers who implement their
platform, and stopped with the asinine "we're trying so hard guys" blog posts.
We get it. You're having a hard time and you want everyone off of your grass.
That's fine, but don't come asking for sugar at my house 'cause you ain't
Beyonce.

~~~
rtpg
Netflix has less "hard problems" in terms of operations than Twitter.

With Twitter everyone's data is interconnected. With Netflix, you could
theoretically have each server just be completed isolated from each other
(modulo account management).

Netflix has operational problems to solve in terms of bandwidth, though.
There's definitely major difficulties there, but they're of a different
variety to the high coupling Twitter deals with

~~~
tomkin
I'll say what I've said in another thread on this matter, which is, no one
outside of this bubble buys this shit. There are global supply-chain networks
that move millions of products, and they don't employ as much as a micro-
blogging platform. It's hubris to think we're so special.

~~~
williamscales
I absolutely agree with you. Modern software "engineering" is much closer to
craftsmanship than true engineering practice.

------
ben174
I always find it humorous that Twitter was the creator of the HTTP response
code:

    
    
        420: Enhance Your Calm
    

Returned when an application is being rate limited for making too many
requests.

[https://developer.twitter.com/en/docs/basics/response-
codes](https://developer.twitter.com/en/docs/basics/response-codes)

~~~
lightbyte
Is this not what 429 is for? Their docs [1] also suggest 429 is what they are
actually returning

[1]: [https://developer.twitter.com/en/docs/basics/rate-
limiting](https://developer.twitter.com/en/docs/basics/rate-limiting)

~~~
ben174
Right, the joke is that 420 is a marijuana reference. "Enhance your calm",
i.e. smoke a joint and chill out a bit.

------
meritt
As the pricing appears to be based on # of requests and/or # of tweets
consumed, does it have a `include_spambots=false` filter or do we have to pay
for the pleasure of wading through spam?

~~~
komali2
I linked this elsewhere in this thread, but I reported this obvious spam
account, with hundreds of spam followers, weeks ago and nothing has been done:
[https://twitter.com/ksfAKBARI](https://twitter.com/ksfAKBARI)

I feel like twitter must not be taking spambots seriously, or has malicious
intent to keep them there, because out of hundreds (thousands?) of employees
they can't get a team onboard to shut these obvious accounts down. I mean, _it
's been reported_ and it still exists. Either the report log is huge or it's
being ignored.

------
passivepinetree
Is there an available history of Twitter's APIs over the years? IIRC it was
initially open, and then they locked it down so many 3rd party apps for
Twitter had to shut down, and now they're sort of opening it up again?

What's the deal?

~~~
porfirium
They never "locked it down", they just rate limited some vital endpoints and
never bothered to add endpoints for new, important features.

My own recommendation: disassemble any official app (for example, the Android
one with apktool) and get the tokens and endpoints and do whatever the hell
you want without paying a cent.

~~~
0x0
In the beginning you could post a tweet just by doing a http basic-
authenticated post containing the tweet body, and there were unauthenticated
rss feeds for most data feeds. These days you have to create app ids, tokens,
etc etc etc. It really was a lot more open earlier.

~~~
herbst
Twitter was part of the reason I fell in love with development. The easy
interaction with the whole platform changed my view on data. It broke my heart
when they introduced v2

------
hw
My experience with Twitter API has been horrible to say the least. Plenty of
API errors, service over capacity, and instability. So now I have to pay to
get access to an API that's stable?

Step 1 - build a product. make sure it has lots of bugs and is unstable.

Step 2 - roll out a premium product that's the same product, just stable, and
less buggy.

Profit.

How is charging for an API not a finger to the developer community? Facebook
for example doesn't charge for their API access. They charge for the products
that the API has access to. If I have access to a product, I shouldn't need to
be charged for the APIs to access that product

~~~
MichaelGG
Interesting. I found Twitter to be vastly better than other social media
companies. Far better documentation (though weird complicated auth where I get
4 pieces of info for some reason). Far more stable and understandable. Twitter
returns a ton of data in each response so there's no need to go making
multiple requests.

Compare to FB which is just a nightmare of trying to guess and figure things
out. It's so bad. If it reflects their underlying model at all, their main
codebase must be very difficult to work on.

Charging is a welcome improvement over the previous situation: Deal with Gnip
or other data brokers. They're "Enterprise". Get no information, have to deal
with a long sales cycle and sales team, have to wade through tons of BS to
figure out what you need. It sucks.

~~~
hw
In terms of documentation I do agree with you, although I suspect that's just
due to the complexity of Facebook and its vast array of products, vs Twitter.

My bad experiences have to do with the actual API calls themselves which
aren't stable and occasionally times out or errors out.

If anything I wish they would improve their API infrastructure as a whole
instead of offering the improvements only on paid accounts. I can see charging
for rate limits but not for system stability.

------
Gabriel_Martin
Absolutely not. Never again will I trust Twitter as a developer.

~~~
toephu2
Because once bitten twice shy and companies never change?

~~~
Gabriel_Martin
Perhaps it's as simple as that. I dumped too many resources into my last
effort: time designing, flying out, sitting down with their dev advocates, all
a waste of my time. I shan't reward them for pulling the rug out, or make the
same mistake again.

Also, it's in solidarity with developers who invested even more than myself,
years of time into apps, to be betrayed.

That's an emphatic "hell no" from me dog.

------
caio1982
Good for them, I honestly hope it works business-wise as they want and it
makes developers satisfied. However, I'm afraid Twitter the company has got
the same sort of trust issues as Google from the developers POV. They need to
address this specifically, if they want people to believe their premium API
move is for real.

~~~
freedomben
Google is obviously a huge company with many different products, but I've had
great success and satisfaction with many of their APIs. What sort of problems
are you referring to?

------
Veratyr
This is a good start. Gnip seems to basically be invite only (I've tried
contacted them multiple times and have not once received a response) and the
public API is so limited it's near useless.

I'd still rather everyone move to Mastodon.

~~~
andypiper
Gnip is Twitter's enterprise API business with annual subscriptions to
firehose and decahose access. This is month-to-month billing for improved API
access and data quality.

~~~
detaro
how does this address anything in the parent post?

~~~
andypiper
It points out that Gnip is not “invite only” - it is an enterprise
subscription product.

~~~
nl
As someone who had a Gnip subscription for years I think "invite only" is
close to a fair characterization.

It took huge amounts of effort to get them to return our inquiries. Given the
huge amount of money we ended up spending that seemed ridiculous.

But I guess that just got us ready for the whole "everyone must move to Gnip
2.0, but no we won't merge the multiple pull requests to make _our official
client work with our new API_ "[1] thing. Wow, that was fun...

[1]
[https://github.com/twitter/hbc/pulls](https://github.com/twitter/hbc/pulls)

~~~
hobofan
Wow... there goes the last shred of sympathy I had for Twitter for not being
profitable.

------
katpas
I immediately checked the date on the article after reading the headline.

Thought it was a relic from the third party client days.

------
farnsworthy
Is everything starting around $149+, or only search?

I suppose that's understandable, but I think it would be cool if they
implemented something more akin to the AWS pricing model (S3, for example, as
tweet streams remind me more of this than of enterprise-level pricing).

~~~
lux
Is that the price for Premium Search access? It's giving me a 404 error
clicking the 'See pricing' link on
[https://developer.twitter.com/en/pricing](https://developer.twitter.com/en/pricing)

~~~
jamestimmins
Same here, both logged in and logged out.

~~~
andypiper
Thanks for the note - working on that.
[https://developer.twitter.com/en/pricing](https://developer.twitter.com/en/pricing)
should work.

There's free sandbox access, and then tiered pricing for increased access and
additional features like counts and the enrichments.

~~~
lux
It's the 'See pricing' link under the Premium / Search Tweets cell on that
page that goes to a 404.

Edit: The link points here:

[https://developer.twitter.com/en/pricing/search-30day](https://developer.twitter.com/en/pricing/search-30day)

~~~
andypiper
thanks - we'll look into that.

------
abootstrapper
Twitter's current plan is to execute on all the ideas their user base was
talking about 5+ years ago.

------
ranman
I'm still waiting for that webhook API they talked about 2 years ago.

~~~
andypiper
the webhook-based Account Activity API is in beta now and on the roadmap for
general availability.

~~~
ranman
I would be interested in that beta. I'm already in the direct messaging beta.

------
danso
Until info about the new endpoints and queries is released, I'm not rushing to
get out my wallet, but at the very least it's encouraging to see _any_
movement regarding the API. This seems like a vote of confidence that the
basic API will be around for at least a few more years.

~~~
andypiper
[https://developer.twitter.com/en/docs/tweets/search/overview...](https://developer.twitter.com/en/docs/tweets/search/overview/premium)

------
_pmf_
Oh boy.

> However, this left a gap that made it painful for growing businesses to
> deliver scalable solutions to customers

How about this: close your monetization gap yourself instead of pushing the
problem downstream?

------
egypturnash
I wonder how many people "we came up with a new way to sell your data!" will
send away from Twitter.

------
ssvss
Anyone knows how much they charge for the gnip enterprise access.

~~~
dotmanish
It varies according to the requirements & level of enrichment (+
negotiations). Typically, a couple of thousand dollars for entry levels.

------
jankotek
Is there a way to get copy of twitter entire dataset?

------
philtar
I have a feeling in my gut that it's too late.

------
nkg
Why is Twitter API so open ? I have been playing with it for a few weeks and I
can find almost every data I want. How does it benefit them ?

~~~
DaiPlusPlus
Ecosystem lock-in. I remember shortly after Twitter launched there were many
copycat microblogging services and the community had grand ideas of an "SMTP
for Microblogging" so Mentions, Retweets and the like would work in a
distributed fashion between service-providers, you could even run your own
fully federated microblog off a Raspberry Pi in your garage if you wanted.

Twitter kept themselves in the lead - not just by delivering a superior user-
experience (Nazis and trolls notwithstanding...) but also cultivating a
developer community in a way that kept people off rival platforms - before
stabbing them in the back in recent years (e.g. no support for polls in third-
party clients).

This announcement is a continuation of the strategy: Twitter is making more
data about _their_ platform available, other platforms aren't as open, so
people will stick with Twitter because their data is available - and it also
benefits them directly because these "Premium" APIs do cost money to access:

> Pricing for these elevated tiers of the Search Tweets API starts at
> $149/month

~~~
cisanti
They somehow bribed the media using it. Without the massive free advertising
from media and President(s) Twitter would have been long dead.

------
benwilber0
twitter is out of ideas.

~~~
bpicolo
API access to twitter is a huge business for a ton of different enterprises.
Everything from customer support/happiness products to finance wants the
twitter firehose. This is a totally reasonable and good business avenue for
them to care about / explore.

~~~
caio1982
Only if they mean it and intend to keep it sane, developers-friendly, in the
long run.

~~~
bpicolo
Charging money for it is a good way to incentivize keeping it.

