

Building a complete Tweet index - ChrisArchitect
https://blog.twitter.com/2014/building-a-complete-tweet-index

======
minimaxir
Will this new index be accessible via the Twitter API? It seems like this only
impacts manual searches via the web/client.

~~~
AznHisoka
Not that I know of. This historical index actually has been in place for many
many months, but Twitter just decided to publish this article recently. The
API sometimes has less than a week of data, as little as 3-4 days I've
noticed. And some days (like once every 3 months) it returns 50% less data,
but so does the web app.

~~~
idbehold
Yeah, this really isn't interesting to anyone except maybe their investors. If
they provided this functionality in their API it would've been big news.

------
lpolovets
Interesting to think that half a trillion tweets is about 70TB of data, and
that can be stored for about $10k/year in Dropbox.

~~~
enf
Tweets contain a lot of metadata too though. In practice a tweet is really
about 3K, not 140 bytes.

~~~
hk__2
Yes, taking the example in the API docs [1] it gives me 3K of JSON, which can
be gzip’d down to 1K.

[1]:
[https://dev.twitter.com/rest/reference/get/statuses/show/%3A...](https://dev.twitter.com/rest/reference/get/statuses/show/%3Aid)

------
jperras
Without sounding facetious: Am I amble to search (or at least paginate)
through all of my own DMs yet?

------
joelrunyon
Didn't backtweet do this? It was the most useful twitter search ever before
Twitter bought them and shut it down...

------
DAddYE
That's just a news for "investors" I don't see it coming to the api so I
wonder why this is an hacker news...

~~~
cmelbye
It's a technical post on their engineering blog about how they implemented it.
Why wouldn't that belong on Hacker News?

