

Ask HN: indexing the revolution - tamersalama

Amidst my feeling of joy and gratitude towards younger generations - I have an idea (among other) for indexing the revolution (tweets, media, articles, posts, etc...)
Any suggestion about a platform, or any prior similar experiences would be appreciated.
======
RobGR
The Internet Archive or the Library of Congress may archive much of the
material:

[http://www.readwriteweb.com/archives/twitters_entire_archive...](http://www.readwriteweb.com/archives/twitters_entire_archive_headed_to_the_library_of_c.php)

<http://www.archive.org/>

If you want to archive it in a searchable way yourself, you may find that a
very technically daunting task. People use large clusters of computers and
pretty advanced technology to attempt stuff like this.

On the other hand, if you restrict the amount of data you want to ingest, you
can get pretty far just by configuring existing software. You could set up
Drupal on a hosting service that include Apache Solr search/indexing, such as
Acquia or Pantheon, and then configure Drupal to pull in information from
selected twitter searches and feeds of various blogs, and archive it.

Eventually you would run into space and volume restrictions. If you picked
enough data sources, you eventually might max out the rate of wrties to MySQL.
But you can go pretty far just by point and click configuring what exists.

------
nyellin
I don't know exactly what you are trying to do, but all Tweets will be
archived anyway. (Source:
[http://www.usatoday.com/tech/news/2010-04-14-library-
congres...](http://www.usatoday.com/tech/news/2010-04-14-library-congress-
twitter_N.htm))

I think a visual collage of the revolution on social media would be very
interesting.

