You own your content. You are responsible for it.
It would be very wise of Twitter to offer a bulk export function - not because they have to, but because it's to Twitter's benefit to not have me thinking twice about whether I should post to Twitter or communicate in some other, more archive-friendly way.
Look at the 2.5 hour Facebook outage from a couple of days ago. If Facebook were to be hit by an airplane or asteroid, a seriously significant portion of 500,000,000 peoples' lives would be lost, if only just in photos. If my house were to have the same thing happen, 150 people would experience a somewhat lesser loss.
I have thousands of emails between close friends in Gmail that I would like to preserve, but no convenient system exists for archiving them away from Google. The point of archive is to have redundant systems on which to rely. Archiving my Gmail with Google is therefore not an archive since it's still only one system.
It's like leaving a tasty pork chop in the 'fridge at work and expecting the cleaning lady to not toss it over the weekend. I really think archive should be the responsibility of the user.
How about IMAP?
I haven't ran it in awhile but when I did it worked and was able to download my emails from the very beginning without crashing. Anyway, you might find it useful...
1. Yes, Twitter's real-time just like the phone, but it's not only real-time like the phone. You can revisit any tweet you like. They support viewing old tweets, they support listing old tweets. My only concern is that they place limits on how much they list.
2. Yes, we're responsible for our content, but the significance of services change over time. Tomorrow's Hacker News startup might be interesting, you might drop some content in, but most people aren't going to immediately create an offsite backup. Likewise, the tweets we made in the first two years of Twitter's life probably seemed worthless to us at the time, but now their value has increased. It just would be nice to have access to that.
This means that if you want a full history of your tweets, then it's your responsibility to maintain it. Would it be a good service to its users if Twitter maintained full history? Yes, but that's not your argument. Your argument is that it's a user's right.
There are other solutions around if you look.
And it's worked without a hitch.
It's nice that it groups tweets by mont and day, so you can pinpoint that one tweet you tweeted that day. Got mine running here: http://jarqu.es/t/
I've been playing around with a project to locally archive a bunch of data sources that interest me (email, instant messaging logs, Twitter, SMS, some blog and social news comments) in a straightforward and open data format. Unfortunately this type of tool might be something that most people don't realize they need until it's too late.
I've seen a lot of talk from Twitter that "you can trust us", "we still have all of your tweets", but it really doesn't help much if there isn't any way to get at them. Has there been any discussion from Twitter that this is even on their radar as a near-term priority?
I was sort of surprised that the Library of Congress work did not include some sort of web.archive equivalent that allowed anyone access to the entire database — Twitter has shown a willingness to let other people solve hard problems it didn't want to, including when it pointed developers to a 3rd-party firehose for a while.
It is much akin to NYT charging for access to old articles.
They may have to store more tweets in the current database or charge a retrieval fee to obtain it from the archive. They could give it away for free too...
ThinkUp can archive your tweets to your server for any reason you want to. Not only does it archive tweets, but it also allows you to view some nice analytics such as a map of where replies to a tweet came from, a chart of your followers over time, who your most active friends, etc.
But wait, there's more! If you check it out right now, we'll throw Facebook integration, for absolutely free! That's right. For absolutely nothing, you can archive your Facebook statuses, and preserve all those precious memories.
You can get all this for $0.00. That's right, $0.00. Just go to http://github.com/ginatrapani/thinkup/ and download the code today. You can create plugins for it, modify it, and do anything to it the GPL allows you to do.
No wonder dozens of services (or even curl-based shell scripts or tutorials) are available to back up the tweets (shameless plug for my own http://sparrw.com/, which focuses on easy searching of past tweets and treating tweeted links as sort of auto-bookmarks, which is what I'm often using Twitter for).
But all this works only if you're quick enough, and set up some sort of backup system before the magic 3200 limit :(
Can you imagine the implications of an account's entire tweet archives being open to the public? A prospective employer could look up every mention you ever made about them or their industry. Venting tweets would be forever inscribed in history. That's not good. It's bad.
My startup http://keepstream.com is involved in this real-time curation (Twitter now, more services later). I would love to chat with anyone interested in the subject; my contact information is in my profile.
There are a lot more features read through the about page (e.g. filter via dates, languages, sort against date, query dependent trends, ...)
(Yes, there's also http://thinkupapp.com/ - but it's more complex and has more features/supported services. I found TweetNest crazy easy to get started with though.)
Thankfully I took a CSV dump of my tweets 2006-2008 but I'm missing quite a few in the gap :-(
I was thinking recently about getting all the tweets from 2006, it should be about 2 million tweets, so should be possible to fetch by sequentially walking through the ids.
WHAT ARE YOU THINKING!!!!!!!!!!!!???????????????
I agree that Twitter can do a better job of allowing us to search tweets by topic, but that's a 3rd party app waiting to happen. In the meantime, take a screen shot and post on your web or in flickr.
Twitter still maintains a database of all
the tweets sent by a user. However, to
ensure performance of the site, this
artificial limit is temporarily in place
Keep on banging on at Twitter about it as users. It is your data, so demand it back from Twitter!
Can you imagine any other service surviving where you cannot lookup older data?
BTW Google has a social search that indexes twitter and facebook but it's not complete by any means.
I reviewed mine a while back and realised they were inconsequential drivel, so I deleted them along with the account. Now I have a read-only account for reading the tweets of some folk who tend to be funny to read. No techies trying to demonstrate value, no not-that-interesting acquaintances, no naive political rants from relatives - much better.