BackBlaze, an online backup company, posted a great blog article a while back about building their custom 67TB servers for just under $8,000. They even include all the how to, and a link to the custom case manufacturer if you wanted to do it yourself: http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-h...
There must be something hard about it. I work at a huge company and constantly get emails about deleting files from my home directory, because there is only 100G of storage for everyone.
All I know is that I have a 1TB 3-way RAID-1 array at home, and it cost less to build that than it costs to pay me to cleanup my inbox. Not My Problem, I suppose...
Where I work we have a super fast SAN for our storage, with a giant cache, fiber everything, and separate channels for backups. The cost works out to something like $1k/GB. Beyond that we're perennially short on rack space, so there's a cost to buying new storage. I used to believe it would be cheaper for the company to add storage than to make me clean up my home directory, but I worked out the numbers one day and it's simply not the case.
The artifically low file storage is a plausibly deniable tactic to get you to delete your old files, minimizing the chance of legally discoverable material being subpoenaed in the future.
It's actually because if you give users 100gig shared they will manage to use it and be slightly miffed about having to do housework on it every so often. But if you give users 1TB shared they will manage to use it and be slightly miffed about having to do housework on it every so often.
Specifically, I asked the IT department at a previous employer about the 50MB (no joke) email quota, and they specifically said it was for legal reasons (not on the record, of course).
Document retention policies are very common and rarely hush-hush. The only time they would have a need to make it "off the record" would be when they were subject to discovery regarding a lawsuit or criminal charge.
Re document-retention policies: Typically the real concern is the cost of document review: If there's a lawsuit, a lawyer or paralegal ($$$) will have to review all the potentially-relevant documents to determine whether there are any that should be (i) labeled as confidential, and/or (ii) withheld on grounds of attorney-client privilege.
(Footnote: The latter gives rise to the bigger fear -- if you were to inadvertently give the other side a copy of a privileged document, you might be deemed to have done a "subject-matter waiver," entitling the other side to all other privileged documents covering the subject(s) discussed in the first document. Moreover, the other side might be able to take the depositions of business people and lawyers, which amounts to interviewing them on videotape, to find out just what the client discussed with its lawyers, including who said what. The other side can usually be counted on to play any awkward sound bites for the jury.)
Having enough seeking bandwidth across 7200 RPM or 10K RPM or 15K RPM bottlenecks sometimes contributes to not being able to use 1 TB 7200 RPM disks in enterprise file servers with acceptable performance, but needing more expensive (per GB/TB) options.
Then there are the costs associated with backups.
Not that this necessarily explains away your complaint entirely.
Get a distributed file system and keep it on a number of different machines -- then you don't have to care about the connections and it scales easier to more users.
You could probably build one today, but if you mean for a reasonable cost, then it's probably 15-20 years depending if storage doubles every 18 months or 2 years.
You can already buy 512GB USB drives, because you don't need to power them while not in use you can basically stack them indefinitely the only question is would you conciser a 50 pound USB device a USB stick and would you pay several hundred k to do so?
Storage has experienced rapid exponential growth for a while.
