
Twitpic blocking archive team from backing up pictures - Argorak
https://twitter.com/textfiles/status/523160196672409600
======
stevencorona
I used to be the CTO of Twitpic (I left about 2 years ago).

It sucks to see your work crash into the ground like this.

Just so it's clear, there's just Noah (and his parents) that are left running
the company.

~~~
yuhong
Did you notice the people trying to contact Noah Everett?

~~~
stevencorona
Yes and he won't reply to me either.

------
djloche
Seeing as they're shutting down the company/service in a week, they're
probably out of cash. While it sucks for your customers, that's what happens
when you run out of money. It sucks for everyone. I'm guessing they were
floating the company on personal funds in hopes the acquisition would go
through, but it did not - so now they're doing everything they can to stop
this turning into a personal financial bloodbath from the bandwidth costs
spiral due to everyone rushing to get their photos before the shutdown.

Just an ugly situation for everyone.

~~~
lalwanivikas
Here's a solution:
[https://twitter.com/dcurtis/status/523215681412546562](https://twitter.com/dcurtis/status/523215681412546562)

Dustin Curtis: "Can anyone put me in touch with the founder of Twitpic?(I'll
pay to backup all of the photos, and even host them. Ridiculous situation.)"

~~~
dcurtis
This Noah guy (the founder) is really hard to get in contact with. If anyone
can make an intro--hi@dustincurtis.com

------
idlewords
For context, before archive team was blocked Jason Scott commented on
Facebook:

"Archive Team has 900 IPs aimed at Twitpic. We're grabbing 100 pictures a
second."

He subsequently corrected this figure to 42,000 photos per hour, which works
out to about 11/sec.

Later he adds:

"In some cases, twitpic (before removing all the image access) was banning
entire ISPs to stop archive team backing it up."

~~~
rgbrenner
In fairness, the archive team is attempting to download 120TB of data.. which
is going to cost Twitpic about $15K in outgoing data charges from Amazon.

Maybe they just don't have $15K to pay for this. (and that's not counting all
of the users who are also downloading their data)

~~~
tedivm
I have absolutely no doubt that people would be willing to donate to cover the
archival costs, especially if that donation was overseen by the archive team.
Options like this can only be explored if there is a conversation.

~~~
k-mcgrady
Yes but you have that conversation before you start archiving. If I started
racking up $1000's of dollars in server fees while going bust I'd shut off the
archive team too. Ask and offer to pay first.

~~~
nknighthb
They did.

~~~
k-mcgrady
Source?

~~~
nknighthb
Oh good grief, really? Every thread, including this one, has been full of
links. Here's one I have at my fingertips:
[https://twitter.com/textfiles/status/507595291634040832](https://twitter.com/textfiles/status/507595291634040832)

People have been trying to talk to him since the day the announcement was
made.

------
Mithaldu
Important to the situation is also that, as of ~24hours ago, the users
themselves couldn't download their own pictures:
[https://twitter.com/textfiles/status/522837349676236801](https://twitter.com/textfiles/status/522837349676236801)

Edit: While at the same time twitpic had no announcement on the frontpage and
users were still uploading:
[https://twitter.com/textfiles/status/522835537543974913](https://twitter.com/textfiles/status/522835537543974913)

~~~
SrslyJosh
And that's still the case. Twitpic's "download" link (from current the front
page) is handing out 503s, and has been all day:

[http://twitpic.com/account/settings](http://twitpic.com/account/settings)

I wish I could say this will be a good lesson in not trusting startups with
personal data, but it won't be.

~~~
eropple
Part of it is that almost every service you interact with on the internet is a
"startup". It's not reasonable to expect everybody and their (literal)
grandmother to know who's got an investor-friendly P/E.

~~~
Retric
You are really in a bubble. I can't think of any start-ups's I regularly
interact with. Now older companies that might or might not be profitable sure,
but 'start up's' almost by definition don't support large chunks of the worlds
population.

------
sp332
"you let the visual component of arguably the greatest communication
shift/revolution in the 21st century be under the whim of one idiot". 800
million pictures, just gone. For example
[https://twitter.com/textfiles/status/523160725158903808](https://twitter.com/textfiles/status/523160725158903808)

~~~
toomuchtodo
Assuming ~120TB [1], that would cost ~$1300 in AWS Export charges, plus
whatever the drives cost. Just sent a tweet to Noah offering to cover all AWS
export/drive costs to export the archive.

[1]
[https://twitter.com/textfiles/status/523162867789758464](https://twitter.com/textfiles/status/523162867789758464)

~~~
sethish
Yes. This was discussed at length with the Twitpic founder. He said something
to the effect of: "don't worry, I'm working on a deal, trust me. But stop
downloading the pics"

~~~
toomuchtodo
Was this in the last 48 hours? I'm hanging out in #quitpic as rumbles if
that's a better place to discuss.

~~~
sp332
No, about a month ago. [http://www.itworld.com/article/2834923/twitpic-really-
is-shu...](http://www.itworld.com/article/2834923/twitpic-really-is-shutting-
down-after-a-failed-acquisition.html) After the first time they said they were
shutting down, they said they made a deal and would stay open. Now they're
closing again.

~~~
toomuchtodo
Hence why I made the offer ~30 minutes ago :)

------
brm
This whole thing reads weird. From shutting down, to acquisition, to shutting
down again and seemingly blocking exports.

However, as this is the internet, until news breaks otherwise I will continue
to believe that Hanlon's razor applies here...

Never attribute to malice that which is adequately explained by stupidity.

~~~
IvyMike
Either way, I wonder if the twitpic guys fully understand the damage they may
be doing to their personal reputations. Do they expect to be able to run a web
service in the future? Because if the current situation persists, no one will
ever trust them again.

~~~
gtaylor
If we're being honest, the vast majority of their users don't know who was
behind Twitpic. The HN crowd is probably a lot more likely to know/care than
your typical Twitpic user.

They'll probably be able to start new things without most people noticing. And
they probably know this.

~~~
javajosh
Well, yeah, but it's the HN crowd that they will likely want to work with/hire
from, so reputation is actually important.

~~~
jamesaguilar
Well, a few things. First off, a lot of people on here probably understand
that when you don't have money to pay for bandwidth, banning people who use a
lot of it might be reasonable. It's not like you have a whole lot of other
options. Passing around the hat isn't likely to work, and if they could get
additional funding, they already would have.

Second, even if people didn't understand that, the HN crowd is a tiny slice of
the good hackers out there. Most of the best people I know do not read it, or
even actively disdain it. "Cesspit" and the like are a terms I have heard in
person or on newsgroups used to describe the comments section here.

Third, I'll bet most HN people don't even know the names of Twitpic's
founders. I don't. I wouldn't be able to identify whether another company,
down the line, was founded by them.

Lastly, I doubt most HN people are so principled that a year or two from now
they would turn down an otherwise compelling job offer because the founders of
the company were once associated with banning Archive Team's IPs. It's mildly
annoying at worst.

~~~
malandrew
If someone thinks hacker news is a cesspit of comments on the internet, then I
can only imagine that someone hasn't really spent any time on the internet.
It's not always sunshine and happiness here, but HN is definitely among the
top sites in terms of comment quality, especially considering the broad
variety of topics covered here. You really only end up with better quality
comments by going to sites that only cater to special interest communities.

------
ChuckMcM
This feels like a hostage situation again. The last time they said they were
going away they also prevented people from downloading their images. I'm
having a hard time understanding that (granted S3 'out' bandwidth costs money
but still.)

------
CanSpice
I saw this and thought "eh, that doesn't bother me, I already used the 'export
your photos' thing." So I went to my TwitPic settings page, clicked the link,
and downloaded a 14MB zip file.

The only thing is, it's empty. I don't know how you get a 14MB zip file that's
empty, but Windows is telling me it's empty. That's not cool.

~~~
unimpressive
Advice on your issue from #quitpic:

14:23 <REDACTED> you have to enable the "show hidden folders" in windows,
under their view options

14:24 <REDACTED> each version of windows is slightly different but it is
usually press alt, select tools, folder options, view, and the radio next to
show hidden files and folders and then ok

14:24 <REDACTED> then you close and reopen the zip file

14:30 <REDACTED> oh, and sometimes the first download is corrupt.

14:30 <REDACTED> I've had to help too many people at work with this problem
plus a person or two on this channel

~~~
CanSpice
Thanks, but that doesn't appear to be the problem with my file. I tried to
extract the files, and then Windows tells me that the archive is invalid.

~~~
selectodude
>14:30 <REDACTED> oh, and sometimes the first download is corrupt.

------
dumbfounder
Major headache for Twicsy! We now have to delete about a billion pictures from
our index. Which isn't trivial... such a shame. We reached out to him several
times over the past few weeks to see if we could arrange something, but he
never responded.

~~~
sp332
Do you have a cache of comments or photos from Twitpic? Could you stop by
#quitpic on efnet and say hi? :)

------
pjc50
Remember, any cloud service you use may shut down and delete all your data
tomorrow, whether through bankruptcy, acqui-shutdown, accident, or vandalism.
If it's a free service you will have no comeback. If it's a paid service you
probably won't have a comeback either.

~~~
iamtew
Free service yes, but if you're paying for SaaS and rely on it for your
business you should really get a source code escrow agreement in place. That
will allow you to get the necessary components from the provider in case they
go belly-up or such.

Of course hosting the application is a whole different thing, but hey, at
least you got all your data available.

~~~
pjc50
Has anyone successfully done the escrow thing? That would also only be
available to the very largest customers.

------
wmeredith
It was a free service right? Whatever happened to easy come, easy go? The
indignation around this is alien to me.

~~~
jamesbrownuhh
It's more about having a moral obligation to history. Twitpic has been a home
for hundreds of millions of images for a good chunk of a decade during which
the most monumental shifts in communication and expression have occurred.
Unique and irreplaceable moments have been captured in this place. To turn
around and delete it all just because you didn't get the trademark application
you wanted (and apparently can't be bothered to just think of a new name
instead) is nothing other than an unprecedented act of wanton, petulant
vandalism against history.

It would be like the world's biggest art gallery turning around and saying
'Oh, sorry, they won't let us call our gallery McDonalds, so we're shutting
down and burning all your pictures. After all, easy come, easy go.'

~~~
idlewords
Okay, but we keep getting burned by these irreplaceable droves of stuff going
offline (see GeoCities), and all our indignation doesn't bring them back.

At some point we have to learn this lesson for good, and stop trusting random
entities to be good stewards of things we find meaningful.

~~~
eropple
"We" aren't the problem. The problem is that startups need non-technical users
to use their stuff. People who aren't qualified to, nor remotely interested
in, managing a tech stack simply are not going to care about how random that
entity is. You can't make them care. It falls to awesome people like Jason and
the rest of Archive Team to keep that sort of historical record for posterity.
Because it's important. Trying to thwart them, as Noah Everett is doing, is
just straight-up not good.

~~~
cookiecaper
The Internet Archive should archive stuff _before_ a company enters financial
hardship and can longer afford the outgoing bandwidth costs to have their
entire dataset replicated online. It makes sense to block those attempts.

It strikes me that a similar fate would most likely have occurred if Everett
was hit by a bus and his bank accounts eventually ran dry.

We should identify historically significant single points-of-failure and make
archives _before_ the failures occur, insofar as is possible.

~~~
eropple
Sure, but it's important to note that I'm not talking about blocking the
scrapers. I'm talking about "we're willing to pay for an S3 export, Amazon
will mail out a bunch of hard drives." Everett has steadfastly avoided
contacting Archive Team or anyone else who wants to do this and I can't think
of an ethically not-icky reason for that.

~~~
cookiecaper
The not-icky reason is that he doesn't want to. He's not under any obligation
to do it. Maybe we'd _like_ it if he did this, but I don't think it's fair to
assume bad or even just "ethically icky" motives.

If all of twitpic's data is exported or copied elsewhere, he can no longer
effectively profit off of it. We don't know that his intention is to just nuke
the data and be done, he may be trying to salvage what he can monetarily (and
the "we're getting acquired, just kidding" thing is probably good evidence
that something like this is going on behind the scenes). I don't think there's
anything wrong with this either. It's his service, and if his motive was
profit, it makes perfect sense to shop the data around instead of just
accepting a total loss. There's no reason twitpic should have to be run for
altruistic purposes.

Maybe dcurtis et al would have better luck getting a response if instead of
saying "We'll cover the bandwidth costs", they said "We'll buy the data from
you at a price commensurate with the years of effort and upkeep you've
invested into an apparently massively important historical archive". I don't
think it's fair to pretend like Noah Everett is obligated to sell his archive
for the cost of bandwidth.

In any case, at this point he's made it clear that he wants to wash his hands
of the project in one way or another. It's his personal project and he is and
should be free to do that. This is the worst possible time to try to do a bulk
archive.

I understand that we can't see into the future and can't always predict when
something like this is going to go down, but I'd think we can probably improve
our heuristics so that something as seemingly significant as twitpic can't
just vanish into the ether next time.

~~~
eropple
_> The not-icky reason is that he doesn't want to._

Eight hundred million photos and all their comments. That's historical record.
That's _our_ historical record. No one jerk gets to destroy all of that. The
"hurr, well, don't use it" is completely bonkers--I don't use it, but _my_
history is on there, too. It's a record I didn't put there but matters to me
as a citizen and as a member of society. Destroying a major part of the
historical record for _everybody_ is indefensibly shitty, full-stop, you don't
get to say "well it's mine" when it's _not_.

It's ours. Twitpic was allowed to profit by holding onto it awhile. Now it's
time to do the right thing.

~~~
cookiecaper
It is his. This is what happens in an ethereal format like the internet, and
it's something we'll have to adapt to -- in the history of mass publication,
one wasn't able to just destroy all copies of a thing with the push of a
button. Books and pamphlets could be hunted down, but almost always some would
survive, because you can't remotely change the contents of the paper. In the
digital age, you _can_. If there's a lesson to take from this, it's the
dangers of the non-permanence of the web; we should be looking at solutions
that can mitigate that danger. But I don't blame Everett at all for not just
giving away his company's IP for $15k in bandwidth bills, and I don't believe
he should be shamed into doing so.

If I were him, I'd probably be contacting Sotheby's or another auctioneer that
specializes in high-value goods and seeing about selling off the dataset to
the highest bidder. Then these fellows can put their money where their mouth
is and show us just how important the preservation of that data is.

~~~
eropple
This celebration of plutocracy gone wild is one of the more disgusting things
I've read on Hacker News, and that's saying a lot. His corporation _only
exists_ through our societal forebearance. He has an obligation to the society
that gave him the legal structure and protections thereof. Part of that is not
deleting history just because he's in a snit.

~~~
cookiecaper
a. He's not necessarily deleting it.

b. If it's worth so much, people should be willing to pay a fair price for it.
The implication that he should let it all go for a payment to Amazon is,
frankly, insulting, and I'm sure you'd be singing a different tune if it was
your own startup that you were trying to liquidate.

c. Talk is cheap. It's easy to try to shame someone into letting you take
their intellectual property. It's not plutocracy, it's just business.

d. Getting a lot of users doesn't require your company to convert into a
charity. I am 100% sure that Noah Everett has a price, and I'm sure if people
_actually_ valued the Twitpic corpus he'd be able to get it.

e. All uses of Twitpic were voluntary and done with the implicit
acknowledgement that the images and comments may go away one day. If someone
was not OK with this, they should've saved their data, and many probably did.
Blocking the Internet Archive from devaluing the dataset is not bad, wrong, or
ethically icky in any way. The larger ethical crime lies at the feet of the
people trying to convince Everett that he doesn't deserve fair compensation
for curating this historically significant dataset.

f. If no one is willing to pay the money to preserve the dataset, it's
obviously not as culturally significant as you believe it is. There's plenty
of money out there.

------
ddorian43
I just opened the archiveWarrior and it's downloading from twitpic.

Help them:
[http://archiveteam.org/index.php?title=Warrior](http://archiveteam.org/index.php?title=Warrior)

~~~
asiekierka
It's only downloading comments and metadata, not the images.

------
cperciva
In their defense, it's entirely possible that they simply don't have the
capacity to handle the large volume of requests...

------
nkozyra
Given these points:

\- The whole shutdown / acquisition / shutdown saga \- Tepid response wrt
getting compensated to export data

Someone's still holding out hope for an injection of capital. Because once the
export happens and the data is gone, the value is entirely lost.

I wouldn't bank on getting any traction on paid exports until the last
possible second.

------
serve_yay
I dunno what the hell these people's problem is. Nothing about their actions
makes any sense to me.

------
bond
Every time I read about Twitpic closing I remember this...

[http://mixergy.com/interviews/twitpic-noah-
everett/](http://mixergy.com/interviews/twitpic-noah-everett/)

------
mariusz79
Come on, wasn't twitpic free? What why should they keep paying for hosting if
the business went under?

~~~
sp332
Paying for hosting? They were making money from ads they put next to our
pictures. They were using our content to drive traffic to their site. Without
the user's photos, the site is worthless. They should have been paying users a
cut of advertising revenue like YouTube does. "Free" means they're pocketing
the profit for themselves.

~~~
billmalarky
This is absurd. They were providing a service that users deemed valuable, and
users gave them access to the content (and the ability to monetize that
content) in exchange for their service. You don't actually believe Facebook
should pay you for using their free service do you?

~~~
sp332
This isn't about money. People have offered to cover the costs of a backup.

------
influx
If they are on S3, would they be willing to take money to do a data export?

~~~
IvyMike
Thus far the Twitpic guys haven't been very open or cooperative with the
archive team guys.

[https://twitter.com/textfiles/status/523162680400830464](https://twitter.com/textfiles/status/523162680400830464)

~~~
toomuchtodo
I sent a tweet to Noah offering to covert all costs of an export. We'll see
what happens.

------
craigmccaskill
Maybe they're having issues fulfilling individual users requests, so blocked
the archive team to help reduce stress?

I can't imagine they'll want to spin up new machines, so it makes sense to
throttle/limit/block.

Not sure I'd immediately jump to malice here.

~~~
duskwuff
No, like, right now images hosted on Twitpic are _completely_ inaccessible.
You can't view them at all.

~~~
craigmccaskill
That doesn't mean this isn't an effort to reduce stress on the user collection
download system.

I'm not saying that it _isn 't_ malicious. I just don't think it's wise to
jump to that conclusion straight away.

~~~
dthunt
Name the next plausible explanation and how likely you think it is relative to
'save money, screw your data'. I think you'll have takers on a bet at those
odds, given what you just said.

------
jszymborski
I don't understand why they wouldn't just put the pictures up on BitTorrent?

~~~
Xorlev
That's a LOT of data. 100s of TBs.

Good luck.

~~~
jszymborski
Doesn't the torrent protocol lend itself to huge amounts of data? Plus, you
can select which files to download in the protocol.

~~~
malandrew
I reckon the feature that is missing is not the ability to pick and choose,
but instead to automatically assign people trying to help preserve something
to one chunk, with the idea of getting full coverage of the entire data set.

If there are 100 pieces, and 500 people, you want each piece to be held by 5
people for 5x redundancy. And, if nodes leave the swarm, the system
automatically rebalances which nodes are maintaining a particular piece of the
dataset.

The only solution being worked on out there that does this is IPFS

[https://github.com/jbenet/ipfs](https://github.com/jbenet/ipfs)
[https://github.com/jbenet/node-ipfs](https://github.com/jbenet/node-ipfs)
[https://github.com/jbenet/go-ipfs](https://github.com/jbenet/go-ipfs)

------
jtchang
This whole thing is a fiasco. But maybe if it wasn't Twitpic wouldn't get as
much exposure.

My bet is that this is just a giant media whoring play. If you want to save
the company data then you have to be transparent and ASK for help. If it is
going to cost $15K someone might very well step up. But if you are tightlipped
then people will assume the worst.

------
thesis
I might be unfamiliar with how the archive team operates but shouldn't this
have been ongoing over the years rather than all at once now?

~~~
asiekierka
The Archive Team only backs things up when there is a risk of them being gone.

They cannot do it any other way - the Internet Archive does not have infinite
resources to host things, and they need to prioritize. 120TB is not an easy
thing to host.

------
differentView
What's the big deal about these pictures? It's not the first time a major
service has shut down. I am assuming historically important images have been
saved and hosted elsewhere.

~~~
Argorak
The pictures, yes, but not the conversation about them.

~~~
differentView
Why are those conversations important?

------
paulhauggis
Should we now have this expectation for any free github project that's
popular?

~~~
paulhauggis
I guess the hypocrisy and entitlements will continue....

