Hacker News new | comments | show | ask | jobs | submit login
Store arbitrary files with Flickr's new 1TB storage limit (github.com)
59 points by meltingice on May 21, 2013 | hide | past | web | favorite | 47 comments

This is kind of a dick move to use IMHO. Clearly they are not giving you the space to store arbitrary stuff. It's a neat hack and props to the author but if enough people use this it will not only cause Yahoo! to have to build countermeasures but will also set an ugly precedent for Yahoo! product managers to consider before doing nice things like they did today.

This is hacker news isn't it? Re-purposing things in ways that the original designers did not envision is one of the cornerstones of hacking.

Please don't try to imply that the rest of us here agree with you in any way just because we're on this site. Hanging out on a site called HackerNews and having a basic sense of right and wrong are things that plenty of us can do simultaneously.

There hasn't been a storage facility labeled get 'x' bytes,Kilobytes,Megabytes,Gigabytes or even Terabytes free that did not result in people wondering about if they could use it for general purpose storage.

What surprises me is that a service like this isn't hardened from day one against the most obvious of flaws.

Of course it's wrong, but it is only really wrong when a billion or so people adopt it, and chances are that this will never see widespread adoption, it's just a guy saying 'see what I could do', not an army of people overrunning Flickr.

The cognitive dissonance of seeing a single person perform a neat little trick versus an army of people performing that same trick bringing down a service sits well in my head, I don't feel this is in any way destroying Flickr, nor do I think that it potentially will destroy Flickr.

It didn't happen with Gmail or any of the other services that were 'exploited' in this fashion before. In fact, those are now trying to get me to put as many files on their storage devices as possible (which I really don't want to, the cost of storage is so low I don't need an external service to host my files for me).

Lighten up.

Worth noting that the original comment said:

> This is kind of a dick move to use IMHO.

Didn't say anything about creating the script, just about using it. Possibly, using it at scale was even implied.

Creating this is a hacker move. Using it to store your server backups is a dick move (and also potentially a poor backup policy).

Also: setting up a webserver with over 1TB of storage is probably simpler, faster and more reliable.

I also think in this case the hack is a little useless. But I've seen examples where they use it to transfer a lot of 3D data for use in WebGL. And the author of this script is also linking to a nice usecase where is is used for game data: http://blog.nihilogic.dk/2008/05/compression-using-canvas-an...

I remember playing with storing files in gmail years ago. Unfortunately, they rate limit how quickly you can "upload" emails via IMAP (or at least they did), which prevents you from using it as a general purpose filesystem. Also, it's dog slow, but that was always going to be the case.

Well, it also should be clear to everyone that:

1) The overwhelming majority of Flickr users will never read these stories or even dream about uploading files;

2) Most people who read this POC will not try it, and of those who try it, most will use it only once;

This is a non-problem.

Jacques, please consider for a moment that you can be right, and be an asshole at the same time. I don't consider being an asshole a cornerstone of hacking.

I don't mean literally that you're an asshole. I've always enjoyed your posts, and I get that you're not acting with malice, which is a cornerstone of being an asshole :) But this is one of the core tendencies that contribute to the perception that geeks and hackers are "socially inept".

Consider the following made-up scenario, even if it is contrived. Try to avoid the urge to pick it apart on a literal basis and focus on how this breeches basic social contracts. Then consider that normal people work at Flickr, and that those people are trying to do something good for users.


Me: This is my library. It's for storing books. You're welcome to store some of your books here as well.

You: <Starts bringing in boxes of random stuff and placing them on shelves.>

Me: Hey, what are you doing?

You: You didn't say I couldn't store random stuff here, and you haven't implemented any counter measures to prevent me from storing random stuff, so I figured it was ok.

Me: You're kind of an asshole. Did you know that?


I'm not saying the hack isn't cool. I'm not saying the hack shouldn't exist. I'm just saying that it's OK to point out that using it might be considered a dick move.

Like pooping in urinals?

Imagery aside, no more like building a house in a storage locker.

a.) flickr is a photo sharing site, no? the UI certainly seems to be optimized for mediocre or even pretty photos, not random noise. If I started pasting huge blocks of mime64 in HN comments to store encrypted personal stuff, or unencrypted cat pictures, would that be in the spirit of hacking, too? Or rather in the spirit of laming? (also notice the fact that I know better than to actually DO this even just ONCE, just to prove a point, because one never knows what random thing might incur a hellban)

b.) imagery aside, more like stepping into a storage locker, saying "I could theoretically live here!", everybody having their minds blown, and then not living in the storage locker because that'd be just pointless and awkward.

How about this: a tool to put something into series of high-ISO photos via steganography (and allow it to be updated, too) which can't be detected without the correct key.

Arguably, you could use HN as a storage mechanism because it just marks comments as dead, it doesn't actually delete them.

Hell, you could even write a utility to spuriously create HN accounts, stuff them full of comments that will be deaded but still accessible, and use that as a data store.

Whilst there is some "hacker spirit" to it, it is absolutely, totally a dick move. And yes, it is like shitting in the urinals, then smearing your shit all over the taps and the mirrors, because why the hell should the rest of us be able to use the facilities for what they were supposed to be used for, if you can have your fun for a few days?

We need a new name for this. It's not "Tragedy of the commons", it's more like "Tragedy of the common troll".

If you advertise free storage you can't really complain when people wonder what else you can store there.

If you advertise a forum then people will do what they can with that forum, some will troll, some will try to advertise (even if they have to spend a lot of time to groom their audience) and some will do that much more outright (spam).

If you launch any service on the internet these days you have to start by analyzing it from the angles of possible abuse first. This is frustrating in a way but it is also inevitable, just like in life there are no niches that contain harvest-able energy without some life-form that takes advantage of that by adapting to it.

Storage, CPU, bandwidth are the currencies of the modern age, giving out any one of those currencies is asking for it, especially when it's done in large denominations.

Well, yes. Until you find out that they are analysing your documents and send you an email explaining why you no longer can access them.

I guess 'noise' does not a photo make.

Of course I doubt they will implement that kind of filtering, but this 'invention' is not that original/difficult/interesting, I guess.

The fact that the original designers did not say anything about this does not mean they did not envision it, by the way. Simply that they may very much either do not care or have a recovery plan.

EDIT: this is obviously easily circumvented with a lot of traffic going on but it is just a simple idea.

The problem with Jacquesm is that he doesn't understand the term 'hacking' in general. Hopefully, I will try to explain what I know.


Hackernews isn't for people who hack stuff that is against the law or something like that. Why would PG want to create a site for criminals? Hackernews is for the other new generation 'hackers' (like growth hackers, design hackers) etc. Not for stuff like this.

Imagine, I rent you my apartment and you 'hack it' into a miniature casino or a drug dealership. When I rented you the apartment, I rented it to you in good faith that you wouldn't do these things. But now, you broke my trust and next time when I rent out my apartment, I would go take some extra steps like installing security cameras inside every single room of the apartment because I have to protect my apartment from unethical usage scenarios. That's the same case with Flickr. So, clearly, this is something that is destructive to someone who gives you something with a good intent. Just because you can do it, doesn't mean you should.

Thanks for making it personal, I will refrain from a response on that level.

Making a proof-of-concept of something to show what you could do is not as far as I know against the law, and if it is then it shouldn't be.

Using this - and breaking the terms of service - should lead to account termination, possibly banning but definitely not a brush with law enforcement.

I can imagine all you want but this is what it is, a simple hack to show a proof-of-concept, not an incitement or example of widespread abuse bringing down a popular service.

So clearly, it is not destructive, though in theory it probably could be used as such, but then again that goes for all the hammers in my toolbox too, including the plastic ones.

Just because you can build it does not mean you can't or shouldn't, it means that you shouldn't use it. Any home chemistry set could be used to make bombs (in fact, any supermarket with a reasonable assortment could be used to make bombs). You can demonstrate such by making a small one, and you can cross the line by either making a large one or actually using it against someone or some property.

Knowledge and ability, proof of concept versus actual malicious use, it's not such a hard line to draw.

> Hackernews isn't for people who hack stuff that is against the law or something like that. Why would PG want to create a site for criminals? Hackernews is for the other new generation 'hackers' (like growth hackers, design hackers) etc. Not for stuff like this.

That's a false dichotomy for one there are other kinds of hackers (the real ones, imo, instead of the ones that borrowed a term because it was cool, 'design hacker', 'growth hacker' as the target audience for HN? and you know this how?), and simply wrong besides.

What is legal and what is not depends on many things, intent for one and location second.

> Hackernews [sic] isn't for people who hack stuff that is against the law or something like that. [...] Imagine, I rent you my apartment and you 'hack it' into a miniature casino or a drug dealership.

This doesn't fall onto that side of the fence. This is just using a gratis service creatively. Sure it might eventually get blocked if it becomes a common thing to do, but most things like this don't.

I don't see how the label "hacking" confers anything about ethics.

Years ago I listened to the creator of Sodaplay: http://sodaplay.com/ (http://soda.co.uk/)

His speech was about how you develop something and people start using it completely different from what you imagined or even wanted. His tip: learn and take use of it.

Probably people like to have 1TB of free storage.

... bonus points for whomever gets an HTTP server running on flickr. Maybe I can store my music there too and stream from a webapp that plays these PNG files.

edit: sorry I must be on the wrong message board. Thought this would interest people. Guess I'll take my idea elsewhere.

That ruins the service for the rest of us. The phrase "this is why we can't have anything nice" applies perfectly to this post and comment.

Maybe we should re-brand the service we're posting this on to 'ConsumerNews' then?


Hacking does not have to equal abusing. I could advocate for fair use of a service and still hack it.

What's your position on Daeken's hack of the hotel room locks then?

IIRC, he was responsible about the hole by telling the company first and trying to work with them to get it fixed. He released the info after they were unresponsive. I think that's proper way to handle that.

But this thing with Yahoo isn't really the same. One is a security breach while the other is trying to abuse a service.

I like hacks and fun experiments. The idea of putting extra content in pictures is interesting but OP was talking about setting up a system around it to put it into wide spread use.

While I understand what you are saying, it does imply that people should not try to experiment or think of new usages because "it will ruin it for everyone else".

Yeah, on the one hand this sort of thing seems like a fun project, but on the other hand it's incredibly exploitive.

You'd probably need a VPS to do that. What you could do is have it pull files based off a database on the server, stick them in a cache, then convert them to what they really are (I assume the utility in the OP just adds a PNG header, so the VPS could just strip that) and then stream it.

Maybe you could hack it together with owncloud or something? I'm sorta tempted to try doing something like this but I don't want to screw flickr over.

I remember something similar popping up within days of gmail offering 1GB free :) Good times.

It looks like we'll have a FlickrFS Fuse implementation shortly.

Flickr aside, I am surprised nobody (AFAIK) has come up with a way of sanitizing uploaded image files yet.

I wonder if there's a way for GD/ImageMagick to detect the image data and strip everything else. (And if EXIF data is needed for photographs, import all non-binary EXIF data into the system first.)

Many do, but it isn't always "sanitizing". http://www.iptc.org//site/Home/Media_Releases/IPTC_study_sho...:

"According to a study by the International Press Telecommunications Council (IPTC), major social networks like Facebook, Twitter or Flickr remove copyright information and other useful embedded data from pictures posted by their users"

Imgur does this.

It also compresses big files though so I'm not sure if it can do it without effecting the image data and format itself.

Doesn't Facebook already do this?

mogrify -strip imagename.jpg

But it won't help. You could just make your data a real image.

You might as well change a single byte. A tiny modification and BANG! nobody notices and the service is pretty useless as a general back-up solution.

They could even implement that in their TOS: "whenever you upload a photo you agree to a random byte being modified on one of the pixels on the border."

It would be pretty easy to stick in some redundancy

Decompress and subsequent recompress would do this just fine.

What would do what fine? mogrify -strip removes what he asks...

Doesn't this already exist? Multiple times?

Oh well, obvious 'exploit'. Wondering how Y! will react. They must've forseen this, right?

Probably poorly.

Question though, how large of a file can you hide with steganography in a 300mb picture?

Would that be big enough to hide an MP3?

On a side note, can you upload files to Flickr that have data appended after the end of the image data? Like people were doing on 4chan until moot removed that capability.

I have a wee bit of Stego experience as I've written a couple of implementations. Generally for it to be "undetectable", you shouldn't go with more than 25% of an image file, assuming 24-bit color, being data, as it quickly becomes apparent that there is something fishy going on. Your best bet is to create a kind of "keyed stegonagraphy" where you generate a series of keyed nodes, creating a cycle (in the graph theoretic sense) of nodes, each node corresponding to a pixel, and the entire cycle determined entirely deterministically from the key.

This is akin to key schedulers used in various cryptography schemes, I suppose. The idea is that you REALLY don't want to just shove your data all at the beginning of the file in order, as it becomes really easy to tease out the data with some cursory frequency analysis/bruteforcing. "Oh the first 20 pixels encode the first X bytes of <insert well known file type here>, BALEETED!"

Then you simply have each user pick their own key, stored locally, and have the cycle generated on the fly when encoding and retrieving data.

They'll probably just ban the API key people use for it.

I was wondering just how long it would take for this to pop up...

Couldn't help but plug this on The Changelog: http://thechangelog.com/flickr-store/

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact