

Ask HN: The best way store thousands of Images on a server - barredo

Hello. I've got a server of my own handling the multimedia files for a website. It has around 200Gb of images (all JPG's) stored in nested folders.<p>I have Apache 2 on Linux and nothing else. I don't know if it is the best solution, but I think is not.<p>What system (http server, cache, or anything) would you recommend me?
======
soult
I recommend gatling: <http://www.fefe.de/gatling/> It's the fastest httpd I
know of.

You might also want to try thttpd: <http://www.acme.com/software/thttpd/> I
heard all the porn sites use it to host their images.

With only the http server for the images running it should work quite well to
rely on the filesystem cache.

------
kierank
MogileFS might be what you're looking for:

<http://www.danga.com/mogilefs/>

with an nginx or lighttpd frontend.

The documentation is a little sparse though the mailing list and irc are
useful for help.

Also it would somewhat depend on your traffic profile. If you've got thousands
of random reads (e.g. thumbnails with dozens per page, ssds might be better;
specifically the Intel X-25).

Otherwise MogileFS with an array of fast drives would probably do the job.
Mogile can also do load balancing based on drive I/O because each image is
replicated across different files/machines.

EDIT: Make sure you've got browser caching set up correctly (though admittedly
that's a tiny bit harder with Mogile)

If you can use a expiry time that's far in the future. Also don't send a Last-
Modified nor an ETag Header. Only send Expires.

------
blurry
Check out this Stanford presentation

 _Facebook - Needle in a Haystack: Efficient Storage of Billions of Photos_

<http://www.flowgram.com/p/2qi3k8eicrfgkv/>

------
iuguy
How about Publicfile? - <http://cr.yp.to/publicfile.html>

Super duper secure and very, very quick, especially for smaller files.

I'd try and run some tests against it, thttpd and your existing apache setup.

------
mqatrombone
A smaller web server like lighttpd or cherokee may be a better choice if all
you are serving is images.

------
pclark
<http://aws.amazon.com/cloudfront/>

~~~
barredo
30 days a month * 1024 Gb of transfer a day * 0,17$/Gb = more than 5,000$ a
month. Am I right?

My own 200mbps connection costs me the equivalent of 300$ a month.

~~~
frisco
A /terabyte/ of transfer a day?!?

~~~
barredo
Yes. It has a lot of traffic. Btw, it's not porn! :-)

------
leej
dont forget nginx

------
catch23
Amazon S3

~~~
barredo
I have my own server to do that. With 1TB of transfer a day, I cannot afford
Amazon S3 (I have my own 200mbps low latency connection)

Thanks, but I was looking for technology to install on my own server. Not
moving to Amazon S3

