
Storage for Photographers - PStamatiou
http://paulstamatiou.com/storage-for-photographers
======
famousactress
Yes, PLEASE. We run a wedding photography business, and our numbers are
something like this:

\- 40-60gigs shot in a weekend, need to offsite the RAWs as soon as possible.
In six or seven years we've never needed offsite retrieval. Fine for it to be
slow.

\- Only need around 100-200g of raws locally for jobs in progress

\- Once processed into final jpgs files are approx 8-10x smaller. Offsite
these as well, and need access to last 1-2 yrs of jpgs randomly/sparsely w/in
1-2 days of a print order coming in

I definitely think some well thought out service that uses maybe a hybrid of
glacier and S3 could be really awesome. Especially if the pain is taken out of
getting the bolus 40-50g of photos "uploaded" in the first place.. We'd
happily pay for a service that just sends us an external drive and a postage-
paid box.. we drop it in the mail after a job and get another drive sent to us
right away.

[EDIT: Occurs to me you might wonder what this looks like today. Today we get
home, and immediately back up onto an 8-bay Drobo Pro
([http://bit.ly/1diZUHD](http://bit.ly/1diZUHD)). It's a fancy proprietary
RAID-like system. I think ours has 7 1TB Drives in it currently, and it's at
about 70% capacity. Then we back up the same jobs to an external hard drive
which gets driven offsite (used to be my office, but now that I work at home
it's the in-laws place) and copied to a hard drive for permanent offsite
storage. Once that happens, the cards from the camera can be cleared. After
processing, the final JPGs get copied to the drobo and make it into the
sneakernet offsite process as well.]

~~~
Terretta
> _Occurs to me you might wonder what this looks like today_

I finally hit a sweet spot of performance, archive accessibility, and offsite
safety, by using internal SSD for latest import workspace, direct attached WD
Velociraptor (Thunderbolt) for current projects (can reconnect between Mac
Mini server and laptops as needed), and 2 direct attached LaCie 4big Quadras
(daisy chained via FW800) for archived projects (up to 32 TB).

Interestingly, because these are all direct attached, BackBlaze will happily
back them up offsite at ridiculously low cost if that's your thing. Initial
backup of several terabytes will take several months, though.

For fast cheap offsite backups, partner with a colleague with similar storage
needs who uses the same ISP via the same head end. Just backup to each other.
You can enjoy near LAN latencies at maxed out link speeds. This trick will
save you both time and headaches.

Finally consider a product like the CyberPower OR2220PFCRT2U (this one also
blends nicely in an AV cabinet) to keep an Airport Extreme and Mac Mini along
with all that storage running for about 80 minutes of power outage.

~~~
famousactress
The idea of partnering w/ someone using the same ISP is interesting. It's
really offsite that's the whole challenge. If it takes three days to get 60g
uploaded (setting aside the nuisance of saturating our bandwidth for doing
things like making Skype calls), then I effectively have close to no
solution.. because during a busy season I have at least one job exposed for
half of most weeks. House burns down, and I lose recent work.. which is the
most important work since it hasn't been delivered.

Really, I need something that gets a copy of the images away from my home
within a day to have something I'd be comfortable with.

~~~
scarecrowbob
Would it be feasible to simply dupe the drive and find a secure off-site place
within driving distance to store it or a storage machine, and then mail drives
to glacier (Amazon will import a HDD via mail for around 100USD) as the
storage machine fills?

~~~
xanderstrike
I worked at a wedding photography/videography place a few years ago and we did
a lower tech version of this. The owner had a storage array at the office and
at home, and he'd cart an external back and forth daily. This way there were
two copies of all work, and three of the most recent: one at the office, one
at his home, and the most recent work (that which hasn't been delivered) on
the portable drive.

------
tghw
We applied to YC with exactly this idea. We made it to the interview, but were
turned down. I wrote about that experience: [http://tghw.com/blog/pull-
hard](http://tghw.com/blog/pull-hard)

We pulled our bootstraps hard and tried to get it going, but soon realized
that there's a pretty big flaw with the idea: most photographers have no idea
that they should backup their photos. They know hard drives crash, but they
never expect it to happen to them.

This meant that we'd essentially have to become insurance salesmen, convincing
people that bad things will happen to them.

Eventually, we decided to shut it down, which I also wrote about:
[http://tghw.com/blog/well-that-sucks-what-else-you-
got](http://tghw.com/blog/well-that-sucks-what-else-you-got)

The old homepage is still up at
[http://www.snaposit.com/](http://www.snaposit.com/).

~~~
mfenniak
Professional photographers know they need to backup their photos.

It looks like your product targeted consumers, but professionals seem like
they'd be a better market. And it should have been priced way, way higher; the
more it cost, the more secure it looks. I'd have a hard time trusting my
money-making business assets to a $9/month service.

Of course, I'm not a professional photographer nor do I have your experience
in trying to build this business, so who knows what my opinion is worth. :-)

~~~
tghw
Actually, once we started getting into the market, we found that most do not
know they need to do regular, offsite backups.

Consider the some examples:

* An older photographer who grew up in the days of film who has just transitioned over to digital.

* A homemaker who is trying to add some much needed income for her family, with a passion but no formal training.

* A well-off software developer who moonlights as a photographer for fun and a little extra money.

Of those, only the third has been exposed, formally, to correct data backup
procedures. The others are likely to keep their photos in folders on the
desktop of their laptop.

Furthermore, photography is not a very lucrative profession for most
photographers. That limits the amount you can charge. Our original pricing was
much higher, but we realized that would be problematic, considering the
average photographer brings in around $30k per year.

~~~
mgkimsal
None of your examples contradict

"Professional photographers know they need to backup their photos."

A homemaker or techie moonlighting for extra income aren't really
'professional photographers'. They may take photographs that look
professional, but it's not the same thing.

One might say that professional developers use version control, then have
counter examples of "well, this web design student doesn't ever uer version
control, but she's the only person working on the code, so it doesn't matter".
This is not a professional developer, even if she is capable of producing
results that look like they were done by a professional.

A professional, imo, is defined as much by their habits and practices as by
the output.

~~~
xanderstrike
> doesn't ever use version control, but she's the only person working on the
> code, so it doesn't matter

As a web developer who works on a lot of solo projects, this sentence hurts
me.

~~~
cowpewter
Do you use version control? If you do, that sentence doesn't even apply to
you.

------
esonderegger
I really wish we as a developer community would better differentiate between
"backup" and "preservation". They are both related to storage, but are
fundamentally different problems. The article touches on the desire to have
photos safe for decades, but doesn't really get into the strategy necessary to
make that happen.

If we care about decades, storing the raw NEF or CR2 file is almost certainly
the wrong approach. Those files could easily be as difficult to open in
fifteen years as a ClarisWorks document is today (just to choose one example -
proprietary formats for professional software from the late 90's are all
pretty hard to open today). Also, important metadata is kept in the lrcat
file. Unless we expect to always be able to open our copies of Lightroom, we
need to migrate that data too.

We used to think of "archiving" as putting something on a shelf and forgetting
about it, sometimes taking environmental factors like humidity into account.
With our digital data, we need a plan for periodically checking it, both for
fixity and to make sure at risk file formats are migrated to current best
practices. This will require greater awareness for the complexities involved
(and the need for open file formats) from everyone from creatives to storage
and service providers.

By the way, every creative industry is struggling with this right now.
Recording studios have session files created in ProTools 4. Designers have
really old Quark files, even though they've been working in InDesign for the
past five years. Composers have Finale files. A lot of these file formats
depend on plugins to be able to display properly. I don't mean to sound
discouraging, I think there's plenty of opportunities for viable businesses
here. We just need to start using the right terms.

~~~
graue
In a word, VMs. If you make sure you can open that NEF or CR2 file in a
virtual machine, and the virtual machine image is in a standard, open format,
you're set. You just have to worry about the image format becoming obsolete
and not 10 or 20 different proprietary formats.

That said, I used to be a pretty prolific hobbyist musician and I haven't
pulled this off myself. I have tons of old songs I can't open because they
rely on a fiddly collection of old freeware plugins for freeware music apps
from 11 years ago. I also have songs I can't open because they only play in a
music tracker _I wrote_ , which is open-source, but only runs on OpenBSD.

It is a tricky problem and it seems hard to tackle because the market for
professional software inherently favors closed-source, which has no incentive
to adopt open, non-siloed formats.

------
_delirium
I've been looking into Glacier, but the pricing in the case where I might want
to retrieve a substantial portion of my data is quite complex. As far as I can
tell, the headline $0.01/GB rate for overages they quote is misleading. They
don't, as you might expect from that language, calculate ([GB retrieved] - [5%
of GB stored]) x $0.01 and bill you that.

Rather, they charge you on a monthly basis, according to your highest hourly
retrieval rate from any single hour _applied to the whole month_ (standardized
at 720 hours). So if your peak hourly retrieval in a month was 10 GB/hr above
your pro-rated free quota for that hour, you're charged for _7200 GB_ of
retrieval overage, or $72, even if you retrieved nowhere near 7 TB of data.
For companies with relatively constant retrieval this doesn't matter, but for
small users, someone who has a large overage one day will be charged as if
they were incurring the same overage continually all month.

To put it concretely, if you store 200GB of photos and then do a bulk
retrieval, you aren't charged $1.90 ((200-10) x 0.01). Rather, the xfer is
counted as 4 hours, so you have a peak transfer of 50 GB/hr. Your free daily
retrieval is allocated to the hours in the day you retrieved data, so in this
case 10/4 = 2.5 GB/hr free. So your overage is 47.5 GB/hr. Multiply by 720 and
$0.01, and the fee for retrieving 200GB of photos is $342. Plus $22.88 in
outgoing AWS bandwidth.

Or at least, I think that's what their pricing page says. Maybe someone else
can better interpret this explanation:
[http://aws.amazon.com/glacier/faqs/#How_will_I_be_charged_wh...](http://aws.amazon.com/glacier/faqs/#How_will_I_be_charged_when_retrieving_large_amounts_of_data_from_Amazon_Glacier)

~~~
sreitshamer
Right, you definitely shouldn't request all 200GB at the same time, because
it'd be expensive and also because you probably can't download all 200GB
immediately either.

Arq (a backup app I wrote) takes a transfer rate from you (guesses your max
transfer rate as a default) and then requests enough to be downloaded in 4
hours at that rate. 4 hours later, it requests another 4 hours' worth while it
downloads the first 4 hours' worth, and so on. This mitigates the peak-
transfer fee.

~~~
jasondenizac
Does Arq fit the OP's use case of backing up files to AWS without keeping them
on your local machine?

------
stevewilhelm
Smugmug's SmugVault

"SmugVault is an added service to your SmugMug account that lets you store
almost anything for next to nothing. Including files not normally supported by
SmugMug."

[http://help.smugmug.com/customer/portal/articles/84562-more-...](http://help.smugmug.com/customer/portal/articles/84562-more-
about-smugvault)

~~~
sreitshamer
Although Smugmug is awesome, the issue I'd have with them is the lack of
control. If my objects are in my AWS account, that feels more permanent long-
term. It's just a gut feeling.

------
callmeed
My company serves thousands of professional photographers (wedding,
commercial, fashion, etc.). What you're describing is definitely an issue for
photographers but I think getting them to use a tool like you've described
would be an uphill battle.

Right now, most of them use Drobos or similar devices. You'd be asking them to
add steps to their workflow which is always tough to do.

Also, I think your estimate of 18 mins to upload 6+GB is an order of magnitude
off. We have tools that allow people to upload JPEGs and most customers claim
it takes longer than that just for 100s of MBs.

My (wild) guess is that Adobe is going to get in this space too. It only makes
sense to have your Lightroom catalog in the cloud. Maybe they wont offer an
affordable solution for terabytes of data, but I'm not sure photogs want their
images in two different "clouds".

I think the key is to do customer development from full-time photographers who
aren't hackers. I'd be happy to send out a survey to some clients if you want
do more research.

~~~
bambax
My Lightroom catalog is backed up to a folder in Dropbox, and this works
really well.

You can't really have the current catalog in Dropbox because then each action
and edit become really slow.

But if you set up Lightroom to do a backup every time you quit the program,
and that backup is in Dropbox, you get the best of both worlds (strong backups
+ snappy edits).

Lightroom backups are really "backups", as they are named by date+time, so in
Dropbox I have a history of everything I ever did in Lightroom, not just the
current (last) version.

It's also fast because I suspect Dropbox is able to do some rsync magic and
only re-upload new packets; my Lightroom catalog is around 150 MB and the
upload to Dropbox is almost instantaneous (on a pretty lousy upstream
bandwidth).

Of course this doesn't address the problem of backing up RAWs, but as far as
Lightroom is concerned I feel pretty safe.

~~~
gerardmurphy
Just to be clear here... Your Lightroom catalog is very important but is not
backing you your Photos. (which I know you said at the end. I am just
emphasizing)

Your LR catalog contains your photo edits, metadata and history... but is not
your original files. Backing it up is very important but is only part of the
story. Putting your Lightroom catalog backups on Dropbox is a great and should
be done by everyone. (Again just not your active catalog.)

------
short_circut
This idea is great, but it ignores on very real problem. Namely that you still
have not addressed the issue of bandwidth. Take for example your 4th of july
example. If I were to do that upload the way you describe, I would already be
over my monthly bandwidth allotment with my ISP as would many hobbyists.

~~~
roc
The US bandwidth situation is the elephant in the room.

There really is no good solution to automated offsite backup when it takes
several days of uploading to handle a single day's worth of data capture. To
make any internet-backup service work, you'd first have to pitch the photog on
switching from their residential broadband package to something more expensive
-- if that's even available short of their leasing office space somewhere.

I guess you could try to build out a reverse-CDN sort of network, with local
relay nodes scattered across the various ISP networks, that might achieve
faster uploads from the user (and maybe not get counted against bandwidth
caps) which then use a fat pipe to send that data to larger regional storage
nodes.

Neither seems particularly plausible at scale.

------
joebo
How about git-annex assistant with the glacier remote? [http://git-
annex.branchable.com/special_remotes/glacier/](http://git-
annex.branchable.com/special_remotes/glacier/). It will move files off your
local once they have been transfered to the remote.

~~~
PStamatiou
The large part of this post for me is about the local thumbnail exports of the
RAWs so I actually know what files are what without having to go just by album
names, dates et cetera.

~~~
joebo
You could keep the thumbnails local and annex the raws. Based on the
thumbnails, you could selectively git annex get from the remote glacier

~~~
PStamatiou
good point

------
Hovertruck
I'm actually working on this idea right now!

[https://plover.co/](https://plover.co/)

We want to take backing up, sharing, and monetizing and make them very simple
and enjoyable to do. We've started rolling out slowly already, our goal is to
be open to the public within a month or so.

~~~
hamburglar
I hope your business model doesn't rely too heavily on the "monetizing"
aspect. I know there are lots of photo hosting services with a "sell prints"
option, but I get the impression not very many people actually use those in
enough volume to cover the hosting costs. I'm imagining a non-pro track that's
targeted at the enthusiast who just wants backup and sharing, but won't
actually sell any prints. I'll be watching.

~~~
Hovertruck
We don't make any money off of prints, only the photographers do. The business
model is subscriptions.

~~~
hamburglar
This is good to hear. BTW, it might help if you had a _little_ more info on
your landing page about the service. For example, I can't tell if this is
going to be aimed at true pros who are willing to shell out a decent amount of
money, or if I'm going to be able to afford it for my own pro-sumer needs.
Having to sign up for a mailing list and wait in order to potentially find out
that I'm nowhere near the target audience is frustrating.

~~~
Hovertruck
Heh, it is unfortunate that there's very little info there. We put it up a
while ago and have been so focused on actually building the app that updating
the website seemed unimportant. :)

~~~
ThomPete
Look at it this way. Improving your website (which is already nice btw) means
more people signing up to be notified when you are ready to launch.

------
danso
I'm in a similar situation as the OP...an amateur but avid photographer with
terabytes of RAW image files.

Currently, I have several multi-terabyte drives that I erratically back up
to...I just got a 4TB drive that I'm going to try to backup everything I've
ever shot in the last 3 years...

But I do want to move it to online storage...and I think this requires triage.
For nearly every photoset, I've quickly gone through them with Lightroom and
starred the ones that I kind of like, and then for maybe 1% of them, have
taken the time to fully process, label, and upload them (in JPG form) to
Flickr.

So for online storage, I think what I'll do is write a batch script that dumps
all the _unstarred RAW files_ as JPGs, because they chances that I'll ever
need these photos in RAW is very slim, and then upload them. For photos that
I've given at least one star and/or taken the time to properly edit and
categorize, I'll send up the RAW file along with the processed JPG.

If I have about 3TB of RAW files...and maybe 10% of those are RAW images that
weren't disposable...that's 300GB right there. And then the remaining RAW
"keep em just in case" files would end up being compressed from 24MB to about
2-3 MB (let's say 15% as an average)...so 400GB.

700GB is still quite a big footprint. However, the biggest problem will
be...let's say I give up local storage all together and rely on the
cloud...what's the best way to browse any photoset at any arbitrary time? When
it's all local, it's trivial to pop open Lightroom and do some quick browsing
and filtering. For online storage, I'll have to put a ton of work into proper
folder-naming, at the very least

~~~
PStamatiou
Your last few sentences are exactly why I wrote this post. I want everything
stored in the cloud but ALSO have a decent way of browsing them locally. The
idea is to just store tiny jpegs (you can set how big they are, but something
like 500-1000px wide likely, just for you to get a sense of which photo this
is and allow you to remotely delete, retreive, etc)

~~~
stevesearer
This is basically what I need for Office Snapshots. I keep all of the original
photos submitted so that I can use the full images in the future (if need be)
and would love to store them somewhere other than my laptop.

If I could browse locally with smaller images (as you describe) like I
currently do with Picasa (galleries, tags, etc) and then have the ability to
download the original images when and if I need them, I'd be in heaven.

------
saidajigumi
I have a serious problem with the proposal offered in TFA and most variants in
this thread: it encourages bad backup policy. Critical data needs three (or
more) backups. In my experience, one cloud provider can only ever be
considered as being one backup no matter their technical architecture.

For example, you can't (and in the broad "you", aren't even qualified to)
audit the provider for data-loss or SLA-impacting SPOFs. You just have to
assume that they're there. You also take on non-technical failure modes: the
provider can go out of business, get bought by an uninterested owner (think
Delicious), change focus (Google Reader), or experience myriad other problems.

FWIW, the best success I've had with helping others switch to good backup
policy happened once it became economically feasible to buy a new laptop and
two bus-powered external drives as big as the laptop's internal storage. The
externals become bootable backups, one of which can be preiodically rotated to
an offsite location. This has prevented severe dataloss events for myself and
others far more times than I care to count now.

~~~
famousactress
FWIW, I'm more technical than a typical user might be but I'd be _very_
intrigued by a provider that used multiple cloud service and gave me the keys
to the underlying S3 buckets (or whatever) so I could independently verify.
That's probly a more valuable idea to a company selling backup to nerds
though, I guess.

------
rdl
I still prefer local storage and self-hosted storage, but what I'd like is a
good plugin for Lightroom to manage cloud storage and multiple, versioned USB
drives (or network locations) for images, from within the Lightroom interface.

(for local storage, once you go beyond individual USB drives, I'd go with a
Synology 1813+ NAS ($999 chassis, 8-bay) or an Areca ARC-8050 thunderbolt raid
($1499, 8-bay).)

------
martin_kivi
I believe Photry (www.photry.com, currently in beta, I'm the founder) could be
a possible solution in the long run. Since we started, a lot has changed on
online photo storage market.

With that in mind we've started to look more on how photographers with larger
photo volumes could benefit from our service. From the feedback from few
photographers we've already included some features (RAW photos with smaller
JPG thumbnails, workspaces and personal sharing for example) that will make
the day to day workflow a bit easier. Currently our goal is to build something
that would fit better for professional photographers (public for clients,
ordering prints and possibly integrated payment flows, win/mac client +
plugins for your favorite photo software).

Pricing wise we are currently not the cheapest but we have thoughts on how to
make this better for high volume photographers. If you're one of them then
please send me an email (martin at photry.com) and I would love to talk a bit
further what we plan to do and if we would fit your storage needs.

------
uptown
A service would be great ... but you might be able to "roll-your-own" with
some basic scripts to handle the conversion (down-sizing) and backup sync -
and something like Koken to handle the sharing with friends & family.

[http://koken.me/](http://koken.me/)

For back-ups - just buy a huge disk or set of disks, and keep 'em off-site
wherever you have a trustable location (friend, family, work - if allowed and
bandwidth isn't an issue) and use CrashPlan's free software to sync a copy to
the secondary location. It's free after the initial expense of the drives and
gives you encrypted, but physical access to the content in case of an
emergency. You could find a friend with the same problem, and just agree to
dedicate a certain amount of disk-space to each-other for this purpose.

------
fsckin
I'm the SO of a professional newborn photographer who generated somewhere in
the neighborhood of 20-30TB of RAW photos, backups, LR xmp sidecars, etc in
the last year. JPEGs don't even register. High end DSLR RAWs are gigantic!

I'm the technical support for overall workflow, storage & backup engineer,
etc, etc. I wear many hats.

Right now I've got a backup on import to a FreeNAS (BSD) filer running on a HP
Micro Server. Lightroom backups go to the same filer and are synced as well.
My local server rsyncs the delta every night to an identical machine at a
friends' house.

This isn't perfect. The one place that isn't 100% backed up is current
projects in post-production. That's an acceptable trade off for now.

The most important thing to note is that there are two kinds of photographers:
Those who never lost data, and those who care about backups. As mentioned
elsewhere in this thread, converting members of Group A into Group B is a
tough sell. Video testimonials might be a good option.

A storage service for photographers _MUST_ be invisible to their workflow.
Make it as easy to use as an egg timer. At the _MOST_ , an addition of a
single screen with sensible defaults (just click next) on photo import.

Ignoring error messages that get in their way is to be expected. Care more
about their backups than they do. Send me (tech support guy) an email to
notify if there haven't been backups added in X days, along with weekly
reports of data usage per day -- how many photos, how large, what root
folders, usage trends, etc.

An ideal installation would be automatic detection of what program they're
using, what places files are stored (hint: It's not just on a local drive).
Detect new file locations and back those up too.

An inexpensive (<$25) option of "only backup current projects I'm working on
and anything I shot in the last month", that actually works... I'd sign up in
a heartbeat.

------
ukd1
Seems like [http://www.loom.com/](http://www.loom.com/) would be great?

~~~
antr
I've been on the waiting list of Loom for ages. Thought it was in the
deadpool...

~~~
ukd1
Nope, think they're just perfecting things!

------
azov
Backup is about having _several_ copies. If you delete files after you
uploaded them to the cloud, you don't have a backup - with all the problems
that come with it (say, if you delete a file by accident there will probably
be no way to restore it).

If your files are important to you - you need a back up. If not - just get rid
of them in the name of simplicity :)

PS. Also, uploading many gigabytes over a typical ADSL connection is just
painfully slow... even local NAS is too slow over WiFi for my taste, and if I
have to have a wire - I can just as well connect an external drive.

~~~
apaprocki
I wanted the benefit of Glacier plus backup functionality so I went with Arq.
It encrypts everything and supports Glacier. Works really, really nicely.

~~~
tankbot
I use Arq as well. Easily one of the best backup options for Macs.

------
throwaway420
I wish I could trust the cloud because it would be a lot more convenient to
not have to worry about storing photos, which involves a lot of effort once
you advance beyond a casual hobby and you get into storing Terrabytes of large
RAW files and maintaining adequate backups.

But the idea of storing any files in the cloud is very distasteful to me ever
since the NSA Snowden story broke. Even something as innocent as a
photographer taking pictures of a skyline or some beautiful architecture could
land him in hot water if the government is looking at metadata and figuring
out that he took lots of photos of a government building or a bridge or a
power plant and randomly throwing him on some watch lists or no-fly lists.
People have been hassled by the feds for a lot less.

My preference would be for all of these cloud storage providers like Google,
Facebook, Apple, Dropbox, Amazon, etc to unambiguously and fiercely challenge
the government on this and somehow unambiguously prove to people that they're
respecting peoples' privacy and not complying with this stuff. Their denials
so far have been obvious non-denials that are very carefully worded around "no
direct access" when that's enough wiggle room to do just about anything.

Using some kind of cloud photo storage is a bad idea IMO until this gets
sorted out.

~~~
chadk
FINALLY. I was going through every comment on this thread and I was wondering
why no one was talking about security. This is one of the key features of the
data warehousing business we have been contemplating.

------
qq66
Draft feature set:

\- Local daemon running that manages this process, on PC or Mac. It can't be
done solely from the browser.

\- The software manages the import from the SD/CF cards, you put them all in,
it sorts everything out and wipes the SD cards on the way out.

\- You then use Lightroom to do whatever you're going to do. The software
would do well to have a Lightroom plugin that lets you select particular
images to mark as "high priority" and "low priority."

\- Now the daemon uploads all of your RAWs, Lightroom metadata, and corrected
JPGs to cloud storage.

\- Based on your settings, these RAWs get discarded after some set amount of
time (maybe immediately, maybe never), and the highest-quality JPGs are also
discarded after a longer amount of time for lower-quality JPGs just in case
you need a emergency backup. After 6 months you lose the raws, 1 year the
highest quality JPG, 2 years the medium quality JPG, etc. You are charged per
gigabyte per month so you can decide how much you care about backup. Basically
offload the cognitive effort of discarding old data that isn't worth its
upkeep costs anymore.

\- The files are available via an API that you can use to build your own
gallery/shopping cart site, with the ability to call a certain quality, PROOF
watermarking, etc. on-the-fly. But the service offers a default site right
there for you to do the same, that's stylable and can be implemented on your
own domain.

\- You can get your entire catalog shipped to you on a hard drive at any time
for a few hundred dollars.

------
geuis
"I open the folder where Lightroom imports are stored and drag these shots to
the RAWbox (fake name for this ideal photo app) desktop application."

You shouldn't be touching the directory structure. This tool should be either
a plugin for Lightroom/Aperture/Photoshop or an entirely separate application.

I would lean towards separate application. It could be a background agent that
detects when new files are added, would work for any known editor or user-
defined directory.

------
bigbento
Over the past few years, I worked on a few versions of this as a side project.
The last iteration didn't do the backup itself--it connected with your
Dropbox, sync'd the raw images, and processed them at a preset +-2 EV, and 3
color temperatures. The image results were good, but honestly I never found it
particularly compelling.

In any case, building this sort of thing is a pain. The most obvious problem
is the size of the images. Glacier is an interesting backup solution, and had
just come out around the time I decided to stop working on my app. A less
obvious problem is raw conversion. All the best converters are proprietary,
meaning that I had to piggy back off the Mac OS converter engines. Again, I
never got to the point where I felt the product was worth the trouble of
buying racking up a Mac Mini to run a little market test. Maybe I'm wrong.

I had a (frankly somewhat scattered) draft of a post mortem written, and since
it seems like there's some interest in this subject, I posted it:
[http://scraps.bigbento.com/2013/05/30/hiatus.html](http://scraps.bigbento.com/2013/05/30/hiatus.html)

------
healthenclave
This is exactly what I was thinking about... I want cloud based storage which
I am not locked into with a decent UI , way to browse and basic image
management (something like ThisLife for the fronted)

I think it's possible to have a either S3 or GDrive based system that has a
Angular.js (or some other JS MVC) that can operate without a backend for very
simple browsing features and connecting to the S3 via apis.

I think it's doable ...

------
petercooper
If you want to avoid having any backup devices yourself, would you need to
upload to two or more Glacier-like services?

I also trust Amazon to stay in business but mistakes can always happen (not
just hardware but ID theft, card expiry while you're on a 6 month hike, etc)
and unless you're prepared to say "losing these is OK" using one service would
be risky.

~~~
PStamatiou
This is true, but what other Glacier-like services are there that have
$0.01/GB/mon storage + retrieval?

~~~
dublinben
Offsite tape backup can approach that price, but I don't know of any exact
alternatives. It would certainly fulfill the requirement of having two
different media types and providers.

[http://www.dpbestflow.org/backup/backup-
overview](http://www.dpbestflow.org/backup/backup-overview)

------
mrushton14
Highly recommend
[http://www.mosaicarchive.com/](http://www.mosaicarchive.com/)

------
martingordon
Has anyone used Lightroom 5's new Smart Preview feature? It generates lossy
DNGs from your full-sized files that you can still work with while your
original sits on an external drive. Once you plug your external drive back in
it, it syncs the non-destructive changes back to the RAW file.

------
madao
I did some on the side admin work for a Photographer once, a real big name. He
was in the middle of making up exactly what this was, however he came up
against the same issues everyone has. Firstly space costs money, Hardware
costs alot too, not to mention bandwidth, this guy would easily do 100-200gb
of photos in a couple shoots. all very high res shots (talking about for big
company ads) his idea is to sell a server with a software suit that people can
dump into a datacenter or at home and upload photos to it, personally I liked
the idea but local storage and hardware failure + the ability to grow the
storage was a real killer for his idea.. (not to mention that your playing
with peoples businesses and lives if you lose those photos)

------
Brandon0
I'll jump in here and share our current problem and how we are addressing it
today. Hopefully it proves useful for whomever tackles this project!

So we are an ecommerce company and we do our own photography for products --
and we do a lot of it. We average about 2-3 photoshoots per month, and each
photoshoot can be upwards of 40GB before any processing. We shoot raw images
and are starting to experiment more and more with video. After we do the post-
processing work and start the video editing we can push 70GB a shoot easily.

Our current workflow is like this:

Shoot tethered to a laptop each day (typically each shoot lasts two days). At
the end of the day, we upload the files up to our local NAS (Netgear ReadyNAS,
just shy of 3TB) which is redundant.

At the end of the shoot, we load a copy of the entire shoot on the local
computers that will be editing so that they don't have to work directly from
the NAS. Over the next few weeks while they edit the files, they will push the
new files up to the NAS into specific folders.

(This is where it gets hairy) We also have an offsite NAS that is located in
the business owner's house (about 6TB, but this one is not redundant). We used
to attempt to upload files each night from the local NAS to an offsite NAS via
rsync, but the pipe was just way too small. It would take days to upload a
photoshoot, and everyone in the office suffered from slow internet while that
happened. Now what we do is bring the offsite NAS _onsite_ every few weeks and
do a manual sync! Locally the sync only takes a few hours and then that night
it goes back home with the business owner.

Now since the local NAS is less than 3TB, we can't store all of our shoots on
there forever. When we get close to full on the local NAS, we grab an external
hard drive and archive old shoots (typically 2+ years old) to it, and then
store that in our massive safe (on-site though). If we need access to just one
or two files, we can retrieve them from the offsite NAS, but if we need access
to quite a few of them, we can dig them out of the safe.

So that's our workflow. Probably not bullet proof, but it has been serving us
well for the last few years. If someone came up with a better solution, we
would be all over it!

~~~
josh2600
Out of curiosity, how big is the pipe at the office? The one holding the local
NAS?

~~~
Brandon0
I thought it was a 50/5, but the speed test I ran just showed 65/5.

------
jrochkind1
> I just need to know they are somewhere safe for decades to come.

The thing about Amazon Glacier... unless they are themselves making more than
one copy, and taking hash fingerprints, and storing a couple copies of the
hash fingerprints, and using it all to check for bit rot on a regular basis...

...and I don't think they're doing all (any?) of these things...

...then people are sadly going to find that their stuff _isn't_ neccesarily
safe for decades to come. One copy on a single piece of media in an Amazon
data center somewhere does not actually make for 'safe for decades to come'.

~~~
eurleif
[https://aws.amazon.com/glacier/#highlights](https://aws.amazon.com/glacier/#highlights)

> Amazon Glacier is designed to provide average annual durability of
> 99.999999999% for an archive. The service redundantly stores data in
> multiple facilities and on multiple devices within each facility. To
> increase durability, Amazon Glacier synchronously stores your data across
> multiple facilities before returning SUCCESS on uploading archives. Unlike
> traditional systems which can require laborious data verification and manual
> repair, Glacier performs regular, systematic data integrity checks and is
> built to be automatically self-healing.

------
VikingCoder
I just came across the Plug Kickstarter, which I think might be the basis of a
viable back-up storage for professional photographers.

[http://www.kickstarter.com/projects/cloud-guys/plug-the-
brai...](http://www.kickstarter.com/projects/cloud-guys/plug-the-brain-of-
your-devices)

Specifically, the $129 level, where you get two plugs, which they claim, "to
replicate your data in 2 different places." Talk two of your friends in remote
parts of the country into letting you Plug in to their home networks, and this
might be viable.

------
gerardmurphy
I am the CEO/Co-Founder of Mosaic. We are a company that helps serious
photographers backup their photographers. I am happy to share what we have
learned in the past 2 years. www.mosaicarchive.com

As a background, we now have thousands of users and customers. We also
finished raising a series A in February. (Not that raising money proves
anything)
[http://www.boston.com/business/technology/innoeco/2013/02/mo...](http://www.boston.com/business/technology/innoeco/2013/02/mosaic_storage_systems_collect.html)

We are now growing at about 5%-7% a week. We recently peaked in the top 100
grossing US iPad Photo/Video Apps in iOS - although most of our revenue comes
outside of iOS. (As they take a 30% cut.)

We target serious photographers (pro's and prosumers) who use Adobe Lightroom.
For pure backup customers our sweet spot are those who have outgrown BackBlaze
and Crashplan. (Both great services but geared at consumers.)

On the backup side, we help photographers manage their backups by using Adobe
Lightroom metadata. Customers can choose to backup everything in Lightroom
automatically. However, many photographers use metadata like stars or flags to
designate their best photos. We allow users to automatically backup these
"best" photos. This provides the automation of the best backup tools with more
precision for larger customers.

We use Amazon Glacier behind the scenes to store the original photos. (Yes,
WMF the restoration costs even out for us over all of our users.) (Sidebar -
we actually ran our data center for months before switching to Glacier...very
glad to have made the switch.)

However... we are not an online backup company. Online backup is a feature is
our service... but not the entirety of it.

Once we have have photographers photos, we also want to help photographers
better use, manage, share, and enjoy those photos. At one point I would have
said we are a backup company, now I call us a workflow company.

We also offer a web and iOS App to privately view and share your Lightroom
photos. (We take JPG snapshots of the RAW files so they load quickly and have
the Lightroom edits. This part of the service runs in Amazon S3 not Glacier)
This gives you access to your photos in the same way you are used to
experiencing them in Lightroom. (Lightroom users try this for free -
[https://itunes.apple.com/us/app/mosaic-
archive/id627973694?l...](https://itunes.apple.com/us/app/mosaic-
archive/id627973694?ls=1&mt=8))

The next step which we are working on now is adding two-way sync where you can
manage your photos in the App and it populates back to your desktop files and
Lightroom. We are also working on more collaborative features so photographers
can share this culling and editing process with their clients or family
members.

The bigger problem we are solving is that photographers have loads of large
files... backing them up is painful yes... but managing them all is more
painful. Sifting through 000s of images to find your best stuff frankly
stinks. This problem is only getting bigger as we shoot more and more.

For those interested in this idea, please consider joining us. We are hiring a
couple more software developers this year.
[http://www.mosaicarchive.com/about-mosaic-photo-
storage/care...](http://www.mosaicarchive.com/about-mosaic-photo-
storage/careers/software-engineer/)

Thanks for the discussion and happy to answer all questions.

Best, Gerard www.mosaicarchive.com

~~~
PStamatiou
Thanks for replying Gerard. Do you ever see a version where I can store
directly on my Amazon Glacier account?

Especially given "However... we are not an online backup company." I'd feel
tremendously safer and more trusting of your service. I haven't used Mosaic so
maybe you already do this, I'll take a closer look this week.

Also (in a nice way), your site could benefit from a redesign. Unfortunately I
think quite a few consumers judge the trustworthiness and value of a brand by
their site/landing design.

~~~
gerardmurphy
Hi PStamatiou,

We have certainly thought about giving direct access to customers Glacier
accounts and behind the scenes each customer has their own glacier "vault" for
potentially this reason. But for many customers they would prefer a set-it and
forget it type of service. We didn't want a barrier of first create an glacier
account then call us...

We also then didn't know what to do if a customer then changes their glacier
files... if we are syncing with Lightroom and this gets out of whack... then
it defeats the purpose of the automated sync with your local files. If it just
"read only" access, we are already providing this with our App.

I think the difference in what we are providing in online backup is that we
are not a pure cloud where we encourage you to delete your local file - We
actually want to sync with your local files. Our philosophy on the workflow
side is that changes that you make to your files on the App, should be
reflected locally in your Lightroom catalog file. This way you always own your
own data.

I agree with you on the site redesign... have had our head down on this. Good
news is that the design is approved and being implemented as we speak
(although won't be out for a couple weeks.)

Thanks again and happy to answer any other questions. Best, Gerard

------
cenhyperion
Great post. I appreciate the focus on minimalism and hobbyist photographers. I
think it'd be an interesting project to take on. The technical side doesn't
sound too difficult, plug into Amazon, lightroom/aperture.

While the post mentions avoiding externals I think a handy feature would be to
push to both an external drive, glacier, and then remove the RAWs locally.
That way you have two copies, so the data is actually backed up. Trusting all
of your photography to amazon not losing data doesn't sound ideal.

------
mark_l_watson
I only have a 128GB SSD on my MacBook Air, which is my primary computer. I
used to run out of space until I read somewhere that you can use DropBox
advanced settings to not keep specific directories on specific devices. So, I
am not likely to have directories like, for example, videos2012, pictures2012
synced on my Air but they are on my old Linux and MacBook Pro laptops which
have lots of disk space.

With high network speeds, I can quickly re sync directories, or just power on
one of my old laptops.

------
peteforde
I'd use this. 100% for sure.

In fact, "a plugin for Aperture/Lightroom that backs up to Amazon Glacier" is
on my idea list. Great minds think alike, fools seldom differ.

One thing I'd like to point out is that photographers are unlikely to be as
cost conscious as other consumers. We put a lot of effort into taking these
damn photos, and it'd be nice if I could be confident that they'd outlive me.
I'm willing to pay a premium for that.

------
resistore
Relying on only one storage provider is a little bit risky to me. I started a
small desktop app for backing up my files with my family photos in mind,
leveraging the properties of erasure coding and multiple dropbox/google drive
services. Also adding a bit of crypto to follow the post-prism trend...
Please, take a look at
[http://yleprovost.bitbucket.org/](http://yleprovost.bitbucket.org/)

------
hfanson1
It is important to be careful with how you try to use Glacier. I have been
looking its pricing; It costs $0.01 per GB per month only if you want to
upload your data.

Downloads add significant cost if you want to retrieve more than 0.17% of your
data in a day.

Downloading 5% of your data increases your monthly rate to $0.097 per GB. 10%
increases to $0.187 per GB 50% increases to $0.907 per GB 100% increases to
$1.807 per GB

Downloads are also delayed by 4 hours after you make a request.

~~~
PStamatiou
if I have 2TB stored and only want to retrieve a single 10GB photo shoot one
month (or even just a single photo), it would not be anywhere close to 5% (.5%
in fact) of my total amount stored. The key point here is that we are dealing
with TBs of photos that are there for safe-keeping and they have already been
tinkered with and all photo JPGs needed have been exported and put elsewhere.
This is just for RAWs and you likely won't need to access them again,
definitely not 5%+ of them like this.

------
sreitshamer
I think Filosync (www.filosync.com, currently in beta, I'm the founder) would
work for this. You could have many projects in your cloud and only sync the
one(s) you need to a given computer. We'd just need to add a feature that
stores "previews" of the items in the folder so you could browse without
downloading. But it uses S3 in your AWS account, so would be cost-prohibitive
for a consumer storing terabytes.

------
espinchi
I'm helping build Photofeed, a platform that stores and organizes your private
photos from your different devices in the cloud.

Photographers with the problem described in the article are within the target
audience, even if we didn't implement some of the key features for them yet
(e.g. storage of images in RAW format).

We'd love having some early feedback from you guys! (Didn't officially launch
yet.)

------
foxhop
[http://www.snaposit.com/](http://www.snaposit.com/) tried but the market
didn't pay.

------
prirun
I've been developing a backup program for a few years now, HashBackup, that
encrypts your files locally, with your own private key, and sends them to
offsite storage you control: your own Glacier or S3 account, your offsite
rsync server (rsync.net for example), your Gmail or other IMAP account, an ssh
server, or ftp server. www.hashbackup.com

------
bcl
My solution is to copy them all to Glacier, I also have local copies, Glacier
is only for worst case scenario retrieval. I wrote about the little script I
use here: [http://www.brianlane.com/automatic-backup-of-files-
to-s3-and...](http://www.brianlane.com/automatic-backup-of-files-to-s3-and-
glacier.html)

------
FollowSteph3
And what happens if one glacier decides to close? Do you think you'll get any
notice ahead of time? I wrote about these types of risks at
[http://www.followsteph.com/2012/04/19/what-are-the-risks-
of-...](http://www.followsteph.com/2012/04/19/what-are-the-risks-of-cloud-
services/)

------
sreitshamer
I'm not convinced that retrieval time doesn't matter. If you decide you want
several photos, wait 4 hours to retrieve them from Glacier, then realize you
wanted a few more and have to wait an additional 4 hours, I think that would
get old pretty quickly.

~~~
hadem
I was wondering something similar. Does Glacier have any way to see a preview
of the images you are downloading? If I upload 100 images with serialized file
names (001.jpg, 002.jpg) how do I know which images to download?

~~~
sreitshamer
There's no preview. Actually there's no way to download or even list your
archives through the AWS Console. You need to write (or find) an app that uses
the Glacier API.

------
Paul12345534
Currently using Crashplan to backup my photos. I copied 200+ DVDs of photos to
my hard drive in numbered directories... then deleted them after I had them
uploaded (Crashplan keeps deleted files if you choose that option).

------
jeffasinger
I believe that [https://www.mediacloud.cc/](https://www.mediacloud.cc/) is
trying to solve that right now. However, their website is sparse on details.

~~~
PStamatiou
Definitely quite a few companies are attempting something in this space. I
think the one that will win will be the one that actually has photographers as
founders and understand the need.

Unfortunately it is quite a niche and most photographers are gadget hounds and
don't mind having 2-3 drobos (including an off-site one).

~~~
StacyC
It appears that Loom is founded by photographers.

[http://www.loom.com/about](http://www.loom.com/about)

I'd love to hear more about this service from some users. I am bumping up
against Dropbox limits with my photos as well.

------
mahyarm
I've always wanted to work on this, but didn't want to work on the
stereotypical nth photo sharing/management startup, especially with other
projects taking up my time.

------
adelmand
We (cintrexav.com - a small video conversion service provider) are building
something very much like this right now. Will post here when we get to MVP in
a month or so!

------
gcb0
What about flickr and the insane free space they provide? you it's free now
right?

searching is also a breeze there.

not sure how you'd go about downloading them all though...

~~~
philfrasty
no RAW support...

------
post_break
"I almost always shoot in small (5.5MP) or medium (11MP) on my 22.3MP Canon 5D
Mark III3."

Sorry but what? I actually had to read that a couple of times.

------
rb2k_
Wasn't that the idea behind "bitcasa" ?

-> [https://www.bitcasa.com/](https://www.bitcasa.com/)

------
josephlord
Covering video too would be great but maybe a second phase handling them
separately is a hassle for users.

------
foxhop
snaposit : [http://www.snaposit.com/](http://www.snaposit.com/) tried but the
market wasn't there. I'm going to send this post to the founder to see his
take on it.

------
tambourine_man
Backblaze?

------
hnriot
"I almost always shoot in small (5.5MP) or medium (11MP) on my 22.3MP Canon 5D
Mark III"

Kind of hard to take anything seriously after reading that. He buys a
seriously expensive camera and shoots it in point and shoot size. That's just
insane.

What's so hard with a simple home NAS. I replace the drive in my laptop every
year for whatever is then biggest (sweet spot) and ue a NAS for backups. RAID5

~~~
rfnslyr
Because cloud _this_ and cloud _that_! I feel like I'm the only person firmly
against cloud storage. I'd constantly feel paranoid about my data randomly
disappearing at the hands of a company I don't know run by people I don't know
on hardware I don't trust.

I want everything local.

~~~
icebraining
Local data can randomly disappear too - hard drives don't always fail in a
predictable fashion.

Personally, I prefer to use both local _and_ cloud storage (encrypted, bien
sûr).

~~~
13b9f227ecf0
I periodically mail a DVD of important stuff to my parents and ask them to
toss it in the basement. You could do the same thing with backup tapes, or
whatever.

Drives/tape/flash/dvd is pretty damn cheap and easy. These S3 type solutions
would have to be much cheaper to be interesting.

~~~
icebraining
A DVD holds 4.7GB. Box.net offers 5GB for free.

Besides, how do you verify that those backups can be restored when you need
them, if they're offline in some remote location? That seems rather
untrustworthy to me. I run checks against parity files regularly to verify
their integrity.

~~~
gohrt
10 DVDS hold 47GB. How much does that cost on Box?

And can you download 47GB faster than your mom can mail you a DVD?

~~~
icebraining
Storing 47GB would cost you 47 cents/month on Glacier.

 _And can you download 47GB faster than your mom can mail you a DVD?_

Probably, though I'm not quite sure about the speed of S3. I use a cheap VPS
instead (much more expensive than Glacier, but also much, much more useful),
from which I can easily get 3MB/s, so that would definitively be yes.

