
Canon's cloud platform has lost users' files and can't restore them - chvik
https://www.digitalcameraworld.com/news/canon-websites-held-to-ransom-by-hackers
======
zxcvbn4038
This sounds to me like the classic “replication is not backups” situation
where (at best) all of the user files were stored in RAID array someplace and
that is what the malware ate. If there had been actual backups and effective
backups then it should have been trivial to restore non-corrupted files. It
also sounds like someone made the decision not to backup the raw images
because they were “big” - that is actually the one thing they should have
backed up because all of the smaller files can be regenerated from the raw
ones. I would not be surprised if all of this was running under someone’s
desk.

~~~
ibeckermayer
What’s the difference between replication and backups? Is the distinction that
backups must be stored on separate infrastructure, whereas replication might
still be 1 or 2 points of failure?

~~~
lukevp
Replication is about having data level redundancy to protect from drive
failure. Backups are about having point in time snapshots of the system state,
and about having them tiered from a location perspective. The 3-2-1[1]
principle says to have 3 total copies, 2 of which are local but on different
devices, and 1 of which is offsite. This gives you tiers of recoverability.

It’s important from a backup perspective that it’s point in time as well,
otherwise as soon as you get ransomware that encrypts your file you now have
replicated those changes everywhere.

[1] [https://www.backblaze.com/blog/the-3-2-1-backup-
strategy/](https://www.backblaze.com/blog/the-3-2-1-backup-strategy/)

~~~
mumblemumble
I would add to that, that the offsite one should also be off _line_.

~~~
skoskie
That’s a lot harder to pull off. What methods do you use to accomplish this?

~~~
randoramax
S3 bucket with object lock is as offline and convenient as it gets

~~~
reaperducer
I don't know much about S3, but isn't S3, by definition, online, not offline?

~~~
zxcvbn4038
I’ve always wondered if Amazon backs up S3. I don’t think they explicitly say
but I get the impression that it is the user’s responsibility to replicate to
a second region to guard against data loss so I am guessing not. Object Lock
wouldn’t protect against an S3 failure.

~~~
count
They say it's designed to provide significant durability :
[https://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurabili...](https://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html)
(99.999999999% durable over a year per object, and able to sustain data loss
in 2 facilities).

Given the peeks that amazon has provided into the scale of S3, I don't know if
you CAN 'back it up'.

------
radicaldreamer
This is all because someone thought it would be “easy” to add a stream of
recurring revenue for cloud photo storage but didn’t take the time to design
the service from a technical perspective to be resilient enough to not crash
and burn.

~~~
nabla9
This can be just calculated risk.

Incredible damage for some users can be negligible damage to the company. In
this case just 10TB of data was lost, so maybe thousands of users for the long
term storage option.

Losing customer data has potential for big reputation or brand damage, but
surprisingly often the damage is relatively small. Putting in too much effort
when potential damage to business is minimal may not be worth of the cost.

~~~
kohtatsu
This comment reads as "This is normal; nothing to see here move along".

Which is more of an indictment on the state of things than on the comment
itself.

If you can't protect people's shit, don't offer to hold onto it for money!

~~~
wodenokoto
No, that comment sounds like the insurance job described in Fight Club[1]

> Narrator: A new car built by my company leaves somewhere traveling at 60
> mph. The rear differential locks up. The car crashes and burns with everyone
> trapped inside. Now, should we initiate a recall? Take the number of
> vehicles in the field, A, multiply by the probable rate of failure, B,
> multiply by the average out-of-court settlement, C. A times B times C equals
> X. If X is less than the cost of a recall, we don't do one.

[1] Video link:
[https://www.youtube.com/watch?v=SiB8GVMNJkE](https://www.youtube.com/watch?v=SiB8GVMNJkE)

~~~
jaclaz
Or that actually happened, in the known Ford Pinto Memo "affair":

[https://en.wikipedia.org/wiki/Ford_Pinto#Cost–benefit_analys...](https://en.wikipedia.org/wiki/Ford_Pinto#Cost–benefit_analysis,_the_Pinto_Memo)

~~~
throwaway0a5e
And a little further down the same page you linked....

[https://en.wikipedia.org/wiki/Ford_Pinto#Retrospective_safet...](https://en.wikipedia.org/wiki/Ford_Pinto#Retrospective_safety_analysis)

Turns out the Pinto was within the realm of normal for a car of that type and
era and if you're building something normal it's hard to justify recalling it
at great expense unless everyone else is recalling it to (in which case that
would be normal).

~~~
jaclaz
Yep, but that is an analysis from a Legal viewpoint.

What - I believe - made the Pinto case memorable was not that in a given
subset of accidents it was (or was not) slightly more dangerous, it is the
fact that the matter was looked into by the manufacturers and that it was
waved off on a mere cost/benefit basis (with a supposed cost for increased
security per car of US$ 11).

Mind you this is all in all normal, any safety norm or related technical
specifications is (or should be) based on the "reasonable costs for society to
obtain a reasonable increase in safety".

------
dharma1
Japanese camera companies are really bad at cloud software - Sony Playmemories
probably the worst offender. Also the in-camera "mini-apps" tend to be
terrible.

They make good hardware but horrible software. They should really buy a
computational photography company that also knows how to make good web and
native apps, like [https://skylum.com/](https://skylum.com/)

~~~
irrational
We were actually on NTT (Nipon Telegraph and Telephone) cloud for many many
years up until very recently. At the time we started using them AWS, Azure,
Google Cloud, etc. did not exist. But over the past few years the quality of
service has dropped considerably. My guess is that their best engineers were
being poached by AWS, Azure, etc. and the leftovers couldn't keep it up.

------
SturgeonsLaw
I take nightly images of my computer's primary drive and replicate them
offsite, with alerting if it fails and automated scheduled backup restore
testing.

It blows my mind to think that some random schmuck like me has better backup
procedures than these multibillion dollar global corporations.

Companies that don't invest in their IT end up paying for it tenfold down the
track.

~~~
dannyw
On topic topic, can I get HN’s advice on my backup strategy?

All important files on Synology Ds218+ NAS, which has 2x12TB helium HDDs in
mirror.

Daily HyperBackup to Google Drive. I test restoring files on a monthly basis.

Email alerts on failure.

Anything I can improve?

~~~
DaiPlusPlus
Google, as a company, is not reliable - I refuse to host anything critical on
Google Drive (even G Suite) because there's zero recourse or accountability if
a bug in Google's software or if Google arbitrarily decides to ban you for
life from their services (e.g. due to misidentifying you from a dodgy YouTube
user or repeated Google Play Store policy violations) then they will delete
all of your personal and business accounts and you're SOL.

What you have is fine for short-term recovery - but I'd make sure you have a
long-term and/or cold-storage option set up. It doesn't need to be anything
particularly fancy: I'd get a rugged+ portable+external single-drive USB
enclosure with a single 16TB drive and have the Synology do a backup of your
most critical data onto that drive and store it in your bank's safety deposit
box or better yet: leave it with a trusted friend who lives in rural Minnesota
or similar.

It goes without saying to encrypt that drive as you won't have full custody of
it: but use a simple, proven encryption scheme with a large ecosystem that you
know that you'll be able to decrypt in 10-20 years' time.

For backup drives that I keep local to me, I refuse to encrypt them because
(in my experience) the possibility of being unable to decrypt data in a
desperate or urgent situation just makes me wince.

~~~
MaxBarraclough
What do you make of comparable services from other providers, say, Amazon
Glacier?

~~~
amiga_500
I've used glacier. For stuff you don't need to restore in a rush, it's cheap.

I did however have a recent health scare and it made me wonder how my non tech
wife could possibly have restored the files as the interfaces are all heavy.

Not a factor I'd previously considered when assessing my backup/restore.

~~~
MaxBarraclough
> For stuff you don't need to restore in a rush, it's cheap

That was my thinking. It seems a good fit as a last-resort backup. Low month-
on-month storage costs, high retrieval costs. So we're essentially betting
that we'll never retrieve the data. Which seems fine.

Also, it apparently has strong assurances against data-loss. Lots of nines.
[0]

> how my non tech wife could possibly have restored the files as the
> interfaces are all heavy

It's all web-API-based, right? Is there a a decent FOSS GUI to navigate it?

[0]
[https://aws.amazon.com/glacier/features/](https://aws.amazon.com/glacier/features/)

~~~
amiga_500
If you retrieve the data slowly it's cheap. It's expensive if say you are a
retailer who needs their database restored asap.

I use a Linux perl client!

~~~
MaxBarraclough
I suppose the biggest risk is failure-to-pay, as with all cloud
backups/storage. If I allow my payment card to expire, Amazon aren't obligated
to continue to store my data. If I drop off the grid for an extended holiday,
that could be a real risk.

To my knowledge, Amazon offer no means of prepaying.

~~~
DaiPlusPlus
> If I allow my payment card to expire, Amazon aren't obligated to continue to
> store my data.

Business idea: cloud archive storage where you pay when you upload data and
optionally pay a modest monthly fee for real-time access to stored data, but
they'll guarantee to keep your data for you if you stop paying: you'll just
need to pay to retrieve that data.

As the long-term archival data wouldn't need to be stored in a data-center:
just a commodity tape-library box in a basement in a farm somewhere near a
freeway I imagine it would be kinda cheap to run as a business. You could set-
up a Foundation or other entity to ensure long-term continuity of operations
and have it self-sufficient through an endowment. E.g. a $1m endowment would
easily pay for something like this into perpetuity.

~~~
ValentineC
There was Permanent.org, three months ago:
[https://news.ycombinator.com/item?id=22943620](https://news.ycombinator.com/item?id=22943620)

~~~
MaxBarraclough
I agree with this comment on how that project completely fails to provide the
necessary assurances of dependable longevity:
[https://news.ycombinator.com/item?id=22944681](https://news.ycombinator.com/item?id=22944681)

------
irrational
"We will contact affected users shortly and offer..." I'm not sure how I
expected this sentence to end, but it certainly wasn't anything as useless as
"our deepest apologies".

------
klodolph
Lesson for those designing cold-storage solutions--design your system, as much
as reasonable, to not support deletion operations. The reason why files get
lost is usually due to software bugs, bad configurations, and operator errors.
Design your system to protect against these things.

~~~
mixedbit
Both AWS S3 and Google cloud storage buckets have an option that makes it
impossible to delete stored objects for some period of time. The option was
added for some legal compliance reasons, but I find it useful as an extra
safeguard that important service data is not accidentally or maliciously
deleted.

~~~
chrismatheson
I suppose that feature would be incompatible with user data in a GDPR world ??

~~~
visarga
Git is also incompatible with GDPR, you can't simply delete a file from all
history.

~~~
msh
Git is not really intended for personal data

~~~
Hamuko
Yeah, I don't really understand why you would like to stuff personal data into
a git repository (unless we're strictly speaking about author data). It's
really not the tool for it.

Now for those who decided that a blockchain is the perfect solution for
storing personal data however...

------
jarym
So today I learnt:

1\. Canon don’t have any backup procedures in place for their cloud platform.
Any hacker now will be salivating at the idea of pulling a ransomware hack I
imagine.

2\. Canon developers follow the ‘test in production’ methodology of continuous
integration.

~~~
galoisgirl
How do you know that? Gitlab had 5 backups and could restore none.

~~~
strombofulous
Is it a backup if it can't be used like one?

~~~
galoisgirl
It is and is not at the same time until you try to restore it.

~~~
jarym
Quantum mechanics goes over my head

------
hellofunk
There is one somewhat reassuring lesson here -- No matter the company, no
matter the brand, no matter their worth: software development is wrought with
challenges, and major mistakes for seemingly obvious things are made by
everyone, all the time, at every level.

~~~
hvidgaard
This is not one of them. Ensuring proper data loss protection is something
both Amazon and Google are very good at. A company of Canons size could easily
get the proper guidance to avoid it. But it would not have been as cheap as
storing the files on your own servers. So this happened because Canon did not
want to spend the money on doing it right with offline backups or append only
storage in a cloud with enough nines.

~~~
qntmfred
Companies of Canon's size can be constrained for resources just as much as any
other company. Canon is largely a legacy company at this point, with
significantly declining revenue the last several years.
[https://i.redd.it/lrlfen2x7sm31.jpg](https://i.redd.it/lrlfen2x7sm31.jpg)

~~~
hvidgaard
Even more reason to spend a little bit more to make this right. It's not
expensive to get a proper expert on the matter to figure things out. The last
thing they want is bad publicity for things that could earn them recurring
revenue.

~~~
hellofunk
I don’t think they deliberately thought they were doing something
inadequately, these problems are always obvious in hindsight. Issue is that
eventually most companies will have this moment seeing through the lens of
hindsight, the reasons are always different.

------
henvic
Everyone should be looking for a backup strategy that involves at least a hot
backup and a cold backup, with some sort of offline validation. However this
is easier said than done (because now you've to deal with multiple file
systems, decisions about cryptography, lack of interest in creating something
that you hope you'll never need, and so on). I wish there was a service (and
don't tell me about Backblaze because it's incomplete and doesn't work that
well - from my own experience) that would do all these things for you in a
reliable and trustworthy way (including giving you snapshots in physical media
from time to time - delivering to different addresses, possibly in different
jurisdictions, and so on).

~~~
ChrisMarshallNY
I have a multi-dimensional backup strategy, which includes “cloudy” backups of
my most important assets, using services like GitHub, local hourly NAS
(Synology double-redundant RAID-like) incremental backups of my whole system,
and running “hot” backups of my system and development volumes; automatically
updated every four hours (or on-demand. It's a bootable CCC clone).

I seldom need to use anything more than my “hot” backups, but have
occasionally needed to restore individual files from NAS.

Whether or not I have a backup is never even something I think about at all,
which is a big weight off my shoulders.

But photos and videos are a different matter from code (my main assets). They
require a _lot_ of space. It’s easy for me to be smug about backups; but
photographers have a much more intense set of assets.

~~~
ghaff
A problem with digital media as well is that it encourages you to just save
everything but it's easier that way. But as I'm discovering well into the
digital era, I wish I had more curated photos and video--though that comes
with its own problems.

~~~
fokinsean
I dread the day I decide to go through all my photos and organize them. It's
one reason why I'm so reluctant to move off of google photos, it's sooo easy
to find stuff and it groups photos together nicely.

But I really want to move my photos off of google.

------
dsabanin
From time to time, I have a perisistent feeling that iCloud might be
misplacing my files. It's just a feeling, but I realized that it's quite
challenging to be sure it's not happening.

~~~
bitexploder
It’s not so hard. A few command line commands on a Mac. Store the md5 of each
file, throw it into a JSON file. Load it all up in a dict. I do something
similar every month. Gotta do something with those cores overnight :)

It’s very brute force and inelegant, but also simple and quick enough.

~~~
dsabanin
One day, I'll do it. Honestly, I'm just afraid to find out the answer… :)

------
hxegon
wow this is bad. I know you should keep local backups too but if you shoot RAW
then that quickly becomes cumbersome, and only using cloud backup looks very
convenient.

If I were a professional photographer missing client material or someone who
lost potentially irreplaceable memories I'd want a hell of a lot more then
just "[Cannon's] deepest apologies".

~~~
_Microft
The software and service might have been ... provided "as is", without
warranty of any kind, expressed or implied, including but not limited to...

~~~
DaiPlusPlus
> provided "as is", without warranty of any kind, expressed or implied,
> including but not limited to...

IANAL, but when a service is advertised as being ideal for storing your work
or photos, isn't that directly implying that there is a warranty that their
service is fit-for-purpose? You can't advertise something in big lettering and
then countermand that in the small-print - so those magic words certainly
don't shield the company from liability at all.

I understand that the "without warranty of any kind, expressed or
implied"-line we see in software-licenses and EULAs is when software is
distributed _without consideration_ (e.g. open-source software), but when
there is consideration (i.e. people paying Canon to host their photos...) then
there's a liability if Canon lost peoples' data - so I understand they will be
sued for this if anyone lost anything of value. At the very least a 100%
refund...

They only way I can see Canon getting out of this is if they had prominent
warnings displayed throughout their service's UX advising their users that
their service was not suitable for long-term storage of valuable data.

Again, IANAL - can anyone is is a lawyer chip-in?

~~~
dspillett
Also NAL, but I would say it very much depends on the law in which-ever
jurisdiction it gets tested in - which might be the one you are in, the one
Canon are in, or somewhere else entirely.

Some legal concepts such as "fit for purpose" as defined in the UK's Consumer
Rights Act (2015) certainly seem relevant here, but that states that companies
should offer replacement or refund. Note the word refund, not recompence, is
used: Canon may be required to repay you in full everything you paid for the
service. All £0.00 of it.

 _> i.e. people paying Canon to host their photos_

Is this the case though? Several other comments have mentioned it being a free
service.
[https://image.canon/st/en/faq.html](https://image.canon/st/en/faq.html)
states that too. Unless there is a non-free option too.

 _> but when a service is advertised as being ideal for storing your work or
photos_

I'm not sure what it is advertised as, but that doesn't seem to be what the
service is intended for.

Reading other parts of the FAQ it seems that the service is intended as a
transfer agent, with the convenience of online storage being a useful side
effect. Quoting the FAQ: _" Image.canon is designed to ease your imaging
workflow – whether you are a professional, enthusiast, or casual user.
Wirelessly connecting your camera to the service allows seamless forward of
images not only to your computer and smartphone devices but ..."_. The
implication I'm making being that they can just argue that their service is
designed to move images around your devices and the user should have been
backing them up from there.

Users might try to argue false advertising if "fit for purpose" doesn't fly
because of the purpose being defined differently. But good luck funding such
as case against the lawyers that Canon can afford. It would have to be a
class-action or similar, unless some government body takes the issue up (which
from the users PoV will effectively be the same, and the best they'll get is a
small voucher for a few £ off future Canon offerings). That is what a lot of
things like this boil down to: legally enforceable sometimes doesn't exactly
mean legal, it sometimes means "can be enforced by having a better legal team
than the little guy"!

------
hendry
I was wondering what those weird messages from
[https://image.canon/](https://image.canon/) were about.

IIUC the idea of their new platform was to be ephemeral in any case. Just to
give a pipeline to stream in to other accounts.

I had no idea it was offering long term storage. Better off with S3 Glacier
for that!

------
headmelted
On-site and at least two off-site.

It sucks for their users who maybe don’t know the golden rule above. I
certainly don’t blame non-tech folk who paid a known photography company to
handle a complex problem on their behalf - they did the right thing in handing
this responsibility to people who they believed could be trusted.

To the greater point - this is incredibly damning of Canon’s cloud storage
going forward. As others have said, the amount of data lost overall is not
much in the greater scheme of things, but that’s not the concern. What’s
worrying is that they were able to lose any data at all. How much redundancy
do they have? How are permissions managed?

When I backup photos and videos at home I have a script to _chattr -i_ all the
files independently as they’re stored on top of the redundancy and backups.
You need to protect your data from yourself, too.

------
mad182
There is no cloud, it's just someone else's computer.

------
rasmore
They’re paying for the negligence against software developers. Those two
incidents come down to the design flaws after all. The CEOs of Canon Inc and
Canon USA are too old to admit the fact that software is more powerful than
what it used to be when the hardware was selling well simply owing to its pure
performance and features and the software was just a driver. You would be
surprised if I tell you how many skilled software engineers have left the
companies in the last three years. What makes things worse is that the old men
aren’t aware of why they left.

------
fredthomsen
The cloud is just another backup option and you should have 3. That being
said, you expect more from a paid service.

------
pabo
I can't access the article, but there was a recent ransomware attack on Canon
[1]. I wonder if this could be related to it.

[1]
[https://news.ycombinator.com/item?id=24185734](https://news.ycombinator.com/item?id=24185734)

~~~
waihtis
I think they claim that this was a coding error and not a ransomware case.

The timing is convenient though.

------
obayesshelton
Ouch, I am guessing they are using some on-premise or none major cloud
provider? I feel for them though, without the right resource or budget they
are doing their best. I doubt they intentionally decided to not make a good
product.

------
KingOfCoders
Canon is no software company and doesn't have the right mindset (also see most
other camera manufacturers mobile app reviews). Probably outsourced with a
focus on having low development and operating costs.

------
m0zg
Anyone who relies on any software camera companies produce deserves to lose
their data IMO. Just look at Canon DPP. Almost 20 years old, and it's still a
primitive, unusable, steaming pile of shit.

------
hadrien01
I'm redirected into a refresh loop with this url:
[https://www.digitalcameraworld.com/cc.html](https://www.digitalcameraworld.com/cc.html)

------
bluedino
I think people would be surprised at the amount of backup or file storage
services that keep everything in one data center. Your files might be
redundant across machines but it's not like they have a second data center
somewhere with another copy of all your data, and they certainly don't backup
everything up to tape or something.

------
ponker
The camera companies are so bad at software that the software companies are
building better cameras than them... in the few millimeters thickness of a
phone! I salivate for the day that Google or Apple buys Nikon or Leica or
whatever and shoves all of that computational photography goodness under a
24x36mm camera sensor.

~~~
DaiPlusPlus
As of 2019, Apple sources their iPhone camera sensor components from Sony and
the camera lenses from Largan Precision (and other companies). The software
that Apple contributes is important - yes (after-all, this is software-
defined-photography now, with Portrait Mode and that fancy but fake depth-of-
field stuff in the single-lens iPhones) - but let's not pretend Apple is "a
camera company".

~~~
richardwhiuk
I expect Apple sold more camera than Canon last year.

------
Funes-
Saving your shit on "the cloud": not even once.

~~~
canofbars
I store my data on the nextcloud server sitting in the same room. I guess for
added safety I could also set up automated encrypted backups to backblaze or
something.

------
ReptileMan
With spinning rust so cheap nowadays - anything could be backed up easily both
in the cloud and locally.

------
Aeolun
No backups? Wut?

------
moreorless
People call me crazy for having local backups, cloud backups, and offline
backups for my critical assets.

~~~
system2
3-2-1 backup is not crazy and also is an IT standard.

------
jaylittle
I'm utterly shocked at how many people assume that "The Cloud" is both run by
people infinitely more intelligent and more competent than they are and that
everything there is backed up.

What a crock of shit.

------
saagarjha
Has GDPR compliance gone too far?

~~~
danieka
No.

Losing data is just as bad from a GDPR perspective.

~~~
saagarjha
Oh, really? Can you actually be fined for destroying disks containing customer
information?

~~~
mschuster91
Yes! Accidental loss is a data breach, see page 7ff of
[http://ec.europa.eu/newsroom/document.cfm?doc_id=47741](http://ec.europa.eu/newsroom/document.cfm?doc_id=47741)

~~~
saagarjha
Interesting, they seem to call this an “availability breach”; of course, they
require disclosure if it affects people. Do data storage providers need to
essentially SLA, then?

