
Fired employee wipes out whole season of TV show  - briandear
http://www.msnbc.msn.com/id/42366946/ns/technology_and_science-tech_and_gadgets/?gt1=43001
======
mbreese
The thing that really sucks for the TV show is that it wasn't even their own
employee. It was an employee for their data hosting company. You know... the
guys who you're _supposed_ to outsource to to make sure that this stuff
doesn't happen.

~~~
jrockway
_You know... the guys who you're supposed to outsource to to make sure that
this stuff doesn't happen._

Actually, outsourcing is designed to be cheap, not to lower risk. Trusting the
lowest bidder to an important task usually results in disappointment.

~~~
pero
Actually, big corporations outsource to _transfer_ risk and not because it's
cheap. To compliance/legal departments, paying exorbitant fees to qualified
vendors _is_ saving money in their risk analyses.

~~~
mbreese
Exactly. If you don't have expertise in house, then it can be cheaper to hire
that task out to an outside company rather than hire someone to manage it in
house. This sure sucks for the TV show, but at least they have someone else to
hold accountable. If this had happened and they handled their own IT, there
wouldn't be anyone but themselves to hold accountable. And you can bet that
means a lawsuit with damages.

~~~
jrockway
But of course, if the hosting company had any lawyers, their contract says
they're not liable for this.

~~~
pero
How so? _They were negligent_. Beyond that, you're also assuming the client
didn't have lawyers--there's no way this contract gets signed off on without
the vendor absorbing the risk.

If all the legal work was done properly, the vendor would end up in court with
their insurer.

------
alexqgb
In other news, soon-to-be-fired producer learns the true meaning of 'backup'.
Also, 'redundancy'.

~~~
maushu
Isn't the data hosting company job to do backups and redundancy? Unless they
clearly state that they don't do that.

~~~
gm
I'd say the opposite: Unless they clearly state they _do_ backups, they are
not (legally) responsible for doing so.

~~~
isleyaardvark
That's what data hosting is. If they were doing webhosting, sure, but the
article repeatedly states they were doing "data hosting".

Here's the company's web page detailing their data backup plan:
<http://www.cyberlynk.net/services_corporate/dsp_backup.cfm>

~~~
gm
What I'm saying is that whatever is in the contract is what matters. Data
hosting, web hosting, data storage, whatever terms are used matter little.

EDIT: Case in point: We probably all agree that dropbox could be called "data
storage", but they specifically say that backups and backup costs are to be
done by the client (<http://www.dropbox.com/dmca#terms>), not by them.

------
nopassrecover
Surely the data company has redundancy or backups or data recovery software
available? Surely they considered the risk of a staff member deleting copies /
a hacker / a fire.

Surely the producers have at least working copies somewhere (I can't see
people live-editing on a cloud server). Surely they considered the risk of the
company going bust / data loss / their own staff deleting the copies.

Would assume this was an April Fools thing it's that bad (except for the fact
it quite obviously isn't due to the quite obvious reputation damage that will
effectively destroy this company).

------
pbhjpbhj
>" _According to a lawsuit that was filed last week in Hawaii District Court,
a man named Michael Scott Jewson was terminated from CyberLynk. From his
parents' residence, he allegedly accessed CyberLynk's data and intentionally
wiped it out. Jewson is alleged to have been charged in February with a
federal computer crime violation and admitted his guilt in a plea agreement.

The data breach allegedly knocked out 6,480 WER1 electronic files, or 300
gigabytes of data, comprising two years of work from hundreds of contributors
globally, including animation artwork and live action video production._"

These sort of reports often over do it with the use of "allegedly" as an
attempt to mitigate any libel claim.

"Jewson is alleged to have been charged"

Surely it's a matter of record whether he was charged. If the reporter can't
even confirm this then they don't have a story.

"The data breach allegedly knocked out 6,480 WER1 electronic files"

Again they don't need that "allegedly", it's presumably a reported fact, the
allegation they have to be wary off is the one that says the person
responsible.

This really annoys me, allegedly.

~~~
ubuntuftw
Former TV news person here.

While I agree with you on your first point (that the reporter shouldn't have
to use the word "allegedly" if he/she can confirm the man was charged),
personally, I think using allegedly in the sentence about the files being
knocked out is warranted. "The data breach" is referring to the data breach
the man *allegedly carried out.

TV news reporters != attorneys. So when in doubt, attribution is your best
friend.

~~~
pbhjpbhj
The data breach he allegedly carried out is reported as a known quantity. The
only allegation that needs mitigating is that it was this person that carried
it out. If one is unsure of the details then it would be "reportedly" or an
"unconfirmed source stated" or similar, no?

Thanks for your viewpoint though.

~~~
ubuntuftw
The problem is that the reporter doesn't know that the details of the breach
mentioned in this article are facts. The figures, that there were 6,480 files
totaling 300 GB of data likely came from the prosecution. So the reporter
needs to get himself off the hook and either attribute it ("the prosecution
says") or use words like "allegedly," "reportedly," etc.

------
trafficlight
I just don't understand how there could have only been a single copy of those
videos.

~~~
Retric
Who says he only deleted a single file? It's not hard to delete stuff from
online backup systems.

~~~
Udo
No, I think what trafficlight meant was: how can there be only a single copy
of this material? What about the media they actually shot the episodes on?
Surely there must be some kind of copy on the cutting room computers? It looks
as if they're mainly out to collect damages from the company.

~~~
Retric
_The data breach allegedly knocked out 6,480 WER1 electronic files, or 300
gigabytes of data, comprising two years of work from hundreds of contributors
globally, including animation artwork and live action video production._

Sounds like they hosted everything in the "cloud". Consider what happens when
you edit a Google doc, there is no offline backup just whatever Google
provides. So sure they are not at zero, but plenty of stuff can and was lost.

~~~
Udo
That's what they claim, sure. But does it really work like that? It seems to
me that there would have to be local copies of those videos, if for no other
reason than synching production quality videos to the cloud in realtime is
probably unrealistic. Maybe some HNer working in the TV industry can
elaborate?

~~~
trafficlight
From what little video editing I've been involved in, we had source material
in 2 or 3 different places, plus multiple edits of said material, plus rough
edited timelines, plus the final product.

And keeping everything in the cloud would be insane. In our shop every video
editing workstation had it's own RAID array just for realtime editing. In a
large shop you'd have a SAN of some kind.

------
sabalaba
(Warning anecdotal evidence) Having worked in China on a number of startups
and worked alongside people who have had to fire employees, it's often common
practice to just lock fired employees out of their systems. I've had this
happen to people I know and had to do it once when letting a programmer go
(which is nearly impossible to do after a probationary period due to the labor
laws in the country, it's very very difficult to fire somebody, the company
has to provide reason, pay social benefits, or face the employee in a court--
forget it if you're a foreign company).

I have heard stories of letting people go to find out that all of the work
that employee did for the last month or two deleted, source missing, machines
just wiped clean.

I read that title and kind of laughed because, well, it is a seemingly
commonplace response in some parts of the world.

~~~
yardie
In my case I gave a months notice I was resigning and was locked out
immediately. Imagine how much it sucks to come into work everyday with nothing
to do except go to lunch. When I wasn't offering the other developers advice
on the code I had worked on or APIs I was familiar with I was just shooting
the breeze.

This was the hole in the policy. It was designed to keep disgruntled employees
from damaging the system but the ones leaving on good terms were treated
exactly the same.

~~~
michaelcampbell
This is more/less common in certain industries, and perhaps partially based on
regulatory issues too, although I'm guessing there.

I have worked on financial software the majority of my career; at one point in
a Wall St. investment banking firm. The very explicit protocol there was if
you were going to leave the 2 week notice was customary, but you would
probably be locked out of everything instantly, and would be around for hands-
off consulting at the whim of your manager for the duration. It was all
handled very professionally and friendly, at least for me, and they let me
keep working for the 2 weeks since anything I touched had a _LOT_ of layers to
go through before it ever saw production data.

~~~
Nick_C
Same industry for me, but in the trading area. It was standard for pretty much
all firms that we were escorted off the premises immediately. I was touched
that they let me stay for half-an-hour to say goodbye to people ;). Like you,
it was done professionally and friendlily.

------
motters
This is one thing which can happen when you put all your data into "the cloud"
and hope for the best. One risk which any business has to take into account is
the possibility of a rogue employee or ex-employee running amok. Putting all
your eggs (or data) into one basket, especially when that basket is
administrated by people who don't particularly care either way what happens to
your business, may not be the wisest strategy.

------
scythe
Kind of ironic that this would happen on World Backup Day, perhaps?

<http://worldbackupday.com/>

~~~
pinwale
The real site is <http://worldbackupday.net>

------
hartror
I assume this is them: <http://www.cyberlynk.net/> with the late 90s "new"
graphic against off-site data backup. I hope the show producers sue someone,
either Cyberlynk if they claimed they did redundant backup or their IT
guy/company for setting them up with such a bum deal.

------
r00fus
No backups? 300GB doesn't sound like much. I just processed and shuttled 2.3TB
today (sneakernet backup).

~~~
famousactress
I was just thinking the same thing.. 300gb, mostly of presumably high-res art
and video.. and that's 2 years work? Sounds really small.

~~~
dagw
2D Animation doesn't take up much space. Beyond a few painted high res
background, most files are small and a lot of things are procedurally
generated and take up almost no space at all. I worked on a 2D animated movie
10 years ago, and the data needed to render the final movie weighed in at less
than 100 GB (and that was 2-3 years of work by 30-40 people). Also I
understood from the story that they didn't loose everything, just enough to
make it impossible to recreate the episodes without basically starting from
scratch.

------
dhughes
Can you hire him to wipe Jersey Shore?

~~~
dschobel
Don't watch the show then. Or do you object to other people watching content
you don't personally approve of?

Because there are many terms for that particular personality type, none of
them flattering.

~~~
michaelcampbell
LiteralNet is in full force today, I see.

------
jrockway
That employee may be going to jail, but it's not his fault that this happened.
If your fired employees have access to your servers, you fucked up the firing
horribly. Similarly, if one data center going offline can take out your entire
season of work, perhaps you need to learn what redundancy is.

~~~
ericd
How is this not his fault?

~~~
jrockway
What I'm saying is that relying on crazy people to not go crazy is not a good
data loss prevention strategy.

(In the end, putting someone in a situation where they'll do something bad is
worse than being the person that does something bad. A good example is the
Colgan Air crash from a few years ago. Anybody knows that when your plane is
stalling you're supposed to push the control column forward and apply full
engine power. The plane does the pushing the control column forward part
automatically! But for some reason, the trained airline captain did the exact
opposite, stalling the plane and killing everyone on board. Why? Because he
got paid so little that he couldn't afford a hotel room to sleep in before his
work that day, and had to commute across the country and _then_ work a full
shift. This is the airline's fault for impairing someone to the extent that he
killed 50+ people by doing the exact opposite of what any trained pilot would
do. $300 for a hotel room and everyone on that flight would still be alive
today.

Similarly, when you fire someone that works with critical data, you need to
make sure the guy's access is revoked. It's a precaution that needs to be
taken, as this case shows. When necessary precautions are omitted, bad things
happen. Planes crash, and important data goes away forever.)

~~~
tjmc
I understand your point but the difference in the Colgan case is that the
pilot made a tragic mistake, whereas the disgruntled employee _deliberately_
wiped the show. Yes, his employer is careless for not revoking his
permissions, but the question remains - how is this not his fault?

~~~
davidandgoliath
It is. It's also the fault of everyone else involved who didn't back up.

------
robryan
This is surprising, it would be like sitting my web apps code/ database on one
live server without a local copy, and possibly multiple backups elsewhere. How
can a TV production company justify this? I mean it would cost just about
nothing relative to production costs to put 300gig on S3.

~~~
mbreese
You're assuming that the production company can afford to have someone around
who knows how to put data on S3. My guess is that they outsourced this to the
company who had the fired employee. Still, they should still have some kind of
original files on some other medium somewhere... but who knows, maybe their
production workflow revolved around this data-host. In which case they are
really borked.

~~~
detst
We have tons of decades-old video alive and well today. They may not have
expertise with S3 but if they don't have the expertise to ensure robust
backups, they're doing it wrong.

~~~
robryan
Yeah they may not but you have the weight up the costs of getting a contractor
in to setup an almost fail safe system against losing probably millions of
dollars of production.

I would assume it's more about people assuming something like this would never
happen as to not justify the outlay of protecting against it.

------
shadowpwner
Interesting how this happens on the same day of World Backup day.
([http://blog.reddit.com/2011/03/better-have-backup-plan-
world...](http://blog.reddit.com/2011/03/better-have-backup-plan-world-
backup.html))

------
Gorm-Casper
Am I the only one who found the idea with tiny bubbles at the scrollbar more
interesting than the story itself? Front-end developers' curse :P

------
Splines
Man, that sucks. Hopefully it's a lesson to everyone that you don't leave the
keys to the kingdom to one guy, _especially_ if they've just been fired.

------
daemin
Shouldn't the company have off-site backups of the data its hosting, like,
say, on tape or something?

------
tlrobinson
For once I agree with the editorializing of a headline on Hacker News.

------
dzorz
April 1st?

~~~
ctdonath
Must be. I thought the link would be about Charlie Sheen.

------
monological
Is the guys name really Micheal Scott? April fools!

------
nvictor
april fools?

------
ThomPete
Aprils fool?

