What was deleted, per Loren Carpenter on Quora, was, "in effect the database(s) containing the master copies of characters, sets, animation, etc. The frames were to be computed from that. A few hundred MB, if I remember." (http://www.quora.com/Pixar-Animation-Studios/Did-Pixar-accid...)
A little more detail than provided in the video, where they just called it, "the film", as if the movie's entirety of data were both deleted and duplicated on that home PC.
The sum of all the source files (models, animation, shaders, lighting, sets, etc..) was probably in the hundreds-of-megabytes range at that point in time as Loren pointed out in his comments on Quora.
I've also added a lot more details to that Quora thread too (http://www.quora.com/Pixar-Animation-Studios/Did-Pixar-accid...) if folks want to read more about the whole thing, although it was 14 years ago so my memory is a little foggy on some of the details at this point in time.
"Well, the good news is that we're getting a new backup system." Long pause. "The bad news is that your files are at about the third gigabyte of a two gigabyte tape."
Ever since, I'm always frustrated by people that assume things are backed up. If you don't test your backups, you don't have backups.
I built a cluster which used a disk array for storage. When I was contracted to build this I was assured that they were building several other clusters for use as test, QA and prod and that all I had to do was build the dev system and document so thoroughly that the other systems could be built from the documentation. At the time I wasn't yet security cleared so would not be allowed to build the other environments.
Some time later (6-8 months) I was called back to help solve a performance related problem. When I arrived I discovered that the dev machine was now production, and that no other machines were built. I would have to do the work on production.
Worse though... their backup and failover mechanism wasn't what I'd documented, but involved an identical piece of hardware nearby which was powered off. They hoped to simply turn it on when production failed.
Their daily process was to approach the disk array and remove the 1st hot swappable disk from the live machine, and swap it with the 1st disk in the powered off identical system. When the array had rebuilt the first disk, then move the 2nd disk.
I am seldom speechless. It was only a critical part of a £2bn project. Beyond all of the obvious WTFs, I still wonder whether the 2nd machine could be powered on at all given that all of the disks were pulled at different points in time.
I couldn't agree more! It's amazing how often the question "we take backups, right?" produces "uuhh..". And yes, if you don't test them, they're worthless. Because the rule is, if you haven't tested them, they won't work when the shit's really hitting the fan.
I have a projects folder, which is on Dropbox and additionally on Time Machine. I can confirm they are really in both by browsing them. Anything more I should do?
It may be that your projects folder really is backed up, but that it relies on some stuff that is outside the folder for whatever reason - for example, a data file that didn't fit in dropbox, or some changes to config files outside the folder (where noone remembers exactly what needed to be changed); or a connection that relies on a private key that's on your computer, gets used automagically, but is not backed up.
I mean, it's probably overkill in your case - most likely your project source is quite safe; but for a working production system it's an entirely another matter.
SunOS version mid 1990s - you had to be root to run the backup script AND you could create files in /dev without warning AND / and /home were in the same partition so it had lots of space
He literally, despite being 30, cried his eyes out for about an hour.
That was when I learned not to fuck around with data security and backups...
I wish everyone this experience at least once but without the pain of losing so much. It's really important to always have the backup monkey on your back.
The comment on the cartoon "I remember thinking this is really bad we're going to get our backups" comes from someone who obviously does not have the appropriate fear of what can happen to even begin to understand a good backup system. I'm guessing she is much more careful with her children and anticipates things in advance.
As far as "the backup being bad" where were they stored? Were they stored offsite?
It's very easy to say the backup should be tested, there should be mirrored recovery systems etc. but this is a movie, even in a studio system this is a one shot deal, the systems and even the hardware will be disposed of after the master is cut,the people will leave for other jobs etc.
Now in the middle of crunch to make the movie - tell the producers you are going to double the HW budget for a test system to test the backups and you are going to stop doing any new work until a backup system is built and tested.
Personally I'm amazed they had any sort of backup beyond personal copies on animators machines.
I've been doing Unix backups for about 27 years. Back when it was more of a pain then it was in 1998 and certainly today. Total backup, incremental tapes the whole thing. Taking the tapes offsite. I wasn't a sysadmin but it was my business. So if I didn't have a backup it would be my problem and my loss both in work and money. I can assure you that setting up a backup plan (key word "plan") that evaluates the cost and trouble is worth it. The only reason higher ups don't approve of this is that they don't know about what can happen if something fails.
Ask yourself this question. Do you think Pixar changed their backup process after this event? I think they probably did. Not only because the failure happened to them but also because higher ups saw what could have happen if not for the woman who wanted to be with her child at home who saved the project in process.
That was my thought. Heck, it happened to me once.
Combination of a SCSI cable knocked loose, but not visibly so, and the brand new sysamdin (me in both cases) knowing enough to head and tail the backup log file looking for 'start' and 'complete' but not knowing to look in the middle for what was actually happening.
Lost a month of data in the (thankfully) little used legacy ERP system we used. If the entire plant had still been using it it would have been a disaster, not an inconvenience.
I think we set things up giving her a full tree with her SGI machine on the netwrok, physically in the building. Then we sent the machine home with her and incrementally updated it weekly via rdist over an ISDN line.
I wish we had Dropbox back then!
There were several backup strategies for that machine, but they failed. A few reasons are outlined elsewhere in this thread (running past the 4gig limit on tapes) but, if memory serves, other errors were also occurring that involved drives being out of space so error logging wasn't working properly.
Making matters worse, after our initial restoral, the crew came back to start working on the show and then, a few days later, we discovered that the restoral we thought was good was, in fact, not. So we had, in effect, split the source tree and now had a bad branch with a week of the crew's work on it, including animation.
So we had to untangle the mess as best as we could, which was about 70% script based, and about 30% by hand. Further details are in my post on the Quora thread... (http://www.quora.com/Pixar-Animation-Studios/Did-Pixar-accid...)
And honestly...the larger the system, the more things that can go wrong. Lots of people back up files to RAID arrays, only to find out that their parity drive(s) are hosed at the worst possible moment. The price of a good backup system is eternal diligence.
edit: added "could" to make it clear that I'm suggesting that such was an impossibility.
Still, in 1999 I had a PII 333MHz and 10GB HD (and Windows 98, sigh)
But reading about the history of Pixar, the difficulties (in the first Toy Story) involved the workflow of creating a full-feature film (apart from all the other problems they had)
So I'm guessing, since Toy Story 2 was their 3rd feature film, things were still a bit raw (especially given their fast pace)
Sure, 10GB fits in a usb stick today (or better, fits in a 3 year old one)
And managing the amount of data was certainly a challenge.
If you see the first Toy Story today you can think of that being rendered in real time by a modern graphic card easily.
But still, they had what was available and managed to do all kinds of tricks
Now see what they could do in 1984 http://www.youtube.com/watch?v=TYbPzyvResc
It'd be much easier to just get a giant RAID NAS, and manage it all there, which should in theory be quite sufficient assuming you are keeping proper backups. Many of the render nodes will have copies of the needed data, but the authoritative copy needs to live somewhere.
I'm assuming they were working on it quite a bit before then.
I think you're overestimating the tech available at that time
It was most likely "sudo rm -rf .*" which will actually go BACKWARDS up the tree as well.
$ mkdir -p foo/bar
$ cd foo/bar
$ rm -rf ..
rm: "." and ".." may not be removed
and all local files beginning with .
the BSD version of rm will never remove . or .. see:
checkdot is called against all argv:
any version of rm that does remove '.' or '..' is not POSIX compliant.
The GNU coreutils version of rm does not contain this check:
(edit: it does - thanks, was running from memory and didn't actually check it)
Imagine you place yourself in '/tmp' to throw away all those pesky '.' directories and files that applications leave there. You run that command, it matches '..' and your whole system is gone (btw, 'rm -rf .??*' avoids this).
See for yourself by running:
> /bin/rm -r -f *
Running that command from the top of the directory tree where ToyStory2 lives should delete everything below, which would wipe the show.
pokes Backblaze and grabs a couple of random files
It works! Yay.
The data wasn't gone (only the file system was nuked), but would have required specialist intervention to restore. Or a hacker with the right tools (assuming such tools exist for the *nix file systems - they certainly do for fat32 and NTFS)..
And, at that point in time, introducing more uncertainty into the restoral process wasn't going to help. There was already too much uncertainty everywhere we were looking.
I'll post about that separately...
Doesn't this command, like any other delete-command, require top privileges (admin/root) that are unavailable in normal situations?
On all (?) modern Unix variants, you can set the sticky bit on a directory to also require write access to the file.
*nix file systems purge the file immediately, unless it is still opened by some program. fat32 deletes the entry from the file allocation table, and the data is eventually overwritten as new files fill that same location.
I imagine that Pixar has a lot of backups and some of them are offsite. Back in the late 1990s, I don't think offsite backups of a project like this would work.
I recreated it in about two days, and it was better.
Obviously, artwork != code. But I find a second pass nearly always improves things.
Nothing else I have is that important. Lots of games and recorded TV shows. Either I already finished it, or I'm probably not going to.
I could loose my desktop or my laptop and only be out time. If both died at once I might loose a few important things, but nothing too life-shattering. In general, I can pick up a brand new computer and be productive by the end of the day.