This resulted in an explosion of tools written in odd ways with obscures languages each with their own odd DSL to interact with. (Something that most VFX houses that survived the late 90s/early 00s suffer from.)
Part of the reason that Rhythm and Hues went bankrupt is that they were the last bastion of "we make all our own tools"
The battle Pixar had to rebuild/replace marionette cost millions and took years. It got so bad they were seriously considering Maya instead. (I thought they had actually bitten the bullet and bought a sight license, however it appears that they wrote presto instead.)
So Pixar isn't the chocolate factory of awesome it used to be. The long hours and the rampant wage fixing somewhat tarnishes the image.
I'm curious about the details of this. What made marionette so bad that they couldn't fix it to do what they needed it to do? From my completely naive perspective, it seems like having custom tools should be a competitive advantage -- you can tune your tools to the needs of your artists.
There were/are two big things that are against you if you develop in house animation software(or any software for that matter):
1) The training gap. People are most efficient with the software they are used to. Maya has been the defacto standard in animation for pretty much 7 years (if not more).
2) the R&D cost. VFX is constantly moving. (Possibly much more than normal web software) With the increase in CPU/GPU power you are able to do much more. Take hair for example, In monsters inc, Sulley has something like a million hairs. (monsters U its 5million +) You need to control that hair system some how. There are people that groom digital hairs for a living. Its a constant battle between cheating, simulation and feedback.
The hair system alone could swallow a team of Five for a year, but then you'll be negleting the other systems that need bug fixes, improvements and R&D magic.
Having an outside company, who has the resources to have many teams work on each individual part, along with a UX designer to either make it work more fluidly(or just put annoying transitions in the way.) and most importantly of all, a team of QA, bugfixers and Tech support.
You're right, custom tools give you an edge, Take katana from sony: it allowed an artist to work on an entire city scape (Billions of objects) on a single computer, in 2006 (it was developed for one of the spiderman films) Something no other system could do.
Another example is of course renderman.
But these are only the successful ones. Many custom tools are built, atrophy and die. Sometimes taking the company with them.
But like the rest of IT the march of commoditisation is in VFX as well. Most software can be bought off the shelf (maya, modo, nuke, Premier, renderman, arnold) Its also almost possible to render in the cloud (although its crushingly expensive, and slower.)