Can you cite Sun products that were made OSS that users weren't already replacing with OSS solutions from other sources? I think Sun's problem was that it was in the business of selling proprietary Unix workstations and servers, and most of their potential customers these days are content to use PCs running Linux, Windows or OS X in those roles.
 Apple is a lot like a proprietary Unix vendor in a lot of ways, but Apple hardware is not more than 50% more expensive than PCs with similar capabilities.
Sun's problem wasn't OSS. Sun bet the farm on SPARC and it died out in favor of x86 faster than anticipated, basically. The OSS stuff was great -- they needed to build and sell an IDE or some other enterprisey-product that would drive Java's big corporate users to pay out big license money.
I would go so far as to say Sun never did enough open source.
I had the misfortune of doing some work in a mixed shop where most of the Unix servers were running Solaris 8/9 recently. A couple things that led to its downfall:
1. No package manager. And now that there is a package manager, maintainers are few and far between. I don't think I would trust it in a production environment. It's never going to be as robust as dpkg or yum. if they had fully open sourced Solaris, and supported the community as primary, there would be a well-maintained package manager. I doubt Oracle will do anything to fix this.
2. I'm still not entirely clear on where their utilities like grep, awk, sed, etc. come from. All I know is I was terrified to push /opt/bin ahead of /bin in my PATH for fear of breaking something, and meantime I hated the native shell tools. They should have adopted GNU's tools as first-rate. Solaris' tools suck.
I see that as more of an orthogonal marketing issue of not understanding where the customer growth was coming from. When you have the attitude that you sell to CIOs and your consultants do the dirty work of setting up the systems, you don't care if college sophomores find it to be usable. (Until 10 years later when those former college sophomores are now making purchasing decisions, whoops!)
Open source won in this area because of its "by developers, for developers" approach moreso than the source code ecosystem.
Package managers are an unassailable benefit over proprietary software. That shop I worked in hasn't updated their servers in half a decade because it's too much work to handle dependencies without a robust manager (and even then it's a significant amount of work.) And it's not like it gets easier with time. We have package maintainers because it's a full-time job trying to check all of a piece of software's dependencies and makes sure nothing breaks with an update.
Now yes, proprietary software like Solaris could, in theory do that. But Sun would have had to double its software staff to even come close to the stability you get out of Debian or Red Hat's repositories.
Sun was a smouldering wreck before they started giving away everything they owned. Their OSS policy was an act of desperation -- it did not work out, but to say it killed them would be vastly misrepresenting their last decade.