There are right ways and wrong ways to do a rewrite. Joel Spolsky says you should never do it but he's flat wrong, mostly, he's right in that it's almost never done correctly.
Most rewrites are undertaken too lightly and with too much naivete. They are approached as projects with a lot of major unspoken assumptions. Often assumptions that are counter-factual. Assumptions such as: at code complete we will have a system that is as refined and defect free as a well-worn system that has been in real-world use for years. Or, there will be no problem with forcing every user to use the new system because it will be in every way objectively superior to the old system starting on day 1. Or, taking on the work of a rewrite won't cause a serious shortage of manpower needed to maintain the old system (or vice versa, the rewrite won't face a serious shortage due to the needs of maintaining the old system). And so on.
Yet such obvious fallacies remain as latent assumptions behind the majority of rewrites.
There are different ways to do rewrites effectively and none of them are easy.
One way is to build a parallel product using an entirely separate team, and having that product compete against your older product directly in the marketplace. This is obviously an expensive route, which is why a lot of companies shy away from it, but sometimes it's the best way to go about things. It gives a chance for a product to mature before it replaces the original. Microsoft did this with Windows, having two parallel teams work on different kernels (NT and 9x) and it took over half a decade for the "rewrite" to win out. This is even more astounding when you consider that originally Microsoft had intended for Windows 95 to be simply Windows 4.0 based on the NT kernel. A goal they didn't really achieve until Windows XP was released 6 years later.
An alternate way to do a rewrite is through refactoring style techniques. Start building the new system inside the old and slowly deprecate and migrate components piece by piece until the old system is entirely gone. This is a very effective way of doing things and has the advantage of being able to scale relative to the amount of development effort available but certainly has its fair share of limitations and downsides (which I won't enumerate).
As far as the Netscape example, they ended up coding themselves into annihilation, as their constant rewrites and architectural buffoonery led them to create a very unstable and clunky browser that was markedly inferior to IE 4 when it hit the market, and they just were not able to respond with any degree of innovation software wise. Also, my understanding is that the seed code for the mozilla project was a horrendous mess that didn't even represent a functional browser at the time and required extensive effort to get to that point.
Building a new system inside an old one assumes that you have something to work with. Sometimes the new system needs to be so different from the old one that this is impossible. If you think about this as a multi-dimensional optimization/search problem when you do the incremental approach you are holding N coefficients constant while modifying some other (usually <<N). If your target system is completely different (==architecture) you'll never get to it with an incremental refactoring effort.
True, and hopefully nobody took my 2 examples as the only methods or suitable conditions for a rewrite.
A rewrite is risk, a huge amount of risk. And often taking on risk can be bad, but often taking on risk is married to the possibility of huge wins. It takes good judgment and solid development practices to take on risky projects and persevere, but it can be done, and indeed is done all the time. Going into a rewrite nonchalantly with a thousand casually made false assumptions and all the while squeezing your dev. and QA cycles between two products is a very, very common way to waste a lot of effort and seriously hurt your company, especially in the market. So be aware of what you're taking on when you take on a rewrite.
As a general rule if your team/company is capable of taking on and succeeding at tasks that others typically struggle at then that can be a key competitive advantage.
The 1998 Netscape/Mozilla code drop didn't even compile, and it was nowhere close to being a functional browser.
They should have released the source of Netscape 3. In fact, even now I'd like to see the source of Netscape 3. It's nearly 20 years old but still a classic.
Most rewrites are undertaken too lightly and with too much naivete. They are approached as projects with a lot of major unspoken assumptions. Often assumptions that are counter-factual. Assumptions such as: at code complete we will have a system that is as refined and defect free as a well-worn system that has been in real-world use for years. Or, there will be no problem with forcing every user to use the new system because it will be in every way objectively superior to the old system starting on day 1. Or, taking on the work of a rewrite won't cause a serious shortage of manpower needed to maintain the old system (or vice versa, the rewrite won't face a serious shortage due to the needs of maintaining the old system). And so on.
Yet such obvious fallacies remain as latent assumptions behind the majority of rewrites.
There are different ways to do rewrites effectively and none of them are easy.
One way is to build a parallel product using an entirely separate team, and having that product compete against your older product directly in the marketplace. This is obviously an expensive route, which is why a lot of companies shy away from it, but sometimes it's the best way to go about things. It gives a chance for a product to mature before it replaces the original. Microsoft did this with Windows, having two parallel teams work on different kernels (NT and 9x) and it took over half a decade for the "rewrite" to win out. This is even more astounding when you consider that originally Microsoft had intended for Windows 95 to be simply Windows 4.0 based on the NT kernel. A goal they didn't really achieve until Windows XP was released 6 years later.
An alternate way to do a rewrite is through refactoring style techniques. Start building the new system inside the old and slowly deprecate and migrate components piece by piece until the old system is entirely gone. This is a very effective way of doing things and has the advantage of being able to scale relative to the amount of development effort available but certainly has its fair share of limitations and downsides (which I won't enumerate).
As far as the Netscape example, they ended up coding themselves into annihilation, as their constant rewrites and architectural buffoonery led them to create a very unstable and clunky browser that was markedly inferior to IE 4 when it hit the market, and they just were not able to respond with any degree of innovation software wise. Also, my understanding is that the seed code for the mozilla project was a horrendous mess that didn't even represent a functional browser at the time and required extensive effort to get to that point.