
Lessons learned from the big rewrite - antonpirker
http://www.anton-pirker.at/the-big-rewrite-war-story/
======
troels
My experience from doing rewrites is that if at all possible (and it usually
is, even if you don't think so), do the rewrite incrementally. For example, in
the case described here, they wanted to switch rdbms, change up the schema,
change the application level implementation/framework and change the interface
all at once. That's a lot of risk to combine.

Maybe they could have started by refactoring the schema in-place (That is,
within mysql and webtek). Then change from mysql to pgsql - still behind
webtek. Maybe separating the change of webtek -> django from the interface
change is a bit harder, but you could still try to restrain your self as much
as possible and then make a second iteration of the interface after the
initial change.

I've taken this approach before, when migrating a big product and it worked
quite well. It seems as a lot of extra work at first, but I'm pretty sure that
is not actually case.

~~~
Demiurge
Doing it incrementally is definitely the best. However, in your scenario, the
problem of a 'weird' codebase people have a hard time using is not addressed
by requiring further development in it. Database layout is usually not the
problem if the system was at all functional.

Having done this a few times, when switching from old to new, the new
framework would have a DB agnostic ORM. What works nicely is building the
models, duplicate templating, getting a mirror working. Shut down the old
site, and then it's possible to go on with normal development.

So, it is important to consider rewrite and redesign as separate goals.

~~~
a8da6b0c91d
In this case porting from perl to python makes it much harder and seems
pointless. Why not just keep it in perl but incrementally move to a modern
perl framework?

~~~
Demiurge
Why do we see so many more posts about Django compared to 'modern Perl
framework'? Python has killer features like simple syntax and developer
familiarity. Django has GeoDjango.

This isn't even a question to me, but to the blog author. My. Point is about
order of steps in the rewrite. It is not harder to rewrite first, redesign
second, it is easier. I've done it multiple times in c++/c#, perl/python,
php/python, r/python.

------
ExpiredLink
> _The first thing I will do the next time is taking more time to analyze the
> old system thoroughly and write a technical spec what the new system should
> do. A simple "What the old system is doing now" as a spec for the rewrite
> does not cut it. No one knows what the old system is doing exactly so it is
> necessary to take the time to write a spec._

I had the same experiences last year. There wasn't even a single person who
knew the 'old system' on the whole. A receipt for many detours and much extra
work.

------
dschiptsov
It is not that simple and comes back to the same old maxim - as long as the
only documentation of what the system is doing and why is its source code, you
are doomed.

If you have a written detailed guidelines of what you are doing and why, which
design decisions were made and why, which data representation is used and why
and what corresponding algorithms are and why, then one could re-write parts
of the system in different languages/frameworks just for fun or to explore and
compare.

Coding or rewriting is second to deep, hierarchical understanding of hows and
whys. This is how the most of programming exercises are designed - once you
have understood how it should be, which results each procedure should return
(wishful thinking and tests before code, to define domain and range of a
procedure - ideas as old as SICP) then coding is the most trivial part.

Look at antirez - he is a rare example of a guy of deep understanding before
code.

~~~
thirdtruck
Uncle Bob made this point very salient in one of his videos:

If there was some catastrophic, quirky failure of backup, and you could only
save either (a) all your production code, sans test, or (b) only your tests,
which would you rather save?

Assuming you've written thorough tests, they win out over code every time. The
tests remove all ambiguity.

I'm reminded of a story that illustrates the point:

A young woman always cuts the ends off of her pot roasts before cooking them,
following her mother's example. One day, she asks her mother why they do that.
The mother doesn't recall, so she asks the grandmother. The grandmother looks
at them both, shrugs, and says, "When I was your age, we lived in an apartment
with a very small oven. We had to cut the pot roast down to make it fit!"

~~~
rquirk
Oh come on! Reality check please :)

Tests alone provide little value to anyone. At least if you had the production
code you could keep it running in its current state while you added back those
important tests bit by bit.

Most of the tests wouldn't be needed as good code tends to be churn-free.
Finally if you wrote the code TDD style, then writing tests for even the hairy
bits would be "easy" compared to rewriting the whole system.

~~~
thirdtruck
I'm going to have to respectfully disagree. :) (Or, with apologies to Adam
Savage, "I reject your reality check and substitute my own!")

A _huge_ portion of my day-to-day agony at my current job stems from a lack of
tests. The developers on my team (all lifelong employees, where I'm a contract
hire) are terrified of making more than the absolutely necessary changes to
code. I can't replace literally copy-pasted code with a shared method without
push-back. To even _speak_ of refactoring is verboten, at least in front of
our internal customers. Testing small changes can take days, as it must go
through QA and the customer, and surprising bugs still slip through. As
someone who went through Fowler's _Refactoring_ as bedside reading, this
process and turnover rates drives me batty.

Here's the thing: We have the production code up and running, and I'm already
adding tests where I can, but we constantly run into _implicit_ requirements.
Even the senior developer, who's worked on this code base for years, keeps
discovering things about it. We continue to discover genuine bugs, too. The
production code empirically lacks enough information to reproduce the tests.

And "good code" begs the question. In fact, it begs two questions: (a) How do
you know code is "good" if it isn't tested empirically, and (b) how can code
ever be churn-free when the requirements themselves keep changing?

~~~
rquirk
Heh, that agony description was eerily familiar, and I'd be the employee! The
hard part is not being afraid to make an obviously right change, even if it
means spending ages manually testing things and then continuing to follow up
weeks later when the non-obvious things you didn't think of break :)

Even in the face of changing requirements I've found that there are substrates
of code that do their job without (showing?) major bugs for years. They never
needing fixing. That's your good code. If you had unit tests, you'd never run
them anyway because you never touch that code. They are the "select()"s of
this world, but specific to whatever you're writing.

Now if the whole system had to be rewritten, the old tests would be useless
anyway. You mention requirements that have changed, so those tests you had
would no longer apply. You'd have to write a new system _and_ new tests ;-)

FWIW I've found that peer code review and static analysis tools give you a lot
of what unit tests would anyway, without the overhead of having to write the
tests and update the dead code when you rewrite the affected part of the
system.

I mean... what's the point of code anyway? Is it to be "perfect"? Or is it to
do a job, earn you money, and be more or less maintainable? Constantly
refactoring to stay in the same place is worse than copy-pasting code, from a
business owner's POV. Sometimes it is better to have 2 copy pasted methods
that are deployed and the customers are happy, than trying to have 1 perfect
method sat in development. If those 2 methods never need updating again, you
wasted time refactoring. If you would need a third copied method later on,
that's the time to refactor.

Maybe I'm being a bit devil's advocatey. Unit tests and refactoring are good
tools, but they are not "free" (since you have to test the real system anyway
at some point) and Uncle Bob is a bit of a snake oil salesman anyway :-P

------
adrianhoward
For those who've not already come across it I would _thoroughly_ recommend
folk taking a gander at Michael Feathers book "Working Effectively with Legacy
Code" on this topic.

Lots of solid advice and patterns for approaching thing sort of thing.

------
Fumzz
To number 2 (Migrating the database is hard) I would also add that it is
important to verify the integrity of the migrated database in some way
afterwards. Even a simple assertion like "the number of user accounts (or
whatever) is the same in both databases" will help greatly.

~~~
antonpirker
Yes, right! I forgot about this in the blog post. We counted the rows of some
tables. Unfortunately we could not do this with all tables because we had dead
data in the old database (ex: child rows without parents). But with the new
database we made sure to have all the database features and checks in place to
guarantee referential integrity. We also did spot tests to make sure the data
is as expected.

------
h1fra
After spending 3months rewriting an huge module for our many websites,
initially estimated for 3weeks at most, I can say that it's easy to make
mistake but also incredibly good for the company.

\- Mixing refactoring and new feature.

\- Introducing new technology

\- Changing behavior of old code that were not directly concerned by the
rewrite

...

Many things that should have been done after that. But, all I can say is if
the company can handle this amount of work, it will always be a very good way
to improve the quality and durability of the website.

Because if you simply rewrite code, there is a big chance that your company
will assign to you, new task that are not related at all and you will never be
able to introduce these feature.

------
fsniper
I have an issue with iframe perf test. It is very reckless and a waste of user
bandwidth. One can reroll user requests from web server access logs and it
would be a controlled perf test with real user behaviour.

------
joosters
Lesson 1: Don't ever attempt a big rewrite?

~~~
antonpirker
Sure you can attempt a rewrite! But you must make sure that you know what are
getting into. It is not an easy thing to do, but once the rewrite is done your
live gets way better :)

~~~
arethuza
Having had rewrites of large projects go spectacularly well and also
spectacularly badly, I suspect one factor to be mindful about is if the
rewrite facilitates the "second-system effect" \- I've had that happen where
the rewrite of a messy but successful suddenly allows the introduction of lots
of cool features you've wanted to add for ages and you actually end up with a
"simplified" system that is actually a lot more complex than the original.

NB I've only had one project go badly wrong that way and I guess it was a
useful learning experience :-)

------
LeonidBugaev
For testing performance i would recommend
[https://github.com/buger/gor](https://github.com/buger/gor) (i'm author).

~~~
nkuttler
Nice, I was just going to point out that using an iframe really seems like a
bad idea.

------
Kiro
Any drawbacks of #3?

~~~
antonpirker
It's a really easy thing to do. But you need enough server resources for this.
A mirror of the current production system would be the best thing. If you only
have one server where the live system is running and there is not enough
resources on this server you can not do this.

------
curiously
I rewrote my product 6 or 7 times in the past 6 years. 4 of those years were
written in isolation by myself. 1.5 years I worked as a developer which I
HATED and fired from (refusing to work excessive overtimes and as a result
impossible to find a job in this competitive labor market) and working on my
product again. I open sourced the old code that I "threw" out to start all
over.

Technically, never reached market until 5 months ago. I have of course,
launched after each time the product was finished but would give up due to
being burnt out etc. An entire year spent starting from scratch which would be
my 7th attempt.

3 month in after launch I was going to rewrite it again, I decided, this
would've been my last attempt in the software business. I decided that enough
was enough, this is it. I ALMOST rewrote it again but having done that 7 times
already, instead of doing that, I just ripped out a lot of code, made
architectural changes and reused what I could, whatever was tested. 8 times a
charm....

My lesson learned from past 6 years pursuing insanity, restarting from scratch
means you will have to solve the same problems again and again. A blank canvas
is NOT a solution. Don't do that, it's insane, I was insane but not anymore.

~~~
segmondy
Stop chasing your tail. I have no idea what you are working on, but if you are
not trying to solve a hard CS problem and you are trying to build a business
...

~~~
curiously
NLP from texts detected from the audio in video files. 100% solved? No. Others
have attempted or marketing their own solution? Yes.

I was just really crazy and stupid to pursue my own vision of what the
solution would be when in reality I lacked the experience (I started with
barely knowing how to code).

