I started seeing horrible Rails app initialization time when testing my rails application, but this slowness can be mitigated if you use nice assisting tools like spork and guard. Also ruby 1.9 >= through <1.9.3 does not load requirements in a efficient manner. There is a patch for 1.9.2 that improves requirement loading that helps. There are many ways to improve Rail's test performance. If you spend the time to tune your tests and test environment, Rails test performance should be good.
In Aaron Patterson's recent "Double Dream Hands: So Intense!" presentation[0], he came to the conclusion that Rails 3's slowness is because MRI sucks at GC on a deep stack depth. The average stack depth of Rails 3 applications is ENORMOUS.
He mentions in the video that he rewrote the Rack API to pass a mutated var to the next piece of middleware in the Rack chain instead of calling the next middleware and passing an env var to it. The gains were pretty impressive. Hopefully, they make this change to the Rack API for Rails 3.2.
That change + the 1.9.3 fix for requiring files[1] should result in much faster overall Rails performance.