My speed comes from avoiding consing, my native types are bitfields. Lisp types are usually a class pointer for every cons cell. For me certain casts are permitted, which are not permitted with Typed Racket. So yes, I have to permit traditional code which worked before, esp. with inferred types from literals. Racket has the advantage of defining stricter language subsets, which we only have with Perl 6 also. There you can override even language and type semantics.
Chez has a nice tagging scheme, which should speed up Racket a lot.
Several gradually typed languages don't infer from literals, but that's one of the best speed gains. E.g. the literal 1000 is a union of int32, int64, uint32 and uint64, all possible valid integer types for a certain value. A negative number cannot be unsigned, and a too large number cannot be 32bit.
Problem is that people rarely add types, you need to infer 90% of them.