Hm, I don't get this argument, does not the internal implementation of arrays and hash tables have to do the exact same thing, that is specialize to the type it holds? I get that it potentially significantly slows down compilation, since C++ style templates are antimodular.
But you could easily compile a module containing templates in some intermediate form that allows for efficient instantiation, languages like OCaml demonstrate that you can have highly polymorphic modules through functors and still efficient and fast compilation. Of course it is probably too late for Go to get an ML style module system, because they chose to use much less expressive interfaces.
