What a great technical problem, except for this bit:
> Btw, I didn't try to hide behind "this is scientific computing -- no way to optimize". It's just that the basis for this code is something that comes out of a "black box" where I have no real access to
A research assistantship I held was based on classified data; all of the published work had to be approved by the DoD and the actual data we used wasn't allowed to be published which made our results entirely unreproducable.
I'm certain the methods being used here are just wrong. It looks like it's code generated by a symbolic algebra system. Chances are that using a more scalable methodology would eliminate the code size problem. For example, symbolic manipulations often have exponential space complexity in the depth of the expression tree, but automatic differentiation can do the required operation in polynomial time and space (often linear or log-linear).
Unfortunately, "no way to optimize" is basically a statement that the questioner is not interested in an algorithmic way to make the problem go away.
It seems you don't understand the context of research projects. When you're assigned to write a program that processes data x coming group another group, it's valid to say that that input data is a 'black box' to you. That still doesn't mean that the whole research is 'black box' or somehow not 'real science'.
Science is a blackbox. We make theories about the world we live in based upon the results of experiments. Richard Feynman commented method of thinking about science on one of his videos.
How? You handed me a black box. You wrote some code, and you fed it some data, and you made some prediction, and all I really have is your word for it that your prediction worked, but I sure can't do anything with it. I don't know what it is or how it works, how to extend it, how to build on it, anything.
That's not science by any useful definition of the term.
> Btw, I didn't try to hide behind "this is scientific computing -- no way to optimize". It's just that the basis for this code is something that comes out of a "black box" where I have no real access to
Black box != science.