Somewhat OT, but I've been wondering for a long time… Is the HPC community the only place the word "codes" is used like this? In usual CS parlance programming is done using a substance called "code" ("the high-performance code the community needs"), but in HPC literature the word "codes" is used, as if programming consisted of distinct objects. Does this arise from some divergent history (would I have called my LINPACK library punched card deck a "code"?) or what?
I've also noticed that it seems more common among Europeans, but that might be just personal experience.
One is a concept/idea/format while the other is a set of instructions.
I've noticed we also tend to also write "computer program" the US way, despite writing "TV programme".
It will be poorly documented, have single letter var names everywhere without comments, have a bunch of hidden dependencies on whatever the postdoc had installed on his/her laptop, leak memory but 'terminate before it should be a problem', etc. Oh, and there will almost never be even a single regression test.
Sure, the underlying algorithm might have a nice proof of numerical stability, but the actual software is almost always trash until some poor soul has to re-engineer it if the software turns out to be useful.
> Oh, and there will almost never be even a single regression test.
Bullshit. "Codes" tend to be run against analytical and/or experimental results.
The fact that you think looking pretty has anything to do with sound software engineering further illustrates my point.
>Bullshit. "Codes" tend to be run against analytical and/or experimental results.
A README that says the author ran it against dataset Y and got X is not a regression test. I'm talking about automated tests run on every proposed patch to ensure the thing is never broken. Real automated tests that cover 90%+ of branches are about as rare as unicorns in scientific computing (much to my annoyance).
Don't get me wrong, I completely understand why people running experiments skip testing (fast iteration). But that leads to the very opposite of "higher standards".
At least, the only ones in the last 30 years, I'm figuring - there is a reason we call it "code", and it's an old one!
Otherwise, we have a site with general information on polyhedral compilation http://polyhedral.info/ and Halide has its own site http://halide-lang.org/
By raising from C you don't need to use or learn a new language. Or one can generate C from a succinct notation or framework, and polyhedral optimize from there. That is what was in the R-Stream-TF paper.
Or, if TC's strength is in its generality, then what are the advantages over something like CuPy for Chainer?
Can someone give an example where TC shines?
In short, yes CuDNN is fast for the cases it was tuned for. It is probably faster on power-of-two sizes, but when you operate on a 26 x 1024954 x 3 tensor, TC can generate specialized code. Want 42 x 17 x 5? TC can generate differently specialized code. With almost no effort from the user (or performance engineers).
Can a performance expert do better job than TC optimizer? Very likely yes, but it will very likely take much more time.
TC is not a framework. It can be integrated with any framework of your liking.
But I guess unlike TC / Halide / TVM, that is less generic and much less customization.
Tensor Comprehensions does not try to manage memory and thus can be integrated into DL frameworks easily.
Edit: of course a computer works best with blocks of memory; that doesn't mean a human developer should have the same view. As a simple example, think of the output vector of a classifier. Why is it a vector, and not a structure? Or think of the internals of an LSTM network; there is more structure in there than just tensors.
It's just like with OOP in game programming where performance matters - even if you want a nice object model for programmer convenience, you'd also want to ensure that you can store the object data into a homogeneous array sequentially in memory, as that gets you a major performance impact; I seem to recall that Carmack had an in-detail article about that some time ago, but can't easily find it.
> Variables not defined anywhere, implicitly become index variables.
That seems like a bold choice. Wasn't there a trend in programming languages, even very high level ones, to encourage variable declaration?
C bindings don't seem to be a priority.
Feel free to use the contacts provided in the documentation here: https://facebookresearch.github.io/TensorComprehensions/cont....
I would have expected that operations expressed in the tensor language could be compiled once and for all (for a given target) into a DLL, and then it would be just a matter of passing the right buffer and shape parameters to a function. I see no reason why this should not be easily handled in C (which makes it easier to bind it from most languages).
If I understand correctly, DLPack is the format of choice to express a tensor, and ATen is not required, but I cannot find examples using DLPack
This needs Python bindings, stat!
Also TC doesn't do data allocation itself and it requires tensor library from users to do that. So if you are using Caffe2 for example, you could use the TcOp that we ship inside TensorComprehensions for Caffe2 and using caffe2 pybindings, you can already write TC in python. No work needed for creating python bindings. We welcome PRs on this :)
As for other tensor libraries, like ATen based on TH* used in torch/pytorch, you have the ability to create tensor but these require integrating tensor library into TC and writing pybindings for them. PRs welcome on this as well.