I'm amazed that the implementation was under 1500 LOC! Was that the research prototype or the shipped preview?
Congratulations on the VLDB paper! Hopefully I'll come say "hi" in LA :)
The shipped preview has only a bit more than 1500LOC.
The VLDB paper was presented at Rio in Aug this year already, but I'll try to come over to LA anyways :)
There are definitely some differences between the kind of UDFs that Spark supports and the kind that Froid handles. For one, Spark UDFs cannot invoke a Spark SQL query in their definition AFAIK, whereas TSQL functions can.
But still, some techniques might be applicable. Definitely worth digging further!