

All about lambda functions in RethinkDB queries - mglukhovsky
http://rethinkdb.com/blog/lambda-functions/

======
ot
So if my understanding is correct, the RethinkDB driver translates a Python
function object into its "abstract interpretation" by invoking the function on
a proxy object x that defines operators such that (simplifying) x.__op__()
returns Op(x). Recursively, this yields the computation tree of the function.
This looks very similar to what is done in C++ metaprogramming through
"expression templates", a technique that is taken to its extremes in libraries
such as Boost.Proto. C# solves this more cleanly by having in the language an
explicit concept of "expression trees" that are given as arguments of LINQ
queries.

I wonder if this imposes any limitations on the functions that can be written.
For example, how are if, for statements handled? What happens if I do "lambda
x: y + x" where "y" is an object that defines __add__? If these cases are not
handled, is a well-defined exception raised?

~~~
coffeemug
This is tricky and there are certainly limitations. For example, you can't
translate an if statement, so instead we implement our own branch command
(<http://rethinkdb.com/api/#rb:control_structures-branch>)

We overload operators when we can (unfortunately no operator overloading for
Javascript folks) and it works really well.

You don't always get a great exception, but most of the time this turns out to
be fine. Novices don't need to worry about this too much, and we found that as
people get more into ReQL, they pick everything up extremely quickly, and they
rarely confuse things or make mistakes.

~~~
hosay123
There are two alternatives to what RethinkDB does: parsing the function's
associated source file (yucky and breaky when multiple lambdas per line) or
disassembling its func_code..

I've looked at this problem before (and ran away screaming every time), but
the approach I tended towards was func_code disassembly, pretty much for the
reasons you mention. Was this ever considered?

~~~
coffeemug
We did look into disassembly, but decided that it's far too much work because
it is not portable across languages and drivers, and would require an enormous
engineering effort (both for ourselves and for community driver maintainers).

From a purely theoretical POV, I do like the idea of disassembly better, but
unfortunately we couldn't make this work and remain pragmatic.

------
thom
Well this is just lovely! It's so similar and such a good fit for what we're
doing in Delver (we build very similar abstract queries out of natural
language, but for compilation into SQL etc) that it makes me want to force
everyone to use RethinkDB. Our lives are maybe a little easier cos we do it in
Clojure so it's s-expressions in, s-expressions out.

Can't see much evidence of a clojure driver yet? Is there an existing project
we could contribute to?

~~~
mglukhovsky
I'd love to see a project bring RethinkDB to the Clojure community-- we
haven't seen one emerge yet. If you'd like to contribute (or better yet, start
the project!) shoot me an email at mike [at] rethinkdb.com.

We've done a lot of work to make building a client driver for RethinkDB
easier: we use Protobufs, have built some testing tools (e.g.
<https://github.com/neumino/rethinkdb-driver-development>), have a Google
group for driver developers
([https://groups.google.com/forum/?fromgroups#!forum/rethinkdb...](https://groups.google.com/forum/?fromgroups#!forum/rethinkdb-
dev) ), and our engineers love helping people build new drivers.

------
carterschonwald
and thus we learn that RethinkDB is an elaborate scheme to get programmers to
learn Higher Order Abstract Syntax techniques, not just have a neat db :)

Hoas, and its sibling PHOAS, are great ways of embedding a mini language
inside another language, they're especially powerful in language like Haskell,
Scala, Idris, and the like.

------
kclay
This will help in the Scala driver I've been developing

~~~
dkhenry
I don't know. I have been working on a Java/Scala driver too and thinking how
you could translate a scala lambda to this kind of representation It would be
really cool to do

    
    
        db("tablename").filter( x => x.name == "foo" ) 
    

and be able to translate that to the relevant query, but I can't really think
of a way without using Macros or reflection.

~~~
kclay
I was thinking maybe a path type syntax, yeah you have to use strings for your
field names but this way you can wrap it in a 'get' node .. Seems to have
worked well with Play! json classes

------
JulianMorrison
That is genius.

I have to wonder though, how they plan to do it in less dynamic languages?
Just spell out the s-expr and forget about using the language's own syntax?

~~~
vyrotek
This looks a lot like LINQ. C# is probably one of those "less dynamic
languages" but has had this for years. It powers a lot of ORMs and other
frameworks that need a queryable expression language. It supports writing
custom 'providers' so that you can write LINQ against almost anything.

<http://msdn.microsoft.com/en-us/library/bb397933.aspx>

~~~
mfenniak
I'm the author of a .NET RethinkDB client
(<https://github.com/mfenniak/rethinkdb-net>), and I can add that RethinkDB's
lambda functions were very easy to implement in .NET by using C#'s expression
tree feature (<http://msdn.microsoft.com/en-us/library/bb397951.aspx>). With
that feature, a lambda can automatically be converted to an AST by creating a
function that takes an Expression<Func<...>> parameter, which can easily be
traversed and converted into RethinkDB's expression tree.

Expression trees were introduced at the same time as LINQ, to support LINQ.

------
hoodoof
Deeply painful for the novice developer, good for the computer scientist.

~~~
coffeemug
To be fair, this is an advanced article written by a computer scientist that
designed the innards of the drivers for other computer scientists that want to
understand how the drivers work.

Novices can do most things without having to understand any of these issues,
and even very advanced users don't ever have to learn this if they don't want
to.

Disclaimer: I'm one of the people responsible for designing ReQL.

~~~
hoodoof
Maybe it needs to be more clear that there is a simple and a hard way to work
with ReThinkDB, and whenever someone finds themself in the "hard way"
documentation, that you provide clear instructions that this is the hard way
and the documentation for the easy way is somewhere else. The assumption for
the novice developer is that "here is the documentation, there's often just
one way to do it, so I'd better read this and learn it". It's not clear that
"this is the hard way, you should avert your eyes".

Maybe try renaming "ReQL" to ThinkSimpleQuery and the other one to
ThinkBigQuery. Something more self explanatory.

~~~
coffeemug
Well, the second paragraph says the following:

    
    
      This post covers all aspects of lambda functions in ReQL from
      concept to implementation and is meant for third party driver
      developers and those interested in functional programming and
      programming language design. For a more practical guide to using
      ReQL please see our docs where you can find FAQs, screencasts, an
      API reference, as well as various tutorials and examples.

