

RethinkDB 1.13: pull data via HTTP, push data via changefeeds - coffeemug
http://rethinkdb.com/blog/1.13-release/

======
sync
Huge release this time around. Congrats guys & gals!

`r.args` isn't very flashy but is a huge step forward for ReQL.

Removing protobuf makes deploying to certain platforms (e.g. heroku) so much
easier.

Promises support for the Javascript driver brings it to the modern age.

And of course changefeeds and `r.http`!

If you've been hearing about RethinkDB and thinking about trying it out, now's
a great time to spin the wheels.

~~~
coffeemug
slava @ rethink here. I forgot to add `r.args`
([http://rethinkdb.com/api/javascript/#args](http://rethinkdb.com/api/javascript/#args))
to the blog post! People have been requesting that feature for a few releases
now, and it should make lots of code much less painful.

FYI the upcoming 1.14 release is scheduled to include a distributed file
system and geospacial indexing (probably the most requested RethinkDB features
of all time).

If you have any questions about Rethink, I'm here all day to answer questions.
(We can also grab lunch or coffee if you're in the Bay Area).

------
mglukhovsky
Mike @ RethinkDB here. We're co-hosting a RethinkDB + Firebase meetup on July
1st in San Francisco on building realtime apps. We'll be doing more in-depth
demos of the new 1.13 features, and showing how you can use them to build
realtime apps faster.

RSVP to the meetup here: [http://www.meetup.com/RethinkDB-Bay-Area-Meetup-
Group/events...](http://www.meetup.com/RethinkDB-Bay-Area-Meetup-
Group/events/188950262/)

We'd love to see you come hang out with the RethinkDB and Firebase teams. Even
better, give a lightning talk on how you're using RethinkDB! Christina is
finalizing the schedule, if you'd like to do a lightning talk shoot her an
email (christina@rethinkdb.com).

~~~
piotrkaminski
I must admit this makes me confused -- do RethinkDB and Firebase integrate in
some non-obvious way? Both appear to be JSON databases with real-time change
feeds so it would seem they'd be competitors if anything... Any chance you can
clarify the relationship? Thanks.

~~~
coffeemug
Slava @ Rethink here.

The relationship between RethinkDB and Firebase is based on
conceptual/philosophical agreement about where the world is going (realtime
json sync), and that developer tools should be delightful. That's how we know
each other, and decided to co-host the event. There is no software integration
or anything.

I don't think the companies are really competitors -- Firebase is a service,
Rethink is a product, Firebase has much more hands off ops and smaller query
language, etc. So the use cases where you'd pick one or the other tend to be
quite different.

We just want to give demos about building realtime apps, and get to know
people who care about the subject!

~~~
piotrkaminski
Thanks, that makes sense. The point about service vs product is well taken,
though I find myself wishing somebody supported both modalities: a service to
get a prototype up and running quickly, with the option of moving to a self-
hosted setup later if needed for scaling or security reasons.

~~~
coffeemug
We'd like that too, but it's easier said than done. A good, robust service is
like a whole other company! We're looking at options, but there is still a
dramatic philosophical difference between a product first company and a
service first company.

------
troyk
I really hope I get to use Rethink in anger before I die. Last time, I had to
move on because we are multi-tenant SaaS and it seemed the compound indexes
just were not up to that task, does this release fix any of those issues? (I
seem to recall I was having to use a between query and not being able to sort,
but it was a while ago and I quickly got back to work with old reliable
(postgresql))

~~~
coffeemug
I think you're talking about
[https://github.com/rethinkdb/rethinkdb/issues/1227](https://github.com/rethinkdb/rethinkdb/issues/1227).
It hasn't been fixed yet, and is a big deal for many multitenancy users.

I'll bump it up in the roadmap -- sorry it's taking so long to get this fixed.
It turns out that everyone only needs 5% of RethinkDB features, but it's
always a different 5%. That makes product roadmaps really hard. I think in
this sense databases are a bit like word processors :)

~~~
simonb
another vote for compound indices.

~~~
coffeemug
Just FYI, you compound indexes are already fully supported -- see
[http://rethinkdb.com/docs/secondary-
indexes/python/](http://rethinkdb.com/docs/secondary-indexes/python/).

The particular issue in question is using an index to filter documents, and
then ordering them by another index.

------
dkhenry
It is tough to keep up with all these changes. I mean Rethink was pretty
feature complete for me like four or five releases ago.

I don't know how I feel about the http command but I am digging the other
changes. Keep up the good work.

~~~
neumino
@dkhenry -- Thanks for all your work on the Java driver!

------
vishy1618
Absolutely amazing release! A quarter back we had a small chat with
@coffeemug, to see if RethinkDB would be a good fit for our product
(geospatial indexing, ad-hoc queries, realtime updates, ...). He said then
that it wasn't yet, and promised that it would be in a year's time. I see now
what he means, Rethink is already looking like a very strong contender!

------
transientbug
Damn, rethink won't stop getting better and better! I'm looking forward to
getting to use the changefeeds, as I've currently got a redis list that I use
to get notified of changes to rethinkdb, which is fine but just another cog in
the system to debug.

Seriously guys, thanks for the great product and awesome database experience
(PyRethinkORM author here).

~~~
coffeemug
Thanks for PyRethinkORM! I haven't used it yet, but really looking forward to
taking it for a spin with Django in a few weeks. We'd love to feature it in
our docs when we get through a few more immediate issues.

~~~
transientbug
Awesome! Let me know how it goes and feel free to file as many bugs and issues
as you can. I've been saying that I'm going to be releasing v1 soon but school
and summer internships have been getting in the way of that, but you might
want to take a look at the `dev` branch it is much better than the current
v0.3 thats on PyPi, imo.

------
imslavko
Built-in notifications on writes are really cool! This is something Meteor had
to implement on the application layer because MongoDB never supported it
natively (and the oplog consumed by Meteor didn't really have enough context).
RethinkDB gives you both oldValue and newValue which is super cool!

~~~
coffeemug
You can also query on them, which is really cool IMO! E.g. give me every
document where the score has increased:

    
    
      r.table('games')  \
       .changes()       \
       .filter(r.row['old_val']['score'] < r.row['new_val']['score']) \
       .run(conn)
    

Or, give me every document for user X:

    
    
      r.table('games')  \
       .changes()       \
       .filter(r.row['new_val']['user_id'] == X) \
       .run(conn)
    

So you can write really cool apps out of the box without having to filter
things on the client.

(BTW, we have _a lot_ more planned for this)

------
davidbanham
Change feeds look great. r.http seems like a frippery, though. Not a great
signal to see things like that getting added when there are much bigger fish
yet to fry.

~~~
coffeemug
The r.http command was controversial, even internally. I had to call in a lot
of favors to convince people to get it in, but here is why I was convinced
it's a good idea:

    
    
      - It fits! It just seems to work magically well with the rest of
        ReQL on so many levels! JSON fits, streams fit, lazy evaluation
        fits, even batch prefetching fits! Everything works so
        wonderfully and provides such a great interactive experience,
        it almost would be silly not to add!
      - It makes a use case that's really important to me
        personally (adhoc analytics) an order of magnitude easier. Over
        the past few months I ran into a few other people that also do
        a lot of interactive adhoc data analysis with Rethink, which
        redoubled my resolve to add r.http.
      - It makes example datasets *much* more elegant. Reading
        RethinkDB docs? Just call this command to get the dataset in so
        you can play around with a command. (We haven't updated the
        examples yet, but we will). That makes the first experience
        with the product better, and makes the learning experience much
        more pleasant.
    

So I hear what you're saying, but I respectfully disagree. If you think of it
as a signal, think of it as a signal of deep passion for the product and our
users! Ornamentation for the sake of ornamentation isn't what we do.

(On an unrelated note, I really love your use of 'frippery' \-- I'm a bit of a
word nerd, and this is a really neat word!)

~~~
davidbanham
Thanks for the considered response. I think I have a much better handle now on
why it seemed like a good idea to you and a bad idea to me.

The disconnect is that we see Rethink in different ways. You're looking at it,
in this context, as Rethink + ReQL combining into a "tool for data analysis".

When I look at Rethink I see "Place to put my data".

My viewpoint pretty much ignores ReQL as anything more than "Means to get my
data out of the thing". Which, thinking harder about it, is wrong. You guys
are looking to make ReQL more than that. My viewpoint is too heavily
influenced by the way I think about things like CouchDB.

I normally don't go in for value judgements about people's viewpoints, but I
think in this case yours has to be declared objectively correct. Given that
it's your product and I'm just a guy that's been an interested observer for a
while, but has never actually used it in anger.

So, thanks for helping me get it. Word nerds unite!

------
hrjet
I am not a database expert hence this question.

RethinkDB looks better than MongoDB in many respects. But how does it compare
to Postgresql (latest versions with document support)?

If I were to start on a new project with modest scalability requirements, what
should I choose and why?

~~~
barosl
I think the biggest difference is the scalability built-in. It has a beautiful
Web admin and CLI admin that ease the scailability works including sharding
and adding a node to the cluster.

PostgreSQL also has built-in streaming replication, which is useful and works
nicely. But the scenario it covers is scaling reads, not writes. To scale
writes, you should shard the data manually.

There are efforts to bring write scalability to PostgreSQL. One of them I'm
keeping my eyes on recently is Postgres-XL, which uses statement-based
replication that focus on writability and availability. It is a premature
product that started recently, though. And also, some easy-to-administer
features like automatic failover and configuration propagation
(adding/removing nodes) are not the objective of the project. You should adopt
another solution, or write it manually.

However, instead of getting the scalability, you lose ACID in RethinkDB -
whose impact can be huge on some use cases. But personally I think it is worth
losing to gain scalability easily. And there is two-phase commit which is at
least doable to imitate transactions, though it isn't elegant at all.

If your application is read-intensive, I think staring with PostgreSQL and
adopting Postgres-XL or pgpool later is a good strategy. But if you are to
shard the data to scale writes, or you want an all-on-one solution that helps
the administrative works a lot, I think RethinkDB is highly recommended.

------
orkj
Great stuff as always. I have had a RethinkDB instance in production for 13
months now (since 1.4 - developed it while on 1.2) and looking forward to
upgrading and playing with the new features in other projects.

Thanks again for your awesome work, guys!

------
kclay
Great release, time to get my Scala driver updated. Who wants to help getting
the `changes` api to work with itertees?

~~~
dkhenry
Yeah I think that change is going to be another fun one to implement.

------
Seich
Congratulations on the release guys. The update to the Javascript driver is
huge and will make development a lot less painful. I get to remove a lot of
redundant code thanks to that.

I am a huge fan, keep up the outstanding work!

------
maxpert
Great job! Totally loved it. Thought it would be a good idea to have channels
instead of tables for pub/sub mechanism.

~~~
coffeemug
We debated this for a while and decided not to do channels for now.

Feeds are great because you can use them to integrate with other pieces of the
infrastructure like RabbitMQ or ElasticSearch, or write reactive apps where
clients instantaneously react to changes in other clients. Incidentally, you
can use them to easily get pubsub, but it wasn't the original intention.

There are much better pub/sub services out there, so we decided to stick to
having a really good feed API and avoid implementing channels for the time
being.

------
selvakn
Is the changefeeds feature comparable with changes api of couchdb?

~~~
coffeemug
Unless I'm mistaken RethinkDB changefeeds are more powerful than the couch
API. You can perform filters on changefeeds, do joins, transformations, etc.,
while the Couch API gives you a feed that has to be fully processed on the
client.

------
doug1001
am i correct that the biggest hurdle to a python 3 driver has been the lack of
python 3 support in Google's protocol buffer library? If so then, it seems
that the 1.13 release which removes the protobuf dependency, will
substantially accelerate development of the python3 driver

~~~
coffeemug
That's right! In fact, there is a pull request to make the driver Python 3
compatible that's being reviewed right now (plus some additional testing
changes, etc.) We should be able to get Python 3 support in pretty soon.

------
Goranek
Is using http out of the db really useful? Can someone give me a simple case
when this should be used?

~~~
coffeemug
It's extremely convenient for interactive use. See this tutorial --
[http://rethinkdb.com/docs/external-api-
access/](http://rethinkdb.com/docs/external-api-access/).

You can grab JSON data out of APIs in seconds, filter and manipulate it, and
enrich it with more APIs. I've been using `r.http` for some time (since an
internal beta) and I now find it invaluable for ad-hoc analysis.

~~~
quotemstr
> You can grab JSON data out of APIs in seconds, filter and manipulate it, and
> enrich it with more APIs.

So it's almost as nice as SQL, except bespoke and not specifically designed
for interactive data exploration?

~~~
coffeemug
I think it's best to play with it to get a feel for how it works. For example,
what would the experience be doing the analysis described in
[http://rethinkdb.com/docs/external-api-
access](http://rethinkdb.com/docs/external-api-access) in SQL?

------
miralabs
anyone using rethinkdb in production? Hows the experience like?

------
samstave
No offense, but I found the voice of the engineer difficult to listen to
seriously, found the video too humorous.

~~~
coffeemug
Michael can be an acquired taste, like fine wine or good scotch. If you don't
push past that first sip, you'll never know the deep, tantalizing world known
to us connoisseurs as @mlucy. (But if you do, please let it be a matter of
public record that we discovered him first)

~~~
samstave
Thanks, apologies for my comment being offensive... but I should be able to
state objectively what I thought detracted from the message of an otherwise
fantastic announcement. I in no way mean to pass any judgement on Michael as a
person...

