
FoundationDB Document Layer - wwilson
https://www.foundationdb.org/blog/announcing-document-layer/
======
rubyn00bie
Soooo stoked. This should make futzing with it to get started a lot easier.

I've been thinking about going to the FoundationDB summit to hopefully learn
more... Does anyone have any good resources for learning about it? I've tried
a few times but find the documentation a bit hard to grok. Maybe I'm just
slow.

Also if you haven't seen it, this is one of the most amazing talks on
distributed systems (because it's about testing them) I've ever seen (by Will
Wilson from FoundationDB pre-Apple merger):
[https://www.youtube.com/watch?v=4fFDFbi3toc](https://www.youtube.com/watch?v=4fFDFbi3toc)

Truly mind blowing how good FoundationDB is supposed to be for its use cases,
and this Mongo layer is just the business when it comes to helping adoption.

~~~
andrewflnr
That talk was glorious. "We want to build a database, but first we need to
build a simulator for the database, but first we need to write a compiler to
help us write the simulator." It would be self-parodying engineering hubris if
it hadn't worked.

------
tinyvm
> Document Layer has a familiar API, and is compatible with the MongoDB®
> protocol

Was wondering how I would replace MongoDB due to the shift in Licensing become
more and more agressive.

Well , I think I found it.

Farewell MongoDB.

~~~
justaaron
more like a scaling issue, as mongo is master-slave and a distributed
horizontally scaled world can't live with such a bottleneck.

~~~
merlynn
Well that’s just a ridiculously uninformed statement. Of course mongodb scales
horizontally. I can share many success stories of massive scale with mongodb.

------
mikece
I'm a little confused. Does the "layers" concept mean the data which is stored
in the core can be accessed in any valid manner that the layers allow? Or
stated differently: if I've got a dumpster fire of data in a MongoDB that
should have been in an RDBMS but moving to SQL would be too risky/slow would
this allow me to schlep my Mongo data to FoundationDB and have the app
continue to use the Mongo access methods while I start switching APIs over to
querying by SQL in batches instead of One Big Volatile Move?

~~~
sigstoat
foundationdb is just a KV store that doesn't even authenticate clients
connecting to it. the idea is you put layers in front of it to take nice
structured requests (of some sort) and turn them into transactional KV
manipulations, while also doing things like authentication, access control,
logging, whatever else you need.

so you could (hypothetically, if such software existed), have this mongo
layer, and then another layer off to the side which took SQL queries, and
figured out how to turn them into KV queries against the data that the mongo
layer has stored.

more realistically, you could have your SQL clients know how to talk to
foundationdb directly (making them into the layer; presumably they're a web
backend or something already), and they could know how to execute the high
level operation they want to perform against the KV data stored by the mongo
layer. conveniently skipping all of the SQL translation that there isn't
software for.

------
spullara
Really excited that they were able to release this layer. With the changes to
the license of MongoDB it could give people a real alternative.

------
monstrado
This is massive for adoption. Can't wait to give it a try.

------
andrewflnr
Is there any reason to expect fundamental inefficiencies with this approach vs
a vertically integrated approach like a conventional document store? Any other
downsides? I ask because this seems like to obviously good an idea to be true,
so I need to look for the flip side before I get to excited. Otherwise, this
laying seems like the obvious approach, and I can only lament that it's taken
this long.

~~~
ryanworl
Yes, the layered approach will always consume more bandwidth unless you just
happen to hit the case where your transaction is read-only and the layer
process is colocated with the storage server where your data is.

The upsides far outweigh this downside in my opinion. The fact that writes are
buffered locally before committing the transaction (improving latency) is
probably enough by itself to overcome that downside.

~~~
bithavoc
Interesting, TiDB also three components [0](KV, DB and PD), I guess this
MongoDB layer in FoundationDB is the DB piece of the cluster.

[0] [https://github.com/pingcap/docs#tidb-
server](https://github.com/pingcap/docs#tidb-server)

------
rhacker
I don't know much about foundationDB, but when they say API compatible, are
they referring to the technical TCP/IP connection stack, or ALSO the concepts
of collections and the query language of MongoDB (such as aggregations, find,
limit, etc..)

In other words can I throw a giant aggregation query with all mongo's various
crazy query language syntax and it will.. work the same?

~~~
sbr464
It doesn't support the aggregation framework currently.

Here is a list of differences:

[https://foundationdb.github.io/fdb-document-layer/known-
diff...](https://foundationdb.github.io/fdb-document-layer/known-
differences.html)

~~~
spullara
Can someone who uses MongoDB at scale (many machines, single db) talk about
how big gap is between what FDB supports and what their application uses?

------
janemanos
Sounds kind of "old" tech to have different layers for supporting different
data models. Like the native multi-model approach of ArangoDB much better.

Their aggregation framework is pretty neat and the database also has full
graph capabilities. Heck, I can even combine joins and graph traversal in the
same query.

------
justaaron
i think we are going to generally see more and more object to key-value
mapping layers as the latter are widely available via cloud providers, and one
basically needs something in the spirit of persistent-volume-claims from
kubernetes land in the database layer that can be fulfilled by some key-value
backend...

------
sfilargi
FoundationDB always sounded amazing, BUT, the way they disappeared when it was
bought by Apple left a bitter taste in my mouth and I don't think I would ever
trust them enough to build my business around their product.

~~~
dilap
It's all open-source and out there now though, so I don't think you really
need to trust them, right?

------
monstrado
Are there any plans to extend the API compatibility of the document layer to
be compatible with other APIs, such as, DynamoDB?

------
justaaron
That's huge. I could have used this a few years ago

------
yoava
Just wandering how will foindationDb compare to mongo in terms of the non
functional s - performance, scale of a collection, number of collections,
queries with and without indexes, aggregated, etc.

------
thefounder
Can someone write a Google Datastore layer so that I can finish my
appengine/datastore adventure once and for all?

~~~
wsh91
Is there anything we could be doing to serve you better?

~~~
thefounder
Yes, let me run datastore on my servers. Surely I may use your cloud offer but
in certain cases I need to run it on my servers. I can't build/invest in
tools/drivers for a proprietary cloud service.

------
aseipp
Wonderful. I just updated the FoundationDB packages in NixOS to 6.0.15 at the
release earlier this month. Will try packaging this up soon as well...

