
Realm is a mobile database: an alternative to SQLite and key-value stores - viebel
https://github.com/realm/realm-js
======
danbruc
OT: What happened in the database space and why? In 2000 you would have had a
hard time naming enough database systems to run out of fingers for counting.
But in recent years database system seem to pop out of the ground like
JavaScript frameworks.

So what changed? Did the idea of a general purpose database system break
because the usage scenarios became so diverse that we can no longer pretend
that one size fits almost all? Is this just a perception thing because
everybody started uploading his weekend hack toy database to GitHub and blogs
about it?

And what really irritates me the most, database systems are really complex
pieces of software requiring years and years of development time, where does
this time come from and who pays for it? Especially having more and more
different databases system to choose from does not really promise that
developing yet another one will be an easy commercial success.

~~~
jinjin2
Realm has been available for many years and is actually really different from
what else is out there by the fact that it is a real object database. It has
just been flying under the radar a bit given that it has only been available
for mobile devices.

I for one think that it is awesome that it is now available for node.js as
well. Especially with the new replication functionality that allows you to
share live data between node instances running on different machines.

~~~
lobster_johnson
"Real" object databases — "OODBMS" — uses to be a thing, too. They still
survive in some niches, like finance and health, though largely as legacy
systems. Products like Objectivity, ObjectStore, POET, Caché (which is built
on MUMPS), GemStone (Smalltalk) etc. were once big, expensive products that
seemed poised to take over from old-hat SQL databases.

Those databases failed not necessarily because the technology was wrong,
although they certainly made some choices that didn't help; those databases
were often tightly coupled with opaque binary object serialization formats
(with no way of looking into a database without going the binary route) that
were in turn tightly coupled with the app's class hiearchy. For example,
several of these actually post-processed your code (e.g. your Java bytecode)
to insert serialization code, so that when you followed a relationship (e.g.
"library.books[0].author"), it would automatically query under the hood and
return the author as an object. They ended up being clunky to work with as a
result; for example, in the beginning, very few of them had a query language,
so the only way to query the data was by following relationships through your
graph of objects.

Realm looks interesting as a new approach to the object database design, but
at the same time seems a bit limited. It doesn't seem to have schemas or any
kind of role-based permissions, and seems to make the same mistake of not
including a query language (so you can't build a REPL for it, and every
language binding needs its own query builder). The transaction semantics also
seem to be entirely undefined (is it read-committed?). From what I can tell,
it's also "offline first", in the sense that you have a local database that
can sync to/from a server, but you're technically always working on an offline
copy of the master data; while nice to have for a mobile app, it does limit
what you can use it for. I also don't see any support for fine-grained
updates. Every write you do seems to overwrite, which I assume means the last
write always wins; how do you deal with conflicts, or implement things like
counters, maps or arrays that need to be incrementally updated?

~~~
brmunk
Realm certainly has a schema - anything that made you think it doesn't? It's
ACID and Serializable.

It also has a query language. But as one of the core design principles is to
be as native to the language and platform as possible, the query language
takes different forms. In Java it's a fluent API, in Cocoa it's NSPredicates
string based. And the JS string based is roughly identical to that as well.

It's absolutely designed to be offline first. I'm curious as to which
limitations you see with that?

Operational Transform is used to resolve conflicts at the property level, so I
would say it's much more fine-grained than most other systems that would
overwrite entire objects. The specific resolution rules are very simple and
intuitive and how to deal with custom needs are described here:
[https://realm.io/docs/realm-object-server/#conflict-
resoluti...](https://realm.io/docs/realm-object-server/#conflict-resolution).
There is ordered lists already and explicit counters are being exposed in the
SDKs at the moment.

You might find this article useful as well: [https://realm.io/news/eventually-
consistent-making-a-mobile-...](https://realm.io/news/eventually-consistent-
making-a-mobile-first-distributed-system/)

~~~
lobster_johnson
I couldn't, and still can't, find any documentation or "list of features" that
mention schema support, query language or ACID semantics. The documentation is
pretty sparse and disorganized.

(By query language I don't mean an opaque builder based on NSPredicates or
whatever, I mean a query _language_ , written as text, which is portable
across clients.)

OT is great, but it doesn't have that until it's actually exposed. My pet use
case is string edits, which absolutely need to be cleanly conflict-resolvable.

The inconvenience with offline-first is that by definition you're never
operating on current data: You have to sync to a central server before the
data is visible to others, and you're editing potentially stale data. The
latency of inbound syncing is particularly dependent on the volume of changes
being sent from upstream, which might require that a client, so as not to be
overwhelmed with updates, subscribe to a very small subset of the entire
dataset. For example, if you open up a UI that edits a single document, you
might want to subscribe to changes to _only_ that document, in order to update
the UI in real time. I don't know if that's even possible with Realm, or if
you can in fact run it in "classic client/server" (i.e. "online first") mode.

~~~
SandyAndyPerth
Why would you want a text-based query language if you are an idiomatic C#
programmer? Every database product on that platform is queried through the
standard LINQ syntax - that's why we implemented it that way.

~~~
lobster_johnson
I'm not an idiomatic C# programmer, but having a text-based language is
_really_ useful in all sorts of situations:

* Interactive REPL (think psql)

* Use outside of C#, e.g. small bash scripts for automation of small tasks such as deleting old records in a cron job

* Copy/paste a into Slack, Github issues etc. without having to drag along any dependent code or worrying that it's not a complete query

Query builders are nice, but eventually they have to compile down to something
"neutral" and portable.

------
azinman2
I've had some nasty bugs happen in production with Realm, as is expected with
any new database store. Newer version seem better in this regard, but if you
read the changelogs you might be shocked at how fundamental the fixes can be.

It also times some time getting use to -- my biggest gripe is that the
notification API won't tell you what actually has been modified, just that an
object changed _somehow_. That makes a pure reactive design quite hard if
those changes have non-trivial consequences, and not just a quick re-rendering
of some element on the screen.

Looking forward to finding time to go back to SQLite where I can actually do
complicated queries, relational joins, triggers, etc on a fast and mature code
base.

~~~
BoorishBears
SQLBrite adds reactivity to SQLite
([https://github.com/square/sqlbrite](https://github.com/square/sqlbrite))

But in general I rely on RxJava to provide fine grained notifications. For
example (psudeocode):

    
    
        someUserQueryThatReturnsObservable()
          .map(user->user.firstName)
          .distinct()
          .subscribe(newFirstName->newFirstName)
    

It's messy because you end up having to repeat everything after map for every
property, not really an ideal solution

~~~
unsoundInput
I started using sqlbrite (in combination with sqldelight) and stopped treating
the database as an object store. Instead I create an interface for the data
the customer needs (e.g. an item interface for elements in a listview), write
a select query whose result fulfills this interface, and let the rest happen
via databinding. This seems to work fine up to a couple of thousand of
elements for me.

------
theamk
Is there any technical information on how the database works? For a supposed
Sqlite replacement, there is a surprising lack of the information. The only
reference to concurrency mode I have found is this:

> All changes to an object (addition, modification and deletion) must be done
> within a write transaction.

> Write transactions incur non-negligible overhead - you should architect your
> code to minimize the number of write transactions.

So is there a single write lock? If you try to start two transactions, will
the second one be delayed or retried? Can reads proceed while the transaction
is processing?

> Results instances are live, auto-updating views into the underlying data

So if I am doing x.first, x.second while the object is being updated in a
second thread, will I get mixed state?

Is database crash proof? If the phone runs out of battery while in write
transaction, will the database get damaged?

... so many questions

~~~
brmunk
It's ACID & Serializable. A write will block all other writes, but multiple
reads can occur without overhead or blocking due to MVCC. Reads are consistent
within the runloop, but updates to your objects occurs when you start a write
transaction, so you know you always work on the latest version of the data.
Basically, you are in an implicit read transaction all the time, except when
you are in a write transaction :-).

> So if I am doing x.first, x.second while the object is being updated in a
> second thread, will I get mixed state? So no you will not get mixed state.

> Is database crash proof? If the phone runs out of battery while in write
> transaction, will the database get damaged?

Yes it is crashproof. Fully ACID.

Any other questions?

This article goes a bit more into the details of threading:
[https://realm.io/news/threading-deep-dive/](https://realm.io/news/threading-
deep-dive/) And this one more into the core design: [https://realm.io/news/jp-
simard-realm-core-database-engine/](https://realm.io/news/jp-simard-realm-
core-database-engine/)

------
wmblaettler
Realm has been discussed on HN quite a bit for the past 3 years:
[https://hn.algolia.com/?query=realm&sort=byPopularity&prefix...](https://hn.algolia.com/?query=realm&sort=byPopularity&prefix&page=0&dateRange=all&type=story)

It's been really nice to use so far for me, plus the Realm team puts out a lot
of helpful deeper dive blog posts:
[https://realm.io/news/](https://realm.io/news/)

------
akeruu
I have dabbled a bit with it quite a few times and was quite happy about it,
but my main issue is that AFAIK there is not "pure" Java API for Realm.

It seems there have been attempts at removing Android dependencies but it
seems they were never finished, meaning you can't have it running on desktop
apps.

This is (sadly) a showstopper for me.

~~~
bigfish24
Adam from Realm, thanks for the feedback. We do plan to offer a pure Java API,
but it hasn't been a top priority. However, with the launch of Realm Mobile
Platform we have been getting more requests for this because teams want to use
Java vs. Node on the server-side to build integrations. My best guess would be
Q3 or Q4 of this year, since we still have higher priority features to work
on.

------
cyberferret
It would be really cool to see this working in Ionic/Cordova. I've played with
Firebase and CouchDB (and also SQLite) in Ionic, so would be good to see how
Realm could work in the same context.

Sometimes I just want a simple persistent on board storage mechanism for my
mobile apps without all the problems that localStorage exhibits. I wonder if
Realm could be a lightweight alternative?

~~~
bigfish24
There is unofficial Realm Cordova support being done by the community:
[https://github.com/airamrguez/cordova-plugin-
realm](https://github.com/airamrguez/cordova-plugin-realm)

------
david-given
The documentation links on the github page are broken; see
[http://realm.io](http://realm.io) to find out what it is. (AFAICT it's
basically Firebase.)

~~~
citruspi
There are two products - Realm Mobile Database[0] and Realm Mobile
Platform[1].

The former is, as the HN post suggests, basically a database for mobile
applications which you can use in place of SQLite or as a key value store.

The latter combines the database software which runs on the mobile devices
with the Realm Object Server[2] which runs server side to provide something
like Firebase (disclaimer: I've used the Mobile Database but haven't used the
Mobile Platform (and Object Server) yet).

[0]: [https://realm.io/products/realm-mobile-
database/](https://realm.io/products/realm-mobile-database/)

[1]: [https://realm.io/products/realm-mobile-
platform/](https://realm.io/products/realm-mobile-platform/)

[2]: [https://realm.io/docs/realm-object-server](https://realm.io/docs/realm-
object-server)

~~~
lpga510
What are the differences with firebase? There must be something.

~~~
bigfish24
The biggest would be the Realm Mobile Platform is not a service, but a server-
side application, Realm Object Server, that you can run on your own alongside
the mobile client SDKs embedded in your apps. Another key difference, is that
with our Node.js SDK you can create a level of integration with any existing
backend infrastructure that would be harder with Firebase.

------
netdur
guys I think this is new, the title is misleading, this is realm for
javascript

------
laex
Shameless plug: If you're looking for a simple key-value store for Swift,
please checkout
[https://github.com/hemantasapkota/SwiftStore](https://github.com/hemantasapkota/SwiftStore)

Underneath it's LevelDB.

