
PouchDB, the JavaScript Database That Syncs - _qc3o
https://pouchdb.com/
======
PaulMest
I've been using PouchDB in a React Native app for about 6 months with SQLite
as its backing store so that we can use more than 50MB of storage on the
device. It has been working pretty well to persist data into an offline cache
and then sync to a CouchDB 2.0 database in Digital Ocean.

Getting it to work inside React Native was initially very challenging. Keeping
our shim up to date with the recent changes to PouchDB has also been
challenging. We are currently using PouchDB 5.4.5 because there was a breaking
change in 6.x and I haven't had a chance to dive into it to figure out what is
going wrong.

The PouchDB community (especially Nolan Lawson) does a great job of showing
examples, answering questions, and responding to feedback.

~~~
m_mueller
Did you also consider Couchbase Lite? I'd be interested in your decision
making.

~~~
davidbanham
I've used pouch on a few projects and also attempted to use cb lite. I found
pouch dramatically easier to get up and running with.

Cb lite was complicated since it needs a native plugin rather than being just
js. That wasn't too bad.

The thing that put the nail in it for me was all the setup on the server side
with sync gateways, multiple databases that then sync with one another, etc
etc.

~~~
m_mueller
CBL has worked OK so far for us with just plain CouchDB serverside, no sync
gateways. But I can see where you're coming from, if your stack is centered
around JS I think pouchDB makes more sense. We have potentially large DBs
(depends on usecase) where browser context isn't enough and/or provides a bad
user experience.

~~~
davidbanham
Oh so you're using CBL with a couchdb backend? I didn't even realise that was
possible. I thought it only synced with Couchbase which is where all the
complexity came in.

~~~
m_mueller
Yes it's possible. Here and there there might be a quirk but so far nothing
that set us back too much. I'm not a fan of Couchbase's Syncgateways either,
I'm not sure the problem they're solving is a very common one.

~~~
stock_toaster
sync_gateway is basically a couchdb protocol proxy, for couchbase-lite to use,
that sits in front of couchbase.

couchbase-lite apparently also works with couchdb (protocol compliant) and
cloudant.

I am currently working at a gig where the use of couchbase-lite with sync
gateway and couchbase are required for a project.

------
ojanik
We've been running PouchDB in production for ~15 months now. We chose it
because it was a greenfield project and it gave us 2 things: Easy offline
support and real-time syncing that makes it easy to create collaboration a-la
Google Docs. Because the entire thing is a web app with app cache manifest
deploying new versions is very little hassle.

In terms of architecture we have about 250 tenants with separate Couch
databases per each. We're still running Couch 1.6. We have yet to evaluate
Couch 2.0.

It's been mostly smooth ride for the most part but this being a very unusual
architecture we had to tackle few interesting problems that came along.

1\. Load times. Once you get over certain db size the initial load time from
clean slate takes ages due to PouchDB being super chatty. I'm talking about
15-30 mins to do initial sync of 20-30mb database. We had to resort to pouch-
dump to produce dump files periodically. That helped a lot. I think this issue
has been rectified with Couch 2.0 and sync protocol update.

2\. Browser limits. Once we hit the inherent capacity of some browsers (namely
Safari on iOS, 50mb) we had to get creative. Now we're running 2 CouchDB
databases for each tenant where 1 has full data and the other only contains
last 7-8 days. Pouch syncs to the latter one. We run filtered replications
between the full db and the reduced db and do periodic purging. On the client
side if a customer tries to go back more than 7 days we just use the Pouch in
online only mode where it acts as a client library to remote couch and doesn't
sync locally.

3\. Dealing with conflicts. This might matter or it might not depending on the
domain but you have to be aware of data conflicts. Because CouchDB/PouchDb is
eventually consistent multi-master setup and you will get data conflicts where
people update the same entity based on the same source revision. PouchDB has
nice hooks to let you deal with this but you have to architect for it.

4\. Custom back-end logic. Because Pouch talks directly to Couch you can't
exactly execute custom back-end logic when needed. We had to introduce a REST
back-channel to make sure our back-end runs extra logic when needed.

5\. We had some nasty one-off surprises. Last one was with an object that had
1700 or so revisions in couch and once it synced to PouchDB it would crash the
Chrome tab in a matter of seconds. Due to the way PouchDB stores revision tree
(lot's of nested arrays) Chrome would choke during JSON.parse() call and eat
up memory until crash. We resolved this one by reducing the revision history
limit that is kept.

~~~
davidbanham
That chattiness is what has driven me away from Pouch, sadly. It's a flaw in
the Couch replication protocol design that won't be fixed until the spec is
changed.

~~~
willholley
The chattiness is mostly addressed with _bulk_get in CouchDB 2.0 - Pouch will
automatically use it if the server supports it. Another option is to stick a
HTTP/2 proxy in front of your CouchDB instance - the chatter to the db is
ultimately still there but it significantly reduces the latency cost to the
PouchDB client. There are plans to add first class HTTP/2 support to Couch but
for remote client architectures just adding a proxy should be a significant
improvement. Projects like [https://github.com/cloudant-
labs/envoy](https://github.com/cloudant-labs/envoy) take this a step further
and provide an extensible proxy (e.g. you can do sub-database access control,
etc).

------
daleharvey
Surprised to see this on HN, but I am one of the maintainers of this (I expect
Nolan will end up finding this too) so happy to answer any questions about it.

~~~
grizzles
Is the project sponsored by Couchbase? If not, have you thought about
publishing a storage API that the replication protocol calls so that it could
be adapted to work with any backend db? What about P2P sync - (eg. serverless)
has anyone thought of making PouchDB do that?

~~~
fredguth
AFAIK, Couchbase is a CouchDB as a service tuned for speed.

PouchDB is a CouchDB implementation in the browser. I don't think your sync
with other db would be easy and I believe there is no such desire in the
project roadmap.

~~~
rakoo
No, Couchbase is more or less a merge of Membase and Couchdb. With Couchdb it
only shares its lineage and its creator; apart from that they are entirely
different beasts.

~~~
fineline
It also shares more than half its name. The recognisable bit. The other bit is
the generic and largely interchangeable "base" / "db". I don't know if it's
intentional, but there's no wonder people get them confused.

------
aeharding
I've been running PouchDB + CouchDB 2.0 in production for a while now
(financier.io). It's worked great, and I really recommend you to check it out.
A couple things:

1\. CouchDB 2.0 is still rough around the edges, particularly with its new
Fauxton interface (ex: completely broken when proxied into subfolder).

2\. CouchDB 2.0 brings the _bulk_get API which has improved sync by an order
of magnitude(s).

3\. I do have custom logic for logging in overriding the _session API in order
to do rate limiting. (I proxy with nginx for IP rate limiting and node.js for
failed password attempts rate limiting.) I also have custom logic for
provisioning a new CouchDB database per user and setting up permissions.

4\. I host on Digital Ocean but I use their new block storage solution so that
a growing db does not become unwieldy/expensive.

5\. My SaaS subscription system is kind of unique: When your subscription
expires you'll simply lose write access to the server (CouchDB), but you can
still pull down your data to PouchDB.

------
awjr
For one of the best uses of PouchDB that blew my mind when I read about it,
read this
[http://www.pocketjavascript.com/blog/2015/11/23/introducing-...](http://www.pocketjavascript.com/blog/2015/11/23/introducing-
pokedex-org)

~~~
awjr
The article I linked has been discussed on here before
[https://news.ycombinator.com/item?id=10619933](https://news.ycombinator.com/item?id=10619933)

------
rahilsondhi
I'm interested in PouchDB to make my JavaScript app easily sync to the server,
but I don't want to switch my server's database from Postgres to CouchDB.
Surely I'm not the only one in this situation?

~~~
daleharvey
You aren't, its mentioned in our faq -
[https://pouchdb.com/faq.html#sync_non_couchdb](https://pouchdb.com/faq.html#sync_non_couchdb).

A lot of people would like their current data to just be able to sync, but it
almost always needs changes in the way data is stored and complementary
changes to the application code

------
azr89
We've been using Pouch in a progressive web app designed to be used on the
field in remote locations, and while there was a learning curve in
understanding how the replication protocol works, and as highlighted in
another comment the way Chrome stores data for a web app - we can't be happier
with pouch/couch.

Additionally, moving out of Cloudant and into CouchDB with an openresty based
reverse proxy has made things even better, and really fun. This is one of
those stacks that feels easy and simple at the same time.
(Ref:[https://www.infoq.com/presentations/Simple-Made-
Easy](https://www.infoq.com/presentations/Simple-Made-Easy)).

~~~
karmelapple
Any guidance on moving from Cloudant to CouchDB? Are you hosting it yourself?
If so, has the amount of maintenance been more than you expected, or was it
mostly setup time and then forget about it?

~~~
azr89
Yup, hosting it ourselves. Its a peach. There are few things that it doesnt
come with out of the box - clustering, Full text search, geoindexing, chained
map reduce, auto compaction, index auto-updation. Once thats done, if anything
it was more forget about it than Cloudant, which bills on requests /
thoroughput. This can catch you out because continuous replications between
databases on the same cloudant account are also counted as requests and billed
as such. And continuous replication is very chatty. So if you have a
particularly creative multi-master setup, like a per user db -> masterdb kind
of thing going, this can eat up your thoroughput / push up your bills with no
practical benefit.

Its really openresty + couch that does it for me. The idea of writing security
/ validations / routing etc right into ngnix combines beautifully with the
CouchDB way of thinking.

~~~
kocolosk
Ah, yeah, you weren't the only one bitten by that. We actually went and
changed the Cloudant metering model recently so that you're billed on
_provisioned throughput_ rather than total request volume. You get
dramatically more predictable billing, with the tradeoff that clients need to
handle a 429 Too Many Requests response if the Cloudant instance is under-
sized. More here:

[https://www.ibm.com/blogs/bluemix/2016/09/new-cloudant-
lite-...](https://www.ibm.com/blogs/bluemix/2016/09/new-cloudant-lite-
standard-plans-are-live-in-bluemix-public/)

------
mike-cardwell
Anyone use this system in production? Care to share your experiences?

~~~
rglullis
The current project I am working on has been developing an offline-first,
messaging/calling app that started as a webapp but was decided to also have
native apps for Android/iOS. But because the web app came first, the web-
centered architecture was kept, and the javascript developers have chosen
PouchDB/CouchDB as the single data storage/sync mechanism.

I was peripherally involved in the part of the services that dealt with user
data - basically, massaging data from the address book and call history,
dealing with the backend.

This is a collection of things I _believe_ I can say after working on this for
almost two years:

\- PouchDB + CouchDB works well, as long as you are already used to model your
application in terms of CouchApps. No SQL, no E-R representation of your data,
etc. If you are not comfortable with the couchDB model and don't know your way
around views, you are not going to have a good time.

\- This is not an issue with PouchDB, but it if you are planning on having
native apps, I'd strongly advise against. All of our native app developers
struggled with the change in mindset and the not so mature state of couchbase
for Android/iOS. And because they can use SQLite, forcing them to adopt
couchbase was not but pain in our team.

\- You need to figure out security and authentication: if you take the direct
approach of using one couchdb per user and just replicate that, you need to
figure out on your own how to secure access. CouchDB only supports OAuth 1 out
of the box. We are using oAuth 2 for our services, and I basically had to
implement a proxy server that checked oauth tokens before passing requests to
our couchdb server.

\- CouchDB that does not allow cross-database views, so if you take the "one-
database-per-user" approach and you need to check anything that spans more
than one database, you are on your own to create more
databases/replicate/index/aggregate the data.

\- The solution for sync helps, and continuous replication makes for a good
demo, but it is not magic. Imagine if you have already accumulated some good
amount of data in your database. The moment you start your application on a
different browser, PouchDB will start desperately to sync everything you have.
You need to know how filtered replication works.

\- Very resource intensive, more so if you keep continuous replication.
Constant usage of CPU and network I/O can make your app feel sluggish.

~~~
nolanl
(PouchDB contributor here.) Just a quick note that slow replication is one of
the things that is largely fixed in CouchDB 2.0 thanks to the new _bulk_get
API
([https://issues.apache.org/jira/browse/COUCHDB-2310](https://issues.apache.org/jira/browse/COUCHDB-2310))
and is set to get even better in CouchDB 3.0 once it has HTTP 2.

As for your app feeling sluggish, my hunch is that you're seeing this in a
Chromium or Gecko browser (e.g. Android WebView) in which cases IndexedDB does
quite a lot of heavy operations on the UI thread
([http://nolanlawson.com/2015/09/29/indexeddb-websql-
localstor...](http://nolanlawson.com/2015/09/29/indexeddb-websql-localstorage-
what-blocks-the-dom/)), which can be mitigated by moving it to a Web Worker
(as Pokedex.org does) or a Service Worker (as HospitalRun.io does).

------
krrishd
My bad if the answer is obvious but (besides backward compatibility/shims +
abstraction), what's the benefit of using PouchDB as opposed to vanilla
localStorage functionality?

~~~
pokstad
It can sync data between CouchDB servers (e.g. IBM Cloudant) and the browser.

~~~
erikb
Honestly I didn't know that this is a feature people want (no criticism
intended). So, you have a database object in your browser and that takes care
of getting the data to the server himself? Otherwise every website syncs data
between its browser instances and its database, right? That's how we get
state.

~~~
candeira
The sync happens in an unmanaged fashion, without the user or the application
programmer having to care about the state of the connection.

Your PouchDB application works locally on your device, whether the connection
is up and down, and the data is synched with the remote database whenever
there is connection.

The alternatives to this PouchDB to CouchDB synching mechanism would have to
be either:

\- the user checks whether the connection is up, and manually manages the
sync, or

\- the application programmer saves the user the trouble by adding code that
checks whether the connection is up, and automatically manages the sync

------
cyberferret
Been experimenting with PouchDB for a while and really liking the simplicity
of the project. Looking to implement it in a mobile hybrid app to allow users
to take data 'offline' then sync up later.

The only downside I've found so far is that the PouchDB Inspector on my Chrome
browser tends to go rogue from time to time and suck up > 50% of the CPU time
and has to be shutdown manually.

------
imtringued
Could I in theory host my own pouchdb instance to keep using applications that
are no longer supported by their vendors? Imagine an IoT vendor uses pouchdb
shuts down. My devices don't work anymore but I can host my own pouchdb
instance. A javascript based database is a bit heavy but it's an acceptable
tradeoff to me.

~~~
nolanl
In principle yes any CouchDB/PouchDB database can sync with one another
because they all share the same protocol, but OTOH I've seen very few examples
of apps that offer users a simple "export to your own CouchDB/PouchDB"
feature.

------
espeed
Datomic-ClojureScript in the browser backed by PouchDB would be killer.

Datomic already has a CouchDB-compatible datastore via its support for
Couchbase. That would mean Datomic-ClojureScript on PouchDB could run in the
browser and sync with Datomic-Clojure backed by CouchDB/Couchbase on the the
server -- that would be killer and give Datomic massive reach.

Has the Datomic team considered PouchDB as a possible datastore for Datomic-
ClojureScript in the browser?

NB: Just posted the question to the Datomic discussion group:
[https://groups.google.com/d/msg/datomic/uqBQE4QlnzI/VuZ14pqO...](https://groups.google.com/d/msg/datomic/uqBQE4QlnzI/VuZ14pqOBQAJ)

------
ioquatix
Since when did arguments become functions?

    
    
        db.replicate.to('http://example.com/mydb');
    

IMHO, should be

    
    
        db.replicate(to: 'http://example.com/mydb');

~~~
Spivak
I'm personally really a fan of this syntax mostly because I like how it reads
when you chain them together.

    
    
        db.replicate
          .from('http://example.com/mydb1')
          .to('http://example.com/mydb2')
          .on('complete', function() {
              
          })
          .on('error', function() {
             
          })
          .then(function() {
            
          });

~~~
zepolen
What happens when you do:

    
    
      db.replicate
          .from('http://example.com/mydb1')
    

or

    
    
      db.replicate
          .from('http://example.com/mydb1')
          .from('http://example.com/mydb2')
          .to('http://example.com/mydb3')
    

or

    
    
      db.replicate
          .from('http://example.com/mydb1')
          .to('http://example.com/mydb2')
          .to('http://example.com/mydb3')
    

? serious question

~~~
azr89
1: replicates from mydb1 into the pouchdb object represented by 'db'

2 & 3: I'm pretty sure chaining replications doesn't work in that way,
although thats a pretty interesting thought.

If you have to chain replications and achieve the structure in 2, you'd have
to define the pouch objects as:

db = new PouchDB('localDB');

mydb1 = new
PouchDB('[http://example.com/mydb1');](http://example.com/mydb1'\);)

etc ..

and then do:

mydb2.replicate.to('mydb3',{live:true});

mydb2.replicate.to('mydb1',{live:true});

mydb1.replicate.to('db',{live:true});

the 'live' flag, as you'd imagine, makes the replication live/continuous, as
opposed to one-time.

------
crooked-v
PouchDB's replication capability is interesting, but is there a way to make it
lazy load to the local DB instead of doing everything up front? I hesitate to
use it for a web project with 10+ MB of docs where it would otherwise be
ideal.

~~~
WorldMaker
You can provide a server-side filter function to replication and progressively
filter partial replications until eventually everything gets replicated. At
that point it becomes a question of architecture of your documents: how much
is needed to replicate before a user may be productive?

You can also explore pouchdb-replication-stream to build bundles that PouchDB
can bootstrap from a little bit faster than a chatty replication.

That said, I've found initial replications of large databases (one I've worked
with this week is a 25+ MB CouchDB database full of photos) is quick enough
(and mostly bandwidth constrained) that I haven't had much in the way of
concern over it.

------
dmihal
Cool library! I'm a huge fan of Meteor.js, which provides a similar isomorphic
database. Cool to see this implemented in such a lightweight package!

------
daurnimator
I really love the project, but I never used it in production due to the large
file size :(

~~~
fiatjaf
Your solution has arrived:
[https://pouchdb.com/custom.html](https://pouchdb.com/custom.html)

