

Datastore API (beta) - gluegadget
https://www.dropbox.com/developers/datastore

======
peterkelly
The biggest problem right now with online file storage services (sorry, I
meant "The Cloud") is that while they all do basically the same stuff, there's
no standard protocol for accessing them. It's a bit like if every website used
a custom protocol instead of HTTP, and web browser vendors had to explicitly
collaborate with each web site (or use the website's SDK) in order to be able
to retrieve content.

Imagine this: All the major players in the space (Dropbox, Box, Google Drive,
Skydrive, Sugarsync etc.) as well as app developers got together and came up
with a standard way of doing this that makes it possible to use a single
protocol & API for all online storage providers. I see this being a good thing
for everyone - users can choose whatever service they want, app developers can
easily provide their customers with a ton of options, and storage providers
don't have to explicitly produce their own SDK or court developers to do
integration with their service. We did have WebDAV, but that doesn't seem to
be widely adopted by any of the major players.

I've been facing this problem myself with my own app; it's a lot of work to
support all the major services and this is really just plumbing stuff that
everyone would rather be taken care of so they can focus on the more important
stuff. I had a meeting with Box recently where I proposed the idea of an open
source framework for iOS which would provide a common native API that
abstracts over all the different REST APIs used by different storage
providers. The people I spoke to seemed quite open to the idea and agreed that
it would be useful. I don't think a library like this would necessary be
considered a bad thing by any of the major players, as it evens out the
playing field and makes it easy to support different services.

If you're interested in taking part and/or using such an API in your iOS app,
I'd love to hear your thoughts on the matter. I have some code already (which
I haven't released yet) and am looking to clean up the API a bit, but I think
it would be useful to a lot of developers.

~~~
icebraining
Big players aren't interested in anything that reduces lock-in. Seamless
integration is one of the keypoints they have to get you to use their
services.

Obligatory presentation by Bruce Schneier on the online feudalism:
[http://www.infoq.com/presentations/cloud-
security](http://www.infoq.com/presentations/cloud-security)

~~~
peterkelly
I found Box to be the opposite of this. While I obviously can't speak on their
behalf, and I was discussing the issue with their developer relations people
(not senior executives), they said they didn't see a problem with this idea.

------
munificent
I'd like to coin a modern update of Zawinski's Law[1]:

    
    
        Every software company attempts to expand until it
        offers a cloud-based platform API. Those companies
        which cannot do so are replaced by ones which do.
    

[1]:
[http://en.wikipedia.org/wiki/Jamie_Zawinski#Zawinski.27s_law...](http://en.wikipedia.org/wiki/Jamie_Zawinski#Zawinski.27s_law_of_software_envelopment)

------
mmahemoff
Something I don't get about Dropbox API: Why can't I just throw URLs at it?

If a user wants to save <thing at URL>, I have to pull it to my server and
push it into Dropbox. Dropbox could just have grabbed it directly and in many
cases, won't even have to thanks, to system-wide caching.

(This is based on the last time I looked at the API; maybe this is supported
now, but I don't think so.)

~~~
timdorr
You're describing the Saver Drop-In:
[https://www.dropbox.com/developers/dropins/saver](https://www.dropbox.com/developers/dropins/saver)

~~~
natch
Can the Saver Drop-In be used with arbitrary third-party URLs? To me this is
what was being described, so if not, then no, it's not describing the Saver
Drop-In.

~~~
rpearl
Yes, you can provide arbitrary URLs to the Saver and our (Dropbox's) servers
will download them, assuming they are publicly available.

(Note: I am an engineer at Dropbox and worked on this product.)

~~~
natch
Thanks. Off topic but slightly related if I'm going to rely on your service as
a developer: as an engineer at Dropbox, can you tell me why I am seeing random
photos that are not mine in my dropbox Photos folder? And does this mean that
other people get to see my random photos? Edit: only on iOS. And maybe it's
not a "folder" per se, it's the second tab from the left at the bottom of the
screen.

~~~
gkurtz1
That photos tab shows you photos from anywhere in your Dropbox account, so
these photos may be from shared folders you're a member of. (Likewise, other
people can only see your photos if you shared them with them.) Another thing
that may happen is someone "camera uploaded" their photos to your account
(e.g., by plugging into your computer and choosing to import photos when
Dropbox asked.) If you're still not sure where they're coming from, you should
open a ticket:
[https://www.dropbox.com/support/s/92/3201098/c/2](https://www.dropbox.com/support/s/92/3201098/c/2)

~~~
natch
Thanks, very informative answer. I think they must be from a certain shared
folder I have access to, which would explain it. Quite a relief, actually.

------
oellegaard
Such an awesome idea. However, in light of the recent events the first thing
to come up in my mind when I see "datastore" and a US corporation is "PRISM
integration". I'd like to see proper encryption of Dropbox as the next
feature, so that we - innocent users- are not uploading our data to the US
government.

~~~
redblacktree
I combine dropbox with TrueCrypt for sensitive stuff. For me, that's just
digital copies of important documents.

~~~
aasarava
So are you just syncing entire truecrypt .tc files to Dropbox? Does that work
incrementally or does the entire .tc file have to get resent every time you
modify a file within?

~~~
Afforess
The entire file is resent.

~~~
sigil
No, only changed blocks are resent. The scattering effect of Truecrypt
encryption plus the relatively large blocksize does amplify the number of
changed blocks somewhat.

Previous investigation and results:

[http://news.ycombinator.com/item?id=2546626](http://news.ycombinator.com/item?id=2546626)

------
jonaldomo
After quickly clicking through the api documentation I don't understand why I
would use this over a hosted NoSql database like mongohq. I still like having
additional options though.

~~~
forgotAgain
Off-line access.

------
jimbokun
"List – A special value that can contain other values, though not other
lists."

[https://www.dropbox.com/developers/datastore/tutorial/ios#re...](https://www.dropbox.com/developers/datastore/tutorial/ios#records)

Seems like a very significant limitation. No mechanism for storing
hierarchical data structures. For example, this is certainly not capable of
storing arbitrary JSON data as a Record.

~~~
est
There's also limit on Postgres hstore and arrays. No nested sturcture.

------
makeramen
As someone who has been playing around with Parse and Firebase SDKs (on
Android), this seems to be the first one that offers full 2-way sync. I
haven't actually used it yet but it seems that way due to the presence of
conflict resolution [1]. I'm personally really excited about that for mobile
side-projects and MVPs.

Parse has an offline cache, that only caches query results and not actual
objects. So any local saves/updates are not reflected until you get network
access again, unless you manually implement your own cache.

Firebase is better at handling realtime changes, but is lacking in any
realtime capabilities at the moment. Also Firebase's mobile authentication SDK
still isn't released yet so no way to handle users yet either.

[1]:
[https://www.dropbox.com/developers/datastore/tutorial/androi...](https://www.dropbox.com/developers/datastore/tutorial/android#conflicts)

~~~
pavpanchekha
Yeah, the Datastores API does two-way synchronization. Also, offline caches
are done for you and offline changes are propagated (with conflict resolution)
when the internet comes back up.

------
obilgic
What app.net wanted to accomplish.

~~~
WalterGR
At first glance it also looks like WinFS for the Cloud.

------
jimsilverman
this is basically what iCloud was supposed to be.

------
sigil
So now app developers have a choice: they can use iCloud or Dropbox to manage
distributed app state. Any iOS app developers (I'm not one) care to compare &
contrast the two options?

It looks like Dropbox may be using Operational Transformation [0] to merge
changes to lists:

 _Lists are special field values. They contain an ordered list of other
values, though not other lists. Lists can be manipulated via four list
operations: put (i.e. replace), move, insert, and delete. These list
operations allow Dropbox to handle merging changes to the structure of the
list automatically._ [1]

iCloud, on the other hand, uses the coarser, state-centric approach of whole
document versioning. [2] (Edit: okay, key-value property versioning is also
available.)

But iCloud lets the app control conflict resolution, while the Dropbox
Datastore API only lets you choose from a predefined set of conflict
resolution strategies. [3]

Which approach provides greater value to the app developer? Has Dropbox
realized here that developers are just not comfortable implementing a custom
merge strategy for entire documents, and not having to worry about this is
worth some loss of control?

[0]
[http://en.wikipedia.org/wiki/Operational_transformation](http://en.wikipedia.org/wiki/Operational_transformation)

[1]
[https://www.dropbox.com/developers/datastore/tutorial/ios#re...](https://www.dropbox.com/developers/datastore/tutorial/ios#records)

[2]
[http://developer.apple.com/library/ios/#documentation/DataMa...](http://developer.apple.com/library/ios/#documentation/DataManagement/Conceptual/DocumentBasedAppPGiOS/ResolveVersionConflicts/ResolveVersionConflicts.html)

[3]
[https://www.dropbox.com/developers/datastore/tutorial/ios#co...](https://www.dropbox.com/developers/datastore/tutorial/ios#conflicts)

~~~
hboon
Not answering your question, but developers can also use
[http://parse.com](http://parse.com) or similar.

------
alexobenauer
For the iOS library specifically, I wonder if it would be possible to build a
library that would (nearly) automatically map a core data store in to this
Datastore API. If so, including such functionality would make the library a
no-brainer for many.

~~~
ZitchDog
I think with NSIncrementalStore it would be totally doable, and a great
alternative to Core Data Sync.

------
fireboi
are there any kind souls that could help please explain the difference between
dropbox datastore, firebase and parse? thank you.

~~~
sankethkatta
In comparison to firebase, this solves the offline sync problem, if your user
is offline, you can still save to dropbox datastore locally, when you get a
connection, it will automatically sync and take care of conflict resolution.

~~~
mikelehen
Firebase dev here. :-) Firebase does handle offline syncing / conflict
resolution.

If you have a Firebase app open and you lose network connectivity, the app
still continues to work fine and any modifications you make will be synced
back to Firebase when you regain connectivity.

What we don't (quite yet) do is handle the case where you make changes, kill
the app (without regaining network connectivity), restart the app, and then
get network connectivity, but that's coming soon.

As for other differences, Dropbox Datastore seems to be tightly tied to
Dropbox, so the end user for your app must have a Dropbox account (and if they
want to collaborate with other users, those users must also have Dropbox
accounts, etc.). So it's really for building apps on top of Dropbox.

Firebase and Parse are just generic backends, with no ties to other services,
so your end users don't need anything to use your app.

As for differentiating Parse and Firebase, Firebase deals with data in real-
time, pushing updates to apps as soon as data changes. Parse is a traditional
request/response model, where your app has to explicitly "refresh" to get new
data.

~~~
sankethkatta
Thanks mike, I didn't realize firebase did offline syncing now!

------
joeblau
Does anyone know of an OSS alternative to this similar to
[http://helios.io/](http://helios.io/) that supports Android/iOS?

~~~
jchrisa
Couchbase Lite is an embedded JSON database with sync capability. It's Apache
licensed and available for iOS and Android:

[https://github.com/couchbase/couchbase-lite-
ios](https://github.com/couchbase/couchbase-lite-ios)

[https://github.com/couchbase/couchbase-lite-
android](https://github.com/couchbase/couchbase-lite-android)

~~~
joeblau
Thanks for the link to this library. It seems like I would still need to
implement a lot of features like login, but it's a great start.

~~~
jchrisa
We ship with support for login with Mozilla Persona. Hope to have baked-in
support for Facebook Connect soon. Of course you can use it with any auth you
want, but you'll end up writing some code...

------
BadassFractal
What is the long term vision here? Is the idea that consumers will have little
apps here and there that will procedurally interact with the owner's data on
Dropbox? Or, is Dropbox making a product along the lines of S3 so that
backends can consider using it too? I guess given how expensive space is at
this point, that's not really an option.

~~~
chacham15
dhouston has said more than once that he wants dropbox to be the filesystem of
the internet.

------
frankcaron
I wonder how Apple will react to @dropbox's new Datastore sync in the long
run. Hopefully it won't pull the rug out a la IAP, UDID, OF, ...

~~~
jemeshsu
I would rather they react positively by opening up their iCloud API for web
application for third parties. Apple already has such in place as can be seen
from their cloud apps at icloud.com.

~~~
duaneb
Apple is a hardware company.

------
joejohnson
So Dropbox is becoming a more-direct competitor with iCloud. Meh.

I'd rather they work on an end-to-end encryption system with a focus on ease
of use.

~~~
Afforess
Hacker News is a tiny audience. I don't see any business sense in focusing on
features that 0.1% of users want.

~~~
albertyw
Not to sound self-righteous, but the HN audience (or more plausibly, some
subset within) has a high influence on what the rest of the world sees and
wants.

Any Dropbox employees read HN?

------
johnrob
Grammar check: "These days, your app need to store and sync more than just
files."

------
modarts
Why would I use this over something like Parse?

~~~
joeblau
I haven't looked into it but it may be cheaper. That being said, both Parse
and Azure support more platforms than this offering. I think that even
Firebase has an service in this space as well.

