
Dropbox: the new file system of the web - ericingram
http://collaborable.com/blog/dropbox-the-new-file-system-of-the-web
======
peteforde
I understand and in some ways share your enthusiasm. I think Dropbox could be
the file I/O layer of the social web... if only they would publish a proper
events API.

As it stands, the only way to receive updates is to poll for changes. This is
a kludge at best (and takes away from the magic in a significant way in
practice) and completely unrealistic at scale. That is, if I have a million
users and I want to update every 15 seconds, Dropbox is going to (rightfully)
shut my IP down.

If developers could define WebHooks to receive events, that'd be a great
start. I'd happily pay for that.

If developers could subscribe to a WebSockets or event XMPP notification
stream of updates, that'd be freaking amazing. I'd stand in line to pay for
that. And that's when you'd see an explosion of applications (web, mobile and
desktop) implementing Dropbox as their filesystem.

We've been asking nicely for years. If there's anyone here with pull or
influence at Dropbox, please give us devs some real-time love. Otherwise,
please add your voice to the choir:

[http://forums.dropbox.com/topic.php?id=20226&replies=52#...](http://forums.dropbox.com/topic.php?id=20226&replies=52#post-127706)

~~~
brlewis
I too am hoping for an improvement that will scale to a million users. But
here's what I do that should scale to tens of thousands decently:

I run dropboxd on my web server. Instead of using the API, users share a
folder with my app. Dropboxd gets the stream of changes, and my app polls the
local filesystem for changes instead of polling across the net.

~~~
peteforde
How do you handle quotas?

~~~
brlewis
I tell users to copy photos in, and they're deleted after upload.

------
mvanveen
I really like Dropbox. In fact, it's fair to say I love Dropbox. It's made so
many use cases so much simpler. However, I am f __ __*g sick of this
perception that Dropbox is the silver bullet for every storage solution. This
sort of sensationalism is just stupid.

Dropbox is a cloud-hosted, distributed system and just like any other large,
distributed system, it's subject to some very real constraints in terms of
latency and reliability.

In other words, it's not fair to claim that Dropbox is some magic silver
bullet that will solve every scaling problem you're subject to, act like a
rock solid filesystem which behaves identically to the one you have locally,
or solve your love life problems.

Maintaining this perception is just going to create frustration for people who
expect it to act this way, or worse, create a living hell for the developers
maintaining the glue code between your system and the Dropbox API.

You try figuring out what the fuck is going on with your system when the
"spooky" or "weird" version regressions start occurring. The arms will raise
in a giant, collective shrug. Watch in horror as your http requests timeout
because the roundtrip time between your box and Dropbox is that bad. The
sodden faces of your product customers and the million "(so-and-so's
conflicted copy of " files laying around your filesystem which JUST. KEEP.
COMING. BACK. will be enough to make you wish you never drank the kool aid.

Now, I'm not saying that the Dropbox REST API isn't the best thing since
sliced bread. All I'm saying is it has it's purposes and there are use cases
and times when it simply isn't an appropriate choice.

In short, please respect the fact that distributed storage is a hard problem
with hard constraints, stop proselytizing Dropbox as the 2nd coming, and relax
and enjoy your backed up media and documents on your iPad already. Seesh.

~~~
pak
I'm with you; I've already experienced various spooky things with desktop
applications on top of Dropbox (like the excellent Papers2 by mekentosj)
because of file conflicts and latency. And that's not Papers2's fault, since
it would have to be redesigned to be tolerant of such things. But it goes a
long way toward showing that Dropbox is not a syncing FS that can put any
local application data in the cloud. In particular, Dropbox is bad for typical
database use patterns (simultaneous writes, transactions) and the versioning
and locking are not tunable.

------
Someone
AFAIK, Dropbox is a 'filesystem' that does not support partial file
overwrites, does not do locking in a meaningful way, etc.

Because of that, anything built along the lines outlined in this post will be
brittle. For example, if one updates that 'products.csv', one must be
absolutely sure that one applies those updates to the last version saved. A
solid tool would have to detect races and offer merging of the files. I would
rather use some SQL in the cloud or, lacking that, some SCM. Those, at least,
come with tools to manually/automatically handle such race conditions.

~~~
pavpanchekha
To be fair, opportunistic locking is implemented in the API (you can pass a
revision ID and the call fails if this is not the current version, or
something like that).

~~~
talaketu
That would be _optimistic_ locking.

------
JoshTriplett
This seriously disturbs me. Dropbox has several fundamental flaws that make it
entirely unsuitable for widespread adoption as anything other than a personal
file store for people who don't care about privacy. A "filesystem of the web"
would need security, encryption, fine-grained sharing controls, reliability,
and numerous other features. Meanwhile, the whole sync mechanism of Dropbox
doesn't really matter for a "filesystem of the web", because changes can just
occur directly in the remote filesystem rather than locally with syncing.

Now, if someone called S3 "the new filesystem of the web", that would make
more sense. Numerous services build on S3, including Dropbox itself, and it
provides enough tools and API to allow those services to include all the
critical features mentioned above. Add a few features for users to hook up
their own S3 data to services rather than the S3 accounts of those services,
and you'd have something even more interesting. (The same thing applies to any
equivalent service to S3, and many such services exist.)

~~~
extension
_Meanwhile, the whole sync mechanism of Dropbox doesn't really matter for a
"filesystem of the web", because changes can just occur directly in the remote
filesystem rather than locally with syncing_

Excel can't make changes directly to the remote filesystem. That's the novelty
of this scheme. It isn't yet another cloud database, it's a way to share local
files with cloud apps.

I have mixed feelings about this.

It could serve to bring data out of the silos, and back under user control.
That would be an epic win.

But it's not federated. And I anticipate problems with access control, which
needs to be per app, per file, and separate read/write.

Aside: I recently had a nasty scare when a shitty iOS password manager
_deleted_ my Keepass database from my Dropbox. If hadn't found a copy in the
Dropbox cache on my laptop, I would have been in a world of shit.

------
bprater
Dropbox has now officially replaced "My Documents" on all my computers,
Windows and Linux. I recently reinstalled Windows 7 on my gaming PC. When it
finished up, I installed DropBox, logged in to my account and went to watch a
movie. A couple hours later, I was synced up. Too simple.

So, yes, I can't wait for more web apps to allow me to save my data to my
Dropbox!

~~~
ericingram
I'm building one

~~~
keithpeter
OK, at work we use Sharepoint to collaboratively edit and push out data to
colleagues as Word and Excel files.

Will your app let me do the same kind of thing at home with outside work
friends and the voluntary work I do?

~~~
ericingram
I think Dropbox would already enable the collaborative edit and push thing you
described with outside work friends, depending on how you define
"collaboratively edit".

On top of that, it should be easy to build a web app to accomplish the same
thing, plus some extra elegant features.

The question is, would you pay for it? ;)

~~~
kenjackson
Can you check files out with DropBox, so you don't have two people working on
a file (since presumably they don't have merge working for Word/Excel docs).

~~~
nahname
I seriously hate your use case. The locked file is the most troublesome thing
that seems to negatively impact everyone using these systems so one person
can, once in a while, ensure no conflicts happen. A better solution would be
smart merging. Just look at source control to see how half-assed locking is as
a solution to this problem.

/end-rant

~~~
rhizome
I think this sentiment is reason enough to always leverage source-code control
for something like this. They've been working on the problem for an ultra-long
time in computer terms, so adapting one's use cases to the capabilities of
existing foundational components is probably the quickest way to long-term
happiness in the revisable storage world.

------
trotsky
Drag and Drop file uploads/downloads in HTML5:
[http://tutorialzine.com/2011/09/html5-file-upload-jquery-
php...](http://tutorialzine.com/2011/09/html5-file-upload-jquery-php/)

WebDAV mounts as a network share out of the box in Win/OSX/Lin

I love dropbox but I could see it being much worse than usual for a fair % of
users

~~~
freshhawk
I wonder if anyone has the ability to understand and use dropbox this way but
isn't able to mount network shares. I feel like anyone who would use this
would see it as worse than usual. Dropbox and it's clones are great, and it's
good to see this convenience available to a more mainstream audience.

But as someone who has used sshfs for years to get seamless access to all my
boxes from wherever I am I'm certainly not getting excited about some kind of
crippled filesystem-over-http fad. It took mobile to remind people that the
internet is good for more than just the web but I don't see that trend
stopping (even if we do have to start calling software "native apps")

The web's great, but this everything-in-the-browser silliness means people are
using crappier tools than they used to have. I find it annoying but
practically it's just not a stable state of affairs because it's so easy to
change.

~~~
ericingram
This seems to conflict: "The web's great, but this everything-in-the-browser
silliness means people are using crappier tools than they used to have."

... with your sentiment on the original post; that it was pointing out how
Dropbox integration lets people get out of the browser and use the tools they
might be more comfortable with...

Are you for, or against APIs to connect web and desktop and mobile?

~~~
freshhawk
That was unclear. The reason I grouped the dropbox API in there is that it is
an HTTP REST API. Good for a lot of things, but to layer a filesystem on top
of it is complex and doesn't fit well.

It's a natural extension but it's reinventing the wheel poorly. That happens
all the time as things progress. Wheels are reinvented on the new platforms.
Normally the new method either pulls ahead quickly or is replaced by an update
of the old one. I don't think a good enough implementation of a filesystem
over HTTP is easy, and it will at least have to be bloated. A good web aware
wrapper around existing filesystem sharing stuff would be much better, sooner
of later someone will do that or something even better because they're so
clever.

tldr; I like the idea of API's and having internet aware applications for
things a browser isn't suited for, and non-http protocols for the things HTTP
isn't well suited for. This API strikes me as a poor direction for major
development on top of, but makes sense since dropbox is already there and
useful.

------
collint
This is not how the Dropbox API works at this moment.

There is no mechanism to push updates to your application. Only the desktop
client gets async updates. It will be a glorious day when such an API exists.

But don't hold your breath, it's been a topic on their forums for years.

~~~
ericingram
Push events to the web app isn't important to me. Syncronization with a web
app every few minutes would work well, and you could login to push a button if
you needed the changes pulled immediately.

~~~
chubs
That's the way i implemented it in an iphone app i recently made: a 'sync now'
button to pull anything new and push any updates to the server. Works quite
well - I recently deleted my app (iphone died) and reinstalled it, one 'sync
now' later and all my data was back.

I'm a big fan of using dropbox like this, and am hoping to explore this
approach in a lot of productivity apps i plan to make in the future. I like
the idea that the user can 'see' their data in a readable format
(json,csv,yaml - i still haven't decided which i prefer!) in their
dropbox\apps\MyApp folder.

It's like icloud but you feel in control, rather than 'my data is out there
somewhere but i can't see it'.

~~~
brlewis
My users would not be this excited if a "sync now" button on the web needed to
be pushed. The point of Dropbox integration is to upload photos without
leaving Lightroom/Aperture/etc. and not having to install a plugin:
[http://pgunupudi.blogspot.com/2009/09/dropbox-support-for-
ou...](http://pgunupudi.blogspot.com/2009/09/dropbox-support-for-ourdoings-
photo.html)

------
daniel_levine
I think this is right on. Dropbox is a virtual file system and not just for
the web also for offline. And it works on every platform, I'd much prefer to
have things like my contact book and email on Dropbox than where they are
today.

~~~
ericingram
I'm guessing the fact that it 'works on every platform' has a lot to due with
it's past and future success.

It's a jump back to where we came from in a sense. The desktop file system
works well for a lot of business people, which is partly why Dropbox is
growing so fast.

------
mattberg
giving the user that much control, using his example of a spreadsheet on his
computer, makes me cringe thinking about how much the user could mess things
up with the data.

~~~
ericingram
By contrast, it makes me excited thinking about the possible solutions to the
problems you just described. What could be done? Maybe there is a revision
system in the web app that shows the changes which must be confirmed before
applying? This also would give you web app the ability to easily version
control data.

Awesome

~~~
molecule
"How do I undelete files or recover old versions of files?"

<http://www.dropbox.com/help/11>

------
gordonc
We at <http://views.fm> completely agree

~~~
trotsky
Looks like your TOS grants you a perpetual all rights license to all content
in the user's dropbox without any of the usual qualifications about only as
far as it's needed to provide the service?

views.fm: _By providing any content to this web site: (a) you agree to grant
the site editor a worldwide, royalty-free, perpetual, non-exclusive right and
license (including any moral rights or other necessary rights.) to use,
display, reproduce, modify, adapt, publish, distribute, perform, promote,
archive, translate, and to create derivative works and compilations, in whole
or in part. Such license will apply with respect to any form, media,
technology already known at the time of provision or developed subsequently;_

Dropbox: _By using our Services you provide us with information, files, and
folders that you submit to Dropbox (together, “your stuff”). You retain full
ownership to your stuff. We don’t claim any ownership to any of it. These
Terms do not grant us any rights to your stuff or intellectual property except
for the limited rights that are needed to run the Services, as explained
below.

We may need your permission to do things you ask us to do with your stuff, for
example, hosting your files, or sharing them at your direction. This includes
product features visible to you, for example, image thumbnails or document
previews. It also includes design choices we make to technically administer
our Services, for example, how we redundantly backup data to keep it safe. You
give us the permissions we need to do those things solely to provide the
Services. This permission also extends to trusted third parties we work with
to provide the Services, for example Amazon, which provides our storage space
(again, only to provide the Services)._

I assume that's a cut and paste job gone horribly wrong, but that is pretty
gross.

~~~
rhizome
I would imagine that a storage company would have something like that if they
planned public/social features, such as an imgur for pix you're already
storing with them.

------
drivebyacct2
In terms of these sorts of developer oriented features, I'm much more excited
to see Camlistore take off. Portable, better sync options, open source,
support for any backend, etc. Just something to look at if this topic is
interesting.

In fact, their semi-slogan is "your home directory for the web". Very privacy
focused (an encryption layer for backends is in progress based on their issues
log), since that seems to get buzz these days.

<http://camlistore.org/>

~~~
ericingram
Camlistore looks really interesting

------
wavephorm
Dropbox is merely providing the blueprint for how a cloud a file system should
work. I hope they're using their funding to develop something completely above
and beyond what they're currently offering, because the network storage space
is going to become awfully crowded in a few years.

~~~
ericingram
"...merely providing the blueprint for how a cloud a file system should work."

That sounds like kind of a big deal to me, but hey call me an optimist.

