
Dropbox: Going Deeper with Project Infinite - samber
https://blogs.dropbox.com/tech/2016/05/going-deeper-with-project-infinite/
======
nimish
So instead of using a FS shim to userspace on security grounds they decided to
distribute a custom kext that's closed source -- ie a great way for normal
bugs to turn into dangerous ones?

I'm not sure I understand the logic here

~~~
danieldk
Exactly, we should be moving in the opposite direction: make as much software
as possible user-space and sandboxed. I assume a party as large as Dropbox can
request Apple to extend the APIs where necessary?

~~~
coldtea
> _I assume a party as large as Dropbox can request Apple to extend the APIs
> where necessary?_

As large? Dropbox is like a fly to Apple. Not to mention competitors. And they
haven't budged for much larger parties.

So, unless it's something that Apple intends to do anyway, Dropbox's request
will don't have much success.

~~~
toomuchtodo
> So, unless it's something that Apple intends to do anyway, Dropbox's request
> will don't have much success.

You'd think Apple would be happy to add extensions they themselves could use
in the future to replace Dropbox with their own iCloud extension.

~~~
coldtea
On the contrary. Why would they make them open in that case?

They could just add them internally and use them for an "upgraded
Finder/iCloud" and not expose them for Dropbox to use.

Implementing them for a competitor so that they "can use them themselves in
the future" doesn't make sense as a strategy -- except if they are too
benevolent.

------
urza
Thanks to dropbox inifit I discovered Infinit.sh -
[http://infinit.sh/](http://infinit.sh/)

which I got really excited about. I wonder what is their experience with using
FUSE and DOKANY..

~~~
skrowl
I have a similar success story. Thanks to Dropbox spamming me with "OMG
UPGRADE!" every time I clicked anything, I found out about SyncThing
(specifically SyncTrayzor for Windows).

Thanks, Dropbox!

~~~
aw3c2
Also checkout Seafile if you want something more Dropboxy.

~~~
bravura
But that seems to only use Seafile's servers, not S3.

~~~
GordonS
you can also install seafile on your own VM and use your own storage

------
ja30278
So..the cost of an extra kernel/userspace switch was too much for a file
request that is going to be serviced by a server on the other side of the
internet?..really?

~~~
ori_b
Pretty much, it's not going to matter:
[http://www.csl.sri.com/users/gehani/papers/SAC-2010.FUSE.pdf](http://www.csl.sri.com/users/gehani/papers/SAC-2010.FUSE.pdf).

Putting ext3 into userspace led to slightly less than 10% performance
degradation on postmark. Fuse really isn't an issue in most cases, and I can't
imagine that this makes a huge difference on most workloads.

~~~
olavgg
There may be other reasons that we are not aware of, anyway learning how to
build your own vfs most likely have huge advantages in the long run.

------
rcarmo
Not really enthusiastic about file syncing moving into kernel space. I get
that this is necessary for on-demand fetching of files, but I worry about
stability and failure modes - I can see apps stalling and/or getting confused
when file open calls fail because the network drops, etc.

Reminds me of the Coda/AFS2 days, really (those weren't as much fun as you'd
think, back when we opened files over X.25 links...)

Bottom line: I hope it's configurable and that I can switch it off (preferably
by default).

------
seanalltogether
I'm a bit scared to figure out what this means for my users scanning their
hard drives with Space Gremlin. I go through standard file apis to scan all
the folders on the users system. Will it end up scanning the entire users
online filesystem and not just the actual local files? Will getting file meta
data on all those online files end up causing a fetch to DB and grind the
whole scan to a halt?

I've been resisting building a custom HFS+ reader for years but apples
expanded use of hard links plus whatever DB is about to unleash might require
a new approach.

~~~
astrange
Do you use getattrlistbulk() or do you only use cross platform APIs?

------
danieldk
As a paying Pro customer, I am a bit worried how intent they seem to be on
pushing the business/enterprise products.

First of all, Dropbox the web application is spammed with Dropbox Business
advertising. I am already paying. And, no, my employer (a European university)
is unlikely to roll out Dropbox Business. So please stop bugging me :(.

Secondly, more and more features are rolled out to business users first. Why?
Are regular paying customers going to be treated as second-class citizens to
bully them into business accounts?

~~~
radicaldreamer
I killed my Dropbox Pro account and upgraded iCloud storage because the
integration with the Photos app in iOS is way superior and the lack of
something like Project Infinite, which results in a lot of manual folder
management for my long tail of old docs. Dropbox's core sync technology is
still unmatched, but the Pro offering leaves a lot to be desired over some of
the better integrated experiences of Google and Apple's photos products.

I expect this to become even more of an issues after WWDC.

~~~
bad_user
Going for iCloud, that's an odd decision, frankly. I would have understood
Google Drive, but not iCloud.

When it comes to my personal photos archive, I care about 2 things: (1)
occasional sharing and (2) keeping my huge archive safe. Dropbox does both,
iCloud does neither.

iCloud sharing only works with people using iDevices and I don't have many
acquaintances or family members with a preference for Apple. Dropbox allows
for publishing to a web link, with or without a password, with or without an
expiration date. And that's cool, because I can send that link to anybody.
Cross-platform and all that.

In terms of safety, I keep an extra offline backup by means of a home Linux
server that's almost always on, synchronizing my Dropbox and does the
occasional local backup. It has a big hard-drive of course. Dropbox works on
Linux, iCloud does not. Also Dropbox provides a 30-days history of all changes
and a 1-year extended history for extra cost. I got the 1-year extended
history. With iCloud get some Ransomware and watch in horror how all of your
photos are gone.

~~~
oarsinsync
> When it comes to my personal photos archive, I care about 2 things: (1)
> occasional sharing and (2) keeping my huge archive safe. Dropbox does both,
> iCloud does neither.

iCloud definitely does occasional sharing. Keeping your archive safe is
definitely questionable though.

> iCloud sharing only works with people using iDevices and I don't have many
> acquaintances or family members with a preference for Apple. Dropbox allows
> for publishing to a web link, with or without a password, with or without an
> expiration date. And that's cool, because I can send that link to anybody.
> Cross-platform and all that.

I can do that with iCloud Photos. I select a bunch of photos I want to share,
I select an existing (or create a new) iCloud shared photo library, and then I
enable a web link for them and share that. Works on fruits, robots, glass
panes and penguins.

------
darwingr
I suppose "deeper" also being the strategic direction they're taking with
this?

~~~
jxy
For a real "deeper" blog they could have given us some benchmarks comparing
good implementations on FUSE and kernel space. They instead gave a meaningless
diagram to argue about performance. Toward closed source binary kernel blob
they go, and out I jump.

------
cs2818
Well I just became one of the dozens of people that monitors loaded kernel
extensions.

------
tbrock
This seems incredibly over engineered. One less system call? Great, but I'd
rather the service just be cheaper. Skip the blog posts and just ship it
already!

------
wineisfine
On box.com only admins can move & rename folders, which makes alot more sense
to start off with. With this solution, it seems everyone can still rename the
folders. Too bad their (box.com) support is the worst customer experience I've
ever whitnessed.

------
hemancuso
If you want to try out a FUSE-style Dropbox filesystem, I write one. It's
available on Mac, Windows and soon Linux. [shameless plug]

[http://www.expandrive.com](http://www.expandrive.com)

The bit I don't quite understand about Project infinite is that you still have
to manually decide what gets sync'd or not. It's also not a network volume, so
things like virus scanners or search indexers can just page in-data? And if
you want to offload data, you just move it from one folder to another which
still eats space.

~~~
blindfly
I've looked at your software a few times before. I've also tried to reach out
to you before. To date I've never seen a response from you and I've moved on.
I'm not stoked to see you're available to drop your product name in a thread
about another company but inquiries to your own go unanswered.

~~~
spydertennis
Maybe he doesn't want to talk to you. I'm not stoked to see you commenting
about your personal issues.

~~~
blindfly
Well if he didn't want to talk to me, he might rethink sending messages to
people blindly asking them to get in touch.

~~~
spydertennis
Fair.

------
sneak
Lest we forget: this is the company that let anyone on the internet log in to
any Dropbox user account they wanted with any arbitrary value for the
password.

[https://blogs.dropbox.com/dropbox/2011/06/yesterdays-
authent...](https://blogs.dropbox.com/dropbox/2011/06/yesterdays-
authentication-bug/)

~~~
flyt
Hi Sneak! As you're no doubt aware, people change over the years. They learn
from mistakes, improve how they live their lives, and become better over time.

Companies tend to be the same way, learning along the way and maturing,
especially when it comes to business processes and risk-related parts of the
business.

It's entirely possible that the Dropbox of 2016 isn't like the Dropbox of five
years ago in many concrete ways. For example, they could have hired new
people, improved testing and release processes, and become more serious about
engineering discipline.

Many startups early-on make dumb mistakes and go on to great success and
professionalization, but we should have both empathy and forgiveness for them
in the long term. Dropbox has recently demonstrated a focused attention on
large scale, challenging engineering projects (building a replacement for S3
in-house from scratch, writing kernel extensions, etc) and a reasonable
observer might conclude that they've learned from the mistakes of 2011.

~~~
sneak
Regardless of how much they've changed, 2011 Dropbox was decidedly _not_ two
guys in a garage and their complete and total lack of security engineering
diligence and multiple overlapping process failures that must have occurred to
lead to that incident call every future "At Dropbox we take security
seriously" into question. (See also: "goto fail".)

At some point, Dropbox clearly _didn 't_ take security seriously. They claim
otherwise now. The question is now "at what point should we believe them?"
It's subjective and my opinion is that the 2011 management that didn't take
security seriously then probably still doesn't take it that seriously now -
they've simply hired underlings to worry about it.

I have experiences with companies that have security in their DNA from day
one, and I've {observed, worked with, been a customer of} a whole fuckton more
who bolt it on later once time and money permit. Most of the latter do _not_
actually care one whit about security, it's just one more "avoid existential
threat x" box they have to tick as their business grows.

Google falls into the former. Dropbox and Slack and LinkedIn fall into the
latter.

[https://www.troyhunt.com/we-take-security-seriously-
otherwis...](https://www.troyhunt.com/we-take-security-seriously-otherwise/)

There is no reasonable amount of time that needs to pass until I willingly let
a Dropbox or a Slack or a LinkedIn run code in my workstation's kernel. Maybe
that makes me a jerk - if it does, I apologize.

PS: That's not how you spell my username.

~~~
rrdharan
Google most certainly does _not_ fall into the former.

Google has had multiple security incidents during the lifetime of the company
that resulted in an increasing investment in upping their security profile.
Operation Aurora
([https://en.wikipedia.org/wiki/Operation_Aurora](https://en.wikipedia.org/wiki/Operation_Aurora))
was one of them (which of course bit a number of companies and was quite a
sophisticated attack), but they have had other screwups, like the SRE spying
incident ([http://gawker.com/5637234/gcreep-google-engineer-stalked-
tee...](http://gawker.com/5637234/gcreep-google-engineer-stalked-teens-spied-
on-chats)) and others.

For Dropbox, the password incident did result in major and serious change; it
was a turning point resulting in significant investment in product and
infrastructure security. In my admittedly biased opinion Dropbox now has one
of the best security teams out there. For example the product security team
invests heavily in the XSS protections on Dropbox's website that are top of
class, and stronger than those on many of Google's own first party properties
(I'll demur on details here at the risk of likely violating one or more NDAs,
but I encourage you to read
[https://blogs.dropbox.com/tech/category/security/](https://blogs.dropbox.com/tech/category/security/)).

Source: I've worked as a software engineer at both Google and Dropbox and I'm
reasonably familiar with engineering, infrastructure/operational and physical
security practices at both organizations.

~~~
sneak
Will you run the closed-source Dropbox kext on your machine?

~~~
flyt
I will. My system runs lots of closed source code already.

~~~
hughw
Surely you have higher standards for kernel space.

~~~
flyt
nah. I trust Dropbox.

