This feature is implemented at a low level, and works on the command line.
For example if you have a directory that is all stored in the cloud you can `cd` to it without any network delay, you can do `ls -lh` and see a list with real sizes without a delay (e.g., see that an ISO is 650 MB), and you can do `du -sh` and see that all the files are taking up zero space.
If you open a file in that directory, it will open, even from command line, then do `du -sh` and see that that file is now taking up space, while all the others in the directory are not.
You can right-click to pin files and directories to be stored locally, and right-click to send them back to the cloud so they don't take up space.
This is actually very different than traditional network file systems like SMB, NFS, WebDAV, and SSHFS. With a normal network file system over the WAN you would have major latency problems trying to `cd` and `ls` the remote file system. Most of them also don't have any ability to cache files locally when offline, or the ability to manually select which files are stored locally and which are remote. AFS does have some similar capabilities.
This is how Bitcasa is working I think.
You would see all the files, in a fake hard drive of unlimited space. Some magic (machine learning and smart decisions) would try to figure out ahead of time my data access pattern.
So I would see all my pictures list, if I decide to open the first one, it would take a few seconds to download, and then I start browsing the pictures, it would figure out I plan to look at all of them, and pre-fetch them from the cloud, so there would be on average no perceived latency.
(First off, as a disclaimer, I no longer work for Dropbox, I don't speak on their behalf. I've only used the feature as a user.)
I don't know a common search/find system that open()s or read()s files during the search by default. AFAIK Spotlight and Windows search are indexed searches. As for the indexing operations, I don't know how that is handled, they could disable indexing for remote files, or they could somehow integrate with indexing.
Based on my testing of a pre-released version of the feature (it isn't released yet), if you were to do something like `find ~/Dropbox -type f -exec md5 {} +`, it would download files.
As a user it did exactly what I expected. I was truly amazed. It was totally seamless and amazing.
Compared to the complexity of what has already been implemented, solving the problem of "I want to recursively open/read every file in my Dropbox, but I don't want it to download terabytes of data and fill by hard drive" seems fairly simple. For example there could be a setting for the maximum about of space Dropbox will use up, e.g., 40 GB, plus Dropbox could be smart enough to detect disk usage. If you `grep -R` it may download/open/read the files, once you reach 40 GB or near your disk capacity, Dropbox could start removing local copies of files that are not pinned to be local, i.e., remove the files that were downloaded because of the open()/read(), not the files you explicitly told it to keep local. I don't know how the team will choose to implement these features, but I'm confident that it will be well-thought-out and tested.
Remember, Dropbox is the company that especially monkey patched the Finder to get the sync icons (http://mjtsai.com/blog/2011/03/22/disabling-dropboxs-haxie/). They will go to great lengths for a seamless user experience, and do a ton of testing. I have no doubt that when Project Infinite is widely available it will be amazing, seamless, and have functionality many people thought wasn't possible or only dreamed existed.
Valid question. I wouldn't be happy if I ctrl-f on "My Documents", do a search and a 1 TB download starts up invisibly in the background filling my hard drive.
I suppose any company that is giving all their encrypted data to Dropbox to begin with may be OK with it. But most companies are already sketched out by the mere fact that their data is accessible to anyone outside the company.
In any event, if they were to index and provide search as a service as well, I wouldn't think it's something they do quietly. It would most likely include it's own huge marketing campaign.
Could Dropbox detect repeated access patterns from the same process, and/or whitelist processes as known "searchers," and start returning blank files? This seems like the kind of problem only a unicorn would dare to tackle, but as luck would have it...?
You want to save space by not having data on your local system but use a local search to look in the contents of files not on that system? You can't have your cake and eat it too.
I believe this is not the case here. In order for the files to start taking space on your drive you would actually need to right click that folder and choose "Save a local copy".
This is what I was wondering - we'd have to be careful writing a script that happened to traverse into the dropbox folder, because it might try to inflate all the files. It still seems like a cool idea, but I wonder if they have a workaround.
Spotlight is enabled by default _and_ left enabled on basically all the Macs and Mac users are actually a big userbase for Dropbox. It is very unlikely Dropnox team will forget that Spotlight indexing is running in the background.
Does not mean the files will get indexed, but there is no chance that Spotlight will trigger a unexpected terabyte download in the background.
"This is actually very different than traditional network file systems like SMB, NFS, WebDAV, and SSHFS. With a normal network file system over the WAN you would have major latency problems trying to `cd` and `ls` the remote file system."
Is that still true for sshfs ?
People used to ask us if they should rsync to us directly or sshfs mount and then rsync to the mount, and we told them not to do that since the original rsync stat would basically download all files simply to look at them / size them.
But I don't think that's the case anymore. I think sshfs (or perhaps something about FUSE underneath) is smart about that now ... isn't it ?
I haven't used FUSE SSHFS in around 8 years. I'm sure it has improved a lot since then. I could imagine it handling file listing and stat better than other network protocols (cd/ls over ssh works well over most WAN connections). It looks like it now caches directly contents too (https://github.com/libfuse/sshfs). It probably wasn't fair for me to include SSHFS in the list since I haven't used it in so long. I was troublesome when I used it.
"Are you guys allowing full access to the machine now through LXC containers or some sort of VM?"
No - it is the customer, on the client side, that creates an sshfs mount representing their rsync.net account.
It works very well and it is very nice to have a plain old mount point that represents your rsync.net account - especially since you can just browse right into your historical ZFS snapshots, etc.
But in the past, people did that and they got the bright idea to rsync to that local mount point, to do their backups, and that didn't work well.
But my understanding is that nowadays it would work better - you wouldn't download every single file that rsync simply stat'd or listed ...
We still don't recommend it, though. No reason to add that complexity.
git-annex also has the advantage of letting you keep data in multiple destinations (including in cold storage), which I think is becoming increasingly important.
> and you can do `du -sh` and see that all the files are taking up zero space.
That seems wrong to me. It would violate the assumptions of software that does stat() on directory entries and not only verifies presence but also non-zero size.
So it's risking buggy behavior to gain a latency edge over other networked filesystems. I think smart prefetching while preserving correctness would be better.
So, is it working off some kind of always-in-sync (assuming a live network connection) manifest?
I confess, I did not watch the video, and only briefly skimmed the announcement.
Edit: I'm asking a genuine question of real technical interest here. How can this be implemented with no latency and real file sizes immediately available for inspection, while taking up no disk space? I went back and read the announcement again, and there are no hard details I can see that I missed in my initial skim. There has to be something stored locally, right? Hell, I'm running gigabit fiber here, and I still notice latency in the CLI for anything that requires a network connection. Perhaps I misunderstood the parent?
If any of the dropbox product managers are following - I would REALLY appreciate this in the individual paid version as well.
Given a laptop with an SSD<1TB , I still want to be able to make use of the 1TB I'm paying for.
Its currently possible by letting an entire folder upload to DB, and then unsyncing it.
It stays on DB, but gets removed from the local drive.
I would love to be able to see those unsynced folders locally, with the same cloud icon announced here.
+1 – I use Dropbox for things like photos and music which are massive in total but where I rarely access more than a percentage or two on my laptop. It would be really useful to have my entire storage pool accessible without having to round-trip through the selective sync settings each time.
+2 - I feel like this is a (potentially intentional) blind spot for Dropbox and its ilk. I'm paying for 1 TB because I need that capacity, for my photo library, music, etc. - and I want to pay dropbox to make this work well with small fast SSDs. While there's various tricks you can employ to get this working, they're all a pain in one way or another.
"A user who wants to back up and sync lots of media, and selectively offload it onto cloud storage to save local disk space" is probably not such a rare animal. And if I'm not feeling charitable, I would guess that this isn't explicitly supported because if it was, many more Dropbox users would start eating a lot more of their quota than they currently do.
Yeah, I use camera sync to backup my phone's pictures but it's too much for my small SSD in my laptop. I have to periodically move files from "Camera Uploads" to "Camera Uploads Archive" (which is setup to not sync to my laptop) but to do that I have to be on a machine that can hold all of that (which luckily I have one but still it's a huge PITA. I've ALWAYS wanted dropbox to work the way outlined in Project Infinite and I'll be really disappointed if it doesn't come to personal as well. Seems like it would be pretty stupid not to as it's just a client change and it's not like it really costs them anymore AFAICT.
Ditto - in particular, I'm curious whether it has any concept of partial staging so e.g. enough of a RAW file could be downloaded to display an embedded thumbnail but the transfer would halt at the next block if the file handle was closed
+1 This feature would be incredibly helpful for photographers / videographers and would be much more convenient for pseudo-cold storage than glacier, etc.
+1 for this! It'd be very handy for me as well as a paying personal account. E.g. as in useful enough that I f any competitors bring this out I'd be switching to them. Hope Dropbox keeps us happy pro users.
This was really exciting to read. The ability to seamlessly integrate with the OS. We already do it with SSHFS and NFS; but bringing Dropbox's sync, versioning, and offline caching would be great.
Then I saw that it's not available on Linux. Which is really surprising, since supporting arbitrary filesystems is IMO dead-easy on Linux, and the easiest of the three desktop OSes. And since our company uses only Linux workstations (except for the designers), this is immediately unavailable to us. This is disheartening.
It is a shame that it went unwritten for an operating system where writing this functionality should be significantly easier (via FUSE or similar user-space file system) than Windows/MacOS.
Yea I was waiting to see if they were going to talk about Linux in the video. They didn't.
This is low level enough that it'd have to be implemented as a FUSE layer, which could add quite a bit of complexity. I wonder how they're going to deal with Linux. Will it still just sync everything?
How would writing it as a FUSE fs add complexity, as compared to whatever (which we don't know) they've done on Windows and OS X? FUSE is built for stuff like this. If anything, it should be easiest with FUSE.
And then they could have used the same codebase for Windows with Dokan (which has FUSE compatibility) and OSXFuse, although those two aren't installed by default. (FUSE is).
An overlay filesystem has a lot of advantages over FUSE. Mostly, if it crashes the user can still see their files. Implementing it with FUSE makes everything much more severe.
Uhhh ... where did the comparison between overlaying fses and FUSE come from? Overlaying of filesystems is for completely different purposes. It's not even suitable for Dropbox's Project Infinite — what would it be overlayed upon? How would dropboxd send/receive file info/data to/from the overlaying fs's driver?
The mere use of FUSE doesn't make an fs complicated or things "severe". It's just a really simple API made available to userspace processes. In fact, you can build an fs in FUSE that does overlaying itself, using multiple, isolated processes. Or you can use a single, simple, small process to implement a simple fs. FUSE is literally just the glue that would allow something like dropboxd to send/recieve file info/data to/from the VFS.
Uh, this is exactly the type of thing that FUSE is good for. Doing this as a FUSE layer would be way way easier than for Windows (unless they used Dokany[1], and they probably didn't) or god knows what you have to do to get OS X to support user-space filesystems.
I've never really been able to get Dropbox to cooperate with Linux, even when I've used Ubuntu. I'm surprised none of these cloud services have made an Electron cloud application for their services now that Electron and nw.js are more common these days.
I've been using Dropbox with Linux for a long time. It's THE reason for why I'm paying for Dropbox, because none of the other mainstream alternatives are supporting Linux. Which is preposterous given that Linux is the most popular choice for servers, including home servers. But then companies like Google or Microsoft don't give a shit, since for them cloud storage is just a complementary to lure you in to their other stuff and Linux users are not really their target.
So if Dropbox doesn't keep supporting Linux, I'll drop them like they are a hot potato. I do hope they won't make that mistake, because their continued Linux support has been the main sign that they care about me, their customer. And I don't want half-baked support either. I want it to be a first class citizen. Because I'm paying 14 EUR per month for my Dropbox account, amounting to 168 EURs per year, which isn't cheap at all. Yeah, yeah, it's the price of 5 coffees, but if I'd pay that price for everything I use, I'd be broke.
> Electron and nw.js
These would take care just for the UI, which for the purpose of doing file synchronization amounts next to nothing.
Seafile supports Linux and works phenomenally well. I've been using it for a couple years now. Don't trust its encryption (just like you should not trust Dropbox's encryption claims), but from a utilitarian point of view, Seafile is great!
Plus, when it comes to data/file sync tools like Dropboox, we need interoperability with existing tools: File Managers, filesystems etc. Not GUIs. Electron et al. are good for cross-platform GUIs, not CLIs or daemons.
I only mention Electron as an option since they don't attempt to get anything working, also I would mostly use these services on my own desktop, on a laptop I rather download files I need via a website interface directly.
This sounds like the exact feature I've been desperately looking for, for quite some time now, watching Microsoft discontinue online-only files in Onedrive because "it confused customers", and shaking my head at the blatant lack of this kind of thing everywhere...
Until just recently, when I found odrive[1]. Not affiliated with them in any way, just a fan - if you don't want to wait until Dropbox decides to launch this publicly, check it out. It works with Dropbox/Google Drive/OneDrive/CreativeCloud/what-have-you and does exactly what Dropbox Infinite promises to do, except that it's seasoned, working, practically bug-free (that I can tell), and, well, available now.
Microsoft had what they called "placeholders" and it worked great, but it did confuse some users who looked for a file, saw it was "there" and happily went offline. Only to find it was only a placeholder. Furthermore, on devices with limited storage (phones, tablets) even the small size of the placeholders becomes a problem when the number of files is big enough. So Microsoft temporarily removed the placeholders functionality, promising to bring it back in the near future using some new (at the time not yet invented?) technique that would solve the problem.
I wonder if Dropbox could just make use of this (e.g. by registering itself as an HSM backend provider to Windows somehow) rather than doing its own logic.
Onedrive had similar same icon overlays to notify if the file was really downloaded or not. Dropbox is going to run into the same problem with placeholders where people go offline thinking they have the entire file.
This is a tools problem. People need to know how to use the tools in a business or let them go. You don't keep carpenter around who blames power tools not working the way he expects when he never charges the batteries.
Dropbox's response could be, "Hey we have training seminars for that".
App compatibility was also an issue with OneDrive placeholders (some apps couldn't handle placeholders). I wonder if Dropbox is going to run into similar issues with Project Infinite/
$99/year ($8.25/mo) to have "Premium" which offers the option to leave files in the cloud and sync on demand. This is too much money to gain a feature addition. No thanks.
Interesting... I just checked and apparently I'm on a grandfathered free plan that can still unsync. Didn't know they changed it.
For me, it's still more than just a feature addition though - I use odrive for all my cloud storage accounts. It's all under one roof. Your mileage may vary, of course.
At first I thought "Well, Dropbox storage is 5x as expensive as hard drive space, so what's the point?". What I didn't see was the team use case: It's actually brilliant to have access to every document of all of my colleagues at all times, without saving all that stuff locally, or taking forever sync. This is really amazing.
The problem with our current shared team folders is that you need to make a deliberate effort to share something with the team. That means when you want to pull information from the shared space, you're very likely to get an outdated copy. Thanks to 1.) no storage constraints and 2.) deep OS-integration, all files can actually be always up-to-date with Dropbox Infinite. That really sounds cool.
> ..or taking forever sync. This is really amazing.
I don't think this solves the problem of taking forever to sync. Nothing has changed there. You are still bound by the same data retrieval and network latencies to get the file stored in a datacenter somewhere to you. That one is a harder problem to crack because it needs a lot of infra investment in expanding your content delivery footprint and replace SSD's with something much faster, like flash.
This solves an important aspect of the sync problem -- the ability to selectively sync individual files without having to sync the whole directory that it is part of (or even to have to know what the directory is in advance).
Dropbox has actually had Selective Sync for years. The whole part about the metadata being cached locally and a seamless experience at file system level is new though.
Dropbox storage is 5x as expensive as hard drive space, so what's the point?
Yes and no. I have about 80 GB in my dropbox. This is no problem on my two workstations or my big laptop. Unfortunately my Macbook Air has a 128 GB drive. That means I keep having to exclude and include folders to be synced to my laptop or it will run out of space. That is a pain in the ass I would love to avoid.
This is huge. The biggest reason for me to not store more content on dropbox, is that it always comes with the trade-off of taking up space on my local drive (128gb) as well.
Sure there's selective sync, but that's an all-or nothing approach.
I like the way Google approaches this with their Google Photos service. X most recent photos are stored locally, and everything else is pulled down on the fly as you search/browse for them.
Not disagreeing, but I'm not seeing that on my wife's iphone. She got a 16 gig phone (big mistake), but we were thinking that getting everything on iCloud would make up for this. We pay every month for iCloud storage.
That's not what we're seeing. Her phone gets to be 100% full, and deleting a photo frees up space. When we delete a photo, we get dire warnings that "this will delete the photo on all of your devices, and in the cloud too." I download them to my computer before doing this, so nothing is really lost. But it's annoying.
By the way, it works that way by design. iOS will attempt to fill all the storage, and will throw stuff away (essentially on an LRU basis) as needed. Free storage is wasted storage, just like having a lot of physical RAM free is wasted in a virtual memory system.
This is why I was amused by the breathless articles last week about how to "free space on your iPhone" -- essentially by clearing the cache by golly!
(If the 16GB is filled up 100% and things fail, that's an actual bug though. You might really have filled it up with stuff that doesn't use iCloud, leaving too little for the iCloud apps to work with)
BTW all of the above is orthogonal to "optimize for space / use lower resolution on portable devices" that others have helpfully mentioned.
In settings you can set it to "optimise iPhone space" (also possible on a Mac in the photos app) which will automatically remove local copies of old photos from the device if space runs low, and then download them again when you open them. For me it works very well, though I have had some trouble where it seems to need a few hundred free megabytes to start the process. If you delete photos with iCloud photo library enabled, they will be deleted from all your devices.
What happens if the corporate virus scanner kicks in and reads all listed files? Also, can I un-download files I just checked once and don't need to have synchronized any more?
Theroretically this is nice, practically it will render som unpleasant surprises.
Indeed! I was looking at using Tahoe-LAFS in a similar manner as this dropbox feature, and happened upon a lot of information about those unpleasant surprises:
One good optimization would be caching the first ~512 bytes of each file—enough to make libmagic-based programs (which includes a lot of search-indexing systems, grep's binary-file-skipping, etc.) happy. With this in place, "opaque" files that an indexer doesn't want to look into could be left in the cloud, while only the files that actually need to be content-indexed would be downloaded.
I think it's an unfortunate situation resulting from the need to be realistic about releases. If something can be released now for > 90% of the users then companies will prioritise doing that and then releasing the remaining 10% later.
The problem is when that remaining 10% takes way longer than expected.
To be fair, that's a decent start. Especially if you're looking at prioritising where your customer base is. Linux is important, but nowhere near as much in comparison for the average consumer/business.
That may be true compared to other synchronization solutions on Linux, but I doubt Linux users make up a disproportionate amount of their customer base.
You're looking at the wrong end of it. Dropbox doesn't have to deal with any filesystem, Dropbox instead needs to build a filesystem.
It doesn't have to deal with any filesystem because Linux abstracts it. The same abstraction allows it to build its own filesystem really, really easily. How easy? I wrote, mounted, and used a filesystem in less than fifteen minutes. In node.js. It's that easy.
> desktop environments
Dropbox doesn't need to support various desktop environments, just a stable API for the desktop environments (actually, just file managers) to use. Isn't this how Nautilus, Thunar, Dolphin et al. already support Dropbox's existing features?
"different filesystems": maybe not. This new Dropbox thing is a network filesystem so it could just use normal syscalls to read and write files on any filesystem the ~/Dropbox directory and be (local) filesystem agnostic.
After all you can bypass the problem by running a Windows VM in VirtualBox with guest additions and a shared filesystem. Is another way to map the Windows Dropbox's network filesystem to the Linux one, whatever it is.
Still, very cumbersome and probably slow. I remember not stellar performances with sharing files between host and guest OSes in that way.
Hopefully Dropbox will release the API and somebody will write a user mode filesystem to interface this new service.
The only feature I want in addition to what Dropbox already gives me is a little box I can check that says "Never Delete Anything."
That's it.
Just never give me the option of permanently erasing an entire folder of photos from every machine I've ever owned just because I mess up your wacky Selective Sync workflow.
I understand that there exist people who actually want all their machines to sync destructive changes, but I've never met one of them. More often, I see people lamenting that they deleted something important a year ago, thinking it was still backed up, but now it's gone.
So yeah, you're not a backup service, you're just sync. I get that. But please please be a little less syncy and a little more backupy and you'll fit the worldview of a lot more people.
It sounds like this is a step towards this (and four thousand steps past it), so at least we're getting close.
I've mistakenly deleted files before, and Dropbox did two things: email me that a large number of files were deleted, and linked to my history page where I could restore them. It was a great experience, and seems like a solution to your problem.
Dropbox has had the "packrat" feature for a long time that keeps deleted and previous versions of files. I've been lucky enough not to have to use it much but the few times I have it's performed as expected and been invaluable.
Microsoft had this feature built in windows 8 / 8.1 and removed it in Windows 10 cause it "confused" customers. Hope this will nudge them to bring it back.
It removed "placeholders" and added a selective Sync feature but its not the same. Placeholders experience felt more intuitive. Files that were only available online were marked so you know. I guess the problem started cause a lot of users would not be able to access files when offline and started complaining to their support service. Too bad though I loved that feature.
The word 'revolutionary' definitely deprecated in the past years. Anyways, this is great news as I'll no longer have to default to the website search for selectively non-synced files.
My original thought was, so its a network drive huh?
But when you think about the fact that everyone has 128GB Surfaces they take everywhere remotely this is really powerful and I can image a lot easier to administer. Love how they have integrated and just added a simple icon. Well done Dropbox
I really hope this is available to personal accounts, and not only business, as the blog post sort of implies. With the average laptop being 256GB and the Dropbox paid account being 1TB, that's badly needed.
I was thinking this as well. I have been really frustrated with selective sync because of how long the process takes to check the folders that make sense on each computer.
A couple of usability questions that don't appear to have been addressed in the video:
- on a Mac, if I press the spacebar to show the quick preview and start navigating down the list, will this trigger downloading of all the files?
- how are older files cached? Do we simply unselect devices to remove them locally? At least with Selective Sync, I can remove files eating up my hard drive (very important on devices with 128GB SSDs/HDs or less). It's much clearer in terms of what's on disk and how to manage it with Selective Sync. With Project Infinite, I'm wondering how one would remove files locally without fully deleting the file across the Dropbox folder for every other team member.
Glad to know Dropbox is supporting streaming (as opposed to sync'ing) model. It didn't make much sense to keep large files shared with me in Dropbox otherwise.
Keybase Filesystem [0] has been also doing this from the beginning.
>“Distributed” means you can access it from any device.
> “Filesystem” means that there is no sync model -- files stream in and out on demand. Among other things, that means that files on KBFS don’t take up space on your devices.
Plus the files are automatically signed, and end-to-end encrypted if in /keybase/private :D
What I want from a network-based file system is some level of control over the caching. What I've found with FUSE-based file systems (on Linux) is that often they lock up my terminal. All it takes is a "cd" or an "ls" and suddenly I can't type anything until the network responds, which is not always instantaneous -- can't even Ctrl-C, because the program is blocked in the driver.
I want to be able to tell when a command is going to block, or respond immediately with an error so that I can re-issue the command in a deferred manner. Having the command line lock-up because of filesystem latency is a bad user experience.
Elsewhere in the thread it's mentioned that it won't lock up directory traversal. Of course when catting a file it would, it'd be hard to get around that...
I also want this feature, and I like open stuff, so I wrote SyncthingFUSE[0]! It's primitive right now, but works for me at the moment.
Bazil[1] has similar goals, but wasn't ready enough for me.
I do very much like sibling /u/pritambaral's suggestion of OwnCloud support. I tried OwnCloud briefly in the past, but ended up switching away. A feature like this might get me to try it again and stay.
I have been watching the bazil project for a while - it's exactly the system I want. Still very alpha right now, so I don't (and one shouldn't) trust it with irreplaceable data.
Yes, I'd like to see OwnCloud develop something like this. At least that way, I'd be able to use it on my system; even if OwnCloud themselves don't build it for Linux, I'd be able to make my own on contribute it, unlike with Dropbox.
No, because everyone knows you can't build a profitable startup around open standards that just work for everyone without being tied to your network as an identifiable user.
Yes, because everyone knows it would be amazing to have this kind of functionality just work for everyone, regardless of their personal OS and technology/device choices.
Yes. Checkout git-annex assistant if you want folder icons and GUIs and such.
A couple days ago I contributed https://github.com/DanielDent/git-annex-remote-rclone to git-annex, so in theory you could use Dropbox as a backend for git-annex if you wanted to (I haven't personally tried it with Dropbox).
I'm glad to see they are finally addressing this problem. One of the largest requested features has been better management of selective sync folders [1] and many users have left for other solutions since they still haven't fixed it in the existing client.
Somewhat related: how reliable are SSHfs and other fuse filesystems on macs and on Linux these days?
- On the Mac I'm worried about using fuse in general. Stability, reliability, Mac OS X being a hostile environment as its not "blessed" by Apple.
- On Linux I'm more worried about the general nature of SSH as the basis for a filesystem. Which also applies to the running it on a Mac.
The reason I ask is because theoretically you could spin up a t2 micro with a 1tb EBS volume and forget the whole Dropbox thing, mount it wherever you please, whatever.
Obviously there are drawbacks, it's not local (fast), it doesn't have all the collaboration features of Dropbox, etc. but for relatively cold storage (colder than I need on a local ssd) that I want to access somewhat frequently, SSHfs seems like it could be fine.
I'm not sure how easy it would be to corrupt, etc and I'm not sure about the two things I mention above. Hoping someone here knows.
Which is an alpha-version feature of a service that's currently in private beta. Looks great and I'd like to try it out, but currently I can't. (edit: missing word)
And let us not forget about the whole Drop Dropbox scenario which was very popular on HN at the time it was released: http://www.drop-dropbox.com
And that Dropbox was one of the companies in the whole PRISM scandal.
I still use it, mostly to transfer files between my devices, but there's no way I'm gonna be using it for anything sensitive. Everything sensitive goes to my organization's ownCloud installation which I completely trust since... well, I'm the one maintaining it.
I originally thought this is how Dropbox functioned years ago. Learning that you had to actually have every file on your hard disk if you wanted it in the cloud really put me off from using Dropbox for any significant amount of data.
As so many have stated before me, they better release this for everyone. It isn't a revolutionary feature, it's pretty basic, even if the behind the scenes technicals are complicated.
Sounds like a similar goal to https://upthere.com. If Dropbox can make this real, and with the reliability I've experienced on Dropbox over the past few years, I'm sold.
This remind me Plan 9's "single most important feature"[1]:
> all mounted file servers export the same file-system-like interface, regardless of the implementation behind them. Some might correspond to local file systems, some to remote file systems accessed over a network, some to instances of system servers running in user space (like the window system or an alternate network stack), and some to kernel interfaces. To users and client programs, all these cases look alike.
Sounds like a great idea, but not exactly revolutionary. In the 80'es I saw at least one implementation of this which migrated between tapes(!), hard drives, and memory. I always wanted this.
CODA [1] implemented parts of this as well.
Unfortunately, Dropbox has unencrypted access to all files and is too expensive, otherwise I'd be all over this. I'd rather allocate funds towards an open source implementation (I think something interesting would be done based on idea from CODA and NFSv4).
The cloud "wheel of reinvention" revolves to bring AFS to the masses? Or is this just showing the files using a file manager plugin, not visible as a real filesystem?
For random stuff that should not be synced, much like .gitignore. That'll vary depending on the person, but for me it's usually for Mac/Windows crap (.DS_Store, Thumbs.db), temporary files of any kind, node_modules, and a few others.
It made me wonder if people are using their remote Dropbox space for archival or as a backup solution. The business plans appear to offer unlimited file recovery back to any point in time.
My entire Prepress department does. Granted, we use DAM systems instead of cloud storage, but some are over the wire. Some assets can easily reach 10GB or more (each).
The phone OSes don't allow arbitrary views to the file system. Every file you open from your Dropbox on mobile, you open through the Dropbox app which will then copy the downloaded file over to the opening application.
Dropbox has full control over which files it downloads and which files it needs to keep around.
On the desktop, the file manager and every single application (by virtue of their file open dialog) get access to any file at any time. You are supposed to be able to open any file in your Dropbox without going through a dedicated Dropbox app.
Dropbox uses phone APIs that were designed to do exactly this. There are not "cloud storage" APIs that simulate a file system in desktop OSes. So Dropbox had to implement the syncing logic. And they also, now, have to implement the cloud file system logic to get this new feature.
The first company to sort of mimic this was bitcasa. they just announced they are shutting their doors to consumer storage, because they were having greater success on the enterprise front. I wonder if this is related somehow.
Well unless it is abandoned - you're pushing it right now. And who is this targeting? For me as a "techie" the softpedia doesn't really instil any confidence and actually kind of makes it look cheap.
Actually, I think for older software it is fine for some reason. But in this case it looks strange.
Have anyone speculated how this is achieved, from a technical standpoint? How would one go about to start creating something like this on their own that works against e.g. a NAS?
This is great news. If it's not part of the normal non-business Dropbox I think I'm going to stop using Dropbox. This should be a common-sense feature for the base product in 2016.
I will curious about the performance. SSHFS/FUSE, expandrive /odrive renderings of your onedrive/cloud drive bucket and so on have been pretty bad in my experience.
The Dropbox Client has a lot of smarts in it when it comes to fast and reliable downloads (threading, chunking etc.), so it might probably be a lot better than many other alternatives.
Feels like this is aaaalmost there. I just didn't see an option to universally keep data storage below a certain amount. If I bring down so many local copies that I exceed the space on my computer, do I then have to peck through local files and individually revert them to being reference files until I get my space back?
Would be great if, once a certain threshold is reached, the oldest downloaded files would revert seamlessly.
Google drive has had this feature for a while. Your Google docs are essentially shortcuts to the cloud.
Im glad Dropbox is releasing this, I think it's very useful, but man is it trumped up. They have officially crossed chasm into the mainstream, when there is more marketing hype than technological innovation in a new release.
Google Drive only 'shortcutizes' actual Google docs, which can't be edited offline anyway. If you put a PDF or a large media file (or anything that takes up space anyway) into Google Drive, you don't get a placeholder, you get your entire video collection.
I consider Dropbox's feature a huge improvement over that.
Actually, some Google Docs can be edited offline. I once tried to figure out what is possible and what isn't. Basically, you can edit Docs and Slides offline in Chrome. Drawings and Sheets can only be viewed, not edited offline. This is 2013 information though, some things may have changed for the better.
And they conveniently don't mention Linux support.... Sigh..
Also interesting thing to note is how everybody came down on keybase FS because the sync is on-demand. Most people citing dropbox and things like "if it was a viable option dropbox would do it and not have full sync". Well now... :D
From a first look on Windows, this is implemented with NTFS sparse files [1] and a file system minifilter. The sparse files act as placeholders so that you can browse your entire Dropbox structure. When you access a file, the minifilter sees that and starts fetching the data from Dropbox servers in the background. You can think of it as HSM [2]
My first guess is decoupled drivers with an abstraction that the actual dropbox client service uses. One driver for windows, one for mac. Though I don't know if you can do user mode filesystems on those, but there is this: https://dokan-dev.github.io/ Or else they may have used some fs event notification library to intercept access to files, but I think that would be harder (impossible? IDK win32/mac) to do than a filesystem.
Everything old is new again: https://www.youtube.com/watch?v=vr3UiKkQ3GE. This does look very interesting though; I feel like this is the storage solution that I've been looking for.
I was using BitCasa Drive but they just announced they are shutting it down. I need a cloud storage solution that stores files in the "cloud" but does not sync them locally. I have about 750GB of files and looking for a replacement for BitCasa Drive. Any recommendations?
Currently, to collaborate on Office-type files, we collaborate on Google drive/docs and then move it to Dropbox. This seems so silly.
Is there a better, more integrated solution?
Wish Dropbox had better Office integration. Currently, it just keeps making copies of a file if two people edit it.
Unless you're explicitly trying to stay away from anything microsoft, onedrive/office online sounds like a perfect solution for you. Been using it for quite some time for similar workstream and love it.
No, I don't have any MS hatred, except we have GIANT media files, engineering logs, technical logs, in the same place as well. OneDrive AFAIK, isn't that great at selective syncing (and now Project Infinite!).
Generally, cloud storage is done best by Dropbox. I'd love to be shown to be wrong.
Curious - you mention you use Google Drive for collaboration - but then move it to Dropbox.
Is there any particular reason you move it to Dropbox afterwards? (Or put another way, is there anything with Google Drive that you feel is missing, in your use case?)
I'm not a Dropbox user, but I sorta assumed that this was already the case. I use sshfs when I'm fully connected, and I don't really know how people could get by being limited by their endpoint's storage up until now.
A bit disappointing, it's about time they finally got around to it and yet "Check back here for more info as we continue to make progress." The remaining question is whether it'll be for pro users only as well...
This is a cool technological advancement but for me "in context" is not in a directory structure, it's in a conversation, my team has mostly replaced Dropbox with Slack with the exception of long form documentation.
That is a good idea. I use OneDrive because a terabyte is so inexpensive (and C. Rice is not on Microsoft's board of directors), but Dropbox is technologically impressive and I should give them another evaluation.
To those of you who are dismissing this because AFS/NFS has had this for years, the point is that this is running over the Internet, not your local network.
They totally just implemented Bitcasa for Dropbox. I hope Google Drive follows suit, I have been asking them to buy Bitcasa for years now, since Bitcasa launched.
For example if you have a directory that is all stored in the cloud you can `cd` to it without any network delay, you can do `ls -lh` and see a list with real sizes without a delay (e.g., see that an ISO is 650 MB), and you can do `du -sh` and see that all the files are taking up zero space.
If you open a file in that directory, it will open, even from command line, then do `du -sh` and see that that file is now taking up space, while all the others in the directory are not.
You can right-click to pin files and directories to be stored locally, and right-click to send them back to the cloud so they don't take up space.
This is actually very different than traditional network file systems like SMB, NFS, WebDAV, and SSHFS. With a normal network file system over the WAN you would have major latency problems trying to `cd` and `ls` the remote file system. Most of them also don't have any ability to cache files locally when offline, or the ability to manually select which files are stored locally and which are remote. AFS does have some similar capabilities.