
Syncthing: Open Source Dropbox and BitTorrent Sync Replacement - ushi
http://syncthing.net/
======
XorNot
Can we please please please make it a standard that synchronization tools
spell out how they handle conflicts on the front page.

Bittorrent Sync just overwrites files based on last mod time (terrible
option). What does this do? Does it support backups? Versioning?

~~~
nodata
Newest change wins. More detail:
[https://github.com/calmh/syncthing/issues/61#issue-27202590](https://github.com/calmh/syncthing/issues/61#issue-27202590)

~~~
nightpool
Specifically, this is not dependent on last mod time, but instead a "version"
field in each file's metadata that is incremented every time a file changes.
I'm not really sure how that's going to help with concurrent (or conflicting)
edits though.

------
stinos
Since we're listing alternatives here: I setup SeaFile
([http://seafile.com/](http://seafile.com/)) a couple of months ago and I'm
loving it so far. Mainly because it has client-side encryption and allows a
single client to sync with different servers and selectively choose which
'libraries' (basically directories that are under sync) to use. Typical
usecase is having a personal server for personal files and another one at the
office for work-related stuff.

~~~
ef4
I like seafile too, but after doing some of my own review of the source
([https://github.com/haiwen/seafile/issues/350](https://github.com/haiwen/seafile/issues/350))
I'm not willing to trust it on the open internet.

For now I run seafile inside a private network behind a vpn. But I'd rather
find an alternative with a better security story. Something from a team that
can justify their cryptographic constructs in detail.

~~~
icebraining
git-annex avoid all except (2), by simply encrypting each file using GnuPG,
and applying an HMACSHA1 to the filenames.

[http://git-annex.branchable.com/encryption/](http://git-
annex.branchable.com/encryption/)

~~~
higherpurpose
SHA1?

~~~
icebraining
It's configurable.

 _The default MAC algorithm to be applied on the filenames is HMACSHA1. A
stronger one, for instance HMACSHA512, one can be chosen upon creation of the
special remote with the option mac=HMACSHA512. The available MAC algorithms
are HMACSHA1, HMACSHA224, HMACSHA256, HMACSHA384, and HMACSHA512._

------
pjkundert
Has anyone else taken a look at Ori
([http://ori.scs.stanford.edu](http://ori.scs.stanford.edu)):

> Ori is a distributed file system built for offline operation and empowers
> the user with control over synchronization operations and conflict
> resolution. We provide history through light weight snapshots and allow
> users to verify the history has not been tampered with. Through the use of
> replication instances can be resilient and recover damaged data from other
> nodes.

It seems well thought out, and competitive with many of the other approaches
mentioned here. It uses Merkle trees (as does Git) that encompasses the file
system structure and full history.

~~~
xwei
Academic software usually for publication. After paper accepted, unless they
could publish another paper using same software/concept/idea, this software
will dis-continued.

------
abalone
"Replacement" is too strong a word here. P2P sync requires at least 2 peers to
be online at once. For the simple case of syncing your work and home computers
or sharing with coworkers that is not always a reliable assumption.

It's only a replacement for a centralized service like Dropbox if you have an
always-connected peer (a de facto central server).

~~~
tjmc
I don't understand why you're being downvoted because you make a valid point.
It has no iOS or Android client, no cloud storage (unless you buy your own),
no versioning and sync conflict behaviour is very different. Ergo, it's not a
replacement.

~~~
krbbltr
> no cloud storage (unless you buy your own)

That's kind of the whole point, isn't it? It's clearly meant to be a Dropbox
replacement not for everybody, but for those who consider the lack of
mandatory cloud storage a feature. So yeah, technically more a replacement for
BitTorrent Sync than Dropbox, but not everybody knows about that.

Missing features can be added. Mobile clients by third party devs will surely
crop up if/when this thing gets traction. The protocol is open, after all.

------
sinkasapa
One of my favorite open source tools of this kind is unison. It works great. I
set it up to go and I don't even notice it is there. It is quick, seems to
have been around for a while and is packaged for most Linux distros. It has a
GUI but you don't need it.

[http://www.cis.upenn.edu/~bcpierce/unison/](http://www.cis.upenn.edu/~bcpierce/unison/)

~~~
sdfjkl
Unison is a under-appreciated gem of file synchronization. The author also
published several papers on the subject:
[http://www.cis.upenn.edu/~bcpierce/papers/index.shtml#File%2...](http://www.cis.upenn.edu/~bcpierce/papers/index.shtml#File%20Synchronization)

------
alyandon
I can't really seem to find this information in the documentation.

Does this support delta/block-level sync for large files (e.g.: does mounting
a 100 GB truecrypt container, modifying a file inside the container and
unmounting it cause the entire 100 GB container to be uploaded)?

Does it utilize the native OS platform APIs for detecting file modification
(e.g. inotify on linux) as opposed to scanning/polling large directories
looking for modified date changes?

~~~
nodata
Protocol information is linked from the homepage:
[https://github.com/calmh/syncthing/blob/master/protocol/PROT...](https://github.com/calmh/syncthing/blob/master/protocol/PROTOCOL.md)

> Does this support delta/block-level sync for large files

Yes.

> Does it utilize the native OS platform APIs for detecting file modification

It uses polling.

Edit: inotify issue is here:
[https://github.com/calmh/syncthing/issues/9](https://github.com/calmh/syncthing/issues/9)
(the author seems to want to implement this feature once go has a decent
cross-platform file watching library)

~~~
alyandon
Thanks for the quick response. I just got through reading PROTOCOL.md and was
about to update my post.

------
frabcus
Fancy being interviewed for
[http://redecentralize.org/](http://redecentralize.org/) ?

If so, email me francis@redecentralize.org! (I couldn't see an email or
contact form for you on the syncthing site)

~~~
ushi
You can find an email at the authors GitHub profile[0].

[https://github.com/calmh](https://github.com/calmh)

~~~
frabcus
Thanks!

------
simbolit
I use [http://owncloud.org](http://owncloud.org) and am quite happy. But also
happy for more competition :-)

~~~
aram
No matter how much I tried, I always had crappy experience using OwnCloud (->
very slow to recognize new files and sync them across other computers). I
tried it maybe 4-5 times in total, last time just couple of weeks ago.

Does it work fast for you? Do you use desktop client or primarily web GUI?

~~~
josteink
I tried it too, but ended up replacing it with something else. It was slow as
a pig, was unable to handle datasync with failures and timeouts on any
reasonable amount of files (a 2000 file user) and constantly made my CPU fan
hit max.

My server was a Core i7 with 12GBs of RAM, SSD backing and a more or less
dedicated NAS. I tried with both SQLite and MySQL backing. MySQL was better,
but still unacceptable. If that's not enough to run it for a _single user_ , I
can't imagine what it's like trying to deploy it for a full family, even less
an organization.

I love the _idea_ behind owncloud, and I admit touting it myself in my own
"honeymoon phase", but in hindsight I have no problem admitting that the
execution is just too poor.

~~~
aram
Same experience. I tried it on DO droplets (1, 2, 4 and then 16GB of RAM to
see if I'm nuts), computer I'm working on (16GB RAM) and from a RPi (that was
for fun though). Every single time it was painfully slow.

It took it about 1h to sync a folder with about 100 files, under 30MB total
across 4 computers. Not to mention that minor modifications appeared only
after 15 minutes (small text files).

BTSync worked great, but I wouldn't trust it with something sensitive.

What did you end up using?

~~~
josteink
> What did you end up using?

Long story short, I went from an "all cloud" (Google + Dropbox) setup to a
"all mine, no NSA" setup. Now I've decided that counter zealotry doesn't have
that much for it, and I'll try to balance privacy with best of breed services,
without trying to end up with all my eggs in one basket. Not doing that
ensures I keep interop and portability in mind.

My end-result is calendar via Google (only solution I can get collaboration
and invites working properly), files via Dropbox, mail with Fastmail (much
better than Gmail) and contacts via Baikal on my own hosted server.

Fastmail is adding support for calendering and contacts via CalDAV and
CardDAV, but I've yet to fully investigate that.

It's a moving target for me, but currently it's been stable for a few months,
and I'm mostly happy.

~~~
aram
Thanks for sharing!

------
taterbase
Git-Annex is a great existing option (made by joeyhess). [https://git-
annex.branchable.com/](https://git-annex.branchable.com/)

~~~
Dylan16807
Can you explain to me why and how I would use git-annex for real time sync?
From what I understand of it, it works well for backups and for managing media
files, but I don't understand how it handles the use case of making two
directories on two computers pretend to be one.

~~~
prutschman
git-annex assistant, by the same author, builds automatic synchronization on
top of git-annex: [https://git-annex.branchable.com/assistant/](https://git-
annex.branchable.com/assistant/)

------
freework
One thing I've never gotten about these "syncing" apps...

Lets say I install this software on my phone, my desktop, and my work
computer. I have 100+ GB free on my work computer and my home desktop, but I
only have 16GB on my phone. If I add 20GB worth of movies to my sync folder,
its going to fill up my phone.

~~~
nuclear_eclipse
Dropbox has "selective sync", so individual PC's can choose to only sync a
subset of folders from your account. Also, phones running Dropbox don't
actually "sync" anything to the phone. You can choose to download individual
files, or stream them straight from the Dropbox app, but it doesn't offer you
any way to keep files on the phone up to date with whats in the "cloud".

~~~
werid
I use dropsync on my phone. I have set up rules to sync screenshots to a
different folder from photos, and periodically it will check a folder on
dropbox and sync files from it to phone.

------
JonAtkinson
I just setup Sparkleshare
([http://sparkleshare.org/](http://sparkleshare.org/)) this weekend. I wanted
something which wasn't Dropbox, and preferably open-source, and while
Sparkleshare has a slightly clunky pair mechanism, it works beautifully.

Syncthing looks similar, and LAN sync'ing is a killer feature for those of us
in offices with poor bandwidth.

~~~
marlin
Dropbox actually have a "Sync over LAN" checkbox. It just never seem to been
implemented/enabled.

I actually just dropped Dropbox for btsync the other week and are very happy,
it's iOS app even serves other clients with files over LAN.

Btsync however uses quite crappy crypto and is closed source etc etc.

Syncthing mentions they use TLS encryption, hopefully i can roll my own 8k
keys.

I tried Sparkleshare last year aswell, but my experience was mixed. Also it
relies on quite a different architecture (git) than a "versioning bittorrent
protocol".

Looking forward to evaluating syncthing when time permits.

~~~
NDizzle
Isn't it always enabled by default?!

The last few times I've set up a new machine locally I've used Dropbox to pull
over stuff. It usually moves at 40-50 MB/s (megabytes) which is certainly
using my gigabit LAN.

~~~
marlin
According to [1], "LAN sync" only works when file has already been uploaded to
the dropbox servers.

If this is accurate, then I guess this is the issue i have noticed.

1: [https://superuser.com/questions/663600/set-dropbox-to-
only-u...](https://superuser.com/questions/663600/set-dropbox-to-only-use-lan-
sync-dont-allow-web-sync)

UPDATE:

Confirmed by [2], that Dropbox LAN sync indeed only works once the file
reached their servers.

"Well, when you add a file to your computer's Dropbox, the file is then synced
with Dropbox servers. Dropbox will then initiate the syncing process as soon
as it determines a change has been made to the file. All linked computers and
shared folders will then download any new version of the file. With LAN
syncing, Dropbox will look for the new file on your Local Area Network first,
bypassing the need to download the file from Dropbox servers, thus speeding up
the syncing process considerably."

2: [https://www.dropbox.com/help/137/en](https://www.dropbox.com/help/137/en)

~~~
nodata
For anyone wondering, Dropbox changed this a while back. The requirement to
sync to Dropbox first was added at that point.

------
popey
I've been using Syncthing for some months now and it's working well for my use
case of keeping laptop/desktop and home server files in sync. I had one
occasion when I lost everything as I'd brought up syncthing on my server
without the "sync" directory mounted. It happily deleted all files from my
synced laptop as a result. That's now fixed, but it was a buttock clenching
moment. Yay backups, and a third machine (desktop) which was suspended and
thus out of sync, so still had my data.

Upstream developer is very friendly and attentive & seems happy to discuss new
features and use cases.

------
sixothree
It appears HN readers are terrible at self-organizing. Threads for articles
like this should include by default a top level node for:

    
    
      "Here's the alternative I use"
    
      "Important question about the technology"
    
      "Pertinent question about the article"

~~~
dj-wonk
How can we organize after threads have already been started?

~~~
skeletonjelly
Do I smell a niche for time travel startups? Disrupt the Big Time monopoly!

~~~
dj-wonk
Well, I can't be sure, but I think colloborative technology for rearranging
threads on-the-fly might be slightly less complicated than monopoly-busting or
time-travel.

------
Wilya
That looks like a promising project in a space which definitely needs
improvement. Owncloud and Sparkleshare are okay, but they are far from
perfect, and there is large room for improvement.

------
r0muald
A better title would be "Syncthing, an open source Dropbox replacement written
in Go".

But seriously, it seems promising.

------
Karunamon
How good is this at traversing firewalls? AFAIK, Dropbox will do some manner
of HTTP trickery to allow syncing when behind overly-restrictive firewalls (so
it just goes out the usually-provided web proxy), but the documentation here
references forwarding ports + UPNP, so I'm guessing that doesn't apply here?

------
zyngaro
"Each node scans for changes every sixty seconds". There isn't any portable
way to get notfications about file changes instead of polling? I know about
jnotify in Java but well it's in java.

~~~
icebraining
I don't think there is. Joey Hess ended up having to support inotify, kqueue,
hfsevents and ReadDirectoryChangesW for his git-annex assistant.

------
marcamillion
I have large media files, multiple TBs.

I deal with a constant stream of these and want to have a distributed network
- connected via the inet - that allows me to sync the drives in all locations.

i.e. I would like to setup a server in my home office, one in my co-founder's
home office and another in my editor's home office.

Whenever my editor runs off a few hundred GB of data to a specific folder or
to their drive, I would love for that to be auto-synced to both my server and
that of my co-founder.

Will Syncthing allow me to do this easily and will it be appropriate for an
application like that?

~~~
nodata
It will do what you describe, but since you are handling large files you
should know that Syncthing currently doesn't do dedupe, so if you rename or
move a file, or have multiple copies of it, it will sync the whole file again.
Changes within a file that don't move are handled efficiently.

~~~
marcamillion
If I am not mistaken, you can't do dedupes on media files, right?

------
aw3c2
This looks very promising. But the documentation is not good. I have not
managed to find a "1 minute" friendly overview of how it works. I mean, what
data gets sent how where and why.

~~~
nodata
> But the documentation is not good.

The linked website is aimed at end-users.

Try the protocol docs:
[https://github.com/calmh/syncthing/blob/master/protocol/PROT...](https://github.com/calmh/syncthing/blob/master/protocol/PROTOCOL.md)

~~~
aw3c2
On, I meant end-user friendly documentation. Like ushi said, it is probably
too early for that. Thanks!

~~~
nodata
You could try the discourse site:
[http://discourse.syncthing.net/category/documentation](http://discourse.syncthing.net/category/documentation)

~~~
aw3c2
I did, it drowned me in other things. ;-)

------
rsync
Here is the original from a year or so ago:

[https://raymii.org/s/articles/Set_up_your_own_truly_secure_e...](https://raymii.org/s/articles/Set_up_your_own_truly_secure_encrypted_shared_storage_aka_Dropbox_clone.html)

"Then all current commercial services drop off, including SpiderOak,
Bittorrent Sync and git-annex. This resulted in a clever combination of EncFS
and dvcs-autosync. Because, in this day and age, you cannot trust any "cloud"
provider with your unencrypted data."

~~~
icebraining
The author was uninformed, though. git-annex has supported encryption for
special remotes (including your own rsync.net) since the early beginning (the
wiki page[1] has been there since 2011). And it's certainly open source.

[1] [http://git-annex.branchable.com/encryption/](http://git-
annex.branchable.com/encryption/)

------
doctoboggan
A few months ago I looked into using Syncthing for my decentralized browser,
Syncnet[0]. At that time it did not seem ready for primetime. Does anyone have
a good feel for its maturity as of late? For example, is there an API?
Syncthing looks very promising and I would love to integrate Syncnet with it.

[0]: [http://jack.minardi.org/software/syncnet-a-decentralized-
web...](http://jack.minardi.org/software/syncnet-a-decentralized-web-browser/)

~~~
RachelF
Syncnet looks great!

------
bankim
Alternative would be AeroFS ([https://aerofs.com/](https://aerofs.com/)) which
also does P2P file sync.

~~~
pekk
Kind of a pig for resources though

------
chrisBob
I am very happy with the timemachine backup on my mac, but I have been looking
for a good offsite backup solution so that I can trade storage with my family
in case something happens to my house. This might finally be the right option.
BT Sync seemed ok, but was more than I wanted my parents to try and setup.

~~~
ant512
Try CrashPlan:

[http://www.code42.com/crashplan/](http://www.code42.com/crashplan/)

It's free for peer-to-peer backup and is designed to do exactly what you want.

~~~
slight
Their cloud storage for backups is really cheap if you have a lot to back up
and want them to handle your off-site too.

------
binaryanomaly
Let's hope this becomes what it is promising and reliefs me of dropbox and the
likes... ;)

~~~
clementmiao
I'm curious, what are your complaints about Dropbox? Security, UI, UX,
features?

------
davidjhall
Does this need to use the web gui? I tried setting this up on a digital ocean
server and it spawns off a webserver on 8080 that I can't reach from my
machine. Is there a "headless" mode for client-less servers? Thanks

~~~
jablan
You can edit ~/.config/syncthing/config.xml and restart.

~~~
davidjhall
This helped me ( though I'm sure the ssh tunnel would've worked, I couldn't
get it running right) -- I changed the web address to my ip:port, and could
then log in, add my host, and then change it back to headless or localhost.
Thanks!

------
ReAzem
I would also like to point out
[https://www.syncany.org/](https://www.syncany.org/)

Syncany can work with any backend (like AWS S3) and is encrypted.

It is more of a dropbox replacement while sycnthing is a btsync replacement.

------
nvk
That's great news, have been looking for a OSS sync app for quite some time.

------
interg12
What's wrong with BitTorrent Sync? The fact that it's a company?

~~~
damncabbage
It's closed-source; we can't fix things like the crappy crypto (shared key),
and can't spot backdoors or see what it's actually doing without a lot of
effort.

------
grey-area
This project's aims seem very similar to the earlier camilstore project, also
written in go:

[https://camlistore.org](https://camlistore.org)

Anyone know how it compares?

------
orblivion
Does somebody fund projects like this? Or is it just that the people in charge
of them understand something about UI and marketing? Seems like a nice trend,
if so.

~~~
AdrianoZum
I've wondered this too. Perhaps they charge extra for more features or get
money consulting?

------
desireco42
Since I installed Bittorrent Sync, my need for such software stopped as it
works really well and provides all I need from it.

I couldn't understand quite advantages and why would I replace BTSync, which
BTW, works really well already and does all this nice things. Plus works on my
Phone and Ipad and Nexus.

To clarify one thing, I have home server which obviously hosts BTSync repos
with ample space. Ability to fine-grained share parts of it is invaluable.

------
twosheep
So this may be as good a thread as any to ask for assistance:

My small business is looking for a combined file collaboration / file backup
service that doesn't cost an excessive amount of money (we're a non-profit on
a budget). Is there a good service for this? For example, Dropbox is mainly
for sharing files, whereas Carbonite is mostly for backing up your computer.
Is there a solution for both?

~~~
encoded
SpiderOak does both. Their interface is, IMO, not nearly as good as Dropbox
(I've never used carbonite), but it does check both boxes (backup and
sharing/sync).

~~~
twosheep
Yeah but SpiderOak is $5/user/MONTH, so with 30 employees we're looking at
$1800/year which is just crazy.

------
akumen
We love to through around the words "alternative" and " replacement" it is
neither until it is as easy to use/deploy as X for the average Joe. You know
90% of people out there who would't be able to out the words 'git', 'deploy'
and 'heroku' in the right order as their eyes glaze over in confusion.

------
Joona
I'm looking for a replacement for Dropbox, but it seems that none support
direct links, like in Dropbox's public folder (example:
[https://dl.dropboxusercontent.com/u/38901452/fox2.jpg](https://dl.dropboxusercontent.com/u/38901452/fox2.jpg)
) Is there one?

~~~
Foxboron
The reason is because most of the options; owncloud, btsync, seafile and
syncthing dosn't upload to a unknown company server. It only shares the files
between your devices. Its a huge pluss when you think about security. The
downside is that a device have to be online to share or update the files.

I use BTSync and to avoid the problem i just upload the files to my server and
display them under a subdomain.
[http://pub.volpe.pw/pub/fox_gl.jpg](http://pub.volpe.pw/pub/fox_gl.jpg)

BtSync also offers a phone client, so to mimc the way Dropbox uploads pictures
to dropbox was rather simple. I backed up a gallery on my phone and synced it
to my server and symlinked it under the pub dir, as BTSync dosn't follow
symlinks.

~~~
Joona
I don't mind uploading them under a subdomain or such, but I really like how
easily I can get a public link from Dropbox. How hard was it to set up?

~~~
Foxboron
Once i had setup the subdomain it wasnt hard at all really. I only had a
problem where nginx couldnt serve the files from my home dir (i preffer to
have the syncing folder there), but it was solved quickly with google and SO.

------
dead10ck
This looks very promising. And it's written in Go! The only major feature I
think it's missing is file versioning.

I am curious, though: what do people use to get their files remotely? And
what's the cheapest solution for hosting your own central server? Would a
simple AWS instance work fine?

~~~
Foxboron
I server them to myself using _h5ai (stylized HTTP index) and nginx,
displaying a portion of my files.

I'd also take a look at digitalocean for a VPS.

------
mark_l_watson
I really like the idea but one thing is stopping me: portability to iOS and
Android devices, and mobile apps that work with Dropbox. Dropbox has a first-
mover advantage.

This is mostly a problem for people like me who use both Android and iOS
devices so alternatives need to support both platforms.

------
emsy
Yet another sync app is Pyd.io The Web UI is super neat, and you can choose
between various backends for storage. Pyd.io offers its own sync app which I
found to be horribly slow. I'd suggest to use Pyd.io as a frontend and
BtSync/Seafile/Syncthing as a backend.

------
ertdfgcb
Unrelated, is one of the best open source project landing pages I've ever
seen.

------
biocoder
Have you checked Hive2Hive? Something similar but not yet there.
[https://github.com/Hive2Hive/Hive2Hive](https://github.com/Hive2Hive/Hive2Hive)

~~~
nicoruti
In contrast to Syncthing, Hive2Hive is more a library, providing an API for
any kind of program that requires file synchronization. Another major
difference is that Hive2Hive is written in Java and not in Go.

------
nl
Is there any mobile support?

I use Dropbox pretty frequently to share stuff between mobile devices and
desktops. If Syncthing can't do that it isn't as useful.

~~~
Andrenid
As soon as someone makes a Dropbox alternative that has no servers, has
selective-sync to my iPhone, and is compatible with Keepass to store the
database, then I'm there. I'd happily pay for it, too.

------
haxxorfreak
I don't see a Solaris build on the download page but it's listed next to the
download button on the home page, am I just missing something?

------
Fede_V
This looks incredibly interesting, and I would very much like to move from
Dropbox to something open source. Thanks, will definitely play with it.

------
Lucadg
another alternative: [http://www.filement.com/](http://www.filement.com/) I
don't use it but friends do and are pretty happy with it. From their home
page:

\- Combine devices and cloud services into a single interface. \- Transfer
data between computers, smartphones, tablets and clouds. \- Manage and use
data directly on the device or cloud it is stored.

------
jms703
++ this effort. I'm looking forward to replacing my current BitTorrent Sync
(btsync) setup with Syncthing.

------
Sir_Cmpwn
I would like to see something like this that does not place trust on the
server hosting the files.

~~~
hnha
seafile lets you have encrypted directories.

~~~
nodata
[https://github.com/haiwen/seafile/issues/350](https://github.com/haiwen/seafile/issues/350)

------
scrrr
Is there a paper / doc explaining, how the synchronisation works in detail?

------
chris123
Can we get a "Bitcoin meets Dropbox meets Airbnb" already? Thks :)

------
scragg
I would of liked the name "synctank" better. :)

------
hellbreakslose
Cool, I always like it when things are open source!

------
downstream1960
So its basically pirating, but its saves across all platforms?

~~~
nanawnator
Not really

