Two things I like about it in particular, are
1. It can and favors syncing over local network. This has a huge cost savings in developing world. Even when you do have massive bandwidth, local network sync still has more throughput and lower latency. My music is shared between devices and if I add a track, it takes less than a second to appear on my other devices.
2. You can set conditions, such that delets can be ignored. Eg. I have a WhatsApp message+media backup going back almost 7 years now, ~65GiB. But I don't need all that on my phone. So I just disable syncing delets on my storage. Now my phone can get away with ~3 GiB of whatsapp data (only because I don't delete very often) and I still have complete backup in case I need it.
3. Absolute rock solid stable. At $DAYJOB we use syncthing to sync multi dozen GiBs of new data every day, total file count in 10s of thousands in about ~300 directories. And in past 5 years we have had about 3 instances where we had something that needed attention, out of which 2 were not syncthing's fault.
I've been using Syncthing for years and have absolutely zero complaints against it. Everyone with 2 or more devices should give it a try, just to see there are better options out there.
I would like to know, how are you using Syncthing to create local backups for WhatsApp database/msgstore (crypt14) and media files. You also mention 65GiB as your complete backup ─ have you tested it via a restore? If so, how?
Not the parent poster, but I also do this; I just configured Syncthing to share the /storage/emulated/0/WhatsApp (aka /sdcard/WhatsApp) directory. WhatsApp stores its daily local backup on the Backups subdirectory of that directory, and the media files are all in the Media subdirectory of that directory.
> have you tested it via a restore? If so, how?
I have actually used it to move all the WhatsApp and Signal data from an old phone to a new phone. Just have Syncthing synchronize these directories (with the same path) on the new phone before installing WhatsApp and Signal, then install and launch WhatsApp and Signal. When first launched, if that directory already contains a backup, both WhatsApp and Signal ask if you want to restore from that backup. Signal then asks you to type a long backup encryption key you should have written down somewhere, while WhatsApp asks its servers for the backup encryption key.
But to add, the message store the where the texts are stored. I usually keep it all. WhatsApp doesn't care about the media more than you do, so you can delete anything not needed, and now you have full text message history + important media on your new phone.
I have tested it while migrating across 3 devices and half dozen phone factory restores, so I'm fairly confident it still works. There are very ugly restrictions coming in future Android version with scoped storage and whatnot and I'm worried Google is finally going to cut me out. But until then, I keep mine.
Kubernetes is today's poster child for this. The Hashicorp stack can do just about all the same things, but it just works so there's no cottage industry of consultants and no market for as-a-service implementations. Why promote something like that?
Agree very much. That's why I never let go of a chance to enavgelise Syncthing and its OSS ilk in other fields.
I was evaluating Syncthing for 1-way, append-only phone photo backup, and my Google searches warned me away from this, talking about "unsupported" and "database corruption."
I assume you have not experienced this, but I guess I should have just read the docs and trusted them, instead of forum posts.
Same here. What do you use instead? I haven't been able to find a good open-source multiplatform project for backups/1-way sync.
They had a pretty reasonable (~$6) in-app purchase to unlock samba access, and the fact that it's a German GmbH means I'm slightly more confident they're not trying to copy all our photos to a dark data-vault.
Happy user so far, though it took some cross-checking against the samba share feel confident it was backing up reliably (I'm just particular about trust-but-verify :)
1) I like to minimize the number of APT repositories I use. Debian's and Ubuntu's are well behind on Syncthing versions. Every so often (much more rarely than before), they have breaking changes. As such, my phone can no longer sync with my computers. Perhaps the answer is that Debian should just not be packaging it, because they have such a conservative policy. I'm considering downgrading my phone's version, or building the latest version from source. Worst case I may add their APT repo but I'm not eager.
2) At least with the version I've been using, opting for local-only sync isn't so straightforward. It's per-instance, not per-folder. This means that I have to have my home server running it as local-only, even for small things that I wouldn't mind getting synced over the wire. It means that if I want to have a local-only directory shared between my phone and laptop, it has to go through my home server first. Unless there's a better way I don't know about.
Otherwise Syncthing has been pretty great. I like your no-delete thing, I did not consider that. I could definitely use it for photos.
What’s wrong with adding a first party apt repo for syncthing (or other software)?
I'm synchronizing less than 100 MiB of data, but it's changing all the time. I've not yet had a single issue with it. I've a few conflicts every week when the same file is changed in different locations at the same time, but Syncthing keeps all versions of a file until I resolve the conflicts; I can't see how it could handle that better.
Long story short: I highly recommend this tool.
On Windows, SyncTrayzor is the way to go.
curl -X POST -H "X-API-Key: xxx" "http://localhost:8384/rest/system/reset?folder=YYYY"
Also good to note my setup has 1 introducer which is my NAS (synology) that is 24/7 online which always has the latest changes. Other devices are mostly mobile or turned on at irregular times. For devices without syncthing support like iOS, i rely on exposing the directory on the NAS via webdav. This way i can still access my data on the go.
And my synology creates regular backups/snapshots of my synced directory. So that i can always recover files. Even outside of syncthing.
You can even set an encryption key per share, and if you want to use an untrusted device in your nodes, you can simply not provide it with the encryption key and it will simply synchronize the encrypted payload across nodes.
So if you have a shady/cheapo VPS, you can use its storage without worrying about the plaintext data being stored there.
I normally do all my work on my laptop, but I went ahead and set up sync between my work folders and my personal desktop. Syncing ~1tb of work data has been no problem.
I turn off my computers when I'm not home, and my phone runs syncthing-fork on a limited schedule, so I did at one point have some issues with stuff not syncing because devices weren't turned on. So I set up syncthing on a raspberry pi with a big external hard drive plugged in, and it sits next to the router just quietly synchronizing everything in the background.
It's all very set-it-and-forget-it. Just a great piece of software.
After several attempts, I ended up dividing the SyncThing folders by device storage and level of trust. Simply put, there are things I don't want on my tablet because it has very little storage space, but I still want small things like my todo list and my keypass database there. Then there is stuff I don't mind having on my work laptop, that's mostly stuff like music or personal things that are not sensitive, like desktop wallpapers (so no private stuff I would minder others at work seeing, and no work stuff I would mind being stolen from my other devices). Then there are things I want on my laptops and desktops, but not on my phone, because they're either too big or only useful on a computer, like downloaded software. That's about it, and this way I only have to think about what to put in those folders, but I don't really add new synced folders a whole lot, and when I get a new device, it's self-evident what folders "should" be on it.
FreeFileSync is for stuff I sync manually, e.g. Firefox and Thunderbird profiles, htdocs folder and MySQL database. The only reason is that if I forget to stop Firefox, Thunderbird, or my webserver before starting it elsewhere -- which I should never do, but it can still happen -- and try to sync, I get a much nicer graphical interface to resolve conflicts en masse.
Aren't you worried that a corrupted keepass database will propagate across your devices and that way you'll lose it? I'm having trouble finding an alternative for git in these cases.
And like the other guy said, Syncthing isn't for backups.
I discovered it yesterday and migrated my backup system to FreeFileSync (well, actually I made a double backup with the old and new system). It is _AWESOME_. I finally feel in control of my backups.
And with that having been said, it’s great. It’s maybe more powerful than you’d expect. My NAS for example has a one-way synced folder from a different box for backup, while having a default shared folder with all of my PCs. It’s also great because not all machines have to always be online; it can gracefully handle deferring stuff. For me, since my NAS is connected all the time, I can use it a bit like a central service, syncing between machines that have no overlap in uptime. Simultaneously, if the NAS needs to be down, it doesn’t interrupt syncing between other nodes. And unlike centralized solutions like Dropbox, I’m not limited by my internet connection, since it’s all local... unless I leave the house. In which case Syncthing appears to continue to stay connected, which is really handy.
I don't remember when was the last time I had to use a flash drive or a cable to move files around.
 Android. iOS is too restrictive to do this.
I mostly use it to keep Windows' Document folders in sync across various machines (through a Linux server). Windows apps still put files in lots of random places but most of the important stuff is in Documents these days. I also use it to sync GPS traces from GPSLogger on an Android phone to my server. The Android app works great!
The setup of a new node is more complicated than it should be, that's the one product weakness.
A compromise of the centralized release process could steal all of your (and everyone else's!) files by updating to a malicious version automatically with zero interaction.
Set the syncthing binary's file ownership as root and run it as a normal user so it can't get overwritten, and set STNOUPGRADE=1 in the environment to disable this dangerous default behavior.
Syncthing: Untrusted (Encrypted) Devices - https://news.ycombinator.com/item?id=27702785 - July 2021 (5 comments)
Open Source Continuous File Synchronization - https://news.ycombinator.com/item?id=27149002 - May 2021 (146 comments)
Syncthing Untrusted Device Encryption - https://news.ycombinator.com/item?id=26424096 - March 2021 (1 comment)
Syncthing is everything I used to love about computers - https://news.ycombinator.com/item?id=23537243 - June 2020 (159 comments)
Do not use Syncthing (2019) - https://news.ycombinator.com/item?id=23116462 - May 2020 (1 comment)
Emacs' Org-Mode and Syncthing = Perfect (2017) - https://news.ycombinator.com/item?id=23058358 - May 2020 (77 comments)
Syncthing: An open source Dropbox replacement - https://news.ycombinator.com/item?id=20466469 - July 2019 (41 comments)
Syncthing graduation day - https://news.ycombinator.com/item?id=18832517 - Jan 2019 (114 comments)
Syncthing Usage Data - https://news.ycombinator.com/item?id=13856552 - March 2017 (119 comments)
Syncthing - https://news.ycombinator.com/item?id=10331031 - Oct 2015 (1 comment)
Syncthing: Open Source Dropbox and BitTorrent Sync Replacement - https://news.ycombinator.com/item?id=7734114 - May 2014 (184 comments)
EDIT: added link
I’ve gone through so many iterations of trying to work out how to do something like this for Home Assistant config.
Do you set it up as realtime 2-way sync? And do you just exclude the .git folder from the sync to HA?
I should note I specifically use this for doing source control of configuration and making it easy to manually edit -- I have an entirely separate process for backups of the server itself.
Like you, I've also gone through several iterations but this has been working for me for almost a year now.
That said, I’m making far fewer YAML changes these days than I used to. I’m doing almost everything via their UI so maybe I need to consider how much I actually need source control now vs just having solid backups.
Originally I was against their move from YAML config to the more user friendly GUI approach but I have to admit it’s a lot easier to make quick on-the-fly changes to my automations these days. I’m begrudgingly coming around to the idea of not firing up VS Code every time I want to tweak something!
That said, I can't recall if I tried excluding node_modules when I last gave this a go. Maybe I'll try it again.
I've read of other HN'ers doing this successfully (syncing git repos between machines) but I just seem to get a lot of issues - git repos often broken temporarily, conflicts etc.
Folder Type: Send Only
Working without a vpn and xmpp is a no-go for me, so I have to sacrifice Syncthing, running it only from time to time.
EDIT: This is a more recent feature so you might need the last one or two versions of Android, but if you have a phone with a notch I bet you're good :)
edit: seems to support live photos and exif metadata… I have a feeling some formats or metadata might be lost though, but not entirely sure…
* iphone -> icloud
* icloud -> mac
* mac -> syncthing to the backup server
Standard backup policies from that point on.
I just found out about this tool from this (recent) thread:
In order to do that, I:
- Set a folders in a device to "send only".
- Set a folder in a different device to "receive only".
- Set the "Ignore delete" advanced option for the desired folders on both devices.
And that's it! I use it mainly to do some manual processing of the received photos and videos with Shotwell (copying and sorting my library into Year/Month/Day directories).
You may run into some issues where you may be promoted to "revert local additions" (I think on the receiver's end)... This mainly happens for two reasons:
- if you deleted a file on the receiver's "sync only once" folder
- if you use a single folder where there may be other incoming files from other synced folders.
It is best to avoid the latter scenario; a quick fix for the first one is to make sure to delete the media first in the sender's device's folder and afterwards delete it on the receiver's device's folder.
After that, you can click "revert local additions", and the message should go away.
Fast forward several years, I needed to restore something (non-critical) and it was a giant pain in the ass. I was able to get it done, but it wasn't immediately clear that I would be able to. Something about CrashPlan not retaining deleted files and some of my paths changing or something.
Anyway, I was so frustrated by the whole experience that I dumped CrashPlan.
I run Syncthing to provide file sync services. A Raspberry Pi acts as the "central server" on my home network. Once a night, an EC2 instance launches, syncs the data up, and then performs an offline backup to S3. I get a report e-mailed to me every night so I can keep tabs on stuff.
I did have to spend time building it out, but it costs something like $5 per month and it's been bulletproof.
Lots of love for Syncthing over here. It's been a champ.
I tried to use it to sync SASS and webpack between developers and it didn't work, it seems to have an issue with node_modules has too many files for it to deal with and it thinks the directory is hundreds of gb and just stalls with a million years eta.
The versioning is cool too but I found it a bit flakey.
If you need reliable versioning and syncing of lots of files use GIT, if you need easy cross platform syncing syncthing is the way to go.
Also it really ate up CPU/memory when I had a lot of files going. It was among the worst "spin up the laptop fans at idle" offenders, even more so than equivalent options like onedrive.
That being said it was really great for a subset of my use case and the problems I had with it border more on problems with the model rather than problems with the tool.
I am looking for an alternative that helps bridge the gap that is Google File Stream where even in windows it mounts as a drive but streams everything in the background as needed. Plex seems like a clunky way to go about it but I am open to any suggestions. The file stream feature was mainly used to stream videos in an uncompressed way.
I mainly do local sync between android and windows and a rpi. I would love to have file streaming type support for the android and maybe an ios device.
Android has a way to do file streaming built into some of the cloud manager apps but a diy solution might not have remote access.
I am open to any suggestions, I understand that syncthing is about p2p syncing and the model is mainly for whole file copy.
My only annoyance with Syncthing is when I reinstall an OS on one of my machines. Let's say I'm syncing files between my main workstation, two laptops, a phone, and a cloud backup. Now I want to reinstall the OS on one of the laptops. When I install syncthing on that machine, it gets a new ID. I can make it join the sync swarm but all the other members will think it's a new machine and won't trust it so I have to go to each of the other machines and manually remove the "old" laptop and bring in the "new" one.
I simply save the syncthing config and remove the database, so that the device stays the same, but doesn't retain any sync data
~/.config/syncthing in Linux, the pem files and config.xml. Similarly in Windows too (from C:\Users\<username>\AppData\Local\Syncthing).
Even in Android, you can export and back-up. When you are switching to new device, copy from the back-up and then import.
One thing I'd like to add is, that syncthing is also configurable in detail. I use it to sync data between a workstation behind restricted network and my laptops (most of $HOME). For this I use ssh-tunnels between the hosts. So syncthing's public discovery would not be necessary. I can directly set the tcp endpoints for syncthing so it find its communictation partners. Nice!
Anyone here who decided to try to sync files at TB scale?
And what about file holes? AFAIK syncthing filles them with zeros. Is this still the case?
I’ve gone through various attempts at keeping things in sync across a couple of Macs but always seem to hit issues.. (the issues not being specific to Syncthing)
See e.g. https://news.ycombinator.com/item?id=20955465 ; https://goteleport.com/blog/ssh-certificates ; and the usual man pages
I don't care much for cloud providers adding unnecessary restrictions, so I manage my own sshd_config on VMs. I'd like to rent a flat, not a hotel room, thank you very much.
> Git hosting platform
It appears GitLab, one of my platforms of choice, does support this in self-hosted setups. Even if they didn't, it'd be trivial to extend their sshd_config by myself anyway.
This is interesting about GitLab, thanks for the pointer. As you mentioned it only appears to be configurable instance-wide, there is no way for a user or organization to set it up for themselves (for example for your gitlab.com organization). That makes it a bad replacement for SyncThing'd private keys as moviuro suggested.
I carried around my 1GB Data Traveler flash drive with great conviction. Then, a massive 4G SanDisk drive I would 'never fill up'.
Then, when my 16GB Nexus 4 filled up, one day while I was MTPing my old pictures to my computer, someone told me about Dropbox and how it would sync my pictures for me. After living out of Dropbox for all my non-cold-storage files...I hit the free limit. But I was a student and didn't have money to pay per month.
Skydrive gave me 25 GB. And 3 successive phones ate that like candy. So start paying up Mister. ugh.
But then, Syncthing came along here on HN. I installed it onto my phone and pc for camera download. then another, and another, and a separate folder for sharing files with a colleague, then my work SAN got laggy and lost files, so syncthing to keep my work files local and synced between work devices to the rescue.
And now, I don't pay for storage. or I do, but it just happens to come with word 365, but I don't bother using it.
And the flash drives? I'm up to a 256GB Sandisk Pro of some kind, but only for 'just in case' scenarios. Syncthing handles all of my LAN files access needs.
Someone indicated they did have issues if you do a case only rename between a case sensitive (Linux) system and a case insensitive (Windows/Mac) one. I've never done this myself, but I guess that's one to run into. Also the usual caveats with any program, like that you won't be able to sync con.py to a Windows system, regardless of which program you use.
So for you-to-you it’s seamless but if you want to sync shared-user directories then you lose most info.
I was really annoyed with it until I realized, it's actually OSX's fault, because I renamed a folder from uppercase to lowercase, and there's no good way for syncthing to handle that when you have case sensitive servers syncing with insensitive ones. (Linux and OSX in my case.)
Moral of the story is, don't rename a file only changing it's case-- add a dumb character or something so it can sync to more civilized servers properly.
It's also fantastic for backups from various VMs amd backing up photos from phones to my NAS, etc.
Now moved to a more powerful machine, but the concept is still the same.
This setup allowed me to conveniently stop using google drive.
I used to run Syncthing on an old mac mini with two 2.5" HDD running FreeNAS (previous name of TrueNAS).
Haven't tried it while out-and-about, though that's not really my use case.
*RPi4 NAS has one external thumb drive for flash storage and two spinning rust drives, one for storage, one for backup; all connected via USB. Occasionally I have to plug-unplug the spinning rust drives if the system reboots, as the drives go to sleep.
I finally got rid of it because of the awful mobile experience, but frankly I shouldn't have been counting on that anyway.
What problems were you experiencing on mobile?
Check it out every now and then to make sure, but it just kinda works
(Anecdata I know).
It works, but it's an hack...
It's not a backup solution though... you really want to use actual backup software.
On Android 10 and older, Syncthing can only "read" form SD cards (read/write works if you're rooted).
In the newest versions of Android you give it Scoped Storage.
In the intermediate, if you want 2-way sync on your phone, your data has to live in sdcard/android/media/com.nutomic.syncthingandroid/<sync folder1 >, <sync folder 2> which it does not need root to write to.
But for just grabbing your pictures, Android 10 has the stupid limitations on external file system access that Syncthing has to abide by or root over.