
Syncer: fast stateful file/disk data syncer written on Go - stargrave
https://github.com/stargrave/syncer
======
josteink
I assume the title has a typo and that it's supposed to say "written _in_ Go".

With that said, as a user of a tool/library/service, why should I care what
language it's written in?

I want to know if a data-syncing tool is reliable or not. I know rsync is, but
OK that's not always super-duper fast.

I know volume-streaming with btrfs and ZFS works wonders, without the need to
do manual book-keeping of a secondary state and hoping you ALWAYS keep that
secondary state up to date. I doubt this out-performs the FS just doing the
job itself, and it sounds much more prone to user error.

So what does this offer over those? It sounds like it has less features and is
not as reliable. Being written in Go offers me nothing I don't already have.

~~~
stargrave
I answered in another comment on that question. The main point of course is
not a language. ZFS incremental snapshots and its send/receive are great, but
my task is to create binary identical image of hard drive with bootloaders,
partitions and so on. I have got SSD that can breake and slow USB connected
HDD. I want to sync them several times a day and be able to trivially switch
them in the case of failure. ZFS is only a filesystem. I need to prepare
harddrive with bootloaders to use ZFS send/receive. I do not want it. Raw dd
is what is needed, but it is slow -- the only reason this utility is written.

~~~
josteink
> ZFS incremental snapshots and its send/receive are great, but my task is to
> create binary identical image of hard drive with bootloaders, partitions and
> so on.

If you use UEFI, bootloader is just a normal file on a normal (FAT) partition
so no special magic is needed, but fair enough.

ZFS typically handles partitions its own way so no need to sync that.

But really: If you just want to keep to full drives in sync, at block-level so
one can at any point be a stand-in replacement for the other, it sounds like
what you want is RAID0. Isn't that what you want?

And with UEFI you can boot those RAID-volumes, even though they are soft-raid
volumes.

Not saying your need doesn't exist, but that existing solutions which are more
standardized and requires less administration seemingly already exists.

~~~
ghthor
I believe if you raid0 an HDD and an SSD you lose fast writes because the OS
waits for it to be written to all disks in the array.

Maybe depends on the implementation, I I'd imagine it would degrade
performance.

------
kevan
It's late and maybe I missed something looking through the code, but what
happens if you don't have write permissions on the directory that the
statefile is in? The copy operation is performed but then the state is lost
when the statefile rename fails.

~~~
stargrave
At first, temporary file created in current directory. So if directory is not
writable -- temporary file creation at the very beginning will fail.

~~~
kevan
Ah, that's what I missed. I guess there's still a possibility of permissions
changing during the transfer, but I don't think there's anything you'd want to
do to handle that other than error out.

~~~
stargrave
In that case I will print the message telling that something went wrong but
temporary statefile is alive:
[https://github.com/stargrave/syncer/blob/master/syncer.go#L2...](https://github.com/stargrave/syncer/blob/master/syncer.go#L205)

------
tete
Suggestion: Don't set a default for dst. That looks really dangerous.

~~~
stargrave
Agreed. Fixed. Thanks for the suggestion.

------
mercora
A former colleague has written something quite similar[0] few years ago, which
we used to backup encrypted disks (actually snapshots of encrypted disks to be
precise).

[0] [http://chunksync.florz.de/](http://chunksync.florz.de/)

------
gionn
Nice, but in which use cases it may be useful?

~~~
stargrave
I have got fast SSD that can break down every day. I have got USB hard drive.
I want to sync data every day, twice a day between them as fast as possible
(raw dd takes an hour). If SSD is broken then I just plugin USB hard drive and
continue working (with some minor data lost).

ZFS has incremental snapshots, it is fast, but I need to restore bootloader,
partitions and everything corresponding to be able to bootup and work again.

~~~
rocky1138
Scratching your own itch is the correct path to success, in my books. Keep up
the good work.

------
perlpimp
this is great. I wonder how it would perform when I need to sync 10mil. files.
Rsync and anything else breaks down at this point, well it does work but it
takes forever and rsync bloats to 10-20GB memory foot print and after a few
hours starts streaming updates.

~~~
stargrave
syncer works on block level, with raw byte sequences. It knows nothing about
file systems. Everything is limited with sequential read/write speeds of your
hard drives.

It takes blocksize-size memory block for each CPU in your system. I have got 4
CPUs and work with 2 MiB blocks: program will take 8 MiB of RAM.

