
Arq 5: Massively faster backup and restore for Mac and Windows - hemancuso
https://www.arqbackup.com/blog/arq-5-massively-faster-backup-and-restore/
======
ghshephard
Arq, (For backups to AWS - though obviously supports every cloud back end
under the sun) and "Data Backup" by ProSoft engineering (For backups to USB)
are my goto backup tools for day-day ensuring all my work documents are kept
up to date.

Yes, I have Crashplan (for the last couple years, backblaze for the three
years before that) - but the constant chewing up of CPU cycles gets annoying
after a while. And both crashplan/backblaze "everything for $5" come with
massive caveats (like deleting backups of Hard Drives that haven't been
plugged in for 6 months - I've got Arq Backups of Hard Drives that I haven't
plugged in for a couple years, safe and sound) - and I've never had an AWS
backup bill in excess of $3.00, ARQ does a wicked good job of keeping your
backups on a tight budget.

Also - awesome win for ARQ - when I moved to Singapore, I simply added a AWS
Singapore S3 Bucket and _wowza_ \- fast backups on my gigabit ($49/month) link
from MyRepublic. Really feel like I'm living in the future.

I think once I switch away from Aperture over to Photos, which presumably has
a rock solid backup to iCloud photos, then simply doing a quarterly backup or
so with CarbonCopyCloner + Arq to AWS + DataBackup to USB key will have my OS
X backups covered.

~~~
statictype
_like deleting backups of Hard Drives that haven 't been plugged in for 6
months_

Wait - what?

If your hard disk gets lost or destroyed, how do you protect those backups?

~~~
0xCMP
You need to keep them connected periodically otherwise if you lose it without
reconnecting... _lost_.

For backblaze it's even less time.

~~~
beernutz
I believe for CrashPlan, this ONLY applies to CLOUD backups. If you backup to
local media or across computers, there is no "auto-delete" timeframe.

~~~
jarrodatCode42
Hello Everyone,

Wanted to jump in here to confirm.

This policy only affects devices that have not connected to CrashPlan Central
in 6 months or longer. This does not affect volumes that have not connected to
the device in that period of time. (i.e. an external hard drive that has not
connected in 6 months.) Additionally, there is no minimum connection time for
local CrashPlan backups.

It’s important for CrashPlan users to consistently connect their device(s).
Part of CrashPlan’s ability to maintain the archive health and integrity
relies upon regular connection from the device. CrashPlan is able to routinely
perform maintenance on the archive by comparing checksums between both device
and CrashPlan Central.

[https://support.code42.com/Administrator/3/Monitoring_And_Ma...](https://support.code42.com/Administrator/3/Monitoring_And_Managing/Archive_Maintenance)

Please let me know if I can provide additional clarity.

Best regards,

Jarrod

~~~
jonaswouters
Is this also the case for crashplan business or PROe? (ceejay in europe).

~~~
jarrodatCode42
Since the infrastructure for CrashPlan's backup engine is the same between our
Business/Consumer clients, we recommend that all users routinely connect their
devices to the backup destinations. That being said, this policy only affects
CrashPlan for Home subscribers at this time.

------
jwr
Arq doesn't get the publicity it deserves. It's a reliable, provider-
independent backup solution that YOU control. Data gets encrypted locally,
then sent over to storage providers. When new storage providers appear, Arq
implements APIs and lets you use them.

Most importantly, when restoring, you don't enter your decryption password/key
into a browser window. I don't understand how online-backup companies can talk
about security while requiring users to give them their passwords in order to
restore data.

I've been using Arq for about two years now and I'm very happy with it. For
the reference, I have previous experience with CrashPlan and Backblaze.

~~~
tomaskafka
Can you compare CrashPlan and Arq performance please?

I use self-hosted CrashPlan for several years, it's great, but restores and
thinning archives feel slow (like half a day for cca 1 TB of backups).

~~~
jwr
I had nothing but performance problems with CrashPlan, so Arq was a
significant improvement. But that's setting the bar low.

------
CameronBanga
Been an Arq user for years now, and recommend it to anyone. Love being able to
choose my own storage solution, control encryption keys, etc.

For Arq team (saw one or two here), is B2 on the roadmap?

~~~
skrause
Arq's author has said on Twitter that the B2 API doesn't support all commands
Arq needs, so even if B2 support was planned (which it isn't) it wouldn't be
possible yet:
[https://twitter.com/arqbackup/status/717756616578301952](https://twitter.com/arqbackup/status/717756616578301952)

~~~
atYevP
Yev from Backblaze -> We're working on a few APIs and some should be aligned
with what ARQ is looking for. We're hoping that they do integrate with us,
giving folks an inexpensive alternative!

------
amelius
How is this, from a technical point of view, more advanced than running rsync
in a regular fashion (with hardlinks to keep cheap snapshots)?

Just wondering about the technical aspects.

~~~
pwenzel
One differentiating factor is that rsync doesn't work with storage services
like Amazon S3, Glacier, Google Drive, and the like.

~~~
gcb0
which is a good thing.

if a few years ago somebody said "just copy all your private data to some
place on the internet... but encrypt it first" you would take it for the crazy
argument it is.

~~~
falcolas
What do you think is crazy about it? If it's encrypted, it can't be read. If
it's on "some place on the internet", it can only be as redundant or more
redundant as any other offsite solution.

~~~
GrinningFool
> "If it's encrypted, it can't be read. "

Yet. It can't be read yet.

~~~
falcolas
Any agent capable of ever reading AES encrypted files with 256 bit keys is
also an agent capable of opening your safe deposit box, requesting your tapes
from a data vault provider, or coming over to your house with a rubber hose.

Short of a vulnerability in AES (in which case we have more problems than a
few copies of the Anarchists Handbook in our backup files), cracking proper
encryption is simply not feasible.

~~~
GrinningFool
What about in 10 years, 20, 50?

------
ejdyksen
Mirror:

[https://gist.github.com/15dde3fb013581b6aa59bf0b0c3a701e](https://gist.github.com/15dde3fb013581b6aa59bf0b0c3a701e)

~~~
njade
thanks

------
DenisM
Given the rise of ransomware in the recent months, what protection does Arq
offer against that? That thing alone would easily tip the scales for me versus
the Time Machine.

I see that the AWS S3 IAM user has both read and write access, so if the
ransomware authors ever bother with it, they can kill the backups.

Would that help if I setup versioning on the bucket? Will Arq be able to
restore backups from the older version of data, before the attack takes place?

Any other ideas?

~~~
philsnow
Bucket versioning should help with this. I don't use Arq and don't know if it
supports s3 bucket versioning, so it might not be convenient but the data
would still exist.

Although if an attacker has control writing to your s3 bucket, they could rack
up a big bill.

~~~
DenisM
There is no monetary incentive to rack up the bill, but there is an incentive
to kill backups - ransom demands are a lot more persuasive when the victim has
no backup.

------
philsnow
I would just like to point out that this is major release 5 and they're just
now adding threading and consumption of filesystem events.

That's _great_ from the standpoint of launching a product. Putting off adding
this complexity probably let them get to market sooner.

If I were releasing something like Arq, I would have to fight myself very,
very hard to not add these to the 0.1 release. I don't know this space very
well, but maybe there are several Arq-alikes who started earlier, but didn't
release until later because it wasn't "done" yet, and they missed their
chance.

~~~
ghshephard
But ARQ has always been fast, and figured out what it needed to backup pretty
much instantly, and just did it's job and got out of the way - unlike things
like spotlight/mdworker, or crashplan/backblaze, that are constantly thrashing
my CPU and causing my fans to spin up.

From my uneducated perspective, having simple software that just _worked_ was
a bonus - who knows, maybe with the addition of threading, and consumption of
filesystem events, ARQ is going to become crummy, and a door will be opened
for someone else to write simple backup software _without those features_ that
gets the job done and doesn't bog up your computer. I guess we'll have to try
Arq 5 for a few weeks to find out. Fingers Crossed.

------
nihonde
I have nothing to add except that I've been a happy customer of Arq for years,
and Stefan and team have provided us very helpful and personal support by
email on the rare occasions when we needed it.

------
kylefox
I've been using Arq for years and absolutely love it. Worth every penny. It is
extremely well-built software — it's FAST, doesn't hog resources, and feels
very polished & reliable.

I like that I can backup to multiple destinations (AWS S3/Glacier, Dropbox,
Google Drive, even my own server via SFTP). IMO you can never have too many
backups.

I use it along with Backblaze (and will be setting up Time Machine & Super
Duper or Carbon Copy Cloner this week, after putting it off forever).

Congrats to the Haystack team!

------
sreitshamer
Bah, I guess hosting a site at digitalocean isn't suitable for HN front page?
Even WP super cache isn't saving me.

~~~
sneak
Stop using wordpress.

~~~
falcolas
At a minimum, add some caching in front of WordPress. Even a 1 minute cache of
pages will result in being able to host your WordPress site just about
anywhere. Throw in a CDN, and you could _almost_ host it on a Raspberry Pi and
throw Reddit at it.

------
matrixagent
Awesome, can't wait to upgrade tonight! Any news on the Linux / CLI version?

~~~
AnthonBerg
PLEASE give us a Linux version!

~~~
AnthonBerg
Can I preorder or sponsor a Linux version somewhere?

~~~
arqq
+1 :)

------
blue1
If I understand well, Arq does not backup the whole computer. I am looking for
a tool that allows for recovery of single files, but also that has a backup of
the whole machine, so that if the HD crashes it is possible to rebuild it
verbatim. What does HN suggest for such a tool? I used Norton Ghost for years
(but it has been discontinued).

~~~
gcb0
for disk deaths I'd just settle with raid. life should be simple.

~~~
wrboyce
Not always feasible (laptops, macs).

~~~
gcb0
my netbook from 2003 accepts two ssd (Asus eeepc 1000). you should stop
getting work equipment because of looks or peer pressure.

------
huhtenberg
Stefan, can you disclose how many people are working on the Arq now?

For some reason I always had an impression that you were impressively managing
the whole thing just by yourself.

------
larrysalibra
Love arq too. Rumor has it arq 5 will also stop eating up free space with its
cache.noindex folder. Excited about that!

~~~
sreitshamer
True! It uses sqlite databases (finally). (I work at Haystack Software :)

~~~
arqq
This update is great. Very happy I bought Arq :).

I was excluding cache.noindex from my other backup system (carbon copy
cloner). Is it still safe to do so?

~~~
sreitshamer
Yes

------
roddux
Google cache link:
[https://webcache.googleusercontent.com/search?q=cache:dy3xEV...](https://webcache.googleusercontent.com/search?q=cache:dy3xEVo6yiwJ:https://www.arqbackup.com/blog/arq-5-massively-
faster-backup-and-restore/)

------
victorhooi
Hmm, there's nothing on the announcement about how to upgrade your existing
backups?

Say you have a large backup stored on S3 or Google Drive - has anybody used
this, and can tell me if it upgrades it seamlessly to take advantage of the
new features (e.g. LZ4 compression), or if you need to do a fresh upgrade?

~~~
igas
Yeah, I also curious about this one.

------
m_eiman
I like Arq, use it as a "real, offsite" backup complement to Time Machine. Not
quite as fire-and-forget as Time Machine, but darn close!

And the format of the stored data is available, which is a nice safety feature
in case of major problems.

------
enoch_r
What compression was used before Arq 5? lz4 is super fast, but not
particularly space-efficient compared to some slower compression algorithms.
Since Arq customers are the ones paying the storage bills, this doesn't seem
like an _entirely_ costless decision--Arq is now faster, but you should expect
your storage bills to go up a bit due to the lower compression.

I use Borg backup with lz4 compression, so I definitely don't think this is
the _wrong_ decision, just something to keep in mind (and, it does seem like
something that could and maybe should be user-configurable).

~~~
skrause
Arq's data format description says that the other supported compression type
is gzip, which I assume was used before:
[https://www.arqbackup.com/arq_data_format.txt](https://www.arqbackup.com/arq_data_format.txt)

------
skrause
Still no support for S3's "Infrequent Access" storage class? It would be
perfect for backups and is a good compromise between normal S3 and Glacier.

~~~
lobster_johnson
It does support that:
[http://i.imgur.com/EC1e83h.jpg](http://i.imgur.com/EC1e83h.jpg).

~~~
skrause
Great! It should have been mentioned in the announcement because I've been
waiting for this option a long time.

------
flurpitude
Has anyone tried both this and Crashplan? My Crashplan 10-computer
subscription comes up for renewal soon so I'd be interested in people's
experiences of how they compare. Generally I've found Crashplan to be pretty
good. I like the way it backs up in frequent small increments and I like the
backup destination options (cloud, folder, other PC, etc.). Does Arq compare
well?

~~~
cpach
Arq can’t send backups directly to a hard drive, but it can use an SSH (SFTP)
server as a destination.

[Edit: Apparently the release notes for Arq 5 says it can now back up to a
local folder.]

I have never tried Crashplan though. The main reason I chose Arq instead is
the impression that Arq has a better thought-out encryption scheme.

------
claar
It appears Arq's being destroyed by HN currently -- I downloaded the trial for
Mac, but when I press "Start Trial" I get "Failed to create trial - A server
with the specified hostname could not be found." I'll try again tomorrow.

------
kayoone
I use Arq to backup all my stuff hourly to my NAS at home through SFTP and the
NAS then backups the most important stuff to AWS daily. That works really well
and keeps the cost really low because i need the NAS anyway for streaming
media an such.

------
fasteo
I am a happy user of duplicity [1]. Discussed here [2]. Some comparison with
Arq here [3]. It is more oriented towards servers and system administrators,
but works pretty well and has a similar feature set.

[1] [http://duplicity.nongnu.org/](http://duplicity.nongnu.org/)

[2]
[https://news.ycombinator.com/item?id=6712244](https://news.ycombinator.com/item?id=6712244)

[3]
[https://news.ycombinator.com/item?id=6712411](https://news.ycombinator.com/item?id=6712411)

------
jibsen
The Mac version has gotten a lot of good response, so I tried installing the
Windows version. It feels a bit like a beta sadly.

    
    
      - Backups to local folder do not work
      - Scheduled backups don't always run until you open the GUI
      - Open log viewer before any backups have been performed crashes the app
      - File-Exit does nothing
      - Wizard adds whole C:\ to first destination
      - Doesn't seem to backup (all) locked files
    

I've reported all of these, and the author seems responsive, so I hope it gets
better.

------
overcast
Awesome, got an email about an hour ago stating I received a free upgrade to 5
from my 4 license. Been using Arq for a couple years now, it's been a perfect
solution for offsite backups.

~~~
sreitshamer
We upgraded all Arq 4 purchases in the prior 90 days to Arq 5 for free. We
even converted the per-computer licenses to per-user licenses, so if you
bought licenses for 3 computers, you now have Arq 5 licenses for 3 users.

~~~
overcast
Fantastic!

------
ancarda
Off topic: Is anyone else getting an SSL error trying to connect? This page
opens fine on my Mac (OS X 10.11.4, Safari 9.1) but my iPhone (iOS 9.3.1) asks
me to trust the certificate (ssl376366.cloudflaressl.com)

Edit: Screenshots: [https://imgur.com/a/wdfDt](https://imgur.com/a/wdfDt)

Also, this appears to not happen on Wi-Fi. Willing to send any information I
can gleam from this -- I'd be interested to know if this is a MITM attack or a
mistake on CloudFlare's side.

------
victorhooi
Argh, the website is down...lol:

"Error establishing a database connection"

~~~
stronglikedan
It appears that there may be an issue with just their blog. The main site[0],
and most of its links, works for me.

[0] [https://www.arqbackup.com/](https://www.arqbackup.com/)

~~~
vog
At the moment, this site takes ~30 seconds for every single request. But at
least it works at all.

------
coffeecheque
I was a user of Arq 3, but it was slow and seemed to eat up resources on my
computer. I eventually turned it off, deleted the backups and deleted it. Not
a great backup plan.

So I'm keen to know how much faster Arq 4 was, and in turn Arq 5. I'd be happy
to try again (I think I'll have to pay full price again as I'm not an Arq 4
user) - but might wait until someone can let me know just how much faster it
really is.

This whole thread has also reminded me to run my backups to local Time
Machine!

------
bhauer
This sounds great. To my mind, it seems an approximation of Tarsnap but on
Windows.

Is it possible to use Arq to backup to network connected disks via Windows
SMB? In other words, is it necessary for me to use one of the supported cloud
providers or can I just use one of my own servers as a destination? Similarly,
is it possible to setup multiple redundant backup destinations (e.g., S3 and
Google Drive, or S3 and my own servers)?

~~~
skrause
The release notes at
[https://www.arqbackup.com/download/arq5_release_notes.html](https://www.arqbackup.com/download/arq5_release_notes.html)
mention that you can now backup to a local directory or network drive. It's
weird that the most interesting features were not actually mentioned in the
main announcement.

~~~
bhauer
Fantastic. I think I'm going to give Arq a trial and see how it works for my
use-case.

------
mwexler
Didn't JungleDisk ([http://jungledisk.com](http://jungledisk.com)) kind of
start this "backup to AWS, control your own data" movement? Does anyone use
them anymore? I recall that they started as pay one price, then switched to a
subscription model too...

I wound up on Crashplan, though it's not perfect either.

------
leejoramo
One long standing issue I have with Arq is that there is no easy way to
duplicate backup selections. I would love to be able to easily copy the
selections between different destinations, or different systems that are being
backed up.

Several years ago, I tried to manually edit the plist preference files, but
that was painful and unsustainable hack.

------
riprowan
Thanks for the tip. I'm checking out Arq now!

FWIW I've been a big fan of Syncovery (formerly SuperFlexible File
Synchronizer) for years. It's a Swiss Army Knife of backup/recovery.
[https://www.syncovery.com/](https://www.syncovery.com/)

~~~
ghshephard
Syncovery looks pretty awesome for what it does, I just installed it to take a
look, - Directory Synchronizaton - but it's achilles heel for file backups (as
opposed to directory/disk cloning) - is appears to be it's lack of any type of
versioning. I.E. If I make changes to a file each day, and do backups each day
- no way to go back to a version of the file a couple weeks ago?

~~~
riprowan
It has a versioning feature IIRC but I can't vouch for how sexy it is.

------
jonfle
Been using Arq since version 2 (2010) on multiple machines. Very happy with
Arq and with Stefan's support. I use it with dual destinations for my offsite
backup - about 9Tb. I also use it to provide family in another country with a
viable offsite backup. Restores just work.

------
c0nsumer
Any of you know how to get your license code for an upgrade? I have the
original email from when I bought Arq 4 in 2014, and it contains a license
file. This is an XML with sections like 'id' and 'licenseKey' but neither of
those are working.

~~~
cpach
I don’t know, but why not just contact the author? He’s very pleasant to deal
with in my experience.

~~~
c0nsumer
To add, they sorted it out by adding an option on the site to retrieve the
license code. I've now got an Arq 5 license and lifetime upgrades.

I think that having Arq back up to spare space at a Linode VM is a great way
to use it.

------
lobster_johnson
As an Arq user, this makes me happy, as Arq is not exactly known for being
fast.

The announcement doesn't say, but I'm hoping they have reduced the amount of
space needed for the client-side cache (currently 18GB (!) on my laptop).

~~~
sreitshamer
Yes. I just added that to the announcement post. It stores stuff in sqlite
databases that are a lot smaller.

~~~
lobster_johnson
It went from 18GB to 600MB! That's awesome, thanks!

------
victorhooi
Trying to buy my upgrade now - but it keeps telling me my Arq 4 license key is
invalid.

I've tried both the licenseKey field, and the hash field from the license XML
file I got when I bought Arq 4.

Has anybody else had success buying an upgrade license?

------
hsshah
Can I use Arq to backup disks connected to my airport extreme router? I don't
have a desktop; will be installing it on my laptop. Can I schedule when to
backup ?

------
frankhofmann
been a user since arq 2. It's the best.

Question to any knowledgeable folks: Why does the tool cause so much download
in normal operation ? (i.e. no restoring of files)

I have about 60gb backed up at rest, and am on a backup-every-2h-cycle. I
generate about 6 GB of download traffic each month. So funnily enough, the
traffic costs me more than the actual storage :) mind you it's still next to
nothing, but I found it just curious.

------
reiichiroh
I guess those people with less complicated needs could use
[https://cryptomator.org/](https://cryptomator.org/) to upload the $60/year
Amazon Cloud Drive.

------
krzrak
$50/year and you have to provide your own storage? Can somebody explain to me
how they are competitive, i.e. in comparison with Crashplan or Backblaze?

edit: my bad, it's $50 for one time purchase, not a year.

~~~
jaxondu
Its $50 one time purchase for new purchase. The upgrade from Arq 4 is $25.
During store checkout, you can also choose to add a "Lifetime Upgrades" at $30
which will give you free upgrades for Arq 6, 7, 8, 9, 10, 11, 12 etc.

~~~
Jenkins2000
I purchased the Lifetime Upgrades in September 2015. Now it tells me there is
a $25 upgrade fee to go to version 5. I emailed support a few hours ago to
confirm. If this is true, I am extremely disappointed and feel cheated. Notice
on the form to buy a new license it doesn't say anything about "Lifetime
Upgrades for this major version _only_ which will be obsolete in a few
months". I sure hope this is just a bug that they will fix.

~~~
leejoramo
There is an apology for the delay on twitter. Looks like your license for Arq
5 should be coming.

[https://twitter.com/arqbackup/status/719924424263249920](https://twitter.com/arqbackup/status/719924424263249920)

~~~
Jenkins2000
Thanks! My faith in them has been restored :)

------
yoz-y
I find it a bit ironic that this particular blog has an inaccessible database
at this particular time.

This being put aside I find their pricing a bit excessive in comparison to
Backblaze for example.

~~~
LVB
How much data are you backing up? The compression and deduplication seem
pretty good in Arq. My monthly AWS (and now Google Nearline) bill is usually
around $2, so I hit the breakeven point vs the $5/month services pretty
quickly.

~~~
yoz-y
My current backup is around 300 Gigs. I see the big advantage of Arq (over
Backblaze) is the fact that you can choose your host. However their own
hosting solution for $10/mo with a 250GB data cap is quite expensive. I can
see that for smaller amounts of data Arq is more interesting, especially since
it also stores history (which Backblaze does not).

~~~
chipotle_coyote
It's worth noting that Arq can back up to Amazon Cloud Drive, which offers
unlimited storage for $59.99 a year. As soon as they offered that I switched
from Glacier storage; Cloud Driver is faster, cheaper and less fiddly.

~~~
mobiuscog
the pricing is great for the US... not so good for the rest of the world (for
example:
[https://www.amazon.co.uk/clouddrive/pricing?ref_=cd_home_pla...](https://www.amazon.co.uk/clouddrive/pricing?ref_=cd_home_plans_pricing))

~~~
cpach
That’s weird. I live in Sweden and I was able to sign up for the $59.99
unlimited plan just by surfing to
[https://www.amazon.com/clouddrive/home](https://www.amazon.com/clouddrive/home)

------
brightball
This. Looks. Awesome.

------
PaulHoule
"Error establishing a database connection"

~~~
sreitshamer
Fixed now. Sorry. The load from HN was too much for a while.

~~~
reiichiroh
Can't start a trial account.

Failed to activate. The remote name could not be resolved. store.arqbackup.com

