
A snapshot of your computer with dd, pv and gzip - Part 1 - adito
http://allgood38.github.io/a-snapshot-of-your-computer-with-dd-pv-and-gzip-part-1.html
======
networked
Consider using Clonezilla [1] instead. While dd will produce a sector-by-
sector image of your medium (which can be terribly inefficient if you're using
considerably less than 100% of its available space) Clonezilla is file system-
aware for a number of common file systems and has built-in support for
compression as well. It resembles the recently discontinued Norton Ghost this
way but is fully FOSS.

I normally run it off a live CD/USB of Parted Magic [2], another tool I can
wholeheartedly recommend.

[1] [http://clonezilla.org/](http://clonezilla.org/)

[2] [http://partedmagic.com/doku.php](http://partedmagic.com/doku.php)

~~~
res0nat0r
I'd also recommend SysRescueCD that has these types of tools ready to be used
as a bootable USB/CDrom image.

[http://www.sysresccd.org/System-tools](http://www.sysresccd.org/System-tools)

I've used the imaging tools on the boot cd a few times to clone/restore hosts
a few times without any issues. Very handy.

~~~
WestCoastJustin
I created a SystemRescueCd screencast a while ago if anyone is interested [1].
Although, it does not look like SysRescueCD includes Clonezilla, there are
tools like rsync, dd, and scp for moving files around in a pinch.

[1]
[http://sysadmincasts.com/episodes/3-systemrescuecd](http://sysadmincasts.com/episodes/3-systemrescuecd)

------
hapless
Just by the by, this is a _terrible_ idea to do online. The _best case_ is a
crash-consistent replica. The more likely case is an inconsistent one that may
or may not be repairable, because of the length of time it takes to do the
dump.

If you want a bit-for-bit dump of an online filesystem, use Linux LVM to
create a snapshot, then make an image of that (100% known crash-consistent)
copy.

------
rsync
I don't always backup an entire system with 'dd', but when I do:

dd if=/dev/da0 | ssh user@rsync.net "dd of=backup.dd"

or maybe:

pg_dump -U postgres db | ssh user@rsync.net "dd of=db_dump"

~~~
icebraining
Unless you have a really big upload pipe, better use ssh -C.

~~~
wmf
I don't know if ssh -C is strong enough these days. I use pigz or maybe pxz
-1.

------
j_s
I used dd booting from a USB stick to transfer a Windows install from hard
drive to SSD this weekend - there are a number of steps that can be done
beforehand to tighten up the size of the final output. These steps are also
useful for virtual machines, but they tend to take quite a while...

(1) Defrag - built-in / maybe SysInternals contig
[http://technet.microsoft.com/en-
us/sysinternals/bb897428](http://technet.microsoft.com/en-
us/sysinternals/bb897428)

(2) Defrag the page file - SysInternals PageDefrag
[http://technet.microsoft.com/en-
us/sysinternals/bb897426](http://technet.microsoft.com/en-
us/sysinternals/bb897426)

(3) Zero unused space - SysInternals SDelete [http://technet.microsoft.com/en-
us/sysinternals/bb897443.asp...](http://technet.microsoft.com/en-
us/sysinternals/bb897443.aspx)

(4) Shrink volume (if backup is filesystem-aware, and disk is one large
volume) - [http://technet.microsoft.com/en-
us/magazine/gg309169.aspx](http://technet.microsoft.com/en-
us/magazine/gg309169.aspx)

(5) After transfer, expand volume to fit target disk

Pretty much any utility automating these steps is acquired by the various
virtualization vendors, or heads directly to the enterprise market (kind of
like how inexpensive screencast creation software always disappears!).

~~~
straight_talk_2
Why would you possibly want to copy a pagefile over :D

Actually with the current RAM prices you don't need one at all.

------
loser777
Hmm, I'm a bit wary of this kind of backup, because it seems like it's
nontrivial to recover part of your data if half of the backup goes bad. While
the old fashioned rsync/copy all your files may be less efficient, you'll
still have whatever it managed to copy if something goes wrong partway
through.

This reminds be of doing a tar archive of all your files--if the archive is
corrupted during that process, what then?

~~~
keithpeter
I use clonezilla to image the whole hard drive including OS once I have a
'clean' setup with an empty home and all my software installed.

Then I use rsync to copy all my 'stuff' back off a large external hard drive
to the 'clean' installation. I _try_ not to change OS to often :-)

Subsequently I use rsync with --delete set to sync the home drive only to an
external drive of similar capacity to the computer drive for daily changes.
This drive is ex4 and preserves ownership and permissions, and includes
dotfiles.

Once a week there is rsync without --delete to a large external hard drive as
2nd backup. This drive is NTFS and does not preserve permissions and excludes
dotfiles.

I have a free dropbox account for really crucial 'working' files. That gets
rsync'd to a FAT32 stick each day using the 'time window' command so it does
not write everything each time.

Basically the clonzilla whole drive backup is just to save time/bandwidth with
re-installing and setting up. The rsync backup to the large drive is file
readable on work PCs that use Windows should disaster strike. The 'sync' to
the small hard drive is readable at file level on anything I can boot from a
live CD.

Now, a _bootable_ whole disc image would be of interest now and again.

------
bcoates
If you want to do something less tricky and low level than dd just remount the
drive in question read-only and use tar. (be root to keep permissions, --use-
compress-program=pigz)

On the restore side, you can just use pv like cat, and don't have to provide a
size param:

    
    
      pv my.img.gz | pigz -d | dd (or hopefully tar xf - instead)...
    

In an ideal world a tool like pv could just look at /proc/ _fd_ /fdinfo/
instead of shuffling all the data through itself but I haven't found a tool
that clever yet.

------
WizzleKake
This is an incredibly naive way of backing up your system.

I use rsync to back up certain folders. I'm OK with re-installing Linux if my
hard drive crashes.

~~~
relaxitup
It's not naive at all. There are a plethora of situations/reasons where one
may want or need a block level image backup.

~~~
seunosewa
For example...

~~~
allgood38
Well, it provides a perfect copy of the drive, not just a copy on another file
system.

I've actually been able to restore to a drive, and when it boots its as though
nothing ever happened. Plus its fast.

Its eerie to turn the computer on after knowing you formatted the drive for
whatever reason, and your desktop pops up again.

~~~
gizmo686
It stops being eerie after a while. Several months ago, my brother was putting
together a computer and wanted to test it (ie. the motherboard) before
attaching any peripherals (other than the monitor). Obviously, installing the
OS on the HD to do this was out of the question, so I just copied my computer
onto a thumb-drive and used that. I didn't dd it, but just copied my
filesystem over and did a grub-install.

------
Yen
The author mentions that removed files stick around on disk after being
removed, and that this can increase the size of a compressed bit-for-bit copy.

They then suggest you can mitigate this by writing zeros to a file before
rm'ing it. However, this is slow in the general case, and doesn't help if
you've already rm'd the file.

As an alternative, the 'sfree' utility, available in the debian package
'secure-delete', can be used to fill the unallocated portions of a disk with
zeroes (or random data).

sfree -llz <disk device>

will write zeroes to the free areas of a disk. -ll limits it to only one pass
over the disk, and -z makes that pass write zeroes, instead of random data.

man page:
[http://manpages.ubuntu.com/manpages/lucid/man1/sfill.1.html](http://manpages.ubuntu.com/manpages/lucid/man1/sfill.1.html)

~~~
allgood38
This is a pretty cool utility, I never really did like just creating a blank
file, I'd rather use an explicit tool

I'll update the article.

------
jaryd
On OSX, CarbonCopyCloner works really well :)

------
sergiosgc
If you want to backup with this kind of technique, use partimage instead, and
do so from a rescue CD. You need a quiescent filesystem for imaging, so it
can't be under use by the running OS.

~~~
sciurus
Partimag development has been dead for years; take a look at partclone
instead. Another program with a similar goal but a different approach is
FSArchiver.

[http://www.partclone.org](http://www.partclone.org)

[http://www.fsarchiver.org/Main_Page](http://www.fsarchiver.org/Main_Page)

------
rlpb
Remember that this cannot be run on a live system, since you'll get a corrupt
image. Thus it's more useful for infrequent maintenance tasks than as part of
a regular regime.

------
allgood38
Hi! Author of this article here,

I've done something completely silly. I originally wrote this site on github,
but then moved it over to a new host on its own domain and totally forgot
about the old one.

If you like this article, please check it out here at:

[http://allgood38.io/a-snapshot-of-your-computer-with-dd-
pv-a...](http://allgood38.io/a-snapshot-of-your-computer-with-dd-pv-and-gzip-
part-1.html)

------
jamesmiller5
At first glance it reminded me of this counter argument which warns about
using `dd`, especially on a failing drive.

[http://rachelbythebay.com/w/2011/12/11/cloning/](http://rachelbythebay.com/w/2011/12/11/cloning/)

------
tlongren
Pigz looks nice. First I've ever heard of it.

I just use BackInTime though, like TimeMachine I guess. It's really, really
useful.

[http://backintime.le-web.org/screenshots/](http://backintime.le-
web.org/screenshots/)

~~~
mercurial
Looks like a graphical rsnapshot. Is this it?

