Richard - ZFS Backups

Greetings,

Long time BSDNow listener as well as TechSnap.  While I run Linux on my desktops and laptops, my critical stuff runs BSD such as FreeNAS and pfSense.  I'm observing an odd pattern when backing up files from FreeNas.  I know this process needs improvement, but there are reasons for the way it is now.

FreeNAS 9.3 Stable - Hardware Specs:

Supermicro MB (MBD-X10SLM-F-O R) with Intel Core i3-4130T (Haswell), Dual Core 2.9 GHz
Intel HD graphics 4400 and 32GB DDR3 1600 Mhz ECC RAM
4 x 4TB WD RED NAS Drives (all drives are in RAIDZ2 volume)

Using the Supermicro on-board gigabit network interface.

Environment - Home Office.  The FreeNas is connected directly to a NetGear ProSafe 24 port gigabit switch (JG524), a drop runs to my home office, to a NetGear Prosafe 5 port gigabit switch (GS105) which my desktop is connected to.

I originally tried making backups via direct connected external USB drives on the FreeNAS but performance was horrible, I assume from the lack of USB3 support.

Backup Strategy:

1) Mount USB drives via external USB 3.0 adapter (Orico 6518 USB3 SATA HDD Docking Station) to Arch based Linux desktop.

2) SSH to the FreeNAS Server.

3) From the FreeNAS server use rsync to send files via ssh back to Linux desktop.

In general this is working well.  Large files such as MKV files, Clonezilla images, etc stream at a steady 50MB/sec to 60MB/sec.

However, as soon as it reaches the VirtualBox vdi files, the speed is all over the map from 10KB/sec to 155MB/sec (reported by rsync on freenas) transferring these files takes an hour instead of minutes.  The files are not in use.  They are on the same physical drives on the NAS (same volume), but different data set than files with much faster transfer rate.

    The dataset for MKV files has compression off
    The dataset for Clonezilla backups files has compression "inherit(lz4)" on (ratio at 1.00)
    The dataset for the VDI files has compression "inherit(lz4)" on (ratio of 1.31x)

Can I assume the slowdown is from the files being decompressed?  I was under the impression that the compression was good thing and would not get in the way.  Could it be some overhead of grabbing blocks from various snapshots over time?  This dataset is snapshot nightly.

Examples:

main/vm_storage/QuickBooksPro/QuickBooksPro.vdi
 49,358,569,472 100%   13.27MB/s    0:59:06 (xfr#10285, to-chk=15/62056)

main/vm_storage/iTunesLibrary/iTunesLibraryDynamic.vdi
 72,868,691,968 100%   11.68MB/s    1:39:08 (xfr#10292, to-chk=6/62056)

movies/films/xxxxxxxxxx.2015.1080p.BluRay.x264.DTS-HD.MA.7.1.mkv
 15,556,955,983 100%   51.27MB/s    0:04:49 (xfr#3, to-chk=536/954)

The specs of the desktop:  Intel Core i7-3770K @ 3.5Ghz, 32GB DDR3-1600Mhz RAM, GeForce GTX 970, 250GB SSD, using on-board network interface (Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)) - CPU is at about 3% utilization during backup process.

One day this will be replaced by a ZFS send to the Linux desktop.... but I don't think I'm ready for that yet (have a good guide?).

-Rich