Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- ## new setup storage:{nfs,smb} perf
- ### Config
- host:
- name: joust
- motherboard:
- name: Asus Maximus VIII Ranger
- rev: '1'
- bios_version: '0508'
- bios_date: 08/10/2015
- storage:
- drives:
- - name: Samsung SSD 850 EVO 1TB
- - name: Samsung SSD 850 EVO 1TB
- network:
- cards:
- - name: ConnectX EN 10GigE
- speed: 10 Gb
- phy: sfp+
- bus:
- type: pcie
- version: 2
- speed: x8
- throughput: 4 GB
- ### Perf
- #### Storage
- Two SATA+ata attached Samsung SSD 850 EVO 1TB
- ata1: SATA max UDMA/133 abar m2048@0xdf44b000 port 0xdf44b100 irq 131
- ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
- ata1.00: supports DRM functions and may not be fully accessible
- ata1.00: disabling queued TRIM support
- ata1.00: ATA-9: Samsung SSD 850 EVO 1TB, EMT02B6Q, max UDMA/133
- ata1.00: 1953525168 sectors, multi 1: LBA48 NCQ (depth 32), AA
- ata1.00: supports DRM functions and may not be fully accessible
- ata1.00: disabling queued TRIM support
- ata1.00: configured for UDMA/133
- sd 0:0:0:0: [sda] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
- sd 0:0:0:0: Attached scsi generic sg0 type 0
- sd 0:0:0:0: [sda] Write Protect is off
- sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
- sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
- sd 0:0:0:0: [sda] supports TCG Opal
- ZFS Striped vdev across two drives
- pool: storage-fast
- state: ONLINE
- scan: none requested
- config:
- NAME STATE READ WRITE CKSUM
- storage-fast ONLINE 0 0 0
- ata-Samsung_SSD_850_EVO_1TB_S2RENX0H518475E ONLINE 0 0 0
- ata-Samsung_SSD_850_EVO_1TB_S3PJNF0J803938R ONLINE 0 0 0
- NAME PROPERTY VALUE SOURCE
- storage-fast type filesystem -
- storage-fast creation Sat Oct 5 15:09 2019 -
- storage-fast used 167G -
- storage-fast available 1.59T -
- storage-fast referenced 167G -
- storage-fast compressratio 1.00x -
- storage-fast mounted yes -
- storage-fast quota none default
- storage-fast reservation none default
- storage-fast recordsize 128K default
- storage-fast mountpoint /storage-fast default
- storage-fast sharenfs rw local
- storage-fast checksum on default
- storage-fast compression off default
- storage-fast atime on default
- storage-fast devices on default
- storage-fast exec on default
- storage-fast setuid on default
- storage-fast readonly off default
- storage-fast zoned off default
- storage-fast snapdir hidden default
- storage-fast aclinherit restricted default
- storage-fast createtxg 1 -
- storage-fast canmount on default
- storage-fast xattr on default
- storage-fast copies 1 default
- storage-fast version 5 -
- storage-fast utf8only off -
- storage-fast normalization none -
- storage-fast casesensitivity sensitive -
- storage-fast vscan off default
- storage-fast nbmand off default
- storage-fast sharesmb off default
- storage-fast refquota none default
- storage-fast refreservation none default
- storage-fast guid 3939091147501142709 -
- storage-fast primarycache all default
- storage-fast secondarycache all default
- storage-fast usedbysnapshots 0B -
- storage-fast usedbydataset 167G -
- storage-fast usedbychildren 2.87M -
- storage-fast usedbyrefreservation 0B -
- storage-fast logbias latency default
- storage-fast dedup off default
- storage-fast mlslabel none default
- storage-fast sync standard default
- storage-fast dnodesize legacy default
- storage-fast refcompressratio 1.00x -
- storage-fast written 167G -
- storage-fast logicalused 166G -
- storage-fast logicalreferenced 166G -
- storage-fast volmode default default
- storage-fast filesystem_limit none default
- storage-fast snapshot_limit none default
- storage-fast filesystem_count none default
- storage-fast snapshot_count none default
- storage-fast snapdev hidden default
- storage-fast acltype off default
- storage-fast context none default
- storage-fast fscontext none default
- storage-fast defcontext none default
- storage-fast rootcontext none default
- storage-fast relatime off default
- storage-fast redundant_metadata all default
- storage-fast overlay off default
- Local no fscache
- joust $ sudo sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'; dd if=/storage-fast/win10.iso bs=1M >/dev/null
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 4.59253 s, 1.1 GB/s
- Local fscache
- joust $ dd if=/storage-fast/win10.iso bs=1M >/dev/null
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 0.788005 s, 6.3 GB/s
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 0.773618 s, 6.4 GB/s
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 0.76383 s, 6.5 GB/s
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 0.761465 s, 6.5 GB/s
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 0.770135 s, 6.4 GB/s
- #### pci-e x16 NVME card
- Local no fscache
- joust $ sudo sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches';sudo dd if=/mnt/win10.iso bs=1M of=/dev/null
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 1.86988 s, 2.6 GB/s
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 1.86305 s, 2.7 GB/s
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 1.87927 s, 2.6 GB/s
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 1.85241 s, 2.7 GB/s
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 1.92342 s, 2.6 GB/s
- Local fscache
- joust $ sudo dd if=/mnt/win10.iso bs=1M of=/dev/null
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 0.403291 s, 12.2 GB/s
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 0.399411 s, 12.4 GB/s
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 0.388781 s, 12.7 GB/s
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 0.391618 s, 12.6 GB/s
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 0.395524 s, 12.5 GB/s
- #### pci-e 3x NVME Samsung 970 EVO in a RAIDZ
- Local no fscache
- joust $ sudo sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches';sudo dd if=/storage-fast/win10.iso bs=1M of=/dev/null
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 1.55233 s, 3.2 GB/s
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 1.41235 s, 3.5 GB/s
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 1.44231 s, 3.4 GB/s
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 1.54468 s, 3.2 GB/s
- Local fscache
- joust $ sudo dd if=/storge-fast/win10.iso bs=1M of=/dev/null
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 0.80141 s, 6.2 GB/s
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 0.802386 s, 6.2 GB/s
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 0.824008 s, 6.0 GB/s
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 0.783524 s, 6.3 GB/s
- #### Network
- Both hosts astrosmash (old) and joust (new) have the same cards
- - name: ConnectX EN 10GigE
- speed: 10 Gb
- driver: mlx4
- firmware: '2.9.1000'
- phy: sfp+
- bus:
- type: pcie
- version: 2
- speed: x8
- Both hosts astrosmash and joust connect through a Quanta LB6M via SFP+
- astrosmash$ iperf -s
- joust$ iperf3 -P4 -c 1.4
- [SUM] 0.00-7.08 sec 7.75 GBytes 9.41 Gbits/sec 0 sender
- #### NFS
- Copy /mnt/win10.iso (4.7GB) from joust to astrosmash with and without fscache.
- NFS+ZFS export `/storage-fast` from joust mounted at `/mnt` on astrosmash:
- astrosmash $ mount -t nfs4 -o proto=tcp joust:/storage-fast /mnt
- joust: no fscache
- astrosmash: no fscache
- joust $ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
- joust $
- astrosmash $ sudo sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'; dd if=/mnt/win10.iso bs=1M >/dev/null
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 4.53154 s, 1.1 GB/s
- joust: fscache
- astrosmash: no fscache
- joust $
- astrosmash $ sudo sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'; dd if=/mnt/win10.iso bs=1M >/dev/null
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 4.24109 s, 1.2 GB/s
- joust: fscache
- astrosmash: fscache
- joust $
- astrosmash $ dd if=/mnt/win10.iso bs=1M >/dev/null
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 0.483307 s, 10.2 GB/s
- astrosmash $ dd if=/mnt/win10.iso bs=1M >/dev/null
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 0.503694 s, 9.8 GB/s
- astrosmash $ dd if=/mnt/win10.iso bs=1M >/dev/null
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 0.51277 s, 9.6 GB/s
- astrosmash $ dd if=/mnt/win10.iso bs=1M >/dev/null
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 0.480163 s, 10.3 GB/s
- astrosmash $ dd if=/mnt/win10.iso bs=1M >/dev/null
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 0.485577 s, 10.2 GB/s
- astrosmash $ dd if=/mnt/win10.iso bs=1M >/dev/null
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 0.483307 s, 10.2 GB/s
- astrosmash $ dd if=/mnt/win10.iso bs=1M >/dev/null
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 0.504308 s, 9.8 GB/s
- #### CIFS
- Copy /mnt/win10.iso (4.7GB) from joust to astrosmash with and without fscache.
- NFS+ZFS export `/storage-fast` from joust mounted at `/mnt` on astrosmash:
- astrosmash $ sudo mount -t cifs -o user=chris //1.11/storage-fast /mnt
- joust: no fscache
- astrosmash: no fscache
- joust $ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
- joust $
- astrosmash $ sudo sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'; dd if=/mnt/win10.iso bs=1M >/dev/null
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 4.56908 s, 1.1 GB/s
- joust: fscache
- astrosmash: no fscache
- joust $
- astrosmash $ sudo sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'; dd if=/mnt/win10.iso bs=1M >/dev/null
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 4.36633 s, 1.1 GB/s
- joust: fscache
- astrosmash: fscache
- joust $
- astrosmash $ dd if=/mnt/win10.iso bs=1M >/dev/null
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 4.36633 s, 1.1 GB/s
- astrosmash $ dd if=/mnt/win10.iso bs=1M >/dev/null
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 4.34369 s, 1.1 GB/s
- astrosmash $ dd if=/mnt/win10.iso bs=1M >/dev/null
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 4.35332 s, 1.1 GB/s
- astrosmash $ dd if=/mnt/win10.iso bs=1M >/dev/null
- 4939528192 bytes (4.9 GB, 4.6 GiB) copied, 4.35294 s, 1.1 GB/s
- astrosmash $ dmesg | grep -i fs-cache | grep cifs
- [1613822.857853] FS-Cache: Netfs 'cifs' registered for caching
- # wtf?
- #### HTTP over 10Gb link to 3xNVME x4 RAIDZ mirror
- |astrosmash:~|lab| $ wget -O/dev/null http://mirror/boot/ubuntu/bionic.squashfs
- --2019-10-20 08:08:30-- http://mirror/boot/ubuntu/bionic.squashfs
- Resolving mirror (mirror)... 1.0.0.11
- Connecting to mirror (mirror)|1.0.0.11|:80... connected.
- HTTP request sent, awaiting response... 200 OK
- Length: 621973504 (593M) [application/octet-stream]
- Saving to: ‘/dev/null’
- /dev/null 100%[===================>] 593.16M 919MB/s in 0.6s
- 2019-10-20 08:08:31 (919 MB/s) - ‘/dev/null’ saved [621973504/621973504]
- |astrosmash:~|lab| $ wget -O/dev/null http://mirror/boot/ubuntu/bionic.squashfs
- --2019-10-20 08:12:10-- http://mirror/boot/ubuntu/bionic.squashfs
- Resolving mirror (mirror)... 1.0.0.11
- Connecting to mirror (mirror)|1.0.0.11|:80... connected.
- HTTP request sent, awaiting response... 200 OK
- Length: 621973504 (593M) [application/octet-stream]
- Saving to: ‘/dev/null’
- /dev/null 100%[===================>] 593.16M 1.08GB/s in 0.5s
- 2019-10-20 08:12:11 (1.08 GB/s) - ‘/dev/null’ saved [621973504/621973504]
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
