RAID stripe & PG performance

FreeBSD 9.1-STABLE amd64 GENERIC

CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (2000.04-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x206d7  Family = 0x6  Model = 0x2d  Stepping = 7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x1fbee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant, performance statistics
real memory  = 25769803776 (24576 MB)
avail memory = 24742514688 (23596 MB)

FreeBSD/SMP: Multiprocessor System Detected: 24 CPUs
FreeBSD/SMP: 2 package(s) x 6 core(s) x 2 SMT threads


PostgreSQL 9.1.9 (block_size = 8 KB - default)

=========================================================================
# mfiutil show adapter

mfi0 Adapter:
    Product Name: ServeRAID M5110e
   Serial Number: 2CL1CD
        Firmware: 23.7.1-0010
     RAID Levels: JBOD, RAID0, RAID1, RAID5, RAID6, RAID10, RAID50
  Battery Backup: present
           NVRAM: 32K
  Onboard Memory: 512M
  Minimum Stripe: 8k
  Maximum Stripe: 1M


# mfiutil show volumes

mfi0 Volumes:
  Id     Size    Level   Stripe  State   Cache   Name
 mfid0 (  557G) RAID-1       1M OPTIMAL Reads    <TEST>


# mfiutil show config

mfi0 Configuration: 2 arrays, 1 volumes, 0 spares
    array 0 of 2 drives:
        drive  6 (  279G) ONLINE <HITACHI HUS156030VLS600 A5D0 serial=JTVDL28L> SCSI-6
        drive  7 (  279G) ONLINE <HITACHI HUS156030VLS600 A5D0 serial=LVY9SJVN> SCSI-6
    array 1 of 2 drives:
        drive  5 (  279G) ONLINE <HITACHI HUS156030VLS600 A5D0 serial=J0YPP62N> SCSI-6
        drive  8 (  279G) ONLINE <HITACHI HUS156030VLS600 A5D0 serial=J0YSE4GN> SCSI-6
    volume mfid0 (557G) RAID-1 1M OPTIMAL <TEST> spans:
        array 0
        array 1


# mfiutil cache 0

mfi0 volume mfid0 cache settings:
             I/O caching: writes and reads
           write caching: write-back
write cache with bad BBU: disabled
              read ahead: adaptive
       drive write cache: disabled

-------------------------------------------------------------------------
# dumpfs /dev/mfid0s1e

magic   19540119 (UFS2) time    Thu Jul  4 22:39:34 2013
superblock location     65536   id      [ 51d4bc0b a09bda61 ]
ncg     220     size    70638592        blocks  69528015
bsize   65536   shift   16      mask    0xffff0000
fsize   8192    shift   13      mask    0xffffe000
frag    8       shift   3       fsbtodb 4
minfree 8%      optim   time    symlinklen 120
maxbsize 65536  maxbpg  8192    maxcontig 2     contigsumsize 2
nbfree  8202122 ndir    245     nifree  35420647        nffree  303
bpg     40195   fpg     321560  ipg     161024  unrefs  0
nindir  8192    inopb   256     maxfilesize     36033195603132415
sbsize  8192    cgsize  65536   csaddr  5064    cssize  8192
sblkno  16      cblkno  24      iblkno  32      dblkno  5064
cgrotor 0       fmod    0       ronly   0       clean   0
metaspace 12856 avgfpdir 64     avgfilesize 16384
flags   soft-updates
fsmnt   /var
volname         swuid   0       providersize    70638592

=========================================================================
Benchmark #1: bonnie++

-------------------------------------------------------------------------
test-32kb-stripe $ bonnie++ -n64 -d /var/tmp -c8

Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   8     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
test-32kb-strip 48G   706  99 308393  40 86048  53  1607  99 291870  23 637.2  13
Latency             12500us   73107us    7250ms   14342us     636ms     171ms
Version  1.97       ------Sequential Create------ --------Random Create--------
test-32kb-stripe    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 64 59258  91 +++++ +++ +++++ +++ 56580  92 +++++ +++ +++++ +++
Latency             38851us      15us     165us   38792us      22us     103us


-------------------------------------------------------------------------
test-1mb-stripe $ bonnie++ -n64 -d /var/tmp -c8

Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   8     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
test-1mb-stripe 48G   711  99 312911  41 130135  43  1611  99 423189  30 671.0  15
Latency             12435us     336ms    6687ms    9230us     258ms     671ms
Version  1.97       ------Sequential Create------ --------Random Create--------
test-1mb-stripe     -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 64 60443  95 +++++ +++ +++++ +++ 60824  94 +++++ +++ +++++ +++
Latency             39072us      17us      77us   38960us      23us      72us

=========================================================================
Benchmark #2: pgbench - previously warmed DB


Methodology:

  1. Init the DB (26 GB DB size):

    pgbench -i -s 2000 pgbench

  2. "Warm" the DB cache for 5 hours:

    pgbench -j 6 -c 6 -T 18000 -S pgbench

  3. Make a couple of test attempts (15 min long each) to make sure the results are stable - see below.

-------------------------------------------------------------------------
test-32kb-stripe $ pgbench -j 10 -c 10 -r -T 900 pgbench

tps = 254.190917 (including connections establishing)
tps = 254.196753 (excluding connections establishing)
statement latencies in milliseconds:
        0.002367        \set nbranches 1 * :scale
        0.000877        \set ntellers 10 * :scale
        0.000817        \set naccounts 100000 * :scale
        0.000844        \setrandom aid 1 :naccounts
        0.000728        \setrandom bid 1 :nbranches
        0.000736        \setrandom tid 1 :ntellers
        0.000699        \setrandom delta -5000 5000
        0.031943        BEGIN;
        13.001537       UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
        0.109910        SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
        0.179934        UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
        0.178669        UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
        0.088774        INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
        25.732963       END;

-------------------------------------------------------------------------
test-1mb-stripe $ pgbench -j 10 -c 10 -r -T 900 pgbench

tps = 626.001747 (including connections establishing)
tps = 626.012862 (excluding connections establishing)
statement latencies in milliseconds:
        0.002093        \set nbranches 1 * :scale
        0.000821        \set ntellers 10 * :scale
        0.000770        \set naccounts 100000 * :scale
        0.000760        \setrandom aid 1 :naccounts
        0.000673        \setrandom bid 1 :nbranches
        0.000699        \setrandom tid 1 :ntellers
        0.000667        \setrandom delta -5000 5000
        0.029973        BEGIN;
        5.243319        UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
        0.114803        SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
        0.132933        UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
        0.132787        UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
        0.088243        INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
        10.219683       END;

=========================================================================
Benchmark #3: pgbench - cold DB


Methodology:

  1. Init the DB on a fresh installed OS (26 GB DB size):

    pgbench -i -s 2000 pgbench

  2. Reboot the server.

  3. Measure pgbench preformance

-------------------------------------------------------------------------
$ pgbench -j 10 -c 10 -r -T 900 pgbench

  * 32 KB (half of the UFS bsize) - tps=198

  * 64 KB - tps=226

  * 128 KB (default for the RAID controller) - tps=298

  * 1 MB (max for the RAID controller) - tps=347

=========================================================================
Benchmark #4: pgbench - cold DB, big DB size


Methodology:

  1. Init the DB on a fresh installed OS (88 GB DB size):

    pgbench -i -s 6000 pgbench

  2. Reboot the server.

  3. Measure pgbench preformance

-------------------------------------------------------------------------
$ pgbench -j 10 -c 10 -r -T 900 pgbench

  * 32 KB (half of the UFS bsize) - tps=161

  * 1 MB (max for the RAID controller) - tps=258

=========================================================================
Benchmark #5: randomio on 48 MB file


Methodology:

  1. Create 48 MB file:

    dd if=/dev/zero of=/var/tmp/test-file bs=1m count=48000

  2. Measurement:

    randomio file 8 0.5 1 8192 10 10

  3. Reboot the server:

  4. Do the measurement with the same parameters.

-------------------------------------------------------------------------

  * 32 KB (half of the UFS bsize) - before the server reboot

  total |  read:         latency (ms)       |  write:        latency (ms)
   iops |   iops   min    avg    max   sdev |   iops   min    avg    max   sdev
--------+-----------------------------------+----------------------------------
  242.6 |  120.7   0.0   38.4  678.9   91.2 |  121.9   0.2   27.2  650.0   45.4
  223.0 |  110.8   0.0   41.2  778.7   96.4 |  112.2   0.2   30.7  791.9   41.8
  196.1 |   96.7   0.0   46.7 1442.9  141.6 |   99.4   0.2   33.9  825.0   54.4
  201.9 |   99.8   0.0   43.5  871.9  102.1 |  102.1   0.2   37.0  695.2   44.5
  179.9 |   87.7   0.0   53.2 1941.9  187.4 |   92.2   0.2   34.7 1836.3   75.9
  175.1 |   89.1   0.0   56.2 1628.5  183.4 |   86.0   0.2   36.3  870.3   59.1
  190.7 |   99.3   0.0   43.5 1923.5  158.1 |   91.4   0.2   38.5 1175.7   58.8
  184.3 |   94.0   0.0   50.1 1433.9  144.6 |   90.3   0.2   35.9 1271.4   62.4
  183.2 |   91.9   0.0   48.8 1039.8  112.4 |   91.3   0.2   41.1 1118.1   56.5
  176.7 |   86.0   0.0   50.2 1021.0  146.8 |   90.7   0.2   38.5  918.0   58.4

  * 32 KB (half of the UFS bsize) - after the server reboot

  total |  read:         latency (ms)       |  write:        latency (ms)
   iops |   iops   min    avg    max   sdev |   iops   min    avg    max   sdev
--------+-----------------------------------+----------------------------------
  127.5 |   63.9   3.7   83.0 1551.1  254.8 |   63.6   4.9   40.8 1518.6  110.8
  130.8 |   65.3   3.7   75.1 1707.4  233.9 |   65.5   5.6   34.8 1565.5   96.3
  122.7 |   60.0   4.4  101.0 2151.3  315.4 |   62.7   0.2   37.2 1629.2  113.8
  120.6 |   58.9   0.1   91.1 2352.9  326.4 |   61.7   5.4   45.7 2222.8  125.2
  121.3 |   61.7   0.1   84.7 2294.2  275.8 |   59.6   0.3   43.0 2214.4  112.4
  118.5 |   57.8   0.1   91.5 3463.5  366.8 |   60.7   0.3   47.0 3369.1  150.8
  118.4 |   58.4   0.0   94.6 1864.9  276.8 |   60.0   0.3   46.9 1719.2  108.8
  112.9 |   54.1   0.0   97.3 2104.2  291.1 |   58.8   0.2   45.1 1962.7  119.5
  111.4 |   55.9   0.0   99.0 1437.7  292.0 |   55.5   5.2   45.9 1411.3  119.0
  108.3 |   56.0   0.1   91.6 3224.1  400.1 |   52.3   5.3   43.7 3106.2  173.2

  * 1 MB (max for the RAID controller) - before the server reboot

  total |  read:         latency (ms)       |  write:        latency (ms)
   iops |   iops   min    avg    max   sdev |   iops   min    avg    max   sdev
--------+-----------------------------------+----------------------------------
  326.2 |  163.5   0.0   25.8  519.6   57.3 |  162.7   0.2   22.8  519.4   35.2
  341.0 |  168.2   0.0   22.1  271.4   40.6 |  172.8   0.2   24.1  199.6   28.3
  329.1 |  160.4   0.0   26.2  405.1   54.5 |  168.7   0.2   23.6  269.5   30.7
  332.6 |  170.2   0.0   25.2  393.2   52.7 |  162.4   0.2   22.6  292.1   28.3
  339.2 |  173.3   0.0   22.9  454.2   47.5 |  165.9   0.2   24.4  287.0   30.7
  329.4 |  162.0   0.0   26.8  493.2   54.1 |  167.4   0.2   21.9  295.6   28.5
  327.8 |  164.3   0.0   23.7  404.7   47.0 |  163.5   0.2   25.2  288.4   28.6
  337.0 |  166.3   0.0   24.3  348.7   42.9 |  170.7   0.2   23.0  246.1   28.1
  341.7 |  170.5   0.0   24.6  428.4   55.8 |  171.2   0.2   22.3  339.5   28.8
  327.6 |  162.1   0.0   24.3  353.6   48.5 |  165.5   0.2   24.2  231.9   27.3

  * 1 MB (max for the RAID controller) - after the server reboot

  total |  read:         latency (ms)       |  write:        latency (ms)
   iops |   iops   min    avg    max   sdev |   iops   min    avg    max   sdev
--------+-----------------------------------+----------------------------------
  198.9 |   98.2   1.4   53.3 1276.1  155.5 |  100.7   1.7   27.4  867.9   59.1
  202.0 |  100.5   1.3   48.6  916.8  136.4 |  101.5   0.2   29.3  884.1   60.3
  198.6 |   99.4   0.1   51.6 1170.8  167.8 |   99.2   1.5   28.4 1136.7   70.2
  197.6 |   96.8   1.4   57.7 2077.8  221.2 |  100.8   1.6   25.8 2085.9   91.2
  201.6 |   97.8   0.0   50.0 1084.7  139.2 |  103.8   1.5   28.1 1084.3   59.6
  192.3 |   98.6   0.0   57.8 2635.7  276.9 |   93.7   1.3   22.4 2503.0  113.8
  200.6 |  103.1   0.0   55.6 1643.6  180.6 |   97.5   1.7   26.6 1439.5   73.8
  196.0 |  100.7   0.0   51.9 2171.5  244.0 |   95.3   0.2   22.7 1585.5   88.2
  190.1 |   94.9   0.0   68.2 3859.9  377.4 |   95.2   0.2   22.5 2631.3  133.2
  198.1 |   95.4   0.0   52.8 1339.4  167.3 |  102.7   0.2   23.8 1302.2   72.7