Solaris 11 guest on VMware ESXI submit only one disk I/O at

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=310468404053887

Solaris 11 guest on VMware ESXI submit only one disk I/O at a time (Doc ID 2238101.1)	To BottomTo Bottom

In this Document
Symptoms
Changes
Cause
Solution
References

APPLIES TO:

Solaris x64/x86 Operating System - Version 11.2 and later
Information in this document applies to any platform.
SYMPTOMS

Hypervisor (ESX 5.5U3) sees I/O from Solaris 11 guest arriving on its virtualized LSI Logic SAS scsi controller as synchronous I/O. Meaning that Solaris 11 only queues 1 disk I/O at a time.

Solaris 10 machines, using the same virtualized LSI Logic SAS scsi controller show multiply disk I/Os queued on these same HBA.

The end result is a disk I/O bottleneck with Solaris 11.

Solaris 11 is performing significantly worse in high disk I/O situations, as it appears to be a bottleneck not being able to send more than 1 outstanding I/O at a time.


Iostat shows the following output for c4 which is connected to local disk.

c4 is a virtual LSI HBA:

r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.4 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t0d0s0
0.0 15.8 0.0 53.8 0.0 0.0 3.1 2.9 1 2 c4t0d0s1
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t0d0p0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t0d0p1
1324.3 45.5 338308.7 330.5 0.1 2.0 0.1 1.5 0 13 c4
0.0 16.2 0.0 53.8 0.0 0.0 3.1 2.9 1 2 c4t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t1d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t1d0s0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t1d0p0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t1d0p1
278.2 1.2 70646.3 12.8 0.0 0.8 0.0 3.0 0 59 c4t2d0 <============= at most, 3 I/O are sent to the HBA/disk
278.2 1.2 70646.3 12.8 0.0 0.8 0.0 3.0 0 59 c4t2d0s0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t2d0s8
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t2d0p0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t2d0p1
102.8 1.4 26275.9 14.5 0.0 0.2 0.0 2.1 0 19 c4t3d0
102.8 1.4 26275.9 14.5 0.0 0.2 0.0 2.1 0 19 c4t3d0s0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t3d0s8
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t3d0p0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t3d0p1
106.8 1.4 27302.2 14.5 0.0 0.2 0.0 2.0 0 19 c4t4d0
106.8 1.4 27302.2 14.5 0.0 0.2 0.0 2.0 0 19 c4t4d0s0
CHANGES

This is a new Solaris 11 installation. The issue only exists with Solaris 11.

So far, the customer was using Solaris 10 guest on its ESXi server. No such issue existed with Solaris 10.

CAUSE

It is caused by a bug in ssd driver of Solaris 11 which wrongly considers SCSI disk properties.

On Solaris 11.3, mpt driver has tagged queuing disabled by default.

This is visible with :

# prtpicl -v | grep TQ which is set to 0
pci15ad,1976 (obp-device, 2c80000025f)
:DeviceID 0x10
:UnitAddress 0
:device-id 0x30
:vendor-id 0x1000
:revision-id 0x1
:class-code 0x10000
:unit-address 10
:subsystem-id 0x1976
:subsystem-vendor-id 0x15ad
...
:model SCSI bus controller
:compatible (2c8000002adTBL)
| pci1000,30.15ad.1976.1 |
| pci1000,30.15ad.1976 |
| pci15ad,1976 |
| pci1000,30.1 |
| pci1000,30 |
| pciclass,010000 |
| pciclass,0100 |
...
:scsi-enumeration 0
:scsi-options 0x107ff8
:scsi-reset-delay 0xbb8
:scsi-watchdog-tick 0xa
:scsi-selection-timeout 0xfa
:scsi-tag-age-limit 0x2
:scsi-poll-timeout 0xa
:scsi-transport-timeout 0x3c
:initiator-interconnect-type SPI
:tracebuffer_autostart 0
:tracebuffer_filename /var/tmp/IOCtrace
:snapshot_autostart 0
:shapshot_filename /var/tmp/IOCsnapshot
:disable-sata-mpxio no
:mpxio-disable yes
:ddi-vhci-class scsi_vhci
...
:driver_chip_revision 1030 b0
:firmware-version 1.3.41.32
:target0-sync-speed 0x4e200
:target0-wide 0x1
:target0-TQ 0 <========================== Tag Queueing is disabled
:target1-sync-speed 0x4e200
:target1-wide 0x1
:target1-TQ 0 <========================== Tag Queueing is disabled
:target2-sync-speed 0x4e200
:target2-wide 0x1
:target2-TQ 0 <========================== Tag Queueing is disabled
...
:devfs-path /pci@0,0/pci15ad,1976@10
:driver-name mpt
:binding-name pci1000,30

 This is also visible in the vmdump file of live kernel :

> ::mpt
mpt_t inst mpxio suspend ntargs power
================================================================================
ffffc1000421f000 0 0 0 16 OFF=D3
>
> ffffc1000421f000::print -t mpt ! grep tag
ushort_t m_notag = 0xffff <============== no tag whatever is the target model
int m_scsi_tag_age_limit = 0
> ::mpt

On Solaris 11, the tagged queuing is disabled. Only HBA queuing is allowed which is limited to 3 I/O at most at the same time.
This is exactly what is reflected by the iostat output because we can see up to 3 I/O at the same time in the iostat outputs.

Comparing Solaris 10 and Solaris 11 kernel data related to disk and mpt gives:

Solaris 10 kernel dump (No issue)

CAT(vmcore.0/10X)> analyze
crash file: /cfm_data/isde_data/data11/SR/3-13298414671/uc/vmdump.oseasddvordb003.0_67387170/this/vmcore.0
user: REMI.COLINET@ORACLE.COM (rcolinet:553459)
release: 5.10 (64-bit)
version: Generic_150401-35
machine: i86pc
node name: oseasddvordb003
system type: i86pc
hostid: 260cc364
dump_conflags: 0x10000 (DUMP_KERNEL) on /dev/zvol/dsk/rpool/dump(1G)
time in kernel: Fri Sep 30 08:04:51 UTC 2016
age of system: 45 days 23 hours 27 minutes 58 seconds
CPUs: 1 (3.99G memory)
panicstr:

CAT(vmcore.0/10X)>

CAT(vmcore.0/10X)> sdump 0xffffffff853ed000 sd_lun | egrep 'throttle|queue|arq'
short un_throttle = 0x100 <============ 256 commands supported
short un_saved_throttle = 0x100
short un_busy_throttle = 0
short un_min_throttle = 8
timeout_id_t un_reset_throttle_timeid = NULL
unsigned int :1 un_f_arq_enabled = 1
unsigned int :1 un_f_opt_queueing = 0 <============== no HBA cmd queueing
unsigned int :1 un_f_use_adaptive_throttle = 0
CAT(vmcore.0/10X)>

But sd inquiry is the same.
Issue lies on the HBA interpretation:

CAT(vmcore.0/10X)> sdump 0xffffffff86092de8 scsi_inquiry
struct scsi_inquiry {
uchar_t inq_dtype = 0
unsigned char :7 inq_qual = 0
unsigned char :1 inq_rmb = 0
unsigned char :3 inq_ansi = 2
unsigned char :3 inq_ecma = 0
unsigned char :2 inq_iso = 0
unsigned char :4 inq_rdf = 2
unsigned char :1 inq_hisup = 0
unsigned char :1 inq_normaca = 0
unsigned char :1 inq_trmiop = 0
unsigned char :1 inq_aenc = 0
uchar_t inq_len = 0x1f
unsigned char :1 inq_protect = 0
unsigned char :1 inq_5_1 = 0
unsigned char :1 inq_5_2 = 0
unsigned char :1 inq_3pc = 0
unsigned char :2 inq_tpgs = 0
unsigned char :1 inq_acc = 0
unsigned char :1 inq_sccs = 0
unsigned char :1 inq_addr16 = 0
unsigned char :1 inq_addr32 = 0
unsigned char :1 inq_ackqreqq = 0
unsigned char :1 inq_mchngr = 0
unsigned char :1 inq_dualp = 0
unsigned char :1 inq_port = 0
unsigned char :1 inq_encserv = 0
unsigned char :1 inq_bque = 0
unsigned char :1 inq_sftre = 1
unsigned char :1 inq_cmdque = 1
unsigned char :1 inq_trandis = 0
unsigned char :1 inq_linked = 0
unsigned char :1 inq_sync = 1
unsigned char :1 inq_wbus16 = 1
unsigned char :1 inq_wbus32 = 1
unsigned char :1 inq_reladdr = 0
char [8] inq_vid = [ 'V' 'M' 'w' 'a' 'r' 'e' ' ' ' ' ]
char [0x10] inq_pid = [ 'V' 'i' 'r' 't' 'u' 'a' 'l' ' ' 'd' 'i' 's' 'k' ' ' ' ' ' ' ' ' ]
char [4] inq_revision = [ '1' '.' '0' ' ' ]
char [0xc] inq_serial = [ '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' ]
... // Other fields are all zero
}
CAT(vmcore.0/10X)>

Solaris 11 kernel dump (Issue)

CAT(vmcore.0/11X)> dev scsi
...
CAT(vmcore.0/11X)> sdump 0xffffc1002968e1f0 scsi_device sd_inq
struct scsi_inquiry *sd_inq = 0xffffc1001881abf0
CAT(vmcore.0/11X)>
CAT(vmcore.0/11X)> sdump 0xffffc1001881abf0 scsi_inquiry
struct scsi_inquiry {
uchar_t inq_dtype = 0
unsigned char :7 inq_qual = 0
unsigned char :1 inq_rmb = 0
unsigned char :3 inq_ansi = 2
unsigned char :3 inq_ecma = 0
unsigned char :2 inq_iso = 0
unsigned char :4 inq_rdf = 2
unsigned char :1 inq_hisup = 0
unsigned char :1 inq_normaca = 0
unsigned char :1 inq_trmiop = 0
unsigned char :1 inq_aenc = 0
uchar_t inq_len = 0x1f
unsigned char :1 inq_protect = 0
unsigned char :1 inq_5_1 = 0
unsigned char :1 inq_5_2 = 0
unsigned char :1 inq_3pc = 0
unsigned char :2 inq_tpgs = 0
unsigned char :1 inq_acc = 0
unsigned char :1 inq_sccs = 0
unsigned char :1 inq_addr16 = 0
unsigned char :1 inq_addr32 = 0
unsigned char :1 inq_ackqreqq = 0
unsigned char :1 inq_mchngr = 0
unsigned char :1 inq_dualp = 0
unsigned char :1 inq_port = 0
unsigned char :1 inq_encserv = 0
unsigned char :1 inq_bque = 0
unsigned char :1 inq_sftre = 1
unsigned char :1 inq_cmdque = 1
unsigned char :1 inq_trandis = 0
unsigned char :1 inq_linked = 0
unsigned char :1 inq_sync = 1
unsigned char :1 inq_wbus16 = 1
unsigned char :1 inq_wbus32 = 1
unsigned char :1 inq_reladdr = 0
char [8] inq_vid = [ 'V' 'M' 'w' 'a' 'r' 'e' ' ' ' ' ]
char [0x10] inq_pid = [ 'V' 'i' 'r' 't' 'u' 'a' 'l' ' ' 'd' 'i' 's' 'k' ' ' ' ' ' ' ' ' ]
char [4] inq_revision = [ '1' '.' '0' ' ' ]
char [0xc] inq_serial = [ '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' ]
... // Only zero values
}
CAT(vmcore.0/11X)>

 Comparing the mpt HBA state gives:

 Solaris 10 (No issue)


> ::mpt
mpt_t inst mpxio suspend ntargs power
================================================================================
ffffffff822c7ac0 0 0 0 16 OFF=D3
> ffffffff822c7ac0::print mpt_t m_notag
m_notag = 0xffc0
>
 Solaris 11 (Issue)

> ::mpt
mpt_t inst mpxio suspend ntargs power
================================================================================
ffffc1000421f000 0 0 0 16 OFF=D3
> ffffc1000421f000::print -t mpt m_notag
ushort_t m_notag = 0xffff
>
We have the:

- same SCSI disk Inquiry
- same FW version, productid for the mpt HBA.

But we have different values for:

Solaris 10 (No issue)

ushort_t m_notag = 0xffc0
ushort_t m_nowide = 0xffc0
 Solaris 11 (Issue)

ushort_t m_notag = 0xffff
ushort_t m_nowide = 0xe080
SOLUTION

The sd (SCSI disk) driver makes wrong assumptions about the SCSI devices. It ignores some capacities of the disks exposed by the VMware platform. The change needed to fix such issue is very small.

The issue is referenced under Bug 24764515 - Tag command queueing disabled for VMware mpt HBA

Cu tested the fix on its systems and the I/O throughput went from 1000 iops to 140000 iops!

Fix has been delivered in Oracle Solaris 11.3.17.5.0.

REFERENCES

NOTE:1580689.1 - Collaborate With Fibre Channel (FC) Storage Area Network (SAN) MOS Community Members
NOTE:1502843.1 - SAN Fibre Channel (FC) Storage Connectivity Issues
NOTE:1393062.2 - Information Center: Troubleshooting the Oracle Solaris 10 Operating System
NOTE:1528697.2 - Information Center: Sun Storage Traffic Manager (MPXIO)
BUG:24764515 - TAGGED COMMAND QUEUING DISABLED FOR SCSI-2 AND SPC TARGETS
NOTE:1929376.1 - My Oracle Support - Automated Troubleshooting
NOTE:1303745.1 - Troubleshooting Solaris[TM] 10 and Above (not for Solaris[TM] 8 & 9) SAN Fibre Channel (FC) HBA connectivity issues.
NOTE:1542438.1 - Discovery of Fibre Channel (FC) Disk (LUN) and Tape Storage Devices from a Solaris host perspective
NOTE:166650.1 - Working Effectively With Oracle Support - Best Practices
NOTE:1550562.2 - Information Center: Oracle Storage Area Network (SAN) Fibre Channel (FC) Card - Overview
BUG:15813747 - SUNBT7195820 MPT DRIVER ISSUING "CAN ONLY START 1 TASK MANAGEMENT COMMAND AT A T
BUG:18496291 - MPT DRIVER ISSUING "CAN ONLY START 1 TASK MANAGEMENT COMMAND AT A TIME"
NOTE:1003635.1 - What Does %b (or %Busy) Actually Mean in the Output of iostat(1M)?
NOTE:1001444.1 - Solaris Storage Driver Troubleshooting SCSI Transport Errors - Command Failed to Complete - Command Timeout
NOTE:1285485.1 - GUDS - A Script for Gathering Solaris Performance Data
NOTE:1595092.1 - SRDC - How to Collect Standard Information for a Database Performance Problem
NOTE:1681652.1 - Oracle Solaris Support on Virtualization Platforms


Was this document helpful?

Yes
No


Document Details

Email link to this documentOpen document in new windowPrintable Page

Type:
Status:
Last Major Update:
Last Update:
PROBLEM
PUBLISHED
03-Mar-2017
03-Mar-2017




Related Products


Solaris x64/x86 Operating System


Information Centers


Information Center: Overview of Sun Patches and Updates [1589780.2]

情報センター: Oracle Explorer Data Collector (STB)の概要 [1675915.2]

Information Center: Overview of the Oracle Solaris 10 Operating System [1372665.2]

Information Center: Overview of the Oracle Explorer Data Collector (STB) [1589529.2]

Information Center: Overview of the Oracle Solaris 11 Operating System [1559480.2]



Document References


Collaborate With Fibre Channel (FC) Storage Area Network (SAN) MOS Community Members [1580689.1]

SAN Fibre Channel (FC) Storage Connectivity Issues [1502843.1]

Information Center: Troubleshooting the Oracle Solaris 10 Operating System [1393062.2]

Information Center: Sun Storage Traffic Manager (MPXIO) [1528697.2]

My Oracle Support - Automated Troubleshooting [1929376.1]

Show More


Recently Viewed

	Solaris 11 guest on VMware ESXI submit only one disk I/O at a time [2238101.1]

	Slow IO Performance when running dd [1931027.1]

	I/O Scheduler Selection to Optimize Oracle VM Performance [2069125.1]

	Solaris Performance Analysis And Tuning, Step by Step [1007447.1]

	Troubleshooting Disk Performance [1010680.1]

Show More

Didn't find what you are looking for?Ask in Community...


Related


Products

Sun Microsystems > Operating Systems > Solaris Operating System > Solaris x64/x86 Operating System > Storage Target Drivers > sd

Keywords

BUG;DISK;IOSTAT;QUEUED;SAS;SCSI;SOLARIS;SUBMIT;SYNCHRONOUS;TAG;THROUGHPUT;VMWARE


-------------------------------------------