isi_dsp_install stuck in Installing state if no_SSD version

Issue	The No_SSD version of the DSP package is only intended for specific OneFS versions, as per the warning displayed:

Drive_Support_v1.11_No_SSD.tgz

Contains updates similar to the previous releases, but does not include SSD firmware updates for the following drives and is applicable only for clusters running OneFS versions 7.1.1.0 through 7.1.1.3 and OneFS versions 7.2.0.0 through 7.2.0.1:
- Any HGST HUSMM16x0ASS20x series
- Any HGST HUSMM80x0ASS20x series
- Any HGST HUSSL40x0ASS600 series
- Any HGST HUSSL40x0BSS600 series
- Any Micron RealSSD P300 series
- STEC M8ISB2B 200UC
- STEC MACH8 IOPS

If customers ignore the warning and attempt to install this DSP variant on OneFS 8.0, it will still attempt to install, but the installation will fail with the following error:

Patch install has been requested for patch Drive_Support_v1.11_No_SSD

WARNING: /ifs/data/Drive_Support_v1.11_No_SSD.tar is not installed
DSP Install Failed, 647.604293108 sec elapsed

When patch status is subsequently checked, the DSP install will continue to show as being in progress, even though it has already failed (relevant lines bolded for emphasis):

# isi_dsp_install --latest
ERROR: The upgrade system is busy processing another request.

# isi upgrade view

Upgrade Status:

   Cluster Upgrade State: committed
Current Upgrade Activity: Package
      Upgrade Start Time: 2016-06-21T15:19:25
            Patch Action: INSTALL
              Patch Name: Drive_Support_v1.11_No_SSD
      Current OS Version: 8.0.0.0_build(37)style(5)
      Upgrade OS Version: N/A

# isi upgrade patches list
Patch Name                 Description           Status
-----------------------------------------------------------
Drive_Support_v1.11_No_SSD Drive Support Package Installing
-----------------------------------------------------------

The isi_pkg log file will show the following errors:

# isi_for_array -s 'tail /var/log/isi_pkg'
X-1: 2016-08-02T11:29:30-04:00 <3.6> RTPIsilon01-1 isi_pkg[3399]: Received new request (action:'install', patch:'/ifs/data/Drive_Support_v1.11_No_SSD.tgz', opts:0,0,0)
X-1: 2016-08-02T11:29:30-04:00 <3.3> RTPIsilon01-1 isi_pkg[3399]: isi_pkg_file_cmd: could not perform '/usr/bin/tar -xp -z -f '/ifs/data/Drive_Support_v1.11_No_SSD.tgz' '+*''
X-1: 2016-08-02T11:29:30-04:00 <3.4> RTPIsilon01-1 isi_pkg[3399]: Failed to unpack patch file /ifs/data/Drive_Support_v1.11_No_SSD.tgz
X-1: 2016-08-02T11:29:30-04:00 <3.3> RTPIsilon01-1 isi_pkg[3399]: New request failed with error: failed to locate the patch: /ifs/data/Drive_Support_v1.11_No_SSD.tgz
X-1: 2016-08-02T11:30:11-04:00 <3.6> RTPIsilon01-1 isi_pkg[3467]: Received new request (action:'install', patch:'/ifs/data/Drive_Support_v1.11_No_SSD.tar', opts:0,0,0)
X-1: 2016-08-02T11:30:25-04:00 <3.6> RTPIsilon01-1 isi_pkg[3496]: Syncing patch data from node 1
X-1: 2016-08-02T11:30:30-04:00 <3.6> RTPIsilon01-1 isi_pkg[3514]: Starting task for request 'INSTALL', task 'INSTALL_INIT', hash 'e0102946ea3e53b5fdb79dec305d383f'
X-1: 2016-08-02T11:30:33-04:00 <3.6> RTPIsilon01-1 isi_pkg[3514]: Running requirements file first for Drive_Support_v1.11_No_SSD..
X-1: 2016-08-02T11:30:33-04:00 <3.3> RTPIsilon01-1 isi_pkg[3514]: install_init: Node failed patch requirements check
X-1: 2016-08-02T11:30:33-04:00 <3.6> RTPIsilon01-1 isi_pkg[3514]: Syncing patch data from node 1
X-2: 2016-06-27T18:17:54-04:00 <3.6> RTPIsilon01-2 isi_pkg[19147]: Syncing patch data from node 2
X-2: 2016-07-05T18:29:50-04:00 <3.6> RTPIsilon01-2 isi_pkg[4792]: Syncing patch data from node 2
X-2: 2016-07-05T18:29:50-04:00 <3.6> RTPIsilon01-2 isi_pkg[4808]: Syncing patch data from node 2
X-2: 2016-07-05T18:29:57-04:00 <3.6> RTPIsilon01-2 isi_pkg[5059]: Syncing patch data from node 2
X-2: 2016-07-06T02:09:46-04:00 <3.6> RTPIsilon01-2 isi_pkg[20815]: Syncing patch data from node 2
X-2: 2016-08-02T11:30:25-04:00 <3.6> RTPIsilon01-2 isi_pkg[17858]: Syncing patch data from node 2
X-2: 2016-08-02T11:30:30-04:00 <3.6> RTPIsilon01-2 isi_pkg[17875]: Starting task for request 'INSTALL', task 'INSTALL_INIT', hash 'e0102946ea3e53b5fdb79dec305d383f'
X-2: 2016-08-02T11:30:34-04:00 <3.6> RTPIsilon01-2 isi_pkg[17875]: Running requirements file first for Drive_Support_v1.11_No_SSD..
X-2: 2016-08-02T11:30:38-04:00 <3.3> RTPIsilon01-2 isi_pkg[17875]: install_init: Node failed patch requirements check
X-2: 2016-08-02T11:30:38-04:00 <3.6> RTPIsilon01-2 isi_pkg[17875]: Syncing patch data from node 2
X-3: 2016-07-05T18:29:33-04:00 <3.4> RTPIsilon01-3 isi_pkg[4256]: Failed to sync data from node 3
X-3: 2016-07-05T18:29:54-04:00 <3.6> RTPIsilon01-3 isi_pkg[4975]: Syncing patch data from node 3
X-3: 2016-07-05T18:30:01-04:00 <3.6> RTPIsilon01-3 isi_pkg[5180]: Syncing patch data from node 3
X-3: 2016-07-06T02:09:51-04:00 <3.6> RTPIsilon01-3 isi_pkg[3012]: Syncing patch data from node 3
X-3: 2016-07-06T02:09:51-04:00 <3.6> RTPIsilon01-3 isi_pkg[3021]: Syncing patch data from node 3
X-3: 2016-08-02T11:30:25-04:00 <3.6> RTPIsilon01-3 isi_pkg[77806]: Syncing patch data from node 3
X-3: 2016-08-02T11:30:30-04:00 <3.6> RTPIsilon01-3 isi_pkg[77816]: Starting task for request 'INSTALL', task 'INSTALL_INIT', hash 'e0102946ea3e53b5fdb79dec305d383f'
X-3: 2016-08-02T11:30:34-04:00 <3.6> RTPIsilon01-3 isi_pkg[77816]: Running requirements file first for Drive_Support_v1.11_No_SSD..
X-3: 2016-08-02T11:30:35-04:00 <3.3> RTPIsilon01-3 isi_pkg[77816]: install_init: Node failed patch requirements check
X-3: 2016-08-02T11:30:35-04:00 <3.6> RTPIsilon01-3 isi_pkg[77816]: Syncing patch data from node 3

Cause	A mechanism was put in place in OneFS 7.x to prevent both the normal and the No_SSD versions of the DSP from installing on their respective unsupported OneFS versions, since using the normal DSP on OneFS versions 7.1.1.0-7.1.1.3 and 7.2.0.0-7.2.0.1 can cause data loss if the cluster is in L3 cache mode (bug 145206/ETA 200097). This bug was resolved before OneFS 8.0 was released, so the normal DSP (with SSD firmware) should be used on all OneFS 8.0 versions. However the mechanism that prevents the unsupported DSP version (in this case, the No_SSD variant) from installing appears to be broken in OneFS 8.0, causing this issue.
Resolution
NOTE: This is a manual workaround to resolve the partially installed patch condition, which requires interaction with the OneFS CLI at an advanced level. If you are not familiar or confident with the OneFS CLI or your ability to enter and execute a provided Python script in real-time, please do not proceed and contact a coach or SME for assistance.

1.  Abort and archive the current patch process:

# isi upgrade patches abort

You are about to attempt to abort the currently running Patch process. Are you sure? (yes/[no]): yes
The Patch Request has been aborted, please run 'isi upgrade cluster archive --clear' to clean up the request.

# isi upgrade cluster archive --clear

You are about to produce an Archive of the Upgrade Frameworks current state. This will interrupt the Upgrade process until done. Would you like to continue?? (yes/[no]): yes

The requested Archive will not only produce an Archive, but will also clear the currently Upgrade process. This is not recoverable once done, are you sure you would like to do this?? (yes/[no]): yes

Archive Request initiated.


2.  Verify the patch process is no longer running:

# isi upgrade view

Upgrade Status:

      Cluster Upgrade State: committed
      Current Upgrade Activity: -
      Upgrade Start Time: 2016-06-21T15:19:25
      Current OS Version: 8.0.0.0_build(37)style(5)
      Upgrade OS Version: N/A


3.  Verify that you still see the patch in the Installing state:

# isi upgrade patches list
Patch Name                 Description           Status
-----------------------------------------------------------
Drive_Support_v1.11_No_SSD Drive Support Package Installing
-----------------------------------------------------------
Total: 1

4.  Start Python, then type in or copy the provided script exactly (do not retype the '>>>', these are prompts generated by the python command):


NOTE: The package name below is an example.  You should use the package name shown in the output in step 3.

*** For OneFS 8.0.0.x, do the following: ***
# python

>>> import os
>>> import isi.misc
>>> os.environ["PKG_ALLOW_RESTRICTED"] = "true"
>>> rc=isi.misc.pkg_request('delete','Drive_Support_v1.11_No_SSD',False, False, False)

** The text Drive_Support_v1.11_No_SSD should be replaced depending on the No_SSD package you are trying to remove, since you might be dealing with a different version other than 1.11

The node will respond with the following output:

Patch delete has been requested for patch Drive_Support_v1.11_No_SSD

Once this is done, exit Python:

>>> quit()

*** For OneFS 8.0.1.x, do the following: ***
# python

>>> import os
>>> import isi.patch
>>> os.environ["PKG_ALLOW_RESTRICTED"] = "true"
>>> rc=isi.patch.pkg_request('delete','Drive_Support_v1.14_No_SSD',False, False, False)

** The text Drive_Support_v1.14_No_SSD should be replaced depending on the No_SSD package you are trying to remove, since you might be dealing with a different version other than 1.14

The node will respond with the following output:

Patch delete has been requested for patch Drive_Support_v1.14_No_SSD

Once this is done, exit Python:

>>> quit()



5.  Verify that patch is no longer listed as being in progress:

# isi upgrade patch list
Patch Name Description Status
-----------------------------
-----------------------------
Total: 0