logost

Tests of TCM Cluster over DLM

Aug 3rd, 2022
528
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Bash 15.06 KB | None | 0 0
  1. # Tests of TCM Cluster over DLM
  2. register, reserve, release, preempt via sg_persist tool.
  3. check that everything is equal on both nodes.
  4.  
  5. ### READ FULL STATUS
  6.  
  7. READ FULL STATUS from SP0:
  8. ```
  9. # sg_persist -s /dev/sda
  10.   YADRO     foo               0250
  11.   Peripheral device type: disk
  12.   PR generation=0x6
  13.     Key=0x123abc
  14.       All target ports bit clear
  15.       Relative port address: 0x1
  16.       << Reservation holder >>
  17.       scope: LU_SCOPE,  type: Write Exclusive
  18.       Transport Id of initiator:
  19.         iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d00000b
  20.     Key=0xabc123
  21.       All target ports bit clear
  22.       Relative port address: 0x2
  23.       not reservation holder
  24.       Transport Id of initiator:
  25.         iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d00000c
  26. ```
  27. READ FULL STATUS from SP1 (the same relsult):
  28. ```
  29. # sg_persist -s /dev/sdb
  30.   YADRO     foo               0250
  31.   Peripheral device type: disk
  32.   PR generation=0x6
  33.     Key=0x123abc
  34.       All target ports bit clear
  35.       Relative port address: 0x1
  36.       << Reservation holder >>
  37.       scope: LU_SCOPE,  type: Write Exclusive
  38.       Transport Id of initiator:
  39.         iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d00000b
  40.     Key=0xabc123
  41.       All target ports bit clear
  42.       Relative port address: 0x2
  43.       not reservation holder
  44.       Transport Id of initiator:
  45.         iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d00000c
  46. ```
  47.  
  48. ### targetcli ls
  49.  
  50. targetcli ls on SP0:
  51. ```
  52.   o- iscsi ............................................................................... [Targets: 1]
  53.   | o- iqn.2017-01.com.yadro:tatlin:sn.090821008b1002 ....................................... [TPGs: 1]
  54.   |   o- tpg1 ........................................................ [no-gen-acls, no-auth, rtpi=0x1]
  55.   |     o- acls ............................................................................. [ACLs: 1]
  56.   |     | o- iqn.1996-04.de.suse:01:ad73852d2452 ..................................... [Mapped LUNs: 1]
  57.   |     |   o- mapped_lun0 ...................................................... [lun0 block/foo (rw)]
  58.   |     o- luns ............................................................................. [LUNs: 1]
  59.   |     | o- lun0 .......................................... [block/foo (/dev/ram0) (default_tg_pt_gp)]
  60.   |     o- portals ....................................................................... [Portals: 1]
  61.   |       o- 0.0.0.0:3260 ........................................................................ [OK]
  62.   o- loopback ............................................................................ [Targets: 0]
  63.   o- remote .............................................................................. [Targets: 1]
  64.   | o- iqn.2017-01.com.yadro:tatlin:sn.090821008b1002 ....................................... [TPGs: 1]
  65.   |   o- tpg1 ................................................................. [no-gen-acls, rtpi=0x2]
  66.   |     o- acls ............................................................................. [ACLs: 1]
  67.   |     | o- iqn.1996-04.de.suse:01:ad73852d2452 ..................................... [Mapped LUNs: 1]
  68.   |     |   o- mapped_lun0 ...................................................... [lun0 block/foo (rw)]
  69.   |     o- luns ............................................................................. [LUNs: 1]
  70.   |       o- lun0 .......................................... [block/foo (/dev/ram0) (default_tg_pt_gp)]
  71. ```
  72. targetcli ls on SP1 (RTPI of local and remote tpg are vice versa comparing to SP0):
  73. ```
  74.   o- iscsi ................................................................................ [Targets: 1]
  75.   | o- iqn.2017-01.com.yadro:tatlin:sn.090821008b1002 ........................................ [TPGs: 1]
  76.   |   o- tpg1 ......................................................... [no-gen-acls, no-auth, rtpi=0x2]
  77.   |     o- acls .............................................................................. [ACLs: 1]
  78.   |     | o- iqn.1996-04.de.suse:01:ad73852d2452 ...................................... [Mapped LUNs: 1]
  79.   |     |   o- mapped_lun0 ....................................................... [lun0 block/foo (rw)]
  80.   |     o- luns .............................................................................. [LUNs: 1]
  81.   |     | o- lun0 ........................................... [block/foo (/dev/ram0) (default_tg_pt_gp)]
  82.   |     o- portals ........................................................................ [Portals: 1]
  83.   |       o- 0.0.0.0:3260 ......................................................................... [OK]
  84.   o- loopback ............................................................................. [Targets: 0]
  85.   o- remote ............................................................................... [Targets: 1]
  86.   | o- iqn.2017-01.com.yadro:tatlin:sn.090821008b1002 ........................................ [TPGs: 1]
  87.   |   o- tpg1 .................................................................. [no-gen-acls, rtpi=0x1]
  88.   |     o- acls .............................................................................. [ACLs: 1]
  89.   |     | o- iqn.1996-04.de.suse:01:ad73852d2452 ...................................... [Mapped LUNs: 1]
  90.   |     |   o- mapped_lun0 ....................................................... [lun0 block/foo (rw)]
  91.   |     o- luns .............................................................................. [LUNs: 1]
  92.   |       o- lun0 ........................................... [block/foo (/dev/ram0) (default_tg_pt_gp)]
  93. ```
  94.  
  95. ### Stress test
  96.  
  97. execute SCSI.ProutPreempt test from 300 different initiators in parallell:
  98. ```
  99. for i in $(seq 1 2 499); do
  100.  test-tool/iscsi-test-cu -t SCSI.ProutPreempt --dataloss -i iqn.1996-04.com.local:$i -I iqn.1996-04.com.local:$(($i + 1))  iscsi://172.20.29.210:3260/iqn.2017-01.com.yadro:tatlin:sn.090821008b1002/0 iscsi://172.20.28.211:3260/iqn.2017-01.com.yadro:tatlin:sn.090821008b1002/0 &
  101. done
  102. ```
  103. UA after PROUT CLEAR on peer node:
  104. ```
  105. [66706.589088] SPC-3 PR [remote] Service Action: UNREGISTER Initiator Node: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000023
  106. [66706.589211] SPC-3 PR [remote] for SINGLE TCM Subsystem iblock Object Target Port(s)
  107. [66706.589270] SPC-3 PR [remote] SA Res Key: 0x000000000000bbbb PRgeneration: 0x00000000
  108. [66706.589333] SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000024
  109. [66706.589422] SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
  110. [66706.589481] SPC-3 PR [iSCSI] SA Res Key: 0x000000000000aaaa PRgeneration: 0x0000000f
  111. [66706.589545] Allocated UNIT ATTENTION, mapped LUN: 0, ASC: 0x2a, ASCQ: 0x03
  112. [66724.010317] -----[ Storage Engine Exception; sense_reason 14
  113. [66724.010411] -----[ cmd 0x0:0x0 with tag 0x1000007a dir NONE i_state 1 t_state PROCESSING len 0 refcnt 2 transport_state active
  114. [66724.010504] [iSCSI]: Releasing UNIT ATTENTION condition with INTLCK_CTRL: 0, mapped LUN: 0, got CDB: 0x00 reported ASC: 0x2a, ASCQ: 0x03
  115. ```
  116.  
  117. ### PREEPMT AND ABORT
  118.  
  119. Initiator, reservation on /dev/sdb, preempt to /dev/sda:
  120. ```
  121. # sg_persist  -is /dev/sda
  122.   YADRO     foo0              0250
  123.   Peripheral device type: disk
  124.   PR generation=0x6
  125.     Key=0xbbbb
  126.       All target ports bit clear
  127.       Relative port address: 0x2
  128.       << Reservation holder >>
  129.       scope: LU_SCOPE,  type: Exclusive Access, registrants only
  130.       Transport Id of initiator:
  131.         iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000023
  132.     Key=0xaaaa
  133.       All target ports bit clear
  134.       Relative port address: 0x1
  135.       not reservation holder
  136.       Transport Id of initiator:
  137.         iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000024
  138. # sg_persist -oA -S 0xBBBB -T 6 -K 0xAAAA /dev/sda
  139.   YADRO     foo0              0250
  140.   Peripheral device type: disk
  141. # sg_persist  -is /dev/sda
  142.   YADRO     foo0              0250
  143.   Peripheral device type: disk
  144.   PR generation=0x7
  145.     Key=0xaaaa
  146.       All target ports bit clear
  147.       Relative port address: 0x1
  148.       << Reservation holder >>
  149.       scope: LU_SCOPE,  type: Exclusive Access, registrants only
  150.       Transport Id of initiator:
  151.         iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000024
  152. ```
  153. Peer node (dev/sdb)
  154. ```
  155. [ 1700.829539] SPC-3 PR [iSCSI] Service Action: implicit RELEASE cleared reservation holder TYPE: Exclusive Access, Registrants Only ALL_TG_PT: 0
  156. [ 1700.829634] SPC-3 PR [iSCSI] RELEASE Node: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000023
  157. [ 1700.829704] SPC-3 PR [remote] Service Action: RESERVE created new reservation holder TYPE: Exclusive Access, Registrants Only ALL_TG_PT: 0
  158. [ 1700.829790] SPC-3 PR [remote] RESERVE Node: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000024
  159. [ 1700.829859] SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000023
  160. [ 1700.829946] SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
  161. [ 1700.830005] SPC-3 PR [iSCSI] SA Res Key: 0x000000000000bbbb PRgeneration: 0x00000004
  162. [ 1700.830066] LUN_RESET: Preempt starting for [iblock], tas: 1
  163. [ 1700.830116] LUN_RESET: Preempt for [iblock] Complete
  164. ```
  165.  
  166. ### REGISTER AND MOVE to remote port
  167.  
  168. Initiator, reservation on /dev/sda (SP0), move to /dev/sde(SP1):
  169. ```
  170. # sg_persist -is /dev/sde
  171.   YADRO     foo0              0250
  172.   Peripheral device type: disk
  173.   PR generation=0x2
  174.     Key=0x1234
  175.       All target ports bit clear
  176.       Relative port address: 0x1
  177.       << Reservation holder >>
  178.       scope: LU_SCOPE,  type: Write Exclusive
  179.       Transport Id of initiator:
  180.         iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000015
  181.     Key=0x5678
  182.       All target ports bit clear
  183.       Relative port address: 0x2
  184.       not reservation holder
  185.       Transport Id of initiator:
  186.         iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000016
  187.  
  188. # sg_persist -oM -K 0x1234 -S 0x5678 -Q 2 -X iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000016 -T 1 /dev/sda
  189.   YADRO     foo0              0250
  190.   Peripheral device type: disk
  191.  
  192.    # sg_persist -is /dev/sda
  193.   YADRO     foo0              0250
  194.   Peripheral device type: disk
  195.   PR generation=0x3
  196.     Key=0x1234
  197.       All target ports bit clear
  198.       Relative port address: 0x1
  199.       not reservation holder
  200.       Transport Id of initiator:
  201.         iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000015
  202.     Key=0x5678
  203.       All target ports bit clear
  204.       Relative port address: 0x2
  205.       << Reservation holder >>
  206.       scope: LU_SCOPE,  type: Write Exclusive
  207.       Transport Id of initiator:
  208.         iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000016
  209. ```
  210. SP0 (Reservation holder) :
  211. ```
  212. [357843.580633] SPC-3 PR REGISTER_AND_MOVE: Extracted Protocol Identifier: 0x05
  213. [357843.580727] SPC-3 PR [remote] Extracted initiator port identifier: iqn.1996-04.de.suse:01:ad73852d2452 00023d000016
  214. [357843.580812] SPC-3 PR REGISTER_AND_MOVE: Found remote dest_node_acl: iqn.1996-04.de.suse:01:ad73852d2452 from TransportID
  215. [357843.580893] SPC-3 PR REGISTER_AND_MOVE: Located remote node iqn.1996-04.de.suse:01:ad73852d2452 LUN ACL for dest_se_deve->mapped_lun: 2
  216. [357843.580986] SPC-3 PR Service Action: implicit RELEASE cleared reservation holder TYPE: Write Exclusive Access ALL_TG_PT: 0
  217. [357843.581065] SPC-3 PR RELEASE Node: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000015
  218. [357843.581125] SPC-3 PR [remote] Service Action: REGISTER_AND_MOVE created new reservation holder TYPE: Write Exclusive Access on object RTPI: 2  PRGeneration: 0x00000003
  219. [357843.581235] SPC-3 PR Successfully moved reservation from iSCSI Fabric Node: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000015 -> remote Fabric Node: iqn.1996-04.de.suse:01:ad73852d2452 00023d000016
  220. ```
  221. SP1 (reservation moved to):
  222. ```
  223. [625160.188585] Allowing explicit CDB: 0x00 for Write Exclusive Access reservation holder
  224. [625172.016288] SPC-3 PR Service Action: implicit RELEASE cleared reservation holder TYPE: Write Exclusive Access ALL_TG_PT: 0
  225. [625172.016406] SPC-3 PR RELEASE Node: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000015
  226. [625172.016466] SPC-3 PR Service Action: RESERVE created new reservation holder TYPE: Write Exclusive Access ALL_TG_PT: 0
  227. [625172.016543] SPC-3 PR RESERVE Node: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000016
  228. ```
  229.  
  230. ### LUN_RESET
  231.  
  232. Trigger LUN RESET on initiator:
  233. ```
  234. # sg_reset -d /dev/sda
  235. ```
  236. Check on /dev/sda:
  237. ```
  238. [864409.548843] LUN_RESET: TMR caller fabric: iSCSI initiator port iqn.1996-04.de.suse:01:ad73852d2452
  239. [864409.548954] LUN_RESET: TMR starting for [iblock], tas: 1
  240. [864409.548998] Allocated UNIT ATTENTION, mapped LUN: 0, ASC: 0x29, ASCQ: 0x03
  241. [864409.549049] LUN_RESET: TMR for [iblock] Complete
  242. ```
  243. Check on /dev/sdb (peer node):
  244. ```
  245. [518762.983893] LUN_RESET: TMR starting for [iblock], tas: 1
  246. [518762.983982] Allocated UNIT ATTENTION, mapped LUN: 0, ASC: 0x29, ASCQ: 0x03
  247. [518762.984033] LUN_RESET: TMR for [iblock] Complete
  248. ```
  249.  
  250. ### Initial Sync after master node died
  251.  
  252. Register on both I_T_L nexuses
  253. ```
  254. sg_persist -oG -S 0x1234 /dev/sda
  255. sg_persist -oG -S 0xBBBB /dev/sdd
  256. ```
  257. Reboot master node (SP0 in this case), check logs on next start:
  258.  
  259. ```
  260. [  120.264874] TARGET_CORE[1]: PR data from cluster is invalid
  261. [  120.267002] SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000027
  262. [  120.267092] SPC-3 PR [iSCSI] registration on Target Port: iqn.2017-01.com.yadro:tatlin:sn.090222008b1002,0x000a
  263. [  120.267169] SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
  264. [  120.267228] SPC-3 PR [iSCSI] SA Res Key: 0x0000000000001234 PRgeneration: 0x00000000  APTPL: 0
  265. [  120.267298] SPC-3 PR [remote] Service Action: REGISTER Initiator Node: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000028
  266. [  120.267377] SPC-3 PR [remote] registration on Target Port: iqn.2017-01.com.yadro:tatlin:sn.090222008b1002,0x0001
  267. [  120.267455] SPC-3 PR [remote] for SINGLE TCM Subsystem iblock Object Target Port(s)
  268. [  120.267513] SPC-3 PR [remote] SA Res Key: 0x000000000000bbbb PRgeneration: 0x00000000  APTPL: 0
  269. ```
  270.  
  271. See that cluster data was invalid, but then it comes from other node. Initiator sees PR info from SP0:
  272. ```
  273. sg_persist -ik /dev/sda
  274.   YADRO     foo0              0250
  275.   Peripheral device type: disk
  276.   PR generation=0x2, 2 registered reservation keys follow:
  277.     0x1234
  278.     0xbbbb
  279. ```
  280.  
  281. ### Test APTPL
  282. create backstore device
  283. put APTPL info to `.../pr/pr_aptpl_metadata file`
  284. export resourse (create TPG/LUN/etc)
  285. check that registration exists
  286. ```
  287. dev-dbogdanov:~ # sg_persist -is /dev/sdd
  288.   YADRO     foo0              0250
  289.   Peripheral device type: disk
  290.   PR generation=0x1
  291.     Key=0x1234
  292.       All target ports bit clear
  293.       Relative port address: 0x1
  294.       not reservation holder
  295.       Transport Id of initiator:
  296.         iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000003
  297. ```
Advertisement
Add Comment
Please, Sign In to add comment