Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- # Tests of TCM Cluster over DLM
- register, reserve, release, preempt via sg_persist tool.
- check that everything is equal on both nodes.
- ### READ FULL STATUS
- READ FULL STATUS from SP0:
- ```
- # sg_persist -s /dev/sda
- YADRO foo 0250
- Peripheral device type: disk
- PR generation=0x6
- Key=0x123abc
- All target ports bit clear
- Relative port address: 0x1
- << Reservation holder >>
- scope: LU_SCOPE, type: Write Exclusive
- Transport Id of initiator:
- iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d00000b
- Key=0xabc123
- All target ports bit clear
- Relative port address: 0x2
- not reservation holder
- Transport Id of initiator:
- iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d00000c
- ```
- READ FULL STATUS from SP1 (the same relsult):
- ```
- # sg_persist -s /dev/sdb
- YADRO foo 0250
- Peripheral device type: disk
- PR generation=0x6
- Key=0x123abc
- All target ports bit clear
- Relative port address: 0x1
- << Reservation holder >>
- scope: LU_SCOPE, type: Write Exclusive
- Transport Id of initiator:
- iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d00000b
- Key=0xabc123
- All target ports bit clear
- Relative port address: 0x2
- not reservation holder
- Transport Id of initiator:
- iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d00000c
- ```
- ### targetcli ls
- targetcli ls on SP0:
- ```
- o- iscsi ............................................................................... [Targets: 1]
- | o- iqn.2017-01.com.yadro:tatlin:sn.090821008b1002 ....................................... [TPGs: 1]
- | o- tpg1 ........................................................ [no-gen-acls, no-auth, rtpi=0x1]
- | o- acls ............................................................................. [ACLs: 1]
- | | o- iqn.1996-04.de.suse:01:ad73852d2452 ..................................... [Mapped LUNs: 1]
- | | o- mapped_lun0 ...................................................... [lun0 block/foo (rw)]
- | o- luns ............................................................................. [LUNs: 1]
- | | o- lun0 .......................................... [block/foo (/dev/ram0) (default_tg_pt_gp)]
- | o- portals ....................................................................... [Portals: 1]
- | o- 0.0.0.0:3260 ........................................................................ [OK]
- o- loopback ............................................................................ [Targets: 0]
- o- remote .............................................................................. [Targets: 1]
- | o- iqn.2017-01.com.yadro:tatlin:sn.090821008b1002 ....................................... [TPGs: 1]
- | o- tpg1 ................................................................. [no-gen-acls, rtpi=0x2]
- | o- acls ............................................................................. [ACLs: 1]
- | | o- iqn.1996-04.de.suse:01:ad73852d2452 ..................................... [Mapped LUNs: 1]
- | | o- mapped_lun0 ...................................................... [lun0 block/foo (rw)]
- | o- luns ............................................................................. [LUNs: 1]
- | o- lun0 .......................................... [block/foo (/dev/ram0) (default_tg_pt_gp)]
- ```
- targetcli ls on SP1 (RTPI of local and remote tpg are vice versa comparing to SP0):
- ```
- o- iscsi ................................................................................ [Targets: 1]
- | o- iqn.2017-01.com.yadro:tatlin:sn.090821008b1002 ........................................ [TPGs: 1]
- | o- tpg1 ......................................................... [no-gen-acls, no-auth, rtpi=0x2]
- | o- acls .............................................................................. [ACLs: 1]
- | | o- iqn.1996-04.de.suse:01:ad73852d2452 ...................................... [Mapped LUNs: 1]
- | | o- mapped_lun0 ....................................................... [lun0 block/foo (rw)]
- | o- luns .............................................................................. [LUNs: 1]
- | | o- lun0 ........................................... [block/foo (/dev/ram0) (default_tg_pt_gp)]
- | o- portals ........................................................................ [Portals: 1]
- | o- 0.0.0.0:3260 ......................................................................... [OK]
- o- loopback ............................................................................. [Targets: 0]
- o- remote ............................................................................... [Targets: 1]
- | o- iqn.2017-01.com.yadro:tatlin:sn.090821008b1002 ........................................ [TPGs: 1]
- | o- tpg1 .................................................................. [no-gen-acls, rtpi=0x1]
- | o- acls .............................................................................. [ACLs: 1]
- | | o- iqn.1996-04.de.suse:01:ad73852d2452 ...................................... [Mapped LUNs: 1]
- | | o- mapped_lun0 ....................................................... [lun0 block/foo (rw)]
- | o- luns .............................................................................. [LUNs: 1]
- | o- lun0 ........................................... [block/foo (/dev/ram0) (default_tg_pt_gp)]
- ```
- ### Stress test
- execute SCSI.ProutPreempt test from 300 different initiators in parallell:
- ```
- for i in $(seq 1 2 499); do
- test-tool/iscsi-test-cu -t SCSI.ProutPreempt --dataloss -i iqn.1996-04.com.local:$i -I iqn.1996-04.com.local:$(($i + 1)) iscsi://172.20.29.210:3260/iqn.2017-01.com.yadro:tatlin:sn.090821008b1002/0 iscsi://172.20.28.211:3260/iqn.2017-01.com.yadro:tatlin:sn.090821008b1002/0 &
- done
- ```
- UA after PROUT CLEAR on peer node:
- ```
- [66706.589088] SPC-3 PR [remote] Service Action: UNREGISTER Initiator Node: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000023
- [66706.589211] SPC-3 PR [remote] for SINGLE TCM Subsystem iblock Object Target Port(s)
- [66706.589270] SPC-3 PR [remote] SA Res Key: 0x000000000000bbbb PRgeneration: 0x00000000
- [66706.589333] SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000024
- [66706.589422] SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
- [66706.589481] SPC-3 PR [iSCSI] SA Res Key: 0x000000000000aaaa PRgeneration: 0x0000000f
- [66706.589545] Allocated UNIT ATTENTION, mapped LUN: 0, ASC: 0x2a, ASCQ: 0x03
- [66724.010317] -----[ Storage Engine Exception; sense_reason 14
- [66724.010411] -----[ cmd 0x0:0x0 with tag 0x1000007a dir NONE i_state 1 t_state PROCESSING len 0 refcnt 2 transport_state active
- [66724.010504] [iSCSI]: Releasing UNIT ATTENTION condition with INTLCK_CTRL: 0, mapped LUN: 0, got CDB: 0x00 reported ASC: 0x2a, ASCQ: 0x03
- ```
- ### PREEPMT AND ABORT
- Initiator, reservation on /dev/sdb, preempt to /dev/sda:
- ```
- # sg_persist -is /dev/sda
- YADRO foo0 0250
- Peripheral device type: disk
- PR generation=0x6
- Key=0xbbbb
- All target ports bit clear
- Relative port address: 0x2
- << Reservation holder >>
- scope: LU_SCOPE, type: Exclusive Access, registrants only
- Transport Id of initiator:
- iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000023
- Key=0xaaaa
- All target ports bit clear
- Relative port address: 0x1
- not reservation holder
- Transport Id of initiator:
- iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000024
- # sg_persist -oA -S 0xBBBB -T 6 -K 0xAAAA /dev/sda
- YADRO foo0 0250
- Peripheral device type: disk
- # sg_persist -is /dev/sda
- YADRO foo0 0250
- Peripheral device type: disk
- PR generation=0x7
- Key=0xaaaa
- All target ports bit clear
- Relative port address: 0x1
- << Reservation holder >>
- scope: LU_SCOPE, type: Exclusive Access, registrants only
- Transport Id of initiator:
- iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000024
- ```
- Peer node (dev/sdb)
- ```
- [ 1700.829539] SPC-3 PR [iSCSI] Service Action: implicit RELEASE cleared reservation holder TYPE: Exclusive Access, Registrants Only ALL_TG_PT: 0
- [ 1700.829634] SPC-3 PR [iSCSI] RELEASE Node: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000023
- [ 1700.829704] SPC-3 PR [remote] Service Action: RESERVE created new reservation holder TYPE: Exclusive Access, Registrants Only ALL_TG_PT: 0
- [ 1700.829790] SPC-3 PR [remote] RESERVE Node: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000024
- [ 1700.829859] SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000023
- [ 1700.829946] SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
- [ 1700.830005] SPC-3 PR [iSCSI] SA Res Key: 0x000000000000bbbb PRgeneration: 0x00000004
- [ 1700.830066] LUN_RESET: Preempt starting for [iblock], tas: 1
- [ 1700.830116] LUN_RESET: Preempt for [iblock] Complete
- ```
- ### REGISTER AND MOVE to remote port
- Initiator, reservation on /dev/sda (SP0), move to /dev/sde(SP1):
- ```
- # sg_persist -is /dev/sde
- YADRO foo0 0250
- Peripheral device type: disk
- PR generation=0x2
- Key=0x1234
- All target ports bit clear
- Relative port address: 0x1
- << Reservation holder >>
- scope: LU_SCOPE, type: Write Exclusive
- Transport Id of initiator:
- iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000015
- Key=0x5678
- All target ports bit clear
- Relative port address: 0x2
- not reservation holder
- Transport Id of initiator:
- iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000016
- # sg_persist -oM -K 0x1234 -S 0x5678 -Q 2 -X iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000016 -T 1 /dev/sda
- YADRO foo0 0250
- Peripheral device type: disk
- # sg_persist -is /dev/sda
- YADRO foo0 0250
- Peripheral device type: disk
- PR generation=0x3
- Key=0x1234
- All target ports bit clear
- Relative port address: 0x1
- not reservation holder
- Transport Id of initiator:
- iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000015
- Key=0x5678
- All target ports bit clear
- Relative port address: 0x2
- << Reservation holder >>
- scope: LU_SCOPE, type: Write Exclusive
- Transport Id of initiator:
- iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000016
- ```
- SP0 (Reservation holder) :
- ```
- [357843.580633] SPC-3 PR REGISTER_AND_MOVE: Extracted Protocol Identifier: 0x05
- [357843.580727] SPC-3 PR [remote] Extracted initiator port identifier: iqn.1996-04.de.suse:01:ad73852d2452 00023d000016
- [357843.580812] SPC-3 PR REGISTER_AND_MOVE: Found remote dest_node_acl: iqn.1996-04.de.suse:01:ad73852d2452 from TransportID
- [357843.580893] SPC-3 PR REGISTER_AND_MOVE: Located remote node iqn.1996-04.de.suse:01:ad73852d2452 LUN ACL for dest_se_deve->mapped_lun: 2
- [357843.580986] SPC-3 PR Service Action: implicit RELEASE cleared reservation holder TYPE: Write Exclusive Access ALL_TG_PT: 0
- [357843.581065] SPC-3 PR RELEASE Node: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000015
- [357843.581125] SPC-3 PR [remote] Service Action: REGISTER_AND_MOVE created new reservation holder TYPE: Write Exclusive Access on object RTPI: 2 PRGeneration: 0x00000003
- [357843.581235] SPC-3 PR Successfully moved reservation from iSCSI Fabric Node: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000015 -> remote Fabric Node: iqn.1996-04.de.suse:01:ad73852d2452 00023d000016
- ```
- SP1 (reservation moved to):
- ```
- [625160.188585] Allowing explicit CDB: 0x00 for Write Exclusive Access reservation holder
- [625172.016288] SPC-3 PR Service Action: implicit RELEASE cleared reservation holder TYPE: Write Exclusive Access ALL_TG_PT: 0
- [625172.016406] SPC-3 PR RELEASE Node: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000015
- [625172.016466] SPC-3 PR Service Action: RESERVE created new reservation holder TYPE: Write Exclusive Access ALL_TG_PT: 0
- [625172.016543] SPC-3 PR RESERVE Node: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000016
- ```
- ### LUN_RESET
- Trigger LUN RESET on initiator:
- ```
- # sg_reset -d /dev/sda
- ```
- Check on /dev/sda:
- ```
- [864409.548843] LUN_RESET: TMR caller fabric: iSCSI initiator port iqn.1996-04.de.suse:01:ad73852d2452
- [864409.548954] LUN_RESET: TMR starting for [iblock], tas: 1
- [864409.548998] Allocated UNIT ATTENTION, mapped LUN: 0, ASC: 0x29, ASCQ: 0x03
- [864409.549049] LUN_RESET: TMR for [iblock] Complete
- ```
- Check on /dev/sdb (peer node):
- ```
- [518762.983893] LUN_RESET: TMR starting for [iblock], tas: 1
- [518762.983982] Allocated UNIT ATTENTION, mapped LUN: 0, ASC: 0x29, ASCQ: 0x03
- [518762.984033] LUN_RESET: TMR for [iblock] Complete
- ```
- ### Initial Sync after master node died
- Register on both I_T_L nexuses
- ```
- sg_persist -oG -S 0x1234 /dev/sda
- sg_persist -oG -S 0xBBBB /dev/sdd
- ```
- Reboot master node (SP0 in this case), check logs on next start:
- ```
- [ 120.264874] TARGET_CORE[1]: PR data from cluster is invalid
- [ 120.267002] SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000027
- [ 120.267092] SPC-3 PR [iSCSI] registration on Target Port: iqn.2017-01.com.yadro:tatlin:sn.090222008b1002,0x000a
- [ 120.267169] SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
- [ 120.267228] SPC-3 PR [iSCSI] SA Res Key: 0x0000000000001234 PRgeneration: 0x00000000 APTPL: 0
- [ 120.267298] SPC-3 PR [remote] Service Action: REGISTER Initiator Node: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000028
- [ 120.267377] SPC-3 PR [remote] registration on Target Port: iqn.2017-01.com.yadro:tatlin:sn.090222008b1002,0x0001
- [ 120.267455] SPC-3 PR [remote] for SINGLE TCM Subsystem iblock Object Target Port(s)
- [ 120.267513] SPC-3 PR [remote] SA Res Key: 0x000000000000bbbb PRgeneration: 0x00000000 APTPL: 0
- ```
- See that cluster data was invalid, but then it comes from other node. Initiator sees PR info from SP0:
- ```
- sg_persist -ik /dev/sda
- YADRO foo0 0250
- Peripheral device type: disk
- PR generation=0x2, 2 registered reservation keys follow:
- 0x1234
- 0xbbbb
- ```
- ### Test APTPL
- create backstore device
- put APTPL info to `.../pr/pr_aptpl_metadata file`
- export resourse (create TPG/LUN/etc)
- check that registration exists
- ```
- dev-dbogdanov:~ # sg_persist -is /dev/sdd
- YADRO foo0 0250
- Peripheral device type: disk
- PR generation=0x1
- Key=0x1234
- All target ports bit clear
- Relative port address: 0x1
- not reservation holder
- Transport Id of initiator:
- iSCSI world wide unique port id: iqn.1996-04.de.suse:01:ad73852d2452,i,0x00023d000003
- ```
Advertisement
Add Comment
Please, Sign In to add comment