API tools faq
paste
gugus

Chelsio connectivity problem

gugus
May 27th, 2015
396
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 15.00 KB | None | 0 0
raw download clone embed print report
  1.  
  2. ===== First test with 2 different servers =====
  3.  
  4.  
  5. 2 servers:
  6. - IBM x3550M3 with Intel 82599EB 10-Gigabit (ix0 and ix1)
  7. - HP proliant with Chelsio T540-CR (cxl0 and cxl1 used)
  8.  
  9. IBM ix0 <===> cxl0 HP
  10. ix1 <===> cxl1
  11.  
  12. ===== HP side NIC status =========
  13.  
  14. [root@hp]~# uname -a
  15. FreeBSD hp 11.0-CURRENT FreeBSD 11.0-CURRENT #3 r283536M: Wed May 27 14:19:39 CEST 2015 [email protected]:/usr/obj/BSDRPcur.amd64/usr/local/BSDRP/BSDRPcur/FreeBSD/src/sys/amd64 amd64
  16.  
  17. [root@hp]~# ifconfig -m cxl0
  18. cxl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  19. options=ec00bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
  20. capabilities=ecc7bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,TOE4,TOE6,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
  21. ether 00:07:43:2e:e4:70
  22. inet 8.8.8.2 netmask 0xffffff00 broadcast 8.8.8.255
  23. inet6 fe80::207:43ff:fe2e:e470%cxl0 prefixlen 64 scopeid 0x5
  24. nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
  25. media: Ethernet 10Gbase-SR <full-duplex>
  26. status: no carrier
  27. supported media:
  28. media 10Gbase-SR mediaopt full-duplex
  29. [root@hp]~# ifconfig -m cxl1
  30. cxl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  31. options=ec00bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
  32. capabilities=ecc7bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,TOE4,TOE6,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
  33. ether 00:07:43:2e:e4:78
  34. inet 9.9.9.2 netmask 0xffffff00 broadcast 9.9.9.255
  35. inet6 fe80::207:43ff:fe2e:e478%cxl1 prefixlen 64 scopeid 0x7
  36. nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
  37. media: Ethernet 10Gbase-SR <full-duplex>
  38. status: no carrier
  39. supported media:
  40. media 10Gbase-SR mediaopt full-duplex
  41.  
  42.  
  43. [root@hp]~# ifconfig -v cxl0
  44. cxl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  45. options=ec00bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
  46. ether 00:07:43:2e:e4:70
  47. inet 8.8.8.2 netmask 0xffffff00 broadcast 8.8.8.255
  48. inet6 fe80::207:43ff:fe2e:e470%cxl0 prefixlen 64 scopeid 0x5
  49. nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
  50. media: Ethernet 10Gbase-SR <full-duplex>
  51. status: no carrier
  52. plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
  53. vendor: Intel Corp PN: FTLX8571D3BCV-IT SN: AS50ETU DATE: 2014-08-06
  54. module temperature: 35.77 C Voltage: 3.20 Volts
  55. RX: 0.52 mW (-2.83 dBm) TX: 0.35 mW (-4.44 dBm)
  56. [root@hp]~# ifconfig -v cxl1
  57. cxl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  58. options=ec00bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
  59. ether 00:07:43:2e:e4:78
  60. inet 9.9.9.2 netmask 0xffffff00 broadcast 9.9.9.255
  61. inet6 fe80::207:43ff:fe2e:e478%cxl1 prefixlen 64 scopeid 0x7
  62. nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
  63. media: Ethernet 10Gbase-SR <full-duplex>
  64. status: no carrier
  65. plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
  66. vendor: Intel Corp PN: FTLX8571D3BCV-IT SN: AS60YV9 DATE: 2014-08-09
  67. module temperature: 37.00 C Voltage: 3.21 Volts
  68. RX: 0.29 mW (-5.30 dBm) TX: 0.45 mW (-3.45 dBm)
  69. [root@hp]~# cxgbetool t5nex0 modinfo 0
  70. ID: SFP/SFP+/SFP28
  71. Vendor Intel Corp
  72. SN AS50ETU
  73. PN FTLX8571D3BCV-I
  74. Rev A
  75. Temp: +36C
  76. Vcc 3.200000V
  77. TX Bias 1.152000uA
  78. TX Power 0.332800mW
  79. RX Power 0.512000mW
  80. [root@hp]~# cxgbetool t5nex0 modinfo 1
  81. ID: SFP/SFP+/SFP28
  82. Vendor Intel Corp
  83. SN AS60YV9
  84. PN FTLX8571D3BCV-I
  85. Rev A
  86. Temp: +37C
  87. Vcc 3.200000V
  88. TX Bias 1.280000uA
  89. TX Power 0.435200mW
  90. RX Power 0.281600mW
  91.  
  92. [root@hp]~# sysctl dev.cxl.0.linkdnrc
  93. dev.cxl.0.linkdnrc: n/a
  94. [root@hp]~# sysctl dev.cxl.1.linkdnrc
  95. dev.cxl.1.linkdnrc: n/a
  96.  
  97. [root@hp]~# cxgbetool t5nex0 modinfo 0 raw
  98. 00: 03 04 07 10 00 00 01 00 .... ....
  99. 08: 00 00 00 06 67 02 00 00 .... g...
  100. 10: 08 03 00 1e 49 6e 74 65 .... Inte
  101. 18: 6c 20 43 6f 72 70 20 20 l Co rp
  102. 20: 20 20 20 20 00 00 1b 21 ...!
  103. 28: 46 54 4c 58 38 35 37 31 FTLX 8571
  104. 30: 44 33 42 43 56 2d 49 54 D3BC V-IT
  105. 38: 41 20 20 20 03 52 00 88 A .R..
  106. 40: 00 3a 00 00 41 53 35 30 .:.. AS50
  107. 48: 45 54 55 20 20 20 20 20 ETU
  108. 50: 20 20 20 20 31 34 30 38 1408
  109. 58: 30 36 20 20 68 fa 03 19 06 h...
  110.  
  111. [root@hp]~# cxgbetool t5nex0 modinfo 1 raw
  112. 00: 03 04 07 10 00 00 01 00 .... ....
  113. 08: 00 00 00 06 67 02 00 00 .... g...
  114. 10: 08 03 00 1e 49 6e 74 65 .... Inte
  115. 18: 6c 20 43 6f 72 70 20 20 l Co rp
  116. 20: 20 20 20 20 00 00 1b 21 ...!
  117. 28: 46 54 4c 58 38 35 37 31 FTLX 8571
  118. 30: 44 33 42 43 56 2d 49 54 D3BC V-IT
  119. 38: 41 20 20 20 03 52 00 88 A .R..
  120. 40: 00 3a 00 00 41 53 36 30 .:.. AS60
  121. 48: 59 56 39 20 20 20 20 20 YV9
  122. 50: 20 20 20 20 31 34 30 38 1408
  123. 58: 30 39 20 20 68 fa 03 17 09 h...
  124.  
  125. [root@hp]~# sysctl -n dev.t5nex.0.misc.devlog
  126. Seq# Tstamp Level Facility Message
  127. 0 327507 INFO CORE log initialized @ 0x200a0000 size 32768 (128 entries) fwrev 0x010b1b00 pcie_fw 0x0014cc10
  128. 1 399888 INFO CORE bootstrap firmware took 27 msecs to run
  129. 2 401726 INFO CORE pcie: npf 7 (pfbitmap 0x7f) nvf 64 (pf 0..3 0x10 0x10 0x10 0x10)
  130. 3 412371 INFO CORE flr_timer_start: flowc_id 100 0x205fff80 buf 0x205fcd40
  131. 4 476831 INFO CF cf_parse: file memtype 0x1 memaddr 0x5e0000 mapped @ 0x205e0000:
  132. 5 476858 INFO CF configuration file parser: sge timer value[5] is too large, changing from 200 to 131usecs
  133. 6 477831 INFO CORE configured with caps nbm|link 0x00000000 switch|nic 0x00030001 toe|rdma 0x00010000 iscsi|fcoe 0x00000000
  134. 7 477848 INFO HW hw_tp_tcp_tunings: tuning for LAN environment
  135. 8 491646 INFO RES le configuration: nentries 2048 route 32 clip 32 filter 1024 server 512 active 448 hash 16384 nserversram 0
  136. 9 627528 INFO RES mpartition_init: moved pmrx_start from 0x00600000 to 0x00540000 (EDRAM)
  137. 10 627530 INFO CORE flr_timer_start: flowc_id 19666 0xa05fff80 buf 0xa0399080
  138. 11 641621 INFO RES le initialization: nentries 2048 route 32 clip 32 filter 1024 server 512 active 448 hash 16384 nserversram 0
  139. 12 708962 INFO PORT module[0]: gpio 11 trans 10G 0x10 1G 0xff (length 0) cable 0xff (length 0) module_type 0x02
  140. 13 709002 INFO TM pktsched channel 0 sets speed (from 0) to 10000000 kbps
  141. 14 710701 INFO PORT module[1]: gpio 15 trans 10G 0x10 1G 0xff (length 0) cable 0xff (length 0) module_type 0x02
  142. 15 710733 INFO TM pktsched channel 1 sets speed (from 0) to 10000000 kbps
  143. 16 710771 INFO TM pktsched channel 2 sets speed (from 0) to 10000000 kbps
  144. 17 710808 INFO TM pktsched channel 3 sets speed (from 0) to 10000000 kbps
  145. 18 11197585 INFO PORT port_link_state_handler[0] powering up
  146. 19 11197593 INFO PORT port[0] update (flowcid 19480)
  147. 20 11297588 INFO PORT port_link_state_handler[1] powering up
  148. 21 11297591 INFO PORT port[1] update (flowcid 19480)
  149.  
  150.  
  151. ============================================= IBM side ========================================
  152.  
  153. [root@R1]~# uname -a
  154. FreeBSD R1 11.0-CURRENT FreeBSD 11.0-CURRENT #3 r283536M: Wed May 27 14:19:39 CEST 2015 [email protected]:/usr/obj/BSDRPcur.amd64/usr/local/BSDRP/BSDRPcur/FreeBSD/src/sys/amd64 amd64
  155.  
  156. [root@R1]~# ifconfig -m ix0
  157. ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  158. options=8400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
  159. capabilities=9507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP>
  160. ether 90:e2:ba:84:20:38
  161. inet 8.8.8.1 netmask 0xffffff00 broadcast 8.8.8.255
  162. inet6 fe80::92e2:baff:fe84:2038%ix0 prefixlen 64 scopeid 0x3
  163. inet6 2001:db8:8::1 prefixlen 64
  164. nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
  165. media: Ethernet autoselect
  166. status: no carrier
  167. supported media:
  168. media autoselect
  169. media 10Gbase-SR
  170. [root@R1]~# ifconfig -m ix1
  171. ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  172. options=8400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
  173. capabilities=9507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP>
  174. ether 90:e2:ba:84:20:39
  175. inet 9.9.9.1 netmask 0xffffff00 broadcast 9.9.9.255
  176. inet6 fe80::92e2:baff:fe84:2039%ix1 prefixlen 64 scopeid 0x4
  177. nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
  178. media: Ethernet autoselect
  179. status: no carrier
  180. supported media:
  181. media autoselect
  182. media 10Gbase-SR
  183. [root@R1]~# ifconfig -v ix0
  184. ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  185. options=8400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
  186. ether 90:e2:ba:84:20:38
  187. inet 8.8.8.1 netmask 0xffffff00 broadcast 8.8.8.255
  188. inet6 fe80::92e2:baff:fe84:2038%ix0 prefixlen 64 scopeid 0x3
  189. inet6 2001:db8:8::1 prefixlen 64
  190. nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
  191. media: Ethernet autoselect
  192. status: no carrier
  193. plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
  194. vendor: Intel Corp PN: AFBR-703SDZ-IN2 SN: AD1420A03LE DATE: 2014-05-20
  195. module temperature: 43.41 C Voltage: 3.30 Volts
  196. RX: 0.34 mW (-4.67 dBm) TX: 0.54 mW (-2.62 dBm)
  197. [root@R1]~# ifconfig -v ix1
  198. ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  199. options=8400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
  200. ether 90:e2:ba:84:20:39
  201. inet 9.9.9.1 netmask 0xffffff00 broadcast 9.9.9.255
  202. inet6 fe80::92e2:baff:fe84:2039%ix1 prefixlen 64 scopeid 0x4
  203. nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
  204. media: Ethernet autoselect
  205. status: no carrier
  206. plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
  207. vendor: Intel Corp PN: FTLX8571D3BCV-IT SN: ARP24BE DATE: 2014-07-01
  208. module temperature: 42.50 C Voltage: 3.31 Volts
  209. RX: 0.34 mW (-4.61 dBm) TX: 0.66 mW (-1.77 dBm)
  210.  
  211.  
  212. ============== New test with the same server =======
  213.  
  214. HP: cxl0 (Intel SFP-SR) ----|
  215. |
  216. cxl1 (Intel SFP-SR) ----|
  217.  
  218. cxl2 (OPT SFP-LR) ----|
  219. |
  220. cxl3 (OPT SFP-LR) ----|
  221.  
  222. ====> still nothing with Intel SFP-SR, but good with OPT SFP-LR:
  223.  
  224. [root@hp]~# ifconfig -v cxl0
  225. cxl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  226. options=ec00bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
  227. ether 00:07:43:2e:e4:70
  228. inet 8.8.8.2 netmask 0xffffff00 broadcast 8.8.8.255
  229. inet6 fe80::207:43ff:fe2e:e470%cxl0 prefixlen 64 scopeid 0x5
  230. nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
  231. media: Ethernet 10Gbase-SR <full-duplex>
  232. status: no carrier
  233. plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
  234. vendor: Intel Corp PN: FTLX8571D3BCV-IT SN: AS50ETU DATE: 2014-08-06
  235. module temperature: 36.93 C Voltage: 3.20 Volts
  236. RX: 0.38 mW (-4.12 dBm) TX: 0.35 mW (-4.51 dBm)
  237. [root@hp]~# ifconfig -v cxl1
  238. cxl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  239. options=ec00bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
  240. ether 00:07:43:2e:e4:78
  241. inet 9.9.9.2 netmask 0xffffff00 broadcast 9.9.9.255
  242. inet6 fe80::207:43ff:fe2e:e478%cxl1 prefixlen 64 scopeid 0x7
  243. nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
  244. media: Ethernet 10Gbase-SR <full-duplex>
  245. status: no carrier
  246. plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
  247. vendor: Intel Corp PN: FTLX8571D3BCV-IT SN: AS60YV9 DATE: 2014-08-09
  248. module temperature: 39.19 C Voltage: 3.20 Volts
  249. RX: 0.32 mW (-4.92 dBm) TX: 0.45 mW (-3.41 dBm)
  250.  
  251. [root@hp]~# ifconfig -v cxl2
  252. cxl2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  253. options=ec00bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
  254. ether 00:07:43:2e:e4:80
  255. inet6 fe80::207:43ff:fe2e:e480%cxl2 prefixlen 64 scopeid 0x9
  256. nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
  257. media: Ethernet 10Gbase-LR <full-duplex>
  258. status: active
  259. plugged: SFP/SFP+/SFP28 10G Base-LR (LC)
  260. vendor: OPT PN: SFP-10G-LR SN: OPT13201279 DATE: 2014-06-01
  261. module temperature: 42.90 C Voltage: 3.11 Volts
  262. RX: 0.30 mW (-5.18 dBm) TX: 0.52 mW (-2.78 dBm)
  263. [root@hp]~# ifconfig -v cxl3
  264. cxl3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  265. options=ec00bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
  266. ether 00:07:43:2e:e4:88
  267. inet6 fe80::207:43ff:fe2e:e488%cxl3 prefixlen 64 scopeid 0xb
  268. nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
  269. media: Ethernet 10Gbase-LR <full-duplex>
  270. status: active
  271. plugged: SFP/SFP+/SFP28 10G Base-LR (LC)
  272. vendor: OPT PN: SFP-10G-LR SN: OPT13201254 DATE: 2014-06-01
  273. module temperature: 42.07 C Voltage: 3.09 Volts
  274. RX: 0.46 mW (-3.33 dBm) TX: 0.52 mW (-2.76 dBm)
  275.  
  276.  
  277. =============== Replacing Intel SFP
  278.  
  279. cxl0: swap "Intel Corp PN: FTLX8571D3BCV-IT" by "vendor: Intel Corp PN: AFBR-703SDZ-IN2"
  280.  
  281. Problem fixed :-)
  282.  
  283. [root@hp]~# ifconfig -v cxl0
  284. cxl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  285. options=ec00bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
  286. ether 00:07:43:2e:e4:70
  287. inet 8.8.8.2 netmask 0xffffff00 broadcast 8.8.8.255
  288. inet6 fe80::207:43ff:fe2e:e470%cxl0 prefixlen 64 scopeid 0x5
  289. nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
  290. media: Ethernet 10Gbase-SR <full-duplex>
  291. status: active
  292. plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
  293. vendor: Intel Corp PN: AFBR-703SDZ-IN2 SN: AD1420A03LE DATE: 2014-05-20
  294. module temperature: 39.68 C Voltage: 3.17 Volts
  295. RX: 0.52 mW (-2.84 dBm) TX: 0.36 mW (-4.33 dBm)
  296.  
  297.  
  298. cxl1: swap "Intel Corp PN: FTLX8571D3BCV-IT" by "
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!