API tools faq
paste
Advertisement
Guest User

Log-possible-bug

a guest
Jun 1st, 2011
87
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 98.17 KB | None | 0 0
raw download clone embed print report
  1. 12:02:00,251 INFO [JGroupsTransport] Received new cluster view: [node01-27066|0] [node01-27066]
  2. 12:02:00,251 INFO [JGroupsTransport] Cache local address is node01-27066, physical addresses are [10.100.0.11:56489]
  3. 12:02:00,252 INFO [GlobalComponentRegistry] Infinispan version: Infinispan 'Pagoa' 5.0.0-SNAPSHOT
  4. 12:02:00,276 INFO [CacheJmxRegistration] MBeans were successfully registered to the platform mbean server.
  5. 12:02:00,276 TRACE [DistributionManagerImpl] Starting distribution manager on node01-27066
  6. 12:02:00,276 DEBUG [DistributionManagerImpl] Setting joinComplete to false for node node01-27066
  7. 12:02:00,276 DEBUG [DistributionManagerImpl] Setting joinComplete to true for node node01-27066
  8. 12:02:00,276 INFO [ComponentRegistry] Infinispan version: Infinispan 'Pagoa' 5.0.0-SNAPSHOT
  9. 12:02:00,276 INFO [InfinispanWrapper] Loading JGroups form: file:/home/pruivo/radargun/plugins/infinispan4/lib/jgroups-3.0.0.Alpha1.jar
  10. 12:02:00,276 INFO [InfinispanWrapper] JGroups version: JGroups 3.0.0.Alpha1
  11. 12:02:00,276 INFO [StartClusterStage] Number of members=1 is not the one expected: 11
  12. 12:02:01,277 INFO [StartClusterStage] Number of members=1 is not the one expected: 11
  13. 12:02:02,277 INFO [StartClusterStage] Number of members=1 is not the one expected: 11
  14. 12:02:02,328 INFO [JGroupsTransport] Received new cluster view: [node01-27066|1] [node01-27066, node04-53049]
  15. 12:02:02,329 TRACE [DistributionManagerImpl] view change received. Needs to re-join? false
  16. 12:02:02,329 INFO [DistributionManagerImpl] Detected a view change. Member list changed from [node01-27066] to [node01-27066, node04-53049]
  17. 12:02:02,329 INFO [DistributionManagerImpl] This is a JOIN event! Wait for notification from new joiner node04-53049
  18. 12:02:02,378 TRACE [DistributionManagerImpl] Allowing node04-53049 to join
  19. 12:02:02,379 TRACE [DistributionManagerImpl] Informed of a JOIN by node04-53049. Starting? true
  20. 12:02:02,379 TRACE [DistributionManagerImpl] Node topology info added(null). Topology info is TopologyInfo{address2TopologyInfo={node04-53049=null}}
  21. 12:02:02,380 TRACE [DistributionManagerImpl] New CH is AbstractConsistentHash{caches=null, topologyInfo=null}
  22. 12:02:02,381 TRACE [DistributionManagerImpl] Informed of a JOIN by node04-53049. Starting? false
  23. 12:02:02,381 TRACE [DistributionManagerImpl] New CH is DefaultConsistentHash{addresses ={3182=node04-53049, 6410=node01-27066}, hash space =10240}
  24. 12:02:02,914 INFO [JGroupsTransport] Received new cluster view: [node01-27066|2] [node01-27066, node04-53049, node05-23599, node06-24999]
  25. 12:02:02,914 TRACE [DistributionManagerImpl] view change received. Needs to re-join? false
  26. 12:02:02,914 INFO [DistributionManagerImpl] Detected a view change. Member list changed from [node01-27066, node04-53049] to [node01-27066, node04-53049, node05-23599, node06-24999]
  27. 12:02:02,915 INFO [DistributionManagerImpl] This is a JOIN event! Wait for notification from new joiner node05-23599
  28. 12:02:02,969 TRACE [DistributionManagerImpl] Allowing node06-24999 to join
  29. 12:02:02,969 TRACE [DistributionManagerImpl] Not alowing node05-23599 to join since there is a join already in progress for node node06-24999
  30. 12:02:02,973 TRACE [DistributionManagerImpl] Informed of a JOIN by node06-24999. Starting? true
  31. 12:02:02,973 TRACE [DistributionManagerImpl] Node topology info added(null). Topology info is TopologyInfo{address2TopologyInfo={node04-53049=null, node06-24999=null}}
  32. 12:02:02,973 TRACE [DistributionManagerImpl] New CH is AbstractConsistentHash{caches=null, topologyInfo=null}
  33. 12:02:03,277 INFO [StartClusterStage] Number of members=4 is not the one expected: 11
  34. 12:02:03,506 INFO [JGroupsTransport] Received new cluster view: [node01-27066|3] [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120]
  35. 12:02:03,506 TRACE [DistributionManagerImpl] view change received. Needs to re-join? false
  36. 12:02:03,507 INFO [DistributionManagerImpl] Detected a view change. Member list changed from [node01-27066, node04-53049, node05-23599, node06-24999] to [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120]
  37. 12:02:03,507 INFO [DistributionManagerImpl] This is a JOIN event! Wait for notification from new joiner node07-42120
  38. 12:02:03,590 TRACE [DistributionManagerImpl] Not alowing node05-23599 to join since there is a join already in progress for node node06-24999
  39. 12:02:03,638 TRACE [DistributionManagerImpl] Not alowing node07-42120 to join since there is a join already in progress for node node06-24999
  40. 12:02:04,167 INFO [JGroupsTransport] Received new cluster view: [node01-27066|4] [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120, node08-968]
  41. 12:02:04,168 TRACE [DistributionManagerImpl] view change received. Needs to re-join? false
  42. 12:02:04,168 INFO [DistributionManagerImpl] Detected a view change. Member list changed from [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120] to [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120, node08-968]
  43. 12:02:04,168 INFO [DistributionManagerImpl] This is a JOIN event! Wait for notification from new joiner node08-968
  44. 12:02:04,259 TRACE [DistributionManagerImpl] Not alowing node08-968 to join since there is a join already in progress for node node06-24999
  45. 12:02:04,277 INFO [StartClusterStage] Number of members=6 is not the one expected: 11
  46. 12:02:04,461 TRACE [DistributionManagerImpl] Not alowing node05-23599 to join since there is a join already in progress for node node06-24999
  47. 12:02:04,580 TRACE [DistributionManagerImpl] Not alowing node07-42120 to join since there is a join already in progress for node node06-24999
  48. 12:02:04,792 INFO [JGroupsTransport] Received new cluster view: [node01-27066|5] [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120, node08-968, node09-42492]
  49. 12:02:04,792 TRACE [DistributionManagerImpl] view change received. Needs to re-join? false
  50. 12:02:04,792 INFO [DistributionManagerImpl] Detected a view change. Member list changed from [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120, node08-968] to [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120, node08-968, node09-42492]
  51. 12:02:04,792 INFO [DistributionManagerImpl] This is a JOIN event! Wait for notification from new joiner node09-42492
  52. 12:02:04,931 TRACE [DistributionManagerImpl] Not alowing node09-42492 to join since there is a join already in progress for node node06-24999
  53. 12:02:05,278 INFO [StartClusterStage] Number of members=7 is not the one expected: 11
  54. 12:02:05,811 TRACE [DistributionManagerImpl] Not alowing node08-968 to join since there is a join already in progress for node node06-24999
  55. 12:02:05,892 INFO [JGroupsTransport] Received new cluster view: [node01-27066|6] [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120, node08-968, node09-42492, node10-3814]
  56. 12:02:05,892 TRACE [DistributionManagerImpl] view change received. Needs to re-join? false
  57. 12:02:05,892 INFO [DistributionManagerImpl] Detected a view change. Member list changed from [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120, node08-968, node09-42492] to [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120, node08-968, node09-42492, node10-3814]
  58. 12:02:05,892 INFO [DistributionManagerImpl] This is a JOIN event! Wait for notification from new joiner node10-3814
  59. 12:02:06,106 TRACE [DistributionManagerImpl] Not alowing node10-3814 to join since there is a join already in progress for node node06-24999
  60. 12:02:06,278 INFO [StartClusterStage] Number of members=8 is not the one expected: 11
  61. 12:02:06,291 TRACE [DistributionManagerImpl] Not alowing node07-42120 to join since there is a join already in progress for node node06-24999
  62. 12:02:06,323 TRACE [DistributionManagerImpl] Not alowing node05-23599 to join since there is a join already in progress for node node06-24999
  63. 12:02:06,555 INFO [JGroupsTransport] Received new cluster view: [node01-27066|7] [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120, node08-968, node09-42492, node10-3814, node11-42115]
  64. 12:02:06,555 TRACE [DistributionManagerImpl] view change received. Needs to re-join? false
  65. 12:02:06,556 INFO [DistributionManagerImpl] Detected a view change. Member list changed from [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120, node08-968, node09-42492, node10-3814] to [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120, node08-968, node09-42492, node10-3814, node11-42115]
  66. 12:02:06,556 INFO [DistributionManagerImpl] This is a JOIN event! Wait for notification from new joiner node11-42115
  67. 12:02:06,712 TRACE [DistributionManagerImpl] Not alowing node09-42492 to join since there is a join already in progress for node node06-24999
  68. 12:02:06,725 TRACE [DistributionManagerImpl] Not alowing node11-42115 to join since there is a join already in progress for node node06-24999
  69. 12:02:06,864 TRACE [DistributionManagerImpl] Not alowing node05-23599 to join since there is a join already in progress for node node06-24999
  70. 12:02:07,233 TRACE [DistributionManagerImpl] Not alowing node10-3814 to join since there is a join already in progress for node node06-24999
  71. 12:02:07,278 INFO [StartClusterStage] Number of members=9 is not the one expected: 11
  72. 12:02:07,545 TRACE [DistributionManagerImpl] Not alowing node05-23599 to join since there is a join already in progress for node node06-24999
  73. 12:02:07,732 TRACE [DistributionManagerImpl] Not alowing node08-968 to join since there is a join already in progress for node node06-24999
  74. 12:02:07,775 TRACE [DistributionManagerImpl] Not alowing node10-3814 to join since there is a join already in progress for node node06-24999
  75. 12:02:07,980 TRACE [DistributionManagerImpl] Not alowing node11-42115 to join since there is a join already in progress for node node06-24999
  76. 12:02:08,222 TRACE [DistributionManagerImpl] Not alowing node07-42120 to join since there is a join already in progress for node node06-24999
  77. 12:02:08,264 TRACE [DistributionManagerImpl] Not alowing node09-42492 to join since there is a join already in progress for node node06-24999
  78. 12:02:08,278 INFO [StartClusterStage] Number of members=9 is not the one expected: 11
  79. 12:02:08,313 TRACE [DistributionManagerImpl] Not alowing node08-968 to join since there is a join already in progress for node node06-24999
  80. 12:02:08,466 TRACE [DistributionManagerImpl] Not alowing node05-23599 to join since there is a join already in progress for node node06-24999
  81. 12:02:09,236 TRACE [DistributionManagerImpl] Not alowing node10-3814 to join since there is a join already in progress for node node06-24999
  82. 12:02:09,279 INFO [StartClusterStage] Number of members=9 is not the one expected: 11
  83. 12:02:09,327 TRACE [DistributionManagerImpl] Not alowing node05-23599 to join since there is a join already in progress for node node06-24999
  84. 12:02:09,502 TRACE [DistributionManagerImpl] Not alowing node11-42115 to join since there is a join already in progress for node node06-24999
  85. 12:02:09,535 TRACE [DistributionManagerImpl] Not alowing node09-42492 to join since there is a join already in progress for node node06-24999
  86. 12:02:09,664 TRACE [DistributionManagerImpl] Not alowing node08-968 to join since there is a join already in progress for node node06-24999
  87. 12:02:09,734 TRACE [DistributionManagerImpl] Not alowing node07-42120 to join since there is a join already in progress for node node06-24999
  88. 12:02:10,167 TRACE [DistributionManagerImpl] Not alowing node09-42492 to join since there is a join already in progress for node node06-24999
  89. 12:02:10,279 INFO [StartClusterStage] Number of members=9 is not the one expected: 11
  90. 12:02:10,365 TRACE [DistributionManagerImpl] Not alowing node08-968 to join since there is a join already in progress for node node06-24999
  91. 12:02:10,456 TRACE [DistributionManagerImpl] Not alowing node07-42120 to join since there is a join already in progress for node node06-24999
  92. 12:02:10,539 TRACE [DistributionManagerImpl] Not alowing node05-23599 to join since there is a join already in progress for node node06-24999
  93. 12:02:10,750 WARN [NAKACK] node01-27066: dropped message from node02-34421 (not in table [node05-23599, node01-27066, node11-42115, node06-24999, node08-968, node10-3814, node09-42492, node04-53049, node07-42120]), view=[node01-27066|7] [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120, node08-968, node09-42492, node10-3814, node11-42115]
  94. 12:02:11,013 TRACE [DistributionManagerImpl] Not alowing node11-42115 to join since there is a join already in progress for node node06-24999
  95. 12:02:11,159 TRACE [DistributionManagerImpl] Not alowing node10-3814 to join since there is a join already in progress for node node06-24999
  96. 12:02:11,249 WARN [NAKACK] node01-27066: dropped message from node03-61513 (not in table [node05-23599, node01-27066, node11-42115, node06-24999, node08-968, node10-3814, node09-42492, node04-53049, node07-42120]), view=[node01-27066|7] [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120, node08-968, node09-42492, node10-3814, node11-42115]
  97. 12:02:11,279 INFO [StartClusterStage] Number of members=9 is not the one expected: 11
  98. 12:02:11,683 WARN [NAKACK] node01-27066: dropped message from node02-34421 (not in table [node05-23599, node01-27066, node11-42115, node06-24999, node08-968, node10-3814, node09-42492, node04-53049, node07-42120]), view=[node01-27066|7] [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120, node08-968, node09-42492, node10-3814, node11-42115]
  99. 12:02:11,898 TRACE [DistributionManagerImpl] Not alowing node09-42492 to join since there is a join already in progress for node node06-24999
  100. 12:02:11,987 TRACE [DistributionManagerImpl] Not alowing node07-42120 to join since there is a join already in progress for node node06-24999
  101. 12:02:12,129 WARN [NAKACK] node01-27066: dropped message from node03-61513 (not in table [node05-23599, node01-27066, node11-42115, node06-24999, node08-968, node10-3814, node09-42492, node04-53049, node07-42120]), view=[node01-27066|7] [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120, node08-968, node09-42492, node10-3814, node11-42115]
  102. 12:02:12,257 TRACE [DistributionManagerImpl] Not alowing node08-968 to join since there is a join already in progress for node node06-24999
  103. 12:02:12,279 INFO [StartClusterStage] Number of members=9 is not the one expected: 11
  104. 12:02:12,290 TRACE [DistributionManagerImpl] Not alowing node05-23599 to join since there is a join already in progress for node node06-24999
  105. 12:02:12,545 TRACE [DistributionManagerImpl] Not alowing node11-42115 to join since there is a join already in progress for node node06-24999
  106. 12:02:12,671 TRACE [DistributionManagerImpl] Not alowing node10-3814 to join since there is a join already in progress for node node06-24999
  107. 12:02:12,702 TRACE [DistributionManagerImpl] Not alowing node07-42120 to join since there is a join already in progress for node node06-24999
  108. 12:02:12,995 TRACE [DistributionManagerImpl] Informed of a JOIN by node06-24999. Starting? false
  109. 12:02:12,995 TRACE [DistributionManagerImpl] New CH is DefaultConsistentHash{addresses ={3182=node04-53049, 6410=node01-27066, 6978=node06-24999}, hash space =10240}
  110. 12:02:13,103 TRACE [DistributionManagerImpl] Allowing node09-42492 to join
  111. 12:02:13,106 TRACE [DistributionManagerImpl] Informed of a JOIN by node09-42492. Starting? true
  112. 12:02:13,106 TRACE [DistributionManagerImpl] Node topology info added(null). Topology info is TopologyInfo{address2TopologyInfo={node04-53049=null, node06-24999=null, node09-42492=null}}
  113. 12:02:13,106 TRACE [DistributionManagerImpl] New CH is AbstractConsistentHash{caches=null, topologyInfo=null}
  114. 12:02:13,128 TRACE [DistributionManagerImpl] Not alowing node08-968 to join since there is a join already in progress for node node09-42492
  115. 12:02:13,280 INFO [StartClusterStage] Number of members=9 is not the one expected: 11
  116. 12:02:13,391 TRACE [DistributionManagerImpl] Not alowing node05-23599 to join since there is a join already in progress for node node09-42492
  117. 12:02:13,523 TRACE [DistributionManagerImpl] Not alowing node10-3814 to join since there is a join already in progress for node node09-42492
  118. 12:02:13,853 TRACE [DistributionManagerImpl] Not alowing node07-42120 to join since there is a join already in progress for node node09-42492
  119. 12:02:13,967 TRACE [DistributionManagerImpl] Not alowing node11-42115 to join since there is a join already in progress for node node09-42492
  120. 12:02:14,280 INFO [StartClusterStage] Number of members=9 is not the one expected: 11
  121. 12:02:14,552 TRACE [DistributionManagerImpl] Not alowing node05-23599 to join since there is a join already in progress for node node09-42492
  122. 12:02:14,834 TRACE [DistributionManagerImpl] Not alowing node10-3814 to join since there is a join already in progress for node node09-42492
  123. 12:02:15,029 TRACE [DistributionManagerImpl] Not alowing node08-968 to join since there is a join already in progress for node node09-42492
  124. 12:02:15,229 TRACE [DistributionManagerImpl] Not alowing node11-42115 to join since there is a join already in progress for node node09-42492
  125. 12:02:15,244 TRACE [DistributionManagerImpl] Not alowing node07-42120 to join since there is a join already in progress for node node09-42492
  126. 12:02:15,280 INFO [StartClusterStage] Number of members=9 is not the one expected: 11
  127. 12:02:15,733 TRACE [DistributionManagerImpl] Not alowing node05-23599 to join since there is a join already in progress for node node09-42492
  128. 12:02:15,986 TRACE [DistributionManagerImpl] Not alowing node10-3814 to join since there is a join already in progress for node node09-42492
  129. 12:02:16,211 TRACE [DistributionManagerImpl] Not alowing node11-42115 to join since there is a join already in progress for node node09-42492
  130. 12:02:16,280 INFO [StartClusterStage] Number of members=9 is not the one expected: 11
  131. 12:02:16,298 WARN [NAKACK] node01-27066: dropped message from node02-34421 (not in table [node05-23599, node01-27066, node11-42115, node06-24999, node08-968, node10-3814, node09-42492, node04-53049, node07-42120]), view=[node01-27066|7] [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120, node08-968, node09-42492, node10-3814, node11-42115]
  132. 12:02:16,299 WARN [NAKACK] node01-27066: dropped message from node02-34421 (not in table [node05-23599, node01-27066, node11-42115, node06-24999, node08-968, node10-3814, node09-42492, node04-53049, node07-42120]), view=[node01-27066|7] [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120, node08-968, node09-42492, node10-3814, node11-42115]
  133. 12:02:16,300 WARN [NAKACK] node01-27066: dropped message from node03-61513 (not in table [node05-23599, node01-27066, node11-42115, node06-24999, node08-968, node10-3814, node09-42492, node04-53049, node07-42120]), view=[node01-27066|7] [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120, node08-968, node09-42492, node10-3814, node11-42115]
  134. 12:02:16,304 WARN [NAKACK] node01-27066: dropped message from node03-61513 (not in table [node05-23599, node01-27066, node11-42115, node06-24999, node08-968, node10-3814, node09-42492, node04-53049, node07-42120]), view=[node01-27066|7] [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120, node08-968, node09-42492, node10-3814, node11-42115]
  135. 12:02:16,345 INFO [JGroupsTransport] Received new, MERGED cluster view: MergeView::[node09-42492|8] [node09-42492, node02-34421, node05-23599, node08-968, node06-24999, node03-61513, node01-27066, node11-42115, node10-3814, node07-42120, node04-53049], subgroups=[[node03-61513|0] [node03-61513], [node02-34421|0] [node02-34421], [node01-27066|7] [node01-27066, node04-53049, node05-23599, node06-24999, node07-42120, node08-968, node09-42492, node10-3814, node11-42115]]
  136. 12:02:16,349 TRACE [DistributionManagerImpl] view change received. Needs to re-join? true
  137. 12:02:16,349 DEBUG [DistributionManagerImpl] Setting joinComplete to false for node node01-27066
  138. 12:02:16,350 TRACE [DistributionManagerImpl] Not alowing node08-968 to join since there is a join already in progress for node node09-42492
  139. 12:02:17,281 INFO [StartClusterStage] Number of members is the one expected: 11
  140. 12:02:17,281 INFO [StartClusterStage] Successfully started cache wrapper on slave 0: org.radargun.cachewrappers.InfinispanWrapper@6155e8ec
  141. 12:02:17,281 INFO [Slave] Finished stage: StartClusterStage {productName='infinispan4', useSmartClassLoading=true, config=dist-sync.xml, slaveIndex=0, activeSlavesCount=11, totalSlavesCount=11}
  142. 12:02:17,281 INFO [Slave] Ack successfully sent to the master
  143. 12:02:42,015 TRACE [DistributionManagerImpl] Informed of a JOIN by node09-42492. Starting? false
  144. 12:02:42,015 TRACE [DistributionManagerImpl] New CH is DefaultConsistentHash{addresses ={73=node09-42492, 848=node02-34421, 1849=node03-61513, 2093=node10-3814, 2380=node08-968, 3182=node04-53049, 5868=node07-42120, 5914=node05-23599, 6410=node01-27066, 6978=node06-24999, 8485=node11-42115}, hash space =10240}
  145. 12:02:52,038 TRACE [DistributionManagerImpl] Informed of a JOIN by node03-61513. Starting? true
  146. 12:02:52,038 TRACE [DistributionManagerImpl] Node topology info added(null). Topology info is TopologyInfo{address2TopologyInfo={node04-53049=null, node03-61513=null, node06-24999=null, node09-42492=null}}
  147. 12:02:52,038 TRACE [DistributionManagerImpl] New CH is AbstractConsistentHash{caches=null, topologyInfo=null}
  148. 12:03:02,057 TRACE [DistributionManagerImpl] Informed of a JOIN by node03-61513. Starting? false
  149. 12:03:02,058 TRACE [DistributionManagerImpl] New CH is DefaultConsistentHash{addresses ={73=node09-42492, 848=node02-34421, 1849=node03-61513, 1850=node03-61513, 2093=node10-3814, 2380=node08-968, 3182=node04-53049, 5868=node07-42120, 5914=node05-23599, 6410=node01-27066, 6978=node06-24999, 8485=node11-42115}, hash space =10240}
  150. 12:03:02,129 TRACE [DistributionManagerImpl] Informed of a JOIN by node05-23599. Starting? true
  151. 12:03:02,129 TRACE [DistributionManagerImpl] Node topology info added(null). Topology info is TopologyInfo{address2TopologyInfo={node04-53049=null, node03-61513=null, node05-23599=null, node06-24999=null, node09-42492=null}}
  152. 12:03:02,129 TRACE [DistributionManagerImpl] New CH is AbstractConsistentHash{caches=null, topologyInfo=null}
  153. 12:03:12,147 TRACE [DistributionManagerImpl] Informed of a JOIN by node05-23599. Starting? false
  154. 12:03:12,147 TRACE [DistributionManagerImpl] New CH is DefaultConsistentHash{addresses ={73=node09-42492, 848=node02-34421, 1849=node03-61513, 1850=node03-61513, 2093=node10-3814, 2380=node08-968, 3182=node04-53049, 5868=node07-42120, 5914=node05-23599, 5915=node05-23599, 6410=node01-27066, 6978=node06-24999, 8485=node11-42115}, hash space =10240}
  155. 12:03:12,684 TRACE [DistributionManagerImpl] Informed of a JOIN by node04-53049. Starting? true
  156. 12:03:12,684 TRACE [DistributionManagerImpl] Node topology info added(null). Topology info is TopologyInfo{address2TopologyInfo={node04-53049=null, node03-61513=null, node05-23599=null, node06-24999=null, node09-42492=null}}
  157. 12:03:12,684 TRACE [DistributionManagerImpl] New CH is AbstractConsistentHash{caches=null, topologyInfo=null}
  158. 12:03:22,703 TRACE [DistributionManagerImpl] Informed of a JOIN by node04-53049. Starting? false
  159. 12:03:22,703 TRACE [DistributionManagerImpl] New CH is DefaultConsistentHash{addresses ={73=node09-42492, 848=node02-34421, 1849=node03-61513, 1850=node03-61513, 2093=node10-3814, 2380=node08-968, 3182=node04-53049, 3183=node04-53049, 5868=node07-42120, 5914=node05-23599, 5915=node05-23599, 6410=node01-27066, 6978=node06-24999, 8485=node11-42115}, hash space =10240}
  160. 12:03:22,910 TRACE [DistributionManagerImpl] Informed of a JOIN by node11-42115. Starting? true
  161. 12:03:22,910 TRACE [DistributionManagerImpl] Node topology info added(null). Topology info is TopologyInfo{address2TopologyInfo={node04-53049=null, node03-61513=null, node11-42115=null, node05-23599=null, node06-24999=null, node09-42492=null}}
  162. 12:03:22,910 TRACE [DistributionManagerImpl] New CH is AbstractConsistentHash{caches=null, topologyInfo=null}
  163. 12:03:32,930 TRACE [DistributionManagerImpl] Informed of a JOIN by node11-42115. Starting? false
  164. 12:03:32,930 TRACE [DistributionManagerImpl] New CH is DefaultConsistentHash{addresses ={73=node09-42492, 848=node02-34421, 1849=node03-61513, 1850=node03-61513, 2093=node10-3814, 2380=node08-968, 3182=node04-53049, 3183=node04-53049, 5868=node07-42120, 5914=node05-23599, 5915=node05-23599, 6410=node01-27066, 6978=node06-24999, 8485=node11-42115, 8486=node11-42115}, hash space =10240}
  165. 12:03:33,316 TRACE [DistributionManagerImpl] Informed of a JOIN by node08-968. Starting? true
  166. 12:03:33,316 TRACE [DistributionManagerImpl] Node topology info added(null). Topology info is TopologyInfo{address2TopologyInfo={node04-53049=null, node03-61513=null, node11-42115=null, node05-23599=null, node06-24999=null, node09-42492=null, node08-968=null}}
  167. 12:03:33,317 TRACE [DistributionManagerImpl] New CH is AbstractConsistentHash{caches=null, topologyInfo=null}
  168. 12:03:43,335 TRACE [DistributionManagerImpl] Informed of a JOIN by node08-968. Starting? false
  169. 12:03:43,335 TRACE [DistributionManagerImpl] New CH is DefaultConsistentHash{addresses ={73=node09-42492, 848=node02-34421, 1849=node03-61513, 1850=node03-61513, 2093=node10-3814, 2380=node08-968, 2381=node08-968, 3182=node04-53049, 3183=node04-53049, 5868=node07-42120, 5914=node05-23599, 5915=node05-23599, 6410=node01-27066, 6978=node06-24999, 8485=node11-42115, 8486=node11-42115}, hash space =10240}
  170. 12:03:43,377 TRACE [DistributionManagerImpl] Informed of a JOIN by node10-3814. Starting? true
  171. 12:03:43,377 TRACE [DistributionManagerImpl] Node topology info added(null). Topology info is TopologyInfo{address2TopologyInfo={node04-53049=null, node03-61513=null, node11-42115=null, node05-23599=null, node06-24999=null, node10-3814=null, node09-42492=null, node08-968=null}}
  172. 12:03:43,377 TRACE [DistributionManagerImpl] New CH is AbstractConsistentHash{caches=null, topologyInfo=null}
  173. 12:03:53,397 TRACE [DistributionManagerImpl] Informed of a JOIN by node10-3814. Starting? false
  174. 12:03:53,397 TRACE [DistributionManagerImpl] New CH is DefaultConsistentHash{addresses ={73=node09-42492, 848=node02-34421, 1849=node03-61513, 1850=node03-61513, 2093=node10-3814, 2094=node10-3814, 2380=node08-968, 2381=node08-968, 3182=node04-53049, 3183=node04-53049, 5868=node07-42120, 5914=node05-23599, 5915=node05-23599, 6410=node01-27066, 6978=node06-24999, 8485=node11-42115, 8486=node11-42115}, hash space =10240}
  175. 12:03:53,883 TRACE [DistributionManagerImpl] Informed of a JOIN by node07-42120. Starting? true
  176. 12:03:53,883 TRACE [DistributionManagerImpl] Node topology info added(null). Topology info is TopologyInfo{address2TopologyInfo={node04-53049=null, node03-61513=null, node07-42120=null, node11-42115=null, node05-23599=null, node06-24999=null, node10-3814=null, node09-42492=null, node08-968=null}}
  177. 12:03:53,883 TRACE [DistributionManagerImpl] New CH is AbstractConsistentHash{caches=null, topologyInfo=null}
  178. 12:04:03,504 TRACE [DistributionManagerImpl] Informed of a JOIN by node07-42120. Starting? false
  179. 12:04:03,505 TRACE [DistributionManagerImpl] New CH is DefaultConsistentHash{addresses ={73=node09-42492, 848=node02-34421, 1849=node03-61513, 1850=node03-61513, 2093=node10-3814, 2094=node10-3814, 2380=node08-968, 2381=node08-968, 3182=node04-53049, 3183=node04-53049, 5868=node07-42120, 5869=node07-42120, 5914=node05-23599, 5915=node05-23599, 6410=node01-27066, 6978=node06-24999, 8485=node11-42115, 8486=node11-42115}, hash space =10240}
  180. 12:04:03,513 INFO [Slave] Executing stage: ClusterValidationStage {isPartialReplication=true, replicationTryCount=60, replicationTimeSleep=2000, wrapper=null, slaveIndex=0, activeSlavesCount=11, totalSlavesCount=11}
  181. 12:04:03,522 INFO [ClusterValidationStage] Number of caches that replicated here is 8
  182. 12:04:03,522 INFO [ClusterValidationStage] Replication test successfully passed. isPartialReplication? true, replicationCount = 8
  183. 12:04:03,522 INFO [ClusterValidationStage] Using partial replication, skipping confirm phase
  184. 12:04:03,522 INFO [Slave] Finished stage: ClusterValidationStage {isPartialReplication=true, replicationTryCount=60, replicationTimeSleep=2000, wrapper=org.radargun.cachewrappers.InfinispanWrapper@6155e8ec, slaveIndex=0, activeSlavesCount=11, totalSlavesCount=11}
  185. 12:04:03,523 INFO [Slave] Ack successfully sent to the master
  186. 12:04:03,560 INFO [Slave] Executing stage: WarmupStage {operationCount=1000, slaveIndex=0, activeSlavesCount=11, totalSlavesCount=11}
  187. 12:04:03,560 INFO [WarmupStage] Performing Warmup Operations
  188. 12:04:03,560 INFO [WarmupStage] Cache launched, performing 1000 put and get operations
  189. 12:04:03,561 INFO [WarmupStage] Joining warmupThreads
  190. 12:04:03,561 WARN [DistLockingInterceptor] xxx entry commit warmup_key_4_-2-2-warmup_bucket4_ => Thread[OOB-16,x,node01-27066,5,Thread Pools]
  191. 12:04:03,561 WARN [DistLockingInterceptor] xxx entry commit warmup_key_3_-2-3-warmup_bucket3_ => Thread[OOB-15,x,node01-27066,5,Thread Pools]
  192. 12:04:03,562 WARN [DistLockingInterceptor] xxx entry commit warmup_key_6_-3-1-warmup_bucket6_ => Thread[OOB-16,x,node01-27066,5,Thread Pools]
  193. 12:04:03,566 WARN [DistLockingInterceptor] xxx entry commit warmup_key_3_-5-4-warmup_bucket3_ => Thread[OOB-15,x,node01-27066,5,Thread Pools]
  194. 12:04:03,568 WARN [DistLockingInterceptor] xxx entry commit warmup_key_5_-9-1-warmup_bucket5_ => Thread[OOB-16,x,node01-27066,5,Thread Pools]
  195. 12:04:03,568 WARN [DistLockingInterceptor] xxx entry commit warmup_key_3_-11-0-warmup_bucket3_ => Thread[OOB-12,x,node01-27066,5,Thread Pools]
  196. 12:04:03,569 WARN [DistLockingInterceptor] xxx entry commit warmup_key_4_-9-1-warmup_bucket4_ => Thread[OOB-14,x,node01-27066,5,Thread Pools]
  197. 12:04:03,569 ERROR [InvocationContextInterceptor] Execution error:
  198. org.infinispan.util.concurrent.TimeoutException: Timed out waiting for 60 seconds for valid responses from either of [node06-24999]
  199. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$FutureCollator.getResponseList(CommandAwareRpcDispatcher.java:446)
  200. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:339)
  201. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:124)
  202. at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:387)
  203. at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:108)
  204. at org.infinispan.distribution.DistributionManagerImpl.retrieveFromRemoteSource(DistributionManagerImpl.java:341)
  205. at org.infinispan.interceptors.DistributionInterceptor.realRemoteGet(DistributionInterceptor.java:147)
  206. at org.infinispan.interceptors.DistributionInterceptor.remoteGetAndStoreInL1(DistributionInterceptor.java:135)
  207. at org.infinispan.interceptors.DistributionInterceptor.visitGetKeyValueCommand(DistributionInterceptor.java:101)
  208. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  209. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  210. at org.infinispan.interceptors.LockingInterceptor.visitGetKeyValueCommand(LockingInterceptor.java:188)
  211. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  212. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  213. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  214. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  215. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  216. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  217. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  218. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  219. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  220. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  221. at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:302)
  222. at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:293)
  223. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  224. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  225. at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:83)
  226. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  227. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  228. at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:87)
  229. at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:58)
  230. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  231. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  232. at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273)
  233. at org.infinispan.CacheDelegate.get(CacheDelegate.java:192)
  234. at org.radargun.cachewrappers.InfinispanWrapper.get(InfinispanWrapper.java:95)
  235. at org.radargun.stressors.WarmupStressor.doGet(WarmupStressor.java:95)
  236. at org.radargun.stressors.WarmupStressor.access$100(WarmupStressor.java:18)
  237. at org.radargun.stressors.WarmupStressor$1.run(WarmupStressor.java:66)
  238. 12:04:03,569 INFO [WarmupStage] Caught exception doing a GET on key warmup_key_0_-6-1-warmup_bucket0_
  239. org.infinispan.util.concurrent.TimeoutException: Timed out waiting for 60 seconds for valid responses from either of [node06-24999]
  240. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$FutureCollator.getResponseList(CommandAwareRpcDispatcher.java:446)
  241. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:339)
  242. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:124)
  243. at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:387)
  244. at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:108)
  245. at org.infinispan.distribution.DistributionManagerImpl.retrieveFromRemoteSource(DistributionManagerImpl.java:341)
  246. at org.infinispan.interceptors.DistributionInterceptor.realRemoteGet(DistributionInterceptor.java:147)
  247. at org.infinispan.interceptors.DistributionInterceptor.remoteGetAndStoreInL1(DistributionInterceptor.java:135)
  248. at org.infinispan.interceptors.DistributionInterceptor.visitGetKeyValueCommand(DistributionInterceptor.java:101)
  249. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  250. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  251. at org.infinispan.interceptors.LockingInterceptor.visitGetKeyValueCommand(LockingInterceptor.java:188)
  252. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  253. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  254. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  255. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  256. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  257. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  258. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  259. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  260. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  261. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  262. at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:302)
  263. at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:293)
  264. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  265. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  266. at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:83)
  267. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  268. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  269. at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:87)
  270. at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:58)
  271. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  272. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  273. at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273)
  274. at org.infinispan.CacheDelegate.get(CacheDelegate.java:192)
  275. at org.radargun.cachewrappers.InfinispanWrapper.get(InfinispanWrapper.java:95)
  276. at org.radargun.stressors.WarmupStressor.doGet(WarmupStressor.java:95)
  277. at org.radargun.stressors.WarmupStressor.access$100(WarmupStressor.java:18)
  278. at org.radargun.stressors.WarmupStressor$1.run(WarmupStressor.java:66)
  279. 12:04:03,579 ERROR [InvocationContextInterceptor] Execution error:
  280. org.infinispan.util.concurrent.TimeoutException: Timed out waiting for 60 seconds for valid responses from either of [node06-24999]
  281. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$FutureCollator.getResponseList(CommandAwareRpcDispatcher.java:446)
  282. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:339)
  283. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:124)
  284. at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:387)
  285. at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:108)
  286. at org.infinispan.distribution.DistributionManagerImpl.retrieveFromRemoteSource(DistributionManagerImpl.java:341)
  287. at org.infinispan.interceptors.DistributionInterceptor.realRemoteGet(DistributionInterceptor.java:147)
  288. at org.infinispan.interceptors.DistributionInterceptor.remoteGetAndStoreInL1(DistributionInterceptor.java:135)
  289. at org.infinispan.interceptors.DistributionInterceptor.visitGetKeyValueCommand(DistributionInterceptor.java:101)
  290. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  291. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  292. at org.infinispan.interceptors.LockingInterceptor.visitGetKeyValueCommand(LockingInterceptor.java:188)
  293. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  294. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  295. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  296. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  297. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  298. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  299. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  300. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  301. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  302. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  303. at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:302)
  304. at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:293)
  305. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  306. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  307. at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:83)
  308. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  309. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  310. at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:87)
  311. at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:58)
  312. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  313. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  314. at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273)
  315. at org.infinispan.CacheDelegate.get(CacheDelegate.java:192)
  316. at org.radargun.cachewrappers.InfinispanWrapper.get(InfinispanWrapper.java:95)
  317. at org.radargun.stressors.WarmupStressor.doGet(WarmupStressor.java:95)
  318. at org.radargun.stressors.WarmupStressor.access$100(WarmupStressor.java:18)
  319. at org.radargun.stressors.WarmupStressor$1.run(WarmupStressor.java:66)
  320. 12:04:03,579 WARN [DistLockingInterceptor] xxx entry commit warmup_key_3_-16-2-warmup_bucket3_ => Thread[OOB-12,x,node01-27066,5,Thread Pools]
  321. 12:04:03,579 INFO [WarmupStage] Caught exception doing a GET on key warmup_key_0_-14-0-warmup_bucket0_
  322. org.infinispan.util.concurrent.TimeoutException: Timed out waiting for 60 seconds for valid responses from either of [node06-24999]
  323. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$FutureCollator.getResponseList(CommandAwareRpcDispatcher.java:446)
  324. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:339)
  325. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:124)
  326. at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:387)
  327. at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:108)
  328. at org.infinispan.distribution.DistributionManagerImpl.retrieveFromRemoteSource(DistributionManagerImpl.java:341)
  329. at org.infinispan.interceptors.DistributionInterceptor.realRemoteGet(DistributionInterceptor.java:147)
  330. at org.infinispan.interceptors.DistributionInterceptor.remoteGetAndStoreInL1(DistributionInterceptor.java:135)
  331. at org.infinispan.interceptors.DistributionInterceptor.visitGetKeyValueCommand(DistributionInterceptor.java:101)
  332. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  333. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  334. at org.infinispan.interceptors.LockingInterceptor.visitGetKeyValueCommand(LockingInterceptor.java:188)
  335. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  336. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  337. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  338. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  339. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  340. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  341. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  342. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  343. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  344. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  345. at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:302)
  346. at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:293)
  347. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  348. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  349. at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:83)
  350. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  351. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  352. at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:87)
  353. at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:58)
  354. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  355. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  356. at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273)
  357. at org.infinispan.CacheDelegate.get(CacheDelegate.java:192)
  358. at org.radargun.cachewrappers.InfinispanWrapper.get(InfinispanWrapper.java:95)
  359. at org.radargun.stressors.WarmupStressor.doGet(WarmupStressor.java:95)
  360. at org.radargun.stressors.WarmupStressor.access$100(WarmupStressor.java:18)
  361. at org.radargun.stressors.WarmupStressor$1.run(WarmupStressor.java:66)
  362. 12:04:03,580 WARN [DistLockingInterceptor] xxx entry commit warmup_key_2_-13-3-warmup_bucket2_ => Thread[OOB-12,x,node01-27066,5,Thread Pools]
  363. 12:04:03,581 ERROR [InvocationContextInterceptor] Execution error:
  364. org.infinispan.util.concurrent.TimeoutException: Timed out waiting for 60 seconds for valid responses from either of [node06-24999]
  365. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$FutureCollator.getResponseList(CommandAwareRpcDispatcher.java:446)
  366. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:339)
  367. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:124)
  368. at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:387)
  369. at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:108)
  370. at org.infinispan.distribution.DistributionManagerImpl.retrieveFromRemoteSource(DistributionManagerImpl.java:341)
  371. at org.infinispan.interceptors.DistributionInterceptor.realRemoteGet(DistributionInterceptor.java:147)
  372. at org.infinispan.interceptors.DistributionInterceptor.remoteGetAndStoreInL1(DistributionInterceptor.java:135)
  373. at org.infinispan.interceptors.DistributionInterceptor.visitGetKeyValueCommand(DistributionInterceptor.java:101)
  374. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  375. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  376. at org.infinispan.interceptors.LockingInterceptor.visitGetKeyValueCommand(LockingInterceptor.java:188)
  377. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  378. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  379. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  380. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  381. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  382. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  383. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  384. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  385. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  386. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  387. at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:302)
  388. at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:293)
  389. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  390. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  391. at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:83)
  392. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  393. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  394. at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:87)
  395. at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:58)
  396. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  397. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  398. at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273)
  399. at org.infinispan.CacheDelegate.get(CacheDelegate.java:192)
  400. at org.radargun.cachewrappers.InfinispanWrapper.get(InfinispanWrapper.java:95)
  401. at org.radargun.stressors.WarmupStressor.doGet(WarmupStressor.java:95)
  402. at org.radargun.stressors.WarmupStressor.access$100(WarmupStressor.java:18)
  403. at org.radargun.stressors.WarmupStressor$1.run(WarmupStressor.java:66)
  404. 12:04:03,581 INFO [WarmupStage] Caught exception doing a GET on key warmup_key_0_-21-2-warmup_bucket0_
  405. org.infinispan.util.concurrent.TimeoutException: Timed out waiting for 60 seconds for valid responses from either of [node06-24999]
  406. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$FutureCollator.getResponseList(CommandAwareRpcDispatcher.java:446)
  407. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:339)
  408. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:124)
  409. at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:387)
  410. at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:108)
  411. at org.infinispan.distribution.DistributionManagerImpl.retrieveFromRemoteSource(DistributionManagerImpl.java:341)
  412. at org.infinispan.interceptors.DistributionInterceptor.realRemoteGet(DistributionInterceptor.java:147)
  413. at org.infinispan.interceptors.DistributionInterceptor.remoteGetAndStoreInL1(DistributionInterceptor.java:135)
  414. at org.infinispan.interceptors.DistributionInterceptor.visitGetKeyValueCommand(DistributionInterceptor.java:101)
  415. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  416. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  417. at org.infinispan.interceptors.LockingInterceptor.visitGetKeyValueCommand(LockingInterceptor.java:188)
  418. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  419. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  420. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  421. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  422. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  423. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  424. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  425. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  426. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  427. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  428. at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:302)
  429. at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:293)
  430. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  431. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  432. at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:83)
  433. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  434. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  435. at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:87)
  436. at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:58)
  437. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  438. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  439. at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273)
  440. at org.infinispan.CacheDelegate.get(CacheDelegate.java:192)
  441. at org.radargun.cachewrappers.InfinispanWrapper.get(InfinispanWrapper.java:95)
  442. at org.radargun.stressors.WarmupStressor.doGet(WarmupStressor.java:95)
  443. at org.radargun.stressors.WarmupStressor.access$100(WarmupStressor.java:18)
  444. at org.radargun.stressors.WarmupStressor$1.run(WarmupStressor.java:66)
  445. 12:04:03,585 WARN [DistLockingInterceptor] xxx entry commit warmup_key_0_-26-4-warmup_bucket0_ => Thread[Thread-22,5,main]
  446. 12:04:03,586 WARN [DistLockingInterceptor] xxx entry commit warmup_key_9_-15-0-warmup_bucket9_ => Thread[OOB-12,x,node01-27066,5,Thread Pools]
  447. 12:04:03,592 ERROR [InvocationContextInterceptor] Execution error:
  448. org.infinispan.util.concurrent.TimeoutException: Timed out waiting for 60 seconds for valid responses from either of [node06-24999]
  449. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$FutureCollator.getResponseList(CommandAwareRpcDispatcher.java:446)
  450. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:339)
  451. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:124)
  452. at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:387)
  453. at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:108)
  454. at org.infinispan.distribution.DistributionManagerImpl.retrieveFromRemoteSource(DistributionManagerImpl.java:341)
  455. at org.infinispan.interceptors.DistributionInterceptor.realRemoteGet(DistributionInterceptor.java:147)
  456. at org.infinispan.interceptors.DistributionInterceptor.remoteGetAndStoreInL1(DistributionInterceptor.java:135)
  457. at org.infinispan.interceptors.DistributionInterceptor.visitGetKeyValueCommand(DistributionInterceptor.java:101)
  458. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  459. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  460. at org.infinispan.interceptors.LockingInterceptor.visitGetKeyValueCommand(LockingInterceptor.java:188)
  461. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  462. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  463. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  464. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  465. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  466. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  467. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  468. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  469. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  470. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  471. at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:302)
  472. at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:293)
  473. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  474. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  475. at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:83)
  476. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  477. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  478. at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:87)
  479. at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:58)
  480. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  481. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  482. at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273)
  483. at org.infinispan.CacheDelegate.get(CacheDelegate.java:192)
  484. at org.radargun.cachewrappers.InfinispanWrapper.get(InfinispanWrapper.java:95)
  485. at org.radargun.stressors.WarmupStressor.doGet(WarmupStressor.java:95)
  486. at org.radargun.stressors.WarmupStressor.access$100(WarmupStressor.java:18)
  487. at org.radargun.stressors.WarmupStressor$1.run(WarmupStressor.java:66)
  488. 12:04:03,592 INFO [WarmupStage] Caught exception doing a GET on key warmup_key_0_-36-2-warmup_bucket0_
  489. org.infinispan.util.concurrent.TimeoutException: Timed out waiting for 60 seconds for valid responses from either of [node06-24999]
  490. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$FutureCollator.getResponseList(CommandAwareRpcDispatcher.java:446)
  491. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:339)
  492. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:124)
  493. at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:387)
  494. at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:108)
  495. at org.infinispan.distribution.DistributionManagerImpl.retrieveFromRemoteSource(DistributionManagerImpl.java:341)
  496. at org.infinispan.interceptors.DistributionInterceptor.realRemoteGet(DistributionInterceptor.java:147)
  497. at org.infinispan.interceptors.DistributionInterceptor.remoteGetAndStoreInL1(DistributionInterceptor.java:135)
  498. at org.infinispan.interceptors.DistributionInterceptor.visitGetKeyValueCommand(DistributionInterceptor.java:101)
  499. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  500. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  501. at org.infinispan.interceptors.LockingInterceptor.visitGetKeyValueCommand(LockingInterceptor.java:188)
  502. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  503. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  504. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  505. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  506. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  507. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  508. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  509. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  510. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  511. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  512. at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:302)
  513. at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:293)
  514. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  515. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  516. at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:83)
  517. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  518. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  519. at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:87)
  520. at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:58)
  521. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  522. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  523. at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273)
  524. at org.infinispan.CacheDelegate.get(CacheDelegate.java:192)
  525. at org.radargun.cachewrappers.InfinispanWrapper.get(InfinispanWrapper.java:95)
  526. at org.radargun.stressors.WarmupStressor.doGet(WarmupStressor.java:95)
  527. at org.radargun.stressors.WarmupStressor.access$100(WarmupStressor.java:18)
  528. at org.radargun.stressors.WarmupStressor$1.run(WarmupStressor.java:66)
  529. 12:04:03,592 WARN [DistLockingInterceptor] xxx entry commit warmup_key_7_-41-1-warmup_bucket7_ => Thread[OOB-12,x,node01-27066,5,Thread Pools]
  530. 12:04:03,593 WARN [DistLockingInterceptor] xxx entry commit warmup_key_2_-35-1-warmup_bucket2_ => Thread[OOB-14,x,node01-27066,5,Thread Pools]
  531. 12:04:03,595 ERROR [InvocationContextInterceptor] Execution error:
  532. org.infinispan.util.concurrent.TimeoutException: Timed out waiting for 60 seconds for valid responses from either of [node06-24999]
  533. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$FutureCollator.getResponseList(CommandAwareRpcDispatcher.java:446)
  534. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:339)
  535. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:124)
  536. at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:387)
  537. at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:108)
  538. at org.infinispan.distribution.DistributionManagerImpl.retrieveFromRemoteSource(DistributionManagerImpl.java:341)
  539. at org.infinispan.interceptors.DistributionInterceptor.realRemoteGet(DistributionInterceptor.java:147)
  540. at org.infinispan.interceptors.DistributionInterceptor.remoteGetAndStoreInL1(DistributionInterceptor.java:135)
  541. at org.infinispan.interceptors.DistributionInterceptor.visitGetKeyValueCommand(DistributionInterceptor.java:101)
  542. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  543. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  544. at org.infinispan.interceptors.LockingInterceptor.visitGetKeyValueCommand(LockingInterceptor.java:188)
  545. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  546. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  547. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  548. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  549. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  550. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  551. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  552. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  553. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  554. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  555. at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:302)
  556. at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:293)
  557. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  558. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  559. at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:83)
  560. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  561. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  562. at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:87)
  563. at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:58)
  564. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  565. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  566. at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273)
  567. at org.infinispan.CacheDelegate.get(CacheDelegate.java:192)
  568. at org.radargun.cachewrappers.InfinispanWrapper.get(InfinispanWrapper.java:95)
  569. at org.radargun.stressors.WarmupStressor.doGet(WarmupStressor.java:95)
  570. at org.radargun.stressors.WarmupStressor.access$100(WarmupStressor.java:18)
  571. at org.radargun.stressors.WarmupStressor$1.run(WarmupStressor.java:66)
  572. 12:04:03,595 INFO [WarmupStage] Caught exception doing a GET on key warmup_key_0_-42-2-warmup_bucket0_
  573. org.infinispan.util.concurrent.TimeoutException: Timed out waiting for 60 seconds for valid responses from either of [node06-24999]
  574. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$FutureCollator.getResponseList(CommandAwareRpcDispatcher.java:446)
  575. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:339)
  576. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:124)
  577. at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:387)
  578. at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:108)
  579. at org.infinispan.distribution.DistributionManagerImpl.retrieveFromRemoteSource(DistributionManagerImpl.java:341)
  580. at org.infinispan.interceptors.DistributionInterceptor.realRemoteGet(DistributionInterceptor.java:147)
  581. at org.infinispan.interceptors.DistributionInterceptor.remoteGetAndStoreInL1(DistributionInterceptor.java:135)
  582. at org.infinispan.interceptors.DistributionInterceptor.visitGetKeyValueCommand(DistributionInterceptor.java:101)
  583. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  584. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  585. at org.infinispan.interceptors.LockingInterceptor.visitGetKeyValueCommand(LockingInterceptor.java:188)
  586. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  587. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  588. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  589. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  590. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  591. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  592. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  593. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  594. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  595. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  596. at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:302)
  597. at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:293)
  598. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  599. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  600. at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:83)
  601. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  602. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  603. at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:87)
  604. at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:58)
  605. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  606. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  607. at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273)
  608. at org.infinispan.CacheDelegate.get(CacheDelegate.java:192)
  609. at org.radargun.cachewrappers.InfinispanWrapper.get(InfinispanWrapper.java:95)
  610. at org.radargun.stressors.WarmupStressor.doGet(WarmupStressor.java:95)
  611. at org.radargun.stressors.WarmupStressor.access$100(WarmupStressor.java:18)
  612. at org.radargun.stressors.WarmupStressor$1.run(WarmupStressor.java:66)
  613. 12:04:03,595 WARN [DistLockingInterceptor] xxx entry commit warmup_key_1_-23-4-warmup_bucket1_ => Thread[OOB-14,x,node01-27066,5,Thread Pools]
  614. 12:04:03,595 WARN [DistLockingInterceptor] xxx entry commit warmup_key_4_-32-2-warmup_bucket4_ => Thread[OOB-11,x,node01-27066,5,Thread Pools]
  615. 12:04:03,600 WARN [DistLockingInterceptor] xxx entry commit warmup_key_3_-44-3-warmup_bucket3_ => Thread[OOB-11,x,node01-27066,5,Thread Pools]
  616. 12:04:03,601 WARN [DistLockingInterceptor] xxx entry commit warmup_key_2_-45-0-warmup_bucket2_ => Thread[OOB-14,x,node01-27066,5,Thread Pools]
  617. 12:04:03,604 WARN [DistLockingInterceptor] xxx entry commit warmup_key_7_-56-4-warmup_bucket7_ => Thread[OOB-14,x,node01-27066,5,Thread Pools]
  618. 12:04:03,605 WARN [DistLockingInterceptor] xxx entry commit warmup_key_7_-58-4-warmup_bucket7_ => Thread[OOB-14,x,node01-27066,5,Thread Pools]
  619. 12:04:03,606 WARN [DistLockingInterceptor] xxx entry commit warmup_key_6_-54-4-warmup_bucket6_ => Thread[OOB-14,x,node01-27066,5,Thread Pools]
  620. 12:04:03,606 WARN [DistLockingInterceptor] xxx entry commit warmup_key_7_-60-4-warmup_bucket7_ => Thread[OOB-11,x,node01-27066,5,Thread Pools]
  621. 12:04:03,610 ERROR [InvocationContextInterceptor] Execution error:
  622. org.infinispan.util.concurrent.TimeoutException: Timed out waiting for 60 seconds for valid responses from either of [node06-24999]
  623. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$FutureCollator.getResponseList(CommandAwareRpcDispatcher.java:446)
  624. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:339)
  625. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:124)
  626. at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:387)
  627. at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:108)
  628. at org.infinispan.distribution.DistributionManagerImpl.retrieveFromRemoteSource(DistributionManagerImpl.java:341)
  629. at org.infinispan.interceptors.DistributionInterceptor.realRemoteGet(DistributionInterceptor.java:147)
  630. at org.infinispan.interceptors.DistributionInterceptor.remoteGetAndStoreInL1(DistributionInterceptor.java:135)
  631. at org.infinispan.interceptors.DistributionInterceptor.visitGetKeyValueCommand(DistributionInterceptor.java:101)
  632. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  633. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  634. at org.infinispan.interceptors.LockingInterceptor.visitGetKeyValueCommand(LockingInterceptor.java:188)
  635. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  636. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  637. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  638. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  639. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  640. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  641. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  642. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  643. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  644. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  645. at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:302)
  646. at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:293)
  647. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  648. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  649. at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:83)
  650. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  651. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  652. at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:87)
  653. at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:58)
  654. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  655. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  656. at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273)
  657. at org.infinispan.CacheDelegate.get(CacheDelegate.java:192)
  658. at org.radargun.cachewrappers.InfinispanWrapper.get(InfinispanWrapper.java:95)
  659. at org.radargun.stressors.WarmupStressor.doGet(WarmupStressor.java:95)
  660. at org.radargun.stressors.WarmupStressor.access$100(WarmupStressor.java:18)
  661. at org.radargun.stressors.WarmupStressor$1.run(WarmupStressor.java:66)
  662. 12:04:03,610 INFO [WarmupStage] Caught exception doing a GET on key warmup_key_0_-65-2-warmup_bucket0_
  663. org.infinispan.util.concurrent.TimeoutException: Timed out waiting for 60 seconds for valid responses from either of [node06-24999]
  664. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$FutureCollator.getResponseList(CommandAwareRpcDispatcher.java:446)
  665. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:339)
  666. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:124)
  667. at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:387)
  668. at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:108)
  669. at org.infinispan.distribution.DistributionManagerImpl.retrieveFromRemoteSource(DistributionManagerImpl.java:341)
  670. at org.infinispan.interceptors.DistributionInterceptor.realRemoteGet(DistributionInterceptor.java:147)
  671. at org.infinispan.interceptors.DistributionInterceptor.remoteGetAndStoreInL1(DistributionInterceptor.java:135)
  672. at org.infinispan.interceptors.DistributionInterceptor.visitGetKeyValueCommand(DistributionInterceptor.java:101)
  673. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  674. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  675. at org.infinispan.interceptors.LockingInterceptor.visitGetKeyValueCommand(LockingInterceptor.java:188)
  676. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  677. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  678. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  679. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  680. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  681. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  682. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  683. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  684. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  685. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  686. at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:302)
  687. at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:293)
  688. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  689. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  690. at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:83)
  691. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  692. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  693. at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:87)
  694. at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:58)
  695. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  696. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  697. at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273)
  698. at org.infinispan.CacheDelegate.get(CacheDelegate.java:192)
  699. at org.radargun.cachewrappers.InfinispanWrapper.get(InfinispanWrapper.java:95)
  700. at org.radargun.stressors.WarmupStressor.doGet(WarmupStressor.java:95)
  701. at org.radargun.stressors.WarmupStressor.access$100(WarmupStressor.java:18)
  702. at org.radargun.stressors.WarmupStressor$1.run(WarmupStressor.java:66)
  703. 12:04:03,611 WARN [DistLockingInterceptor] xxx entry commit warmup_key_3_-63-1-warmup_bucket3_ => Thread[OOB-11,x,node01-27066,5,Thread Pools]
  704. 12:04:03,613 WARN [DistLockingInterceptor] xxx entry commit warmup_key_3_-68-4-warmup_bucket3_ => Thread[OOB-11,x,node01-27066,5,Thread Pools]
  705. 12:04:03,614 WARN [DistLockingInterceptor] xxx entry commit warmup_key_8_-57-4-warmup_bucket8_ => Thread[OOB-14,x,node01-27066,5,Thread Pools]
  706. 12:04:03,616 WARN [DistLockingInterceptor] xxx entry commit warmup_key_8_-61-2-warmup_bucket8_ => Thread[OOB-14,x,node01-27066,5,Thread Pools]
  707. 12:04:03,616 WARN [DistLockingInterceptor] xxx entry commit warmup_key_6_-68-3-warmup_bucket6_ => Thread[OOB-11,x,node01-27066,5,Thread Pools]
  708. 12:04:03,617 WARN [DistLockingInterceptor] xxx entry commit warmup_key_6_-69-1-warmup_bucket6_ => Thread[OOB-11,x,node01-27066,5,Thread Pools]
  709. 12:04:03,618 WARN [DistLockingInterceptor] xxx entry commit warmup_key_3_-73-4-warmup_bucket3_ => Thread[OOB-14,x,node01-27066,5,Thread Pools]
  710. 12:04:03,619 WARN [DistLockingInterceptor] xxx entry commit warmup_key_5_-65-3-warmup_bucket5_ => Thread[OOB-14,x,node01-27066,5,Thread Pools]
  711. 12:04:03,619 WARN [DistLockingInterceptor] xxx entry commit warmup_key_2_-64-3-warmup_bucket2_ => Thread[OOB-11,x,node01-27066,5,Thread Pools]
  712. 12:04:03,622 WARN [DistLockingInterceptor] xxx entry commit warmup_key_3_-80-3-warmup_bucket3_ => Thread[OOB-13,x,node01-27066,5,Thread Pools]
  713. 12:04:03,625 WARN [DistLockingInterceptor] xxx entry commit warmup_key_2_-80-1-warmup_bucket2_ => Thread[OOB-12,x,node01-27066,5,Thread Pools]
  714. 12:04:03,626 WARN [DistLockingInterceptor] xxx entry commit warmup_key_2_-81-0-warmup_bucket2_ => Thread[OOB-12,x,node01-27066,5,Thread Pools]
  715. 12:04:03,627 WARN [DistLockingInterceptor] xxx entry commit warmup_key_2_-82-1-warmup_bucket2_ => Thread[OOB-12,x,node01-27066,5,Thread Pools]
  716. 12:04:03,630 WARN [DistLockingInterceptor] xxx entry commit warmup_key_3_-93-0-warmup_bucket3_ => Thread[OOB-12,x,node01-27066,5,Thread Pools]
  717. 12:04:03,630 WARN [DistLockingInterceptor] xxx entry commit warmup_key_5_-84-1-warmup_bucket5_ => Thread[OOB-13,x,node01-27066,5,Thread Pools]
  718. 12:04:03,631 WARN [DistLockingInterceptor] xxx entry commit warmup_key_3_-95-4-warmup_bucket3_ => Thread[OOB-12,x,node01-27066,5,Thread Pools]
  719. 12:04:03,633 WARN [DistLockingInterceptor] xxx entry commit warmup_key_7_-84-2-warmup_bucket7_ => Thread[OOB-12,x,node01-27066,5,Thread Pools]
  720. 12:04:03,633 TRACE [DistributionManagerImpl] Installing new consistent hash DefaultConsistentHash{addresses ={73=node09-42492, 848=node02-34421, 1849=node03-61513, 1850=node03-61513, 2093=node10-3814, 2094=node10-3814, 2380=node08-968, 2381=node08-968, 3182=node04-53049, 3183=node04-53049, 5868=node07-42120, 5869=node07-42120, 5914=node05-23599, 5915=node05-23599, 6410=node01-27066, 6978=node06-24999, 8485=node11-42115, 8486=node11-42115}, hash space =10240}
  721. 12:04:03,633 WARN [DistLockingInterceptor] xxx entry commit warmup_key_7_-85-4-warmup_bucket7_ => Thread[OOB-11,x,node01-27066,5,Thread Pools]
  722. 12:04:03,634 WARN [DistLockingInterceptor] xxx entry commit warmup_key_7_-88-1-warmup_bucket7_ => Thread[OOB-13,x,node01-27066,5,Thread Pools]
  723. 12:04:03,634 WARN [DistLockingInterceptor] xxx entry commit warmup_key_1_-76-2-warmup_bucket1_ => Thread[OOB-12,x,node01-27066,5,Thread Pools]
  724. 12:04:03,640 WARN [DistLockingInterceptor] xxx entry commit warmup_key_7_-92-1-warmup_bucket7_ => Thread[OOB-15,x,node01-27066,5,Thread Pools]
  725. 12:04:03,641 ERROR [InvocationContextInterceptor] Execution error:
  726. org.infinispan.util.concurrent.TimeoutException: Timed out waiting for 60 seconds for valid responses from either of [node06-24999]
  727. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$FutureCollator.getResponseList(CommandAwareRpcDispatcher.java:446)
  728. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:339)
  729. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:124)
  730. at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:387)
  731. at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:108)
  732. at org.infinispan.distribution.DistributionManagerImpl.retrieveFromRemoteSource(DistributionManagerImpl.java:341)
  733. at org.infinispan.interceptors.DistributionInterceptor.realRemoteGet(DistributionInterceptor.java:147)
  734. at org.infinispan.interceptors.DistributionInterceptor.remoteGetAndStoreInL1(DistributionInterceptor.java:135)
  735. at org.infinispan.interceptors.DistributionInterceptor.visitGetKeyValueCommand(DistributionInterceptor.java:101)
  736. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  737. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  738. at org.infinispan.interceptors.LockingInterceptor.visitGetKeyValueCommand(LockingInterceptor.java:188)
  739. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  740. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  741. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  742. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  743. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  744. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  745. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  746. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  747. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  748. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  749. at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:302)
  750. at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:293)
  751. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  752. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  753. at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:83)
  754. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  755. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  756. at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:87)
  757. at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:58)
  758. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  759. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  760. at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273)
  761. at org.infinispan.CacheDelegate.get(CacheDelegate.java:192)
  762. at org.radargun.cachewrappers.InfinispanWrapper.get(InfinispanWrapper.java:95)
  763. at org.radargun.stressors.WarmupStressor.doGet(WarmupStressor.java:95)
  764. at org.radargun.stressors.WarmupStressor.access$100(WarmupStressor.java:18)
  765. at org.radargun.stressors.WarmupStressor$1.run(WarmupStressor.java:66)
  766. 12:04:03,641 INFO [WarmupStage] Caught exception doing a GET on key warmup_key_0_-90-2-warmup_bucket0_
  767. org.infinispan.util.concurrent.TimeoutException: Timed out waiting for 60 seconds for valid responses from either of [node06-24999]
  768. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$FutureCollator.getResponseList(CommandAwareRpcDispatcher.java:446)
  769. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:339)
  770. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:124)
  771. at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:387)
  772. at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:108)
  773. at org.infinispan.distribution.DistributionManagerImpl.retrieveFromRemoteSource(DistributionManagerImpl.java:341)
  774. at org.infinispan.interceptors.DistributionInterceptor.realRemoteGet(DistributionInterceptor.java:147)
  775. at org.infinispan.interceptors.DistributionInterceptor.remoteGetAndStoreInL1(DistributionInterceptor.java:135)
  776. at org.infinispan.interceptors.DistributionInterceptor.visitGetKeyValueCommand(DistributionInterceptor.java:101)
  777. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  778. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  779. at org.infinispan.interceptors.LockingInterceptor.visitGetKeyValueCommand(LockingInterceptor.java:188)
  780. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  781. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  782. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  783. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  784. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  785. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  786. at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
  787. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  788. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  789. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  790. at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:302)
  791. at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:293)
  792. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  793. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  794. at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:83)
  795. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  796. at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
  797. at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:87)
  798. at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:58)
  799. at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:75)
  800. at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:55)
  801. at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273)
  802. at org.infinispan.CacheDelegate.get(CacheDelegate.java:192)
  803. at org.radargun.cachewrappers.InfinispanWrapper.get(InfinispanWrapper.java:95)
  804. at org.radargun.stressors.WarmupStressor.doGet(WarmupStressor.java:95)
  805. at org.radargun.stressors.WarmupStressor.access$100(WarmupStressor.java:18)
  806. at org.radargun.stressors.WarmupStressor$1.run(WarmupStressor.java:66)
  807. 12:04:03,642 WARN [DistLockingInterceptor] xxx entry commit warmup_key_7_-90-4-warmup_bucket7_ => Thread[OOB-15,x,node01-27066,5,Thread Pools]
  808. 12:04:03,642 WARN [DistLockingInterceptor] xxx entry commit warmup_key_5_-100-0-warmup_bucket5_ => Thread[OOB-10,x,node01-27066,5,Thread Pools]
  809. 12:04:03,644 DEBUG [DistributionManagerImpl] Setting joinComplete to true for node node01-27066
  810. 12:04:03,644 INFO [JoinTask] node01-27066 completed join rehash!
  811. 12:04:03,646 WARN [DistLockingInterceptor] xxx entry commit warmup_key_4_-85-1-warmup_bucket4_ => Thread[OOB-15,x,node01-27066,5,Thread Pools]
  812. 12:04:03,647 WARN [DistLockingInterceptor] xxx entry commit warmup_key_7_-98-2-warmup_bucket7_ => Thread[OOB-12,x,node01-27066,5,Thread Pools]
  813. 12:04:03,647 WARN [DistLockingInterceptor] xxx entry commit warmup_key_5_-107-4-warmup_bucket5_ => Thread[OOB-12,x,node01-27066,5,Thread Pools]
  814. 12:04:03,650 WARN [DistLockingInterceptor] xxx entry commit warmup_key_5_-113-0-warmup_bucket5_ => Thread[OOB-15,x,node01-27066,5,Thread Pools]
  815. 12:04:03,650 WARN [DistLockingInterceptor] xxx entry commit warmup_key_1_-82-2-warmup_bucket1_ => Thread[OOB-12,x,node01-27066,5,Thread Pools]
  816. 12:04:03,650 WARN [DistLockingInterceptor] xxx entry commit warmup_key_9_-66-0-warmup_bucket9_ => Thread[OOB-15,x,node01-27066,5,Thread Pools]
  817. 12:04:03,651 WARN [DistLockingInterceptor] xxx entry commit warmup_key_2_-116-4-warmup_bucket2_ => Thread[OOB-15,x,node01-27066,5,Thread Pools]
  818. 12:04:03,652 WARN [DistLockingInterceptor] xxx entry commit warmup_key_6_-106-4-warmup_bucket6_ => Thread[OOB-15,x,node01-27066,5,Thread Pools]
  819. 12:04:03,653 WARN [DistLockingInterceptor] xxx entry commit warmup_key_5_-116-0-warmup_bucket5_ => Thread[OOB-15,x,node01-27066,5,Thread Pools]
  820. 12:04:03,653 WARN [DistLockingInterceptor] xxx entry commit warmup_key_3_-110-1-warmup_bucket3_ => Thread[OOB-15,x,node01-27066,5,Thread Pools]
  821. 12:04:03,654 WARN [DistLockingInterceptor] xxx entry commit warmup_key_2_-125-3-warmup_bucket2_ => Thread[OOB-15,x,node01-27066,5,Thread Pools]
  822. 12:04:03,657 WARN [DistLockingInterceptor] xxx entry commit warmup_key_5_-121-0-warmup_bucket5_ => Thread[OOB-15,x,node01-27066,5,Thread Pools]
  823. 12:04:03,659 WARN [DistLockingInterceptor] xxx entry commit warmup_key_9_-78-4-warmup_bucket9_ => Thread[OOB-15,x,node01-27066,5,Thread Pools]
  824. 12:04:03,661 WARN [DistLockingInterceptor] xxx entry commit warmup_key_2_-133-1-warmup_bucket2_ => Thread[OOB-15,x,node01-27066,5,Thread Pools]
  825. 12:04:03,662 WARN [DistLockingInterceptor] xxx entry commit warmup_key_6_-118-0-warmup_bucket6_ => Thread[OOB-10,x,node01-27066,5,Thread Pools]
  826. 12:04:03,662 WARN [DistLockingInterceptor] xxx entry commit warmup_key_3_-117-0-warmup_bucket3_ => Thread[OOB-10,x,node01-27066,5,Thread Pools]
  827. 12:04:03,663 WARN [DistLockingInterceptor] xxx entry commit warmup_key_4_-103-2-warmup_bucket4_ => Thread[OOB-15,x,node01-27066,5,Thread Pools]
  828. 12:04:03,667 WARN [DistLockingInterceptor] xxx entry commit warmup_key_10_-42-2-warmup_bucket10_ => Thread[OOB-12,x,node01-27066,5,Thread Pools]
  829. 12:04:03,668 WARN [DistLockingInterceptor] xxx entry commit warmup_key_1_-101-2-warmup_bucket1_ => Thread[OOB-12,x,node01-27066,5,Thread Pools]
  830. 12:04:03,670 WARN [DistLockingInterceptor] xxx entry commit warmup_key_10_-44-2-warmup_bucket10_ => Thread[OOB-15,x,node01-27066,5,Thread Pools]
  831. 12:04:03,672 WARN [DistLockingInterceptor] xxx entry commit warmup_key_3_-127-2-warmup_bucket3_ => Thread[OOB-15,x,node01-27066,5,Thread Pools]
  832. 12:04:03,673 WARN [DistLockingInterceptor] xxx entry commit warmup_key_7_-136-1-warmup_bucket7_ => Thread[OOB-10,x,node01-27066,5,Thread Pools]
  833. 12:04:03,673 WARN [DistLockingInterceptor] xxx entry commit warmup_key_5_-133-0-warmup_bucket5_ => Thread[OOB-12,x,node01-27066,5,Thread Pools]
  834. 12:04:03,677 WARN [DistLockingInterceptor] xxx entry commit warmup_key_10_-49-2-warmup_bucket10_ => Thread[OOB-10,x,node01-27066,5,Thread Pools]
  835. 12:04:03,681 WARN [DistLockingInterceptor] xxx entry commit warmup_key_10_-54-4-warmup_bucket10_ => Thread[OOB-10,x,node01-27066,5,Thread Pools]
  836. 12:04:03,689 WARN [DistLockingInterceptor] xxx entry commit warmup_key_3_-134-0-warmup_bucket3_ => Thread[OOB-10,x,node01-27066,5,Thread Pools]
  837. 12:04:03,690 WARN [DistLockingInterceptor] xxx entry commit warmup_key_9_-99-3-warmup_bucket9_ => Thread[OOB-10,x,node01-27066,5,Thread Pools]
  838. 12:04:03,690 WARN [DistLockingInterceptor] xxx entry commit warmup_key_7_-152-4-warmup_bucket7_ => Thread[OOB-12,x,node01-27066,5,Thread Pools]
  839. 12:04:03,690 WARN [DistLockingInterceptor] xxx entry commit warmup_key_2_-155-4-warmup_bucket2_ => Thread[OOB-10,x,node01-27066,5,Thread Pools]
  840. 12:04:03,691 WARN [DistLockingInterceptor] xxx entry commit warmup_key_7_-153-1-warmup_bucket7_ => Thread[OOB-12,x,node01-27066,5,Thread Pools]
  841. 12:04:03,691 WARN [DistLockingInterceptor] xxx entry commit warmup_key_2_-156-1-warmup_bucket2_ => Thread[OOB-10,x,node01-27066,5,Thread Pools]
  842. 12:04:03,692 WARN [DistLockingInterceptor] xxx entry commit warmup_key_5_-150-1-warmup_bucket5_ => Thread[OOB-10,x,node01-27066,5,Thread Pools]
  843. 12:04:03,692 WARN [DistLockingInterceptor] xxx entry commit warmup_key_10_-58-3-warmup_bucket10_ => Thread[OOB-10,x,node01-27066,5,Thread Pools]
  844. 12:04:03,693 WARN [DistLockingInterceptor] xxx entry commit warmup_key_2_-159-1-warmup_bucket2_ => Thread[OOB-10,x,node01-27066,5,Thread Pools]
  845. 12:04:03,693 WARN [DistLockingInterceptor] xxx entry commit warmup_key_5_-155-3-warmup_bucket5_ => Thread[OOB-10,x,node01-27066,5,Thread Pools]
  846. 12:04:03,695 WARN [DistLockingInterceptor] xxx entry commit warmup_key_2_-163-1-warmup_bucket2_ => Thread[OOB-10,x,node01-27066,5,Thread Pools]
  847. 12:04:03,697 WARN [DistLockingInterceptor] xxx entry commit warmup_key_0_-141-3-warmup_bucket0_ => Thread[Thread-21,5,main]
  848. 12:04:03,697 WARN [DistLockingInterceptor] xxx entry commit warmup_key_8_-116-0-warmup_bucket8_ => Thread[OOB-10,x,node01-27066,5,Thread Pools]
  849. 12:04:03,702 TRACE [DistributionManagerImpl] Informed of a JOIN by node02-34421. Starting? true
  850. 12:04:03,702 TRACE [DistributionManagerImpl] Node topology info added(null). Topology info is TopologyInfo{address2TopologyInfo={node04-53049=null, node03-61513=null, node07-42120=null, node11-42115=null, node05-23599=null, node06-24999=null, node10-3814=null, node02-34421=null, node09-42492=null, node08-968=null}}
  851. 12:04:03,702 TRACE [DistributionManagerImpl] New CH is AbstractConsistentHash{caches=null, topologyInfo=null}
  852. 12:04:03,717 TRACE [DistributionManagerImpl] Informed of a JOIN by node02-34421. Starting? false
  853. 12:04:03,717 TRACE [DistributionManagerImpl] New CH is DefaultConsistentHash{addresses ={73=node09-42492, 848=node02-34421, 849=node02-34421, 1849=node03-61513, 1850=node03-61513, 2093=node10-3814, 2094=node10-3814, 2380=node08-968, 2381=node08-968, 3182=node04-53049, 3183=node04-53049, 5868=node07-42120, 5869=node07-42120, 5914=node05-23599, 5915=node05-23599, 6410=node01-27066, 6978=node06-24999, 8485=node11-42115, 8486=node11-42115}, hash space =10240}
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!