Pastebin launched a little side project called VERYVIRAL.com, check it out ;-) Want more features on Pastebin? Sign Up, it's FREE!
Guest

Untitled

By: a guest on Sep 10th, 2013  |  syntax: None  |  size: 24.03 KB  |  views: 56  |  expires: Never
download  |  raw  |  embed  |  report abuse  |  print
Text below is selected. Please press Ctrl+C to copy to your clipboard. (⌘+C on Mac)
  1. 2013-09-10 16:47:22-0700 [] DEBUG: Trying to connect to node cassa001 over thrift
  2. 2013-09-10 16:47:22-0700 [] DEBUG: Not returning SASL credentials for cassa001
  3. 2013-09-10 16:47:22-0700 []  INFO: Starting factory <opscenterd.ThriftService.NoReconnectCassandraClientFactory instance at 0x3153518>
  4. 2013-09-10 16:47:22-0700 [] DEBUG: Node ping successful: cassa001
  5. 2013-09-10 16:47:22-0700 []  INFO: Adding new cluster 'prod': {u'jmx': {u'username': u'', u'password': u'', u'port': u'7199'}, 'kerberos_client_principals': {}, 'kerberos': {}, u'agents': {}, 'kerberos_hostnames': {}, 'kerberos_services': {}, u'cassandra': {u'username': u'', u'seed_hosts': u'cassa001', u'api_port': u'9160', u'password': u''}}
  6. 2013-09-10 16:47:22-0700 []  INFO: Starting new cluster services for prod
  7. 2013-09-10 16:47:22-0700 [prod]  INFO: Starting services for cluster prod
  8. 2013-09-10 16:47:22-0700 [prod] DEBUG: keyspace type system are {'system': [u'system', u'system_traces', u'system_auth', u'dse_auth']}
  9. 2013-09-10 16:47:22-0700 [] DEBUG: Not using SSL for Thrift communication
  10. 2013-09-10 16:47:22-0700 [] DEBUG: ignored_keyspaces are [u'system', u'system_traces', u'system_auth', u'dse_auth']
  11. 2013-09-10 16:47:22-0700 [] DEBUG: Not using Kerberos authentication for Thrift
  12. 2013-09-10 16:47:22-0700 [prod] DEBUG: Not using separate storage cluster
  13. 2013-09-10 16:47:22-0700 []  INFO: Metric caching enabled with 50 points and 1000 metrics cached
  14. 2013-09-10 16:47:22-0700 []  INFO: Starting PushService
  15. 2013-09-10 16:47:22-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  16. 2013-09-10 16:47:22-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  17. 2013-09-10 16:47:22-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  18. 2013-09-10 16:47:22-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  19. 2013-09-10 16:47:22-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  20. 2013-09-10 16:47:22-0700 [prod] DEBUG: Waiting for describe_version() results
  21. 2013-09-10 16:47:22-0700 [prod]  INFO: Starting CassandraCluster service
  22. 2013-09-10 16:47:22-0700 [prod]  INFO: agent_config items: {'cassandra_log_location': '/var/log/cassandra/system.log', 'thrift_port': 9160, 'thrift_ssl_truststore': None, 'rollups300_ttl': 2419200, 'rollups86400_ttl': -1, 'jmx_port': 7199, 'metrics_ignored_solr_cores': '', 'api_port': '61621', 'metrics_enabled': 1, 'thrift_ssl_truststore_type': 'JKS', 'kerberos_use_ticket_cache': True, 'kerberos_renew_tgt': True, 'rollups60_ttl': 604800, 'cassandra_install_location': '', 'rollups7200_ttl': 31536000, 'kerberos_debug': False, 'storage_keyspace': 'OpsCenter', 'ec2_metadata_api_host': '169.254.169.254', 'provisioning': 0, 'kerberos_use_keytab': True, 'metrics_ignored_column_families': '', 'thrift_ssl_truststore_password': None, 'metrics_ignored_keyspaces': 'system, system_traces, system_auth, dse_auth, OpsCenter'}
  23. 2013-09-10 16:47:22-0700 []  INFO: Stopping factory <opscenterd.ThriftService.NoReconnectCassandraClientFactory instance at 0x3153518>
  24. 2013-09-10 16:47:22-0700 [prod] Thrift pool connection to <CassandraNode cassa001:9160 @0x314afc8> failed
  25.         Traceback (most recent call last):
  26.         Failure: telephus.cassandra.ttypes.InvalidRequestException: InvalidRequestException(why='You have not logged in')
  27.  
  28. 2013-09-10 16:47:22-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 0.15226507186889648 left on connect blackout
  29. 2013-09-10 16:47:22-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.2 seconds.)
  30. 2013-09-10 16:47:22-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 0.15256810188293457 left on connect blackout
  31. 2013-09-10 16:47:22-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.2 seconds.)
  32. 2013-09-10 16:47:22-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 0.15256619453430176 left on connect blackout
  33. 2013-09-10 16:47:22-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.2 seconds.)
  34. 2013-09-10 16:47:22-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 0.15256929397583008 left on connect blackout
  35. 2013-09-10 16:47:22-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.2 seconds.)
  36. 2013-09-10 16:47:22-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 0.15256929397583008 left on connect blackout
  37. 2013-09-10 16:47:22-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.2 seconds.)
  38. 2013-09-10 16:47:22-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  39. 2013-09-10 16:47:22-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  40. 2013-09-10 16:47:22-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  41. 2013-09-10 16:47:22-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  42. 2013-09-10 16:47:22-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  43. 2013-09-10 16:47:22-0700 [prod] Thrift pool connection to <CassandraNode cassa001:9160 @0x314afc8> failed
  44.         Traceback (most recent call last):
  45.         Failure: telephus.cassandra.ttypes.InvalidRequestException: InvalidRequestException(why='You have not logged in')
  46.  
  47. 2013-09-10 16:47:22-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 0.46498751640319824 left on connect blackout
  48. 2013-09-10 16:47:22-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 0.5 seconds.)
  49. 2013-09-10 16:47:22-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 0.46535348892211914 left on connect blackout
  50. 2013-09-10 16:47:22-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 0.5 seconds.)
  51. 2013-09-10 16:47:22-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 0.46535038948059082 left on connect blackout
  52. 2013-09-10 16:47:22-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 0.5 seconds.)
  53. 2013-09-10 16:47:22-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 0.46535229682922363 left on connect blackout
  54. 2013-09-10 16:47:22-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 0.5 seconds.)
  55. 2013-09-10 16:47:22-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 0.46533536911010742 left on connect blackout
  56. 2013-09-10 16:47:22-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 0.5 seconds.)
  57. 2013-09-10 16:47:23-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  58. 2013-09-10 16:47:23-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  59. 2013-09-10 16:47:23-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  60. 2013-09-10 16:47:23-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  61. 2013-09-10 16:47:23-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  62. 2013-09-10 16:47:23-0700 [prod] Thrift pool connection to <CassandraNode cassa001:9160 @0x314afc8> failed
  63.         Traceback (most recent call last):
  64.         Failure: telephus.cassandra.ttypes.InvalidRequestException: InvalidRequestException(why='You have not logged in')
  65.  
  66. 2013-09-10 16:47:23-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 1.2875463962554932 left on connect blackout
  67. 2013-09-10 16:47:23-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 1.3 seconds.)
  68. 2013-09-10 16:47:23-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 1.2878634929656982 left on connect blackout
  69. 2013-09-10 16:47:23-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 1.3 seconds.)
  70. 2013-09-10 16:47:23-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 1.2878634929656982 left on connect blackout
  71. 2013-09-10 16:47:23-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 1.3 seconds.)
  72. 2013-09-10 16:47:23-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 1.2878637313842773 left on connect blackout
  73. 2013-09-10 16:47:23-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 1.3 seconds.)
  74. 2013-09-10 16:47:23-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 1.2878646850585938 left on connect blackout
  75. 2013-09-10 16:47:23-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 1.3 seconds.)
  76. 2013-09-10 16:47:24-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  77. 2013-09-10 16:47:24-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  78. 2013-09-10 16:47:24-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  79. 2013-09-10 16:47:24-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  80. 2013-09-10 16:47:24-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  81. 2013-09-10 16:47:24-0700 [prod] Thrift pool connection to <CassandraNode cassa001:9160 @0x314afc8> failed
  82.         Traceback (most recent call last):
  83.         Failure: telephus.cassandra.ttypes.InvalidRequestException: InvalidRequestException(why='You have not logged in')
  84.  
  85. 2013-09-10 16:47:24-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 3.3992953300476074 left on connect blackout
  86. 2013-09-10 16:47:24-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 3.4 seconds.)
  87. 2013-09-10 16:47:24-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 3.3994712829589844 left on connect blackout
  88. 2013-09-10 16:47:24-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 3.4 seconds.)
  89. 2013-09-10 16:47:24-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 3.3994772434234619 left on connect blackout
  90. 2013-09-10 16:47:24-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 3.4 seconds.)
  91. 2013-09-10 16:47:24-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 3.3994932174682617 left on connect blackout
  92. 2013-09-10 16:47:24-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 3.4 seconds.)
  93. 2013-09-10 16:47:24-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 3.39949631690979 left on connect blackout
  94. 2013-09-10 16:47:24-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 3.4 seconds.)
  95. 2013-09-10 16:47:27-0700 [prod] DEBUG: Waiting for describe_version() results
  96. 2013-09-10 16:47:27-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  97. 2013-09-10 16:47:27-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  98. 2013-09-10 16:47:27-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  99. 2013-09-10 16:47:27-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  100. 2013-09-10 16:47:27-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  101. 2013-09-10 16:47:27-0700 [prod] Thrift pool connection to <CassandraNode cassa001:9160 @0x314afc8> failed
  102.         Traceback (most recent call last):
  103.         Failure: telephus.cassandra.ttypes.InvalidRequestException: InvalidRequestException(why='You have not logged in')
  104.  
  105. 2013-09-10 16:47:27-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 7.79595947265625 left on connect blackout
  106. 2013-09-10 16:47:27-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 7.8 seconds.)
  107. 2013-09-10 16:47:27-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 7.7962954044342041 left on connect blackout
  108. 2013-09-10 16:47:27-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 7.8 seconds.)
  109. 2013-09-10 16:47:27-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 7.7963285446166992 left on connect blackout
  110. 2013-09-10 16:47:27-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 7.8 seconds.)
  111. 2013-09-10 16:47:27-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 7.7963273525238037 left on connect blackout
  112. 2013-09-10 16:47:27-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 7.8 seconds.)
  113. 2013-09-10 16:47:27-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 7.7963294982910156 left on connect blackout
  114. 2013-09-10 16:47:27-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 7.8 seconds.)
  115. 2013-09-10 16:47:32-0700 [prod] DEBUG: Waiting for describe_version() results
  116. 2013-09-10 16:47:35-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  117. 2013-09-10 16:47:35-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  118. 2013-09-10 16:47:35-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  119. 2013-09-10 16:47:35-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  120. 2013-09-10 16:47:35-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  121. 2013-09-10 16:47:35-0700 [prod] Thrift pool connection to <CassandraNode cassa001:9160 @0x314afc8> failed
  122.         Traceback (most recent call last):
  123.         Failure: telephus.cassandra.ttypes.InvalidRequestException: InvalidRequestException(why='You have not logged in')
  124.  
  125. 2013-09-10 16:47:35-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 20.257841348648071 left on connect blackout
  126. 2013-09-10 16:47:35-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 20.3 seconds.)
  127. 2013-09-10 16:47:35-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 20.258150100708008 left on connect blackout
  128. 2013-09-10 16:47:35-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 20.3 seconds.)
  129. 2013-09-10 16:47:35-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 20.258151292800903 left on connect blackout
  130. 2013-09-10 16:47:35-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 20.3 seconds.)
  131. 2013-09-10 16:47:35-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 20.258151292800903 left on connect blackout
  132. 2013-09-10 16:47:35-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 20.3 seconds.)
  133. 2013-09-10 16:47:35-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 20.258150100708008 left on connect blackout
  134. 2013-09-10 16:47:35-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 20.3 seconds.)
  135. 2013-09-10 16:47:37-0700 [prod] DEBUG: Waiting for describe_version() results
  136. 2013-09-10 16:47:42-0700 [prod] DEBUG: Waiting for describe_version() results
  137. 2013-09-10 16:47:47-0700 [prod] DEBUG: Waiting for describe_version() results
  138. 2013-09-10 16:47:52-0700 [prod] DEBUG: Waiting for describe_version() results
  139. 2013-09-10 16:47:55-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  140. 2013-09-10 16:47:55-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  141. 2013-09-10 16:47:55-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  142. 2013-09-10 16:47:55-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  143. 2013-09-10 16:47:55-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  144. 2013-09-10 16:47:55-0700 [prod] Thrift pool connection to <CassandraNode cassa001:9160 @0x314afc8> failed
  145.         Traceback (most recent call last):
  146.         Failure: telephus.cassandra.ttypes.InvalidRequestException: InvalidRequestException(why='You have not logged in')
  147.  
  148. 2013-09-10 16:47:55-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 46.960758924484253 left on connect blackout
  149. 2013-09-10 16:47:55-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 47.0 seconds.)
  150. 2013-09-10 16:47:55-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 46.961021900177002 left on connect blackout
  151. 2013-09-10 16:47:55-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 47.0 seconds.)
  152. 2013-09-10 16:47:55-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 46.96101975440979 left on connect blackout
  153. 2013-09-10 16:47:55-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 47.0 seconds.)
  154. 2013-09-10 16:47:55-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 46.961021661758423 left on connect blackout
  155. 2013-09-10 16:47:55-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 47.0 seconds.)
  156. 2013-09-10 16:47:55-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 46.961022615432739 left on connect blackout
  157. 2013-09-10 16:47:55-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 47.0 seconds.)
  158. 2013-09-10 16:47:57-0700 [prod] DEBUG: Waiting for describe_version() results
  159. 2013-09-10 16:47:59-0700 [] DEBUG: Average opscenterd CPU usage: 0.38%, memory usage: 41 MB
  160. 2013-09-10 16:48:02-0700 [prod] DEBUG: Waiting for describe_version() results
  161. 2013-09-10 16:48:07-0700 [prod] DEBUG: Waiting for describe_version() results
  162. 2013-09-10 16:48:12-0700 [prod] DEBUG: Waiting for describe_version() results
  163. 2013-09-10 16:48:17-0700 [prod] DEBUG: Waiting for describe_version() results
  164. 2013-09-10 16:48:22-0700 [prod] DEBUG: Waiting for describe_version() results
  165. 2013-09-10 16:48:27-0700 [prod] DEBUG: Waiting for describe_version() results
  166. 2013-09-10 16:48:32-0700 [prod] DEBUG: Waiting for describe_version() results
  167. 2013-09-10 16:48:37-0700 [prod] DEBUG: Waiting for describe_version() results
  168. 2013-09-10 16:48:42-0700 [prod] DEBUG: Waiting for describe_version() results
  169. 2013-09-10 16:48:42-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  170. 2013-09-10 16:48:42-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  171. 2013-09-10 16:48:42-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  172. 2013-09-10 16:48:42-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  173. 2013-09-10 16:48:42-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
  174. 2013-09-10 16:48:42-0700 [prod] Thrift pool connection to <CassandraNode cassa001:9160 @0x314afc8> failed
  175.         Traceback (most recent call last):
  176.         Failure: telephus.cassandra.ttypes.InvalidRequestException: InvalidRequestException(why='You have not logged in')
  177.  
  178. 2013-09-10 16:48:42-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 118.9441294670105 left on connect blackout
  179. 2013-09-10 16:48:42-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 118.9 seconds.)
  180. 2013-09-10 16:48:42-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 118.94436931610107 left on connect blackout
  181. 2013-09-10 16:48:42-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 118.9 seconds.)
  182. 2013-09-10 16:48:42-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 118.94437050819397 left on connect blackout
  183. 2013-09-10 16:48:42-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 118.9 seconds.)
  184. 2013-09-10 16:48:42-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 118.94437050819397 left on connect blackout
  185. 2013-09-10 16:48:42-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 118.9 seconds.)
  186. 2013-09-10 16:48:42-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 118.94435453414917 left on connect blackout
  187. 2013-09-10 16:48:42-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 118.9 seconds.)
  188. 2013-09-10 16:48:47-0700 [prod] DEBUG: Waiting for describe_version() results
  189. 2013-09-10 16:48:52-0700 [prod] DEBUG: Waiting for describe_version() results
  190. 2013-09-10 16:48:57-0700 [prod] DEBUG: Waiting for describe_version() results
  191. 2013-09-10 16:48:59-0700 [] DEBUG: Average opscenterd CPU usage: 0.07%, memory usage: 41 MB
  192. 2013-09-10 16:49:02-0700 [prod] DEBUG: Waiting for describe_version() results
  193. 2013-09-10 16:49:07-0700 [prod] DEBUG: Waiting for describe_version() results
  194. 2013-09-10 16:49:12-0700 [prod] DEBUG: Waiting for describe_version() results
  195. 2013-09-10 16:49:17-0700 [prod] DEBUG: Waiting for describe_version() results