2013-09-10 16:47:22-0700 [] DEBUG: Trying to connect to node cassa001 over thrift
2013-09-10 16:47:22-0700 [] DEBUG: Not returning SASL credentials for cassa001
2013-09-10 16:47:22-0700 []  INFO: Starting factory <opscenterd.ThriftService.NoReconnectCassandraClientFactory instance at 0x3153518>
2013-09-10 16:47:22-0700 [] DEBUG: Node ping successful: cassa001
2013-09-10 16:47:22-0700 []  INFO: Adding new cluster 'prod': {u'jmx': {u'username': u'', u'password': u'', u'port': u'7199'}, 'kerberos_client_principals': {}, 'kerberos': {}, u'agents': {}, 'kerberos_hostnames': {}, 'kerberos_services': {}, u'cassandra': {u'username': u'', u'seed_hosts': u'cassa001', u'api_port': u'9160', u'password': u''}}
2013-09-10 16:47:22-0700 []  INFO: Starting new cluster services for prod
2013-09-10 16:47:22-0700 [prod]  INFO: Starting services for cluster prod
2013-09-10 16:47:22-0700 [prod] DEBUG: keyspace type system are {'system': [u'system', u'system_traces', u'system_auth', u'dse_auth']}
2013-09-10 16:47:22-0700 [] DEBUG: Not using SSL for Thrift communication
2013-09-10 16:47:22-0700 [] DEBUG: ignored_keyspaces are [u'system', u'system_traces', u'system_auth', u'dse_auth']
2013-09-10 16:47:22-0700 [] DEBUG: Not using Kerberos authentication for Thrift
2013-09-10 16:47:22-0700 [prod] DEBUG: Not using separate storage cluster
2013-09-10 16:47:22-0700 []  INFO: Metric caching enabled with 50 points and 1000 metrics cached
2013-09-10 16:47:22-0700 []  INFO: Starting PushService
2013-09-10 16:47:22-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:22-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:22-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:22-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:22-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:22-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:47:22-0700 [prod]  INFO: Starting CassandraCluster service
2013-09-10 16:47:22-0700 [prod]  INFO: agent_config items: {'cassandra_log_location': '/var/log/cassandra/system.log', 'thrift_port': 9160, 'thrift_ssl_truststore': None, 'rollups300_ttl': 2419200, 'rollups86400_ttl': -1, 'jmx_port': 7199, 'metrics_ignored_solr_cores': '', 'api_port': '61621', 'metrics_enabled': 1, 'thrift_ssl_truststore_type': 'JKS', 'kerberos_use_ticket_cache': True, 'kerberos_renew_tgt': True, 'rollups60_ttl': 604800, 'cassandra_install_location': '', 'rollups7200_ttl': 31536000, 'kerberos_debug': False, 'storage_keyspace': 'OpsCenter', 'ec2_metadata_api_host': '169.254.169.254', 'provisioning': 0, 'kerberos_use_keytab': True, 'metrics_ignored_column_families': '', 'thrift_ssl_truststore_password': None, 'metrics_ignored_keyspaces': 'system, system_traces, system_auth, dse_auth, OpsCenter'}
2013-09-10 16:47:22-0700 []  INFO: Stopping factory <opscenterd.ThriftService.NoReconnectCassandraClientFactory instance at 0x3153518>
2013-09-10 16:47:22-0700 [prod] Thrift pool connection to <CassandraNode cassa001:9160 @0x314afc8> failed
        Traceback (most recent call last):
        Failure: telephus.cassandra.ttypes.InvalidRequestException: InvalidRequestException(why='You have not logged in')

2013-09-10 16:47:22-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 0.15226507186889648 left on connect blackout
2013-09-10 16:47:22-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.2 seconds.)
2013-09-10 16:47:22-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 0.15256810188293457 left on connect blackout
2013-09-10 16:47:22-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.2 seconds.)
2013-09-10 16:47:22-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 0.15256619453430176 left on connect blackout
2013-09-10 16:47:22-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.2 seconds.)
2013-09-10 16:47:22-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 0.15256929397583008 left on connect blackout
2013-09-10 16:47:22-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.2 seconds.)
2013-09-10 16:47:22-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 0.15256929397583008 left on connect blackout
2013-09-10 16:47:22-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.2 seconds.)
2013-09-10 16:47:22-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:22-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:22-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:22-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:22-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:22-0700 [prod] Thrift pool connection to <CassandraNode cassa001:9160 @0x314afc8> failed
        Traceback (most recent call last):
        Failure: telephus.cassandra.ttypes.InvalidRequestException: InvalidRequestException(why='You have not logged in')

2013-09-10 16:47:22-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 0.46498751640319824 left on connect blackout
2013-09-10 16:47:22-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 0.5 seconds.)
2013-09-10 16:47:22-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 0.46535348892211914 left on connect blackout
2013-09-10 16:47:22-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 0.5 seconds.)
2013-09-10 16:47:22-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 0.46535038948059082 left on connect blackout
2013-09-10 16:47:22-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 0.5 seconds.)
2013-09-10 16:47:22-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 0.46535229682922363 left on connect blackout
2013-09-10 16:47:22-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 0.5 seconds.)
2013-09-10 16:47:22-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 0.46533536911010742 left on connect blackout
2013-09-10 16:47:22-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 0.5 seconds.)
2013-09-10 16:47:23-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:23-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:23-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:23-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:23-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:23-0700 [prod] Thrift pool connection to <CassandraNode cassa001:9160 @0x314afc8> failed
        Traceback (most recent call last):
        Failure: telephus.cassandra.ttypes.InvalidRequestException: InvalidRequestException(why='You have not logged in')

2013-09-10 16:47:23-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 1.2875463962554932 left on connect blackout
2013-09-10 16:47:23-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 1.3 seconds.)
2013-09-10 16:47:23-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 1.2878634929656982 left on connect blackout
2013-09-10 16:47:23-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 1.3 seconds.)
2013-09-10 16:47:23-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 1.2878634929656982 left on connect blackout
2013-09-10 16:47:23-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 1.3 seconds.)
2013-09-10 16:47:23-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 1.2878637313842773 left on connect blackout
2013-09-10 16:47:23-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 1.3 seconds.)
2013-09-10 16:47:23-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 1.2878646850585938 left on connect blackout
2013-09-10 16:47:23-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 1.3 seconds.)
2013-09-10 16:47:24-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:24-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:24-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:24-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:24-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:24-0700 [prod] Thrift pool connection to <CassandraNode cassa001:9160 @0x314afc8> failed
        Traceback (most recent call last):
        Failure: telephus.cassandra.ttypes.InvalidRequestException: InvalidRequestException(why='You have not logged in')

2013-09-10 16:47:24-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 3.3992953300476074 left on connect blackout
2013-09-10 16:47:24-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 3.4 seconds.)
2013-09-10 16:47:24-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 3.3994712829589844 left on connect blackout
2013-09-10 16:47:24-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 3.4 seconds.)
2013-09-10 16:47:24-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 3.3994772434234619 left on connect blackout
2013-09-10 16:47:24-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 3.4 seconds.)
2013-09-10 16:47:24-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 3.3994932174682617 left on connect blackout
2013-09-10 16:47:24-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 3.4 seconds.)
2013-09-10 16:47:24-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 3.39949631690979 left on connect blackout
2013-09-10 16:47:24-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 3.4 seconds.)
2013-09-10 16:47:27-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:47:27-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:27-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:27-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:27-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:27-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:27-0700 [prod] Thrift pool connection to <CassandraNode cassa001:9160 @0x314afc8> failed
        Traceback (most recent call last):
        Failure: telephus.cassandra.ttypes.InvalidRequestException: InvalidRequestException(why='You have not logged in')

2013-09-10 16:47:27-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 7.79595947265625 left on connect blackout
2013-09-10 16:47:27-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 7.8 seconds.)
2013-09-10 16:47:27-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 7.7962954044342041 left on connect blackout
2013-09-10 16:47:27-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 7.8 seconds.)
2013-09-10 16:47:27-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 7.7963285446166992 left on connect blackout
2013-09-10 16:47:27-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 7.8 seconds.)
2013-09-10 16:47:27-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 7.7963273525238037 left on connect blackout
2013-09-10 16:47:27-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 7.8 seconds.)
2013-09-10 16:47:27-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 7.7963294982910156 left on connect blackout
2013-09-10 16:47:27-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 7.8 seconds.)
2013-09-10 16:47:32-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:47:35-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:35-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:35-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:35-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:35-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:35-0700 [prod] Thrift pool connection to <CassandraNode cassa001:9160 @0x314afc8> failed
        Traceback (most recent call last):
        Failure: telephus.cassandra.ttypes.InvalidRequestException: InvalidRequestException(why='You have not logged in')

2013-09-10 16:47:35-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 20.257841348648071 left on connect blackout
2013-09-10 16:47:35-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 20.3 seconds.)
2013-09-10 16:47:35-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 20.258150100708008 left on connect blackout
2013-09-10 16:47:35-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 20.3 seconds.)
2013-09-10 16:47:35-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 20.258151292800903 left on connect blackout
2013-09-10 16:47:35-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 20.3 seconds.)
2013-09-10 16:47:35-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 20.258151292800903 left on connect blackout
2013-09-10 16:47:35-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 20.3 seconds.)
2013-09-10 16:47:35-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 20.258150100708008 left on connect blackout
2013-09-10 16:47:35-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 20.3 seconds.)
2013-09-10 16:47:37-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:47:42-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:47:47-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:47:52-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:47:55-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:55-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:55-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:55-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:55-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:47:55-0700 [prod] Thrift pool connection to <CassandraNode cassa001:9160 @0x314afc8> failed
        Traceback (most recent call last):
        Failure: telephus.cassandra.ttypes.InvalidRequestException: InvalidRequestException(why='You have not logged in')

2013-09-10 16:47:55-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 46.960758924484253 left on connect blackout
2013-09-10 16:47:55-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 47.0 seconds.)
2013-09-10 16:47:55-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 46.961021900177002 left on connect blackout
2013-09-10 16:47:55-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 47.0 seconds.)
2013-09-10 16:47:55-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 46.96101975440979 left on connect blackout
2013-09-10 16:47:55-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 47.0 seconds.)
2013-09-10 16:47:55-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 46.961021661758423 left on connect blackout
2013-09-10 16:47:55-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 47.0 seconds.)
2013-09-10 16:47:55-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 46.961022615432739 left on connect blackout
2013-09-10 16:47:55-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 47.0 seconds.)
2013-09-10 16:47:57-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:47:59-0700 [] DEBUG: Average opscenterd CPU usage: 0.38%, memory usage: 41 MB
2013-09-10 16:48:02-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:48:07-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:48:12-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:48:17-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:48:22-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:48:27-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:48:32-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:48:37-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:48:42-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:48:42-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:48:42-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:48:42-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:48:42-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:48:42-0700 [prod] DEBUG: Adding connection to <CassandraNode cassa001:9160 @0x314afc8>
2013-09-10 16:48:42-0700 [prod] Thrift pool connection to <CassandraNode cassa001:9160 @0x314afc8> failed
        Traceback (most recent call last):
        Failure: telephus.cassandra.ttypes.InvalidRequestException: InvalidRequestException(why='You have not logged in')

2013-09-10 16:48:42-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 118.9441294670105 left on connect blackout
2013-09-10 16:48:42-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 118.9 seconds.)
2013-09-10 16:48:42-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 118.94436931610107 left on connect blackout
2013-09-10 16:48:42-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 118.9 seconds.)
2013-09-10 16:48:42-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 118.94437050819397 left on connect blackout
2013-09-10 16:48:42-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 118.9 seconds.)
2013-09-10 16:48:42-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 118.94437050819397 left on connect blackout
2013-09-10 16:48:42-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 118.9 seconds.)
2013-09-10 16:48:42-0700 [prod] DEBUG: not considering <CassandraNode cassa001:9160 @0x314afc8> for new connection; has 118.94435453414917 left on connect blackout
2013-09-10 16:48:42-0700 [prod]  INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 118.9 seconds.)
2013-09-10 16:48:47-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:48:52-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:48:57-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:48:59-0700 [] DEBUG: Average opscenterd CPU usage: 0.07%, memory usage: 41 MB
2013-09-10 16:49:02-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:49:07-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:49:12-0700 [prod] DEBUG: Waiting for describe_version() results
2013-09-10 16:49:17-0700 [prod] DEBUG: Waiting for describe_version() results