API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Sep 21st, 2015
199
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 149.60 KB | None | 0 0
raw download clone embed print report
  1. 2015-09-21 09:18:16,551 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.EvenGuestDistributionBalancePolicyUnit] (DefaultQuartzScheduler_Worker-68) There is no host with more than 10 running guests, no balancing is needed
  2. 2015-09-21 09:19:02,713 INFO [org.ovirt.engine.docs.utils.servlet.ContextSensitiveHelpMappingServlet] (ajp--127.0.0.1-8702-2) Context-sensitive help is not installed. Manual directory doesn't exist: /usr/share/ovirt-engine/manual
  3. 2015-09-21 09:19:16,597 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.EvenGuestDistributionBalancePolicyUnit] (DefaultQuartzScheduler_Worker-1) [66c583c] There is no host with more than 10 running guests, no balancing is needed
  4. 2015-09-21 09:19:19,647 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginAdminUserCommand internal: false.
  5. 2015-09-21 09:19:19,654 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in.
  6. 2015-09-21 09:21:16,574 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf".
  7. 2015-09-21 09:21:16,578 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) The file "/etc/ovirt-engine/engine.conf" doesn't exist or isn't readable. Will return an empty set of properties.
  8. 2015-09-21 09:21:16,578 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf".
  9. 2015-09-21 09:21:16,579 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-java.conf".
  10. 2015-09-21 09:21:16,579 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf".
  11. 2015-09-21 09:21:16,579 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf".
  12. 2015-09-21 09:21:16,580 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf".
  13. 2015-09-21 09:21:16,580 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_AJP_ENABLED" is "true".
  14. 2015-09-21 09:21:16,581 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_AJP_PORT" is "8702".
  15. 2015-09-21 09:21:16,581 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_APPS" is "engine.ear restapi.war legacy_restapi.war".
  16. 2015-09-21 09:21:16,584 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine".
  17. 2015-09-21 09:21:16,585 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000".
  18. 2015-09-21 09:21:16,585 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000".
  19. 2015-09-21 09:21:16,585 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_DATABASE" is "engine".
  20. 2015-09-21 09:21:16,586 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver".
  21. 2015-09-21 09:21:16,586 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_HOST" is "localhost".
  22. 2015-09-21 09:21:16,586 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100".
  23. 2015-09-21 09:21:16,587 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1".
  24. 2015-09-21 09:21:16,587 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_PASSWORD" is "***".
  25. 2015-09-21 09:21:16,593 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_PORT" is "5432".
  26. 2015-09-21 09:21:16,593 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_SECURED" is "False".
  27. 2015-09-21 09:21:16,594 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False".
  28. 2015-09-21 09:21:16,595 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory".
  29. 2015-09-21 09:21:16,595 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DB_USER" is "engine".
  30. 2015-09-21 09:21:16,595 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DEBUG_ADDRESS" is "".
  31. 2015-09-21 09:21:16,595 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine".
  32. 2015-09-21 09:21:16,596 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_ETC" is "/etc/ovirt-engine".
  33. 2015-09-21 09:21:16,596 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_EXTENSION_PATH" is "/usr/share/ovirt-engine/extensions.d:/etc/ovirt-engine/extensions.d".
  34. 2015-09-21 09:21:16,596 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_EXTERNAL_PROVIDERS_TRUST_STORE" is "/var/lib/ovirt-engine/external_truststore".
  35. 2015-09-21 09:21:16,597 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_EXTERNAL_PROVIDERS_TRUST_STORE_PASSWORD" is "***".
  36. 2015-09-21 09:21:16,597 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_EXTERNAL_PROVIDERS_TRUST_STORE_TYPE" is "JKS".
  37. 2015-09-21 09:21:16,597 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_FQDN" is "ovm-manage.".
  38. 2015-09-21 09:21:16,597 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_GROUP" is "ovirt".
  39. 2015-09-21 09:21:16,598 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HEAP_MAX" is "1968M".
  40. 2015-09-21 09:21:16,598 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HEAP_MIN" is "1968M".
  41. 2015-09-21 09:21:16,598 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTPS_ENABLED" is "false".
  42. 2015-09-21 09:21:16,598 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTPS_PORT" is "None".
  43. 2015-09-21 09:21:16,599 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2".
  44. 2015-09-21 09:21:16,599 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTP_ENABLED" is "false".
  45. 2015-09-21 09:21:16,599 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_HTTP_PORT" is "None".
  46. 2015-09-21 09:21:16,599 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_JAVA_MODULEPATH" is "/usr/share/ovirt-engine/modules".
  47. 2015-09-21 09:21:16,600 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"".
  48. 2015-09-21 09:21:16,600 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine".
  49. 2015-09-21 09:21:16,603 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_LOG_TO_CONSOLE" is "false".
  50. 2015-09-21 09:21:16,603 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_MANUAL" is "/usr/share/ovirt-engine/manual".
  51. 2015-09-21 09:21:16,604 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PERM_MAX" is "256m".
  52. 2015-09-21 09:21:16,604 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PERM_MIN" is "256m".
  53. 2015-09-21 09:21:16,605 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine".
  54. 2015-09-21 09:21:16,605 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem".
  55. 2015-09-21 09:21:16,606 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer".
  56. 2015-09-21 09:21:16,606 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12".
  57. 2015-09-21 09:21:16,606 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1".
  58. 2015-09-21 09:21:16,607 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***".
  59. 2015-09-21 09:21:16,607 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore".
  60. 2015-09-21 09:21:16,607 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***".
  61. 2015-09-21 09:21:16,608 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROPERTIES" is " java.awt.headless=true sun.rmi.dgc.client.gcInterval=3600000 sun.rmi.dgc.server.gcInterval=3600000 jsse.enableSNIExtension=false "java.security.krb5.conf=/etc/ovirt-engine/krb5.conf"".
  62. 2015-09-21 09:21:16,619 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROXY_ENABLED" is "true".
  63. 2015-09-21 09:21:16,619 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443".
  64. 2015-09-21 09:21:16,619 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_PROXY_HTTP_PORT" is "80".
  65. 2015-09-21 09:21:16,619 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_REPORTS_BASE_URL" is "".
  66. 2015-09-21 09:21:16,620 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_REPORTS_DASHBOARD_URL" is "/flow.html?viewAsDashboardFrame=true".
  67. 2015-09-21 09:21:16,620 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_REPORTS_NOT_INSTALLED_URL" is "/ovirt-engine/ReportsNotInstalled.html".
  68. 2015-09-21 09:21:16,620 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_REPORTS_PROXY_URL" is "/ovirt/reports-interface".
  69. 2015-09-21 09:21:16,620 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_REPORTS_READ_TIMEOUT" is "".
  70. 2015-09-21 09:21:16,621 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_REPORTS_RIGHTCLICK_URL" is "".
  71. 2015-09-21 09:21:16,621 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_REPORTS_VERIFY_CHAIN" is "true".
  72. 2015-09-21 09:21:16,621 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_REPORTS_VERIFY_HOST" is "true".
  73. 2015-09-21 09:21:16,621 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_STOP_INTERVAL" is "1".
  74. 2015-09-21 09:21:16,622 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_STOP_TIME" is "10".
  75. 2015-09-21 09:21:16,622 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine".
  76. 2015-09-21 09:21:16,622 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up".
  77. 2015-09-21 09:21:16,623 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_URI" is "/ovirt-engine".
  78. 2015-09-21 09:21:16,623 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_USER" is "ovirt".
  79. 2015-09-21 09:21:16,623 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine".
  80. 2015-09-21 09:21:16,623 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine".
  81. 2015-09-21 09:21:16,629 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "ENGINE_VERBOSE_GC" is "false".
  82. 2015-09-21 09:21:16,629 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "JBOSS_HOME" is "/usr/share/ovirt-engine-jboss-as".
  83. 2015-09-21 09:21:16,629 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "JBOSS_RUNTIME" is "/var/lib/ovirt-engine/jboss_runtime".
  84. 2015-09-21 09:21:16,630 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (MSC service thread 1-3) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD,ENGINE_EXTERNAL_PROVIDERS_TRUST_STORE_PASSWORD".
  85. 2015-09-21 09:21:17,083 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Start initializing Backend
  86. 2015-09-21 09:21:17,409 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (MSC service thread 1-4) Loading file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties
  87. 2015-09-21 09:21:17,552 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Running ovirt-engine 3.5.3.1-1.el6
  88. 2015-09-21 09:21:17,552 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-4) Start initializing dictionaries
  89. 2015-09-21 09:21:17,558 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (MSC service thread 1-4) Finished initializing dictionaries
  90. 2015-09-21 09:21:17,558 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Start initializing AuditLogCleanupManager
  91. 2015-09-21 09:21:17,559 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Setting audit cleanup manager to run at: 35 35 3 * * ?
  92. 2015-09-21 09:21:17,582 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Finished initializing AuditLogCleanupManager
  93. 2015-09-21 09:21:17,583 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Start initializing CommandEntityCleanupManager
  94. 2015-09-21 09:21:17,584 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Setting command entity cleanup manager to run at: 35 35 3 * * ?
  95. 2015-09-21 09:21:17,586 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-4) Finished initializing CommandEntityCleanupManager
  96. 2015-09-21 09:21:17,588 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-4) Start initializing TagsDirector
  97. 2015-09-21 09:21:17,598 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-4) Tag root added to tree
  98. 2015-09-21 09:21:17,608 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-4) Finished initializing TagsDirector
  99. 2015-09-21 09:21:17,614 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-4) Start initializing IsoDomainListSyncronizer
  100. 2015-09-21 09:21:17,619 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (MSC service thread 1-4) Finished initializing IsoDomainListSyncronizer
  101. 2015-09-21 09:21:17,688 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Completed initializing handlers
  102. 2015-09-21 09:21:17,714 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Start initializing ErrorTranslatorImpl
  103. 2015-09-21 09:21:17,724 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table.
  104. 2015-09-21 09:21:17,725 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Finished initializing ErrorTranslatorImpl
  105. 2015-09-21 09:21:17,725 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Start initializing ErrorTranslatorImpl
  106. 2015-09-21 09:21:17,726 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-4) Finished initializing ErrorTranslatorImpl
  107. 2015-09-21 09:21:17,726 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Mark incomplete jobs as UNKNOWN
  108. 2015-09-21 09:21:17,771 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-4) Start initializing JobRepositoryCleanupManager
  109. 2015-09-21 09:21:17,771 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (MSC service thread 1-4) Finished initializing JobRepositoryCleanupManager
  110. 2015-09-21 09:21:17,772 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-4) Start initializing AutoRecoveryManager
  111. 2015-09-21 09:21:17,773 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (MSC service thread 1-4) Finished initializing AutoRecoveryManager
  112. 2015-09-21 09:21:17,774 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-4) Start initializing ExecutionMessageDirector
  113. 2015-09-21 09:21:17,781 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (MSC service thread 1-4) Finished initializing ExecutionMessageDirector
  114. 2015-09-21 09:21:17,830 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) Loading extension 'builtin-authn-internal'
  115. 2015-09-21 09:21:17,831 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) Extension 'builtin-authn-internal' loaded
  116. 2015-09-21 09:21:17,842 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) Initializing extension 'internal'
  117. 2015-09-21 09:21:17,843 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) Extension 'internal' initialized
  118. 2015-09-21 09:21:17,843 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) Start of enabled extensions list
  119. 2015-09-21 09:21:17,843 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) Instance name: 'builtin-authn-internal', Extension name: 'Internal Authn (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true'
  120. 2015-09-21 09:21:17,844 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) Instance name: 'internal', Extension name: 'Internal Authz (Built-in)', Version: 'N/A', Notes: '', License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt Project', Build interface Version: '0', File: 'N/A', Initialized: 'true'
  121. 2015-09-21 09:21:17,844 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (MSC service thread 1-4) End of enabled extensions list
  122. 2015-09-21 09:21:17,996 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (MSC service thread 1-2) Initialization of AsyncTaskManager completed successfully.
  123. 2015-09-21 09:21:17,997 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) Start initializing ResourceManager
  124. 2015-09-21 09:21:18,092 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Entered VdsManager constructor
  125. 2015-09-21 09:21:18,125 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Initialize vdsBroker (ovm-host02.,54,321)
  126. 2015-09-21 09:21:18,248 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) VDS 16bd9688-cb24-40be-9ab7-f8643711afb1 was added to the Resource Manager
  127. 2015-09-21 09:21:18,248 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Entered VdsManager constructor
  128. 2015-09-21 09:21:18,255 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (MSC service thread 1-2) Initialize vdsBroker (ovm-host01.,54,321)
  129. 2015-09-21 09:21:18,258 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) VDS fe41f4d5-bab9-45e6-a8fa-df02e3c93e42 was added to the Resource Manager
  130. 2015-09-21 09:21:18,275 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) Finished initializing ResourceManager
  131. 2015-09-21 09:21:18,275 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (MSC service thread 1-2) Initialization of OvfDataUpdater completed successfully.
  132. 2015-09-21 09:21:18,276 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Start scheduling to enable vds load balancer
  133. 2015-09-21 09:21:18,276 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Finished scheduling to enable vds load balancer
  134. 2015-09-21 09:21:18,276 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Start HA Reservation check
  135. 2015-09-21 09:21:18,277 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Finished HA Reservation check
  136. 2015-09-21 09:21:18,284 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Start initializing MacPoolManagerRanges
  137. 2015-09-21 09:21:18,285 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-2) Init VM custom properties utilities
  138. 2015-09-21 09:21:18,285 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-2) Init device custom properties utilities
  139. 2015-09-21 09:21:18,288 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Initializing Scheduling manager
  140. 2015-09-21 09:21:18,305 INFO [org.ovirt.engine.core.bll.network.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) Finished initializing. Available MACs in pool: 251
  141. 2015-09-21 09:21:18,325 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) External scheduler disabled, discovery skipped
  142. 2015-09-21 09:21:18,326 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (MSC service thread 1-2) Initialized Scheduling manager
  143. 2015-09-21 09:21:18,326 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-2) Initializing DWH Heart Beat
  144. 2015-09-21 09:21:18,334 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (MSC service thread 1-2) DWH Heart Beat initialized
  145. 2015-09-21 09:21:21,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = hosted_engine_1, HostId = fe41f4d5-bab9-45e6-a8fa-df02e3c93
  146. e42, vds=Host[hosted_engine_1,fe41f4d5-bab9-45e6-a8fa-df02e3c93e42]), log id: 1d6a638
  147. 2015-09-21 09:21:21,760 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: 1d6a638
  148. 2015-09-21 09:21:21,986 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) RefreshVmList vm id 37bb9732-544c-4c2a-b0e4-cdb0c54088a3 status = PoweringUp on vds hosted_engine_1 ignoring it in the
  149. refresh until migration is done
  150. 2015-09-21 09:21:22,085 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) Initializing Host: hosted_engine_1
  151. 2015-09-21 09:21:22,201 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [3e4d0dcd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: fe41f4d5-bab9-45e6-a8fa-df02
  152. e3c93e42 Type: VDS
  153. 2015-09-21 09:21:22,232 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-10) domain 9de6a788-ab43-4fd4-adfd-c31783467db5:ISO_DOMAIN in problem. vds: hosted_engine_1
  154. 2015-09-21 09:21:22,240 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-10) Host fe41f4d5-bab9-45e6-a8fa-df02e3c93e42 has reported new storage access problem to the following domains 9de6a78
  155. 8-ab43-4fd4-adfd-c31783467db5 marking it for storage connections and pool metadata refresh (report id: 033fb0cc-5427-46a2-a91b-7b7bd061f1af)
  156. 2015-09-21 09:21:23,420 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-5) Command GetStatsVDSCommand(HostName = hosted_engine_2, HostId = 16bd9688-cb24-40be-9ab7-f8643711afb1, vds=Host
  157. [hosted_engine_2,16bd9688-cb24-40be-9ab7-f8643711afb1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
  158. 2015-09-21 09:21:23,423 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) Host hosted_engine_2 is not responding. It will stay in Connecting state for a grace period of 62 seconds and after that an atte
  159. mpt to fence the host will be issued.
  160. 2015-09-21 09:21:23,428 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) AuditLogType: UNASSIGNED not exist in string table
  161. 2015-09-21 09:21:23,428 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) AuditLogType: VDS_HIGH_NETWORK_USE not exist in string table
  162. 2015-09-21 09:21:23,428 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table
  163. 2015-09-21 09:21:23,429 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table
  164. 2015-09-21 09:21:23,429 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table
  165. 2015-09-21 09:21:23,429 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table
  166. 2015-09-21 09:21:23,430 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) AuditLogType: USER_ADDED_DISK_PROFILE not exist in string table
  167. 2015-09-21 09:21:23,430 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) AuditLogType: USER_FAILED_TO_ADD_DISK_PROFILE not exist in string table
  168. 2015-09-21 09:21:23,430 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) AuditLogType: USER_REMOVED_DISK_PROFILE not exist in string table
  169. 2015-09-21 09:21:23,431 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) AuditLogType: USER_FAILED_TO_REMOVE_DISK_PROFILE not exist in string table
  170. 2015-09-21 09:21:23,431 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) AuditLogType: USER_UPDATED_DISK_PROFILE not exist in string table
  171. 2015-09-21 09:21:23,431 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) AuditLogType: USER_FAILED_TO_UPDATE_DISK_PROFILE not exist in string table
  172. 2015-09-21 09:21:23,431 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) AuditLogType: USER_ADDED_CPU_PROFILE not exist in string table
  173. 2015-09-21 09:21:23,432 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) AuditLogType: USER_FAILED_TO_ADD_CPU_PROFILE not exist in string table
  174. 2015-09-21 09:21:23,432 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) AuditLogType: USER_REMOVED_CPU_PROFILE not exist in string table
  175. 2015-09-21 09:21:23,432 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) AuditLogType: USER_FAILED_TO_REMOVE_CPU_PROFILE not exist in string table
  176. 2015-09-21 09:21:23,433 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) AuditLogType: USER_UPDATED_CPU_PROFILE not exist in string table
  177. 2015-09-21 09:21:23,433 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) AuditLogType: USER_FAILED_TO_UPDATE_CPU_PROFILE not exist in string table
  178. 2015-09-21 09:21:23,523 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [2a745455] Running command: LogoutUserCommand internal: false.
  179. 2015-09-21 09:21:23,521 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [153bb360] Running command: LogoutUserCommand internal: false.
  180. 2015-09-21 09:21:23,523 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [53bdd907] Running command: LogoutUserCommand internal: false.
  181. 2015-09-21 09:21:23,548 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-6) [42aaece4] Running command: LogoutUserCommand internal: false.
  182. 2015-09-21 09:21:23,556 ERROR [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [153bb360] Transaction rolled-back for command: org.ovirt.engine.core.bll.aaa.LogoutUserCommand.
  183. 2015-09-21 09:21:23,563 ERROR [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-6) [42aaece4] Transaction rolled-back for command: org.ovirt.engine.core.bll.aaa.LogoutUserCommand.
  184. 2015-09-21 09:21:23,566 ERROR [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-3) [53bdd907] Transaction rolled-back for command: org.ovirt.engine.core.bll.aaa.LogoutUserCommand.
  185. 2015-09-21 09:21:23,577 ERROR [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp--127.0.0.1-8702-5) [2a745455] Transaction rolled-back for command: org.ovirt.engine.core.bll.aaa.LogoutUserCommand.
  186. 2015-09-21 09:21:23,606 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) [53bdd907] Correlation ID: 53bdd907, Call Stack: null, Custom Event ID: -1, Message: Failed to log User null@N/A
  187. out.
  188. 2015-09-21 09:21:23,609 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) [2a745455] Correlation ID: 2a745455, Call Stack: null, Custom Event ID: -1, Message: Failed to log User null@N/A
  189. out.
  190. 2015-09-21 09:21:23,606 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [153bb360] Correlation ID: 153bb360, Call Stack: null, Custom Event ID: -1, Message: Failed to log User null@N/A
  191. out.
  192. 2015-09-21 09:21:23,606 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) [42aaece4] Correlation ID: 42aaece4, Call Stack: null, Custom Event ID: -1, Message: Failed to log User null@N/A
  193. out.
  194. 2015-09-21 09:21:23,715 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host hosted_engine_2 is not resp
  195. onding. It will stay in Connecting state for a grace period of 62 seconds and after that an attempt to fence the host will be issued.
  196. 2015-09-21 09:21:23,716 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) vds::refreshVdsStats Failed getVdsStats, vds = 16bd9688-cb24-40be-9ab7-f8643711afb1 : hosted_engine_2, error = VDSNet
  197. workException: java.net.SocketTimeoutException: connect timed out
  198. 2015-09-21 09:21:23,717 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTim
  199. eoutException: connect timed out
  200. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
  201. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
  202. at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
  203. at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
  204. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsStats(VdsUpdateRunTimeInfo.java:690) [vdsbroker.jar:]
  205. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:644) [vdsbroker.jar:]
  206. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:495) [vdsbroker.jar:]
  207. at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
  208. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
  209. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
  210. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  211. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  212. at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
  213. at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
  214. at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
  215. Caused by: java.net.SocketTimeoutException: connect timed out
  216. at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_79]
  217. at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_79]
  218. at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_79]
  219. at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_79]
  220. at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_79]
  221. at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_79]
  222. at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625) [jsse.jar:1.7.0_79]
  223. at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:293) [utils.jar:]
  224. at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
  225. at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
  226. at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
  227. at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
  228. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
  229. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
  230. at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
  231. at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
  232. at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
  233. at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
  234. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
  235. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
  236. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
  237. at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
  238. at com.sun.proxy.$Proxy81.getVdsStats(Unknown Source)
  239. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
  240. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
  241. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  242. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  243. at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:263) [vdsbroker.jar:]
  244. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  245. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_79]
  246. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  247. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]
  248. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]
  249. at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
  250.  
  251. 2015-09-21 09:21:23,736 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) Failed to refresh VDS , vds = 16bd9688-cb24-40be-9ab7-f8643711afb1 : hosted_engine_2, VDS Network Error, continuing.
  252. java.net.SocketTimeoutException: connect timed out
  253. 2015-09-21 09:21:24,887 INFO [org.ovirt.engine.docs.utils.servlet.ContextSensitiveHelpMappingServlet] (ajp--127.0.0.1-8702-9) Context-sensitive help is not installed. Manual directory doesn't exist: /usr/share/ovirt-engine/manual
  254. 2015-09-21 09:21:25,335 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) RefreshVmList vm id 37bb9732-544c-4c2a-b0e4-cdb0c54088a3 status = PoweringUp on vds hosted_engine_1 ignoring it in th
  255. e refresh until migration is done
  256. 2015-09-21 09:21:28,362 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-20) [2350bd77] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-00
  257. 02-0002-0002-0000000000f1 Type: StoragePool
  258. 2015-09-21 09:21:28,376 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [2350bd77] Storage Pool 00000002-0002-0002-0002-0000000000f1 - Updating Storage Domain 11e3c468-fd7f-4b92-
  259. 916e-de17a0b46af4 status from Active to Unknown, reason : null
  260. 2015-09-21 09:21:28,379 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler_Worker-20) [2350bd77] Storage Pool 00000002-0002-0002-0002-0000000000f1 - Updating Storage Domain 9de6a788-ab43-4fd4-
  261. adfd-c31783467db5 status from Active to Unknown, reason : null
  262. 2015-09-21 09:21:28,464 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) RefreshVmList vm id 37bb9732-544c-4c2a-b0e4-cdb0c54088a3 status = PoweringUp on vds hosted_engine_1 ignoring it in th
  263. e refresh until migration is done
  264. 2015-09-21 09:21:28,473 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [2350bd77] Correlation ID: 2350bd77, Call Stack: null, Custom Event ID: -1, Message: Invalid status o
  265. n Data Center Default. Setting status to Non Responsive.
  266. 2015-09-21 09:21:28,627 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [2350bd77] hostFromVds::selectedVds - hosted_engine_1, spmStatus SPM, storage pool Default
  267. 2015-09-21 09:21:28,634 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-20) [2350bd77] Initialize Irs proxy from vds: ovm-host01
  268. 2015-09-21 09:21:28,663 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [2350bd77] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager
  269. runs on Host hosted_engine_1 (Address: ovm-host01).
  270. 2015-09-21 09:21:28,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [2350bd77] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000000
  271. f1, ignoreFailoverLimit = false), log id: 3610ca05
  272. 2015-09-21 09:21:28,733 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [2350bd77] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-00000000
  273. 00f1
  274. 2015-09-21 09:21:28,735 INFO [org.ovirt.engine.core.bll.storage.SyncLunsInfoForBlockStorageDomainCommand] (org.ovirt.thread.pool-8-thread-22) [485c9dab] Running command: SyncLunsInfoForBlockStorageDomainCommand internal: true. Entities
  275. affected : ID: 11e3c468-fd7f-4b92-916e-de17a0b46af4 Type: Storage
  276. 2015-09-21 09:21:28,782 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) Command GetCapabilitiesVDSCommand(HostName = hosted_engine_2, HostId = 16bd9688-cb24-40be-9ab7-f864371
  277. 1afb1, vds=Host[hosted_engine_2,16bd9688-cb24-40be-9ab7-f8643711afb1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
  278. 2015-09-21 09:21:28,784 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) Host hosted_engine_2 is not responding. It will stay in Connecting state for a grace period of 62 seconds and after that an att
  279. empt to fence the host will be issued.
  280. 2015-09-21 09:21:28,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [2350bd77] START, HSMGetAllTasksInfoVDSCommand(HostName = hosted_engine_1, HostId = fe41f4d5-bab9-
  281. 45e6-a8fa-df02e3c93e42), log id: e257d9a
  282. 2015-09-21 09:21:28,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [2350bd77] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: e257d9a
  283. 2015-09-21 09:21:28,825 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-20) [2350bd77] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3610ca05
  284. 2015-09-21 09:21:28,826 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-20) [2350bd77] Discovered no tasks on Storage Pool Default
  285. 2015-09-21 09:21:28,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-8-thread-22) [485c9dab] START, GetVGInfoVDSCommand(HostName = hosted_engine_1, HostId = fe41f4d5-bab9-45e6-a8fa-df02e3c9
  286. 3e42, VGID=bJxAxg-Cm1W-McCl-gM6e-XRgC-GvnT-46CmXZ), log id: 41dc22e3
  287. 2015-09-21 09:21:28,853 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTi
  288. meoutException: connect timed out
  289. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
  290. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
  291. at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
  292. at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
  293. at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:]
  294. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:649) [vdsbroker.jar:]
  295. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:495) [vdsbroker.jar:]
  296. at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
  297. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
  298. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
  299. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  300. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  301. at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
  302. at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
  303. at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
  304. Caused by: java.net.SocketTimeoutException: connect timed out
  305. at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_79]
  306. at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_79]
  307. at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_79]
  308. at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_79]
  309. at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_79]
  310. at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_79]
  311. at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625) [jsse.jar:1.7.0_79]
  312. at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:293) [utils.jar:]
  313. at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
  314. at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
  315. at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
  316. at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
  317. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
  318. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
  319. at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
  320. at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
  321. at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
  322. at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
  323. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
  324. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
  325. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
  326. at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
  327. at com.sun.proxy.$Proxy81.getVdsCapabilities(Unknown Source)
  328. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
  329. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
  330. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  331. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  332. at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:263) [vdsbroker.jar:]
  333. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  334. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_79]
  335. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  336. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]
  337. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]
  338. at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
  339.  
  340. 2015-09-21 09:21:28,875 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) Failed to refresh VDS , vds = 16bd9688-cb24-40be-9ab7-f8643711afb1 : hosted_engine_2, VDS Network Error, continuing.
  341. java.net.SocketTimeoutException: connect timed out
  342. 2015-09-21 09:21:29,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-8-thread-22) [485c9dab] FINISH, GetVGInfoVDSCommand, return: [LUNs [id=360014056f79d97cd7701d32f7d9d29d9, physicalVolume
  343. Id=Pajgrj-5RHj-54rL-eCnj-5960-3pGP-tGoySD, volumeGroupId=bJxAxg-Cm1W-McCl-gM6e-XRgC-GvnT-46CmXZ, serial=, lunMapping=1, vendorId=SYNOLOGY, productId=iSCSI Storage, _lunConnections=[{ id: null, connection: 192.168.253.112, iqn: ovirtstore
  344. 01.synology:rackstation.target-1.6d77a3aa6d, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };, { id: null, connection: rackstation, iqn: ovirtstore01.synology:rackstation.target-1.6d77a3
  345. aa6d, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };], deviceSize=1024, vendorName=SYNOLOGY, pathsDictionary={sdf=true, sde=true}, lunType=ISCSI, status=null, diskId=null, diskAlias=null, storage
  346. DomainId=11e3c468-fd7f-4b92-916e-de17a0b46af4, storageDomainName=null]], log id: 41dc22e3
  347. 2015-09-21 09:21:31,534 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-27) RefreshVmList vm id 37bb9732-544c-4c2a-b0e4-cdb0c54088a3 status = PoweringUp on vds hosted_engine_1 ignoring it in th
  348. e refresh until migration is done
  349. 2015-09-21 09:21:33,916 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = hosted_engine_2, HostId = 16bd9688-cb24-40be-9ab7-f864371
  350. 1afb1, vds=Host[hosted_engine_2,16bd9688-cb24-40be-9ab7-f8643711afb1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
  351. 2015-09-21 09:21:33,916 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host hosted_engine_2 is not responding. It will stay in Connecting state for a grace period of 62 seconds and after that an att
  352. empt to fence the host will be issued.
  353. 2015-09-21 09:21:33,965 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) START, SetVmStatusVDSCommand( vmId = 36a7cbc1-ec77-4ce7-a2e8-d267406129c4, status = Unknown, exit status = Normal),
  354. log id: 124f9496
  355. 2015-09-21 09:21:33,977 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) FINISH, SetVmStatusVDSCommand, log id: 124f9496
  356. 2015-09-21 09:21:34,047 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM DebianTest was set to the Un
  357. known status.
  358. 2015-09-21 09:21:34,048 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) START, SetVmStatusVDSCommand( vmId = b399c2d6-549e-4b63-80fb-0507fac90da8, status = Unknown, exit status = Normal),
  359. log id: 3a1b526f
  360. 2015-09-21 09:21:34,057 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) FINISH, SetVmStatusVDSCommand, log id: 3a1b526f
  361. 2015-09-21 09:21:34,091 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM Windows7Pool-1 was set to th
  362. e Unknown status.
  363. 2015-09-21 09:21:34,092 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) START, SetVmStatusVDSCommand( vmId = 14f6cff1-56e4-4df7-a265-a72205c7a3f1, status = Unknown, exit status = Normal),
  364. log id: 600a5a73
  365. 2015-09-21 09:21:34,094 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) FINISH, SetVmStatusVDSCommand, log id: 600a5a73
  366. 2015-09-21 09:21:34,131 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM poolTest-1 was set to the Un
  367. known status.
  368. 2015-09-21 09:21:34,131 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) START, SetVmStatusVDSCommand( vmId = 37bb9732-544c-4c2a-b0e4-cdb0c54088a3, status = Unknown, exit status = Normal),
  369. log id: 48743f96
  370. 2015-09-21 09:21:34,139 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) FINISH, SetVmStatusVDSCommand, log id: 48743f96
  371. 2015-09-21 09:21:34,168 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM HostedEngine was set to the
  372. Unknown status.
  373. 2015-09-21 09:21:34,168 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Server failed to respond, vds_id = 16bd9688-cb24-40be-9ab7-f8643711afb1, vds_name = hosted_engine_2, vm_count = 4, spm_status =
  374. None, non-responsive_timeout (seconds) = 62, error = java.net.SocketTimeoutException: connect timed out
  375. 2015-09-21 09:21:34,195 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java
  376. .net.SocketTimeoutException: connect timed out
  377. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126)
  378. at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
  379. at java.net.Socket.connect(Socket.java:579)
  380. at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625)
  381. at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:293)
  382. at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
  383. at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
  384. at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
  385. at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
  386. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
  387. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
  388. at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227)
  389. at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151)
  390. at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143)
  391. at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56)
  392. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167)
  393. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137)
  394. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126)
  395. at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140)
  396. at com.sun.proxy.$Proxy81.getVdsCapabilities(Unknown Source)
  397. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  398. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  399. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  400. at java.lang.reflect.Method.invoke(Method.java:606)
  401. at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:263)
  402. at java.util.concurrent.FutureTask.run(FutureTask.java:262)
  403. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  404. at java.util.concurrent.FutureTask.run(FutureTask.java:262)
  405. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  406. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  407. at java.lang.Thread.run(Thread.java:745)
  408. , Custom Event ID: -1, Message: Host hosted_engine_2 is non responsive.
  409. 2015-09-21 09:21:34,209 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-8-thread-30) ResourceManager::vdsNotResponding entered for Host 16bd9688-cb24-40be-9ab7-f8643711afb1, ovm-host02
  410. 2015-09-21 09:21:34,216 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTi
  411. meoutException: connect timed out
  412. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
  413. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
  414. at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
  415. at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
  416. at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:]
  417. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:649) [vdsbroker.jar:]
  418. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:495) [vdsbroker.jar:]
  419. at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
  420. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
  421. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
  422. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  423. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  424. at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
  425. at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
  426. at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
  427. Caused by: java.net.SocketTimeoutException: connect timed out
  428. at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_79]
  429. at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_79]
  430. at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_79]
  431. at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_79]
  432. at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_79]
  433. at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_79]
  434. at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625) [jsse.jar:1.7.0_79]
  435. at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:293) [utils.jar:]
  436. at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
  437. at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
  438. at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
  439. at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
  440. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
  441. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
  442. at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
  443. at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
  444.  
  445. at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
  446. at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
  447. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
  448. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
  449. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
  450. at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
  451. at com.sun.proxy.$Proxy81.getVdsCapabilities(Unknown Source)
  452. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
  453. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
  454. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  455. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  456. at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:263) [vdsbroker.jar:]
  457. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  458. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_79]
  459. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  460. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]
  461. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]
  462. at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
  463.  
  464. 2015-09-21 09:21:34,285 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-30) [599e4b17] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 16bd9688-cb24-40be-9ab7-f86437
  465. 11afb1 Type: VDS
  466. 2015-09-21 09:21:34,436 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-30) [599e4b17] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host hosted_engine_
  467. 1 from cluster Default was chosen as a proxy to execute Status command on Host hosted_engine_2.
  468. 2015-09-21 09:21:34,436 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-30) [599e4b17] Using Host hosted_engine_1 from cluster Default as proxy to execute Status command on Host hosted_engine_2
  469. 2015-09-21 09:21:34,466 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-30) [599e4b17] Executing <Status> Power Management command, Proxy Host:hosted_engine_1, Agent:ipmilan, Target Host:hosted_engine_2, M
  470. anagement IP:******, User:administrator, Options:retry_on=2,power_wait=5,delay=5,lanplus, Fencing policy:null
  471. 2015-09-21 09:21:34,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-30) [599e4b17] START, FenceVdsVDSCommand(HostName = hosted_engine_1, HostId = fe41f4d5-bab9-45e6-a8fa-df02e3c93e
  472. 42, targetVdsId = 16bd9688-cb24-40be-9ab7-f8643711afb1, action = Status, ip = ******, port = , type = ipmilan, user = administrator, password = ******, options = 'retry_on=2,power_wait=5,delay=5,lanplus', policy = 'null'), log id
  473. : 1051f02e
  474. 2015-09-21 09:21:34,603 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) RefreshVmList vm id 37bb9732-544c-4c2a-b0e4-cdb0c54088a3 status = PoweringUp on vds hosted_engine_1 ignoring it in th
  475. e refresh until migration is done
  476. 2015-09-21 09:21:34,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-30) [599e4b17] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 1051f02e
  477. 2015-09-21 09:21:34,905 INFO [org.ovirt.engine.core.bll.FenceExecutor] (org.ovirt.thread.pool-8-thread-30) [599e4b17] Executing <Status> Power Management command, Proxy Host:hosted_engine_1, Agent:ipmilan, Target Host:hosted_engine_2, M
  478. anagement IP:******, User:administrator, Options:lanplus,power_wait=4, Fencing policy:null
  479. 2015-09-21 09:21:34,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-30) [599e4b17] START, FenceVdsVDSCommand(HostName = hosted_engine_1, HostId = fe41f4d5-bab9-45e6-a8fa-df02e3c93e
  480. 42, targetVdsId = 16bd9688-cb24-40be-9ab7-f8643711afb1, action = Status, ip = ******, port = , type = ipmilan, user = administrator, password = ******, options = 'lanplus,power_wait=4', policy = 'null'), log id: 5f1561e2
  481. 2015-09-21 09:21:35,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-8-thread-30) [599e4b17] FINISH, FenceVdsVDSCommand, return: Test Succeeded, on, log id: 5f1561e2
  482. 2015-09-21 09:21:35,304 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-30) [599e4b17] Opening SSH Soft Fencing session on host ovm-host02
  483. 2015-09-21 09:21:37,711 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) RefreshVmList vm id 37bb9732-544c-4c2a-b0e4-cdb0c54088a3 status = PoweringUp on vds hosted_engine_1 ignoring it in th
  484. e refresh until migration is done
  485. 2015-09-21 09:21:37,777 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-44) Host fe41f4d5-bab9-45e6-a8fa-df02e3c93e42 no longer storage access problem to any relevant domain clearing its re
  486. port (report id: 033fb0cc-5427-46a2-a91b-7b7bd061f1af)
  487. 2015-09-21 09:21:37,778 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-44) Domain 9de6a788-ab43-4fd4-adfd-c31783467db5:ISO_DOMAIN recovered from problem. vds: hosted_engine_1
  488. 2015-09-21 09:21:37,779 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-44) Domain 9de6a788-ab43-4fd4-adfd-c31783467db5:ISO_DOMAIN has recovered from problem. No active host in the DC is rep
  489. orting it as problematic, so clearing the domain recovery timer.
  490. 2015-09-21 09:21:38,493 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) Command GetCapabilitiesVDSCommand(HostName = hosted_engine_2, HostId = 16bd9688-cb24-40be-9ab7-f864371
  491. 1afb1, vds=Host[hosted_engine_2,16bd9688-cb24-40be-9ab7-f8643711afb1]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
  492. 2015-09-21 09:21:38,494 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) Host hosted_engine_2 is not responding. It will stay in Connecting state for a grace period of 124 seconds and after that an at
  493. tempt to fence the host will be issued.
  494. 2015-09-21 09:21:38,494 ERROR [org.ovirt.engine.core.bll.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-30) [599e4b17] SSH Soft Fencing command failed on host ovm-host02: Failed to get the session.
  495. Stdout:
  496. Stderr:
  497. Stacktrace: org.apache.sshd.common.RuntimeSshException: Failed to get the session.: org.apache.sshd.common.RuntimeSshException: Failed to get the session.
  498. at org.apache.sshd.client.future.DefaultConnectFuture.getSession(DefaultConnectFuture.java:43)
  499. at org.ovirt.engine.core.uutils.ssh.SSHClient.connect(SSHClient.java:294) [uutils.jar:]
  500. at org.ovirt.engine.core.bll.utils.EngineSSHClient.connect(EngineSSHClient.java:59) [bll.jar:]
  501. at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeSshSoftFencingCommand(SshSoftFencingCommand.java:87) [bll.jar:]
  502. at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeCommand(SshSoftFencingCommand.java:53) [bll.jar:]
  503. at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1193) [bll.jar:]
  504. at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1332) [bll.jar:]
  505. at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1961) [bll.jar:]
  506. at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:]
  507. at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:]
  508. at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1356) [bll.jar:]
  509. at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:353) [bll.jar:]
  510. at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:435) [bll.jar:]
  511. at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:416) [bll.jar:]
  512. at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:622) [bll.jar:]
  513. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
  514. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
  515. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  516. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  517. at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
  518. at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
  519. at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
  520. at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:114) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]
  521. at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:125) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]
  522. at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:135) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]
  523. at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
  524. at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
  525. at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
  526. at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
  527. at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
  528. at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:82) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]
  529. at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
  530. at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
  531. at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
  532. at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
  533. at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
  534. at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
  535. at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
  536. at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
  537. at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:211) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
  538. at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
  539. at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:194) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
  540. at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
  541. at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
  542. at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
  543. at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
  544. at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
  545. at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
  546. at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
  547. at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
  548. at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
  549. at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
  550. at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
  551. at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
  552. at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
  553. at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
  554. at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
  555. at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [bll.jar:]
  556. at org.ovirt.engine.core.bll.VdsEventListener$4.run(VdsEventListener.java:252) [bll.jar:]
  557. at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:]
  558. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_79]
  559. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  560. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]
  561. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]
  562. at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
  563. Caused by: java.net.NoRouteToHostException: No route to host
  564. at sun.nio.ch.UnixAsynchronousSocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_79]
  565. at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishConnect(UnixAsynchronousSocketChannelImpl.java:252) [rt.jar:1.7.0_79]
  566. at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:198) [rt.jar:1.7.0_79]
  567. at sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:213) [rt.jar:1.7.0_79]
  568. at sun.nio.ch.EPollPort$EventHandlerTask.run(EPollPort.java:293) [rt.jar:1.7.0_79]
  569. ... 1 more
  570.  
  571. 2015-09-21 09:21:38,553 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteT
  572. oHostException: No route to host
  573. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
  574. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
  575.  
  576. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
  577. at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
  578. at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
  579. at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:]
  580. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:649) [vdsbroker.jar:]
  581. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:495) [vdsbroker.jar:]
  582. at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
  583. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
  584. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
  585. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  586. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  587. at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
  588. at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
  589. at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
  590. Caused by: java.net.NoRouteToHostException: No route to host
  591. at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_79]
  592. at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_79]
  593. at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_79]
  594. at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_79]
  595. at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_79]
  596. at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_79]
  597. at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625) [jsse.jar:1.7.0_79]
  598. at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:293) [utils.jar:]
  599. at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
  600. at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
  601. at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
  602. at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
  603. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
  604. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
  605. at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
  606. at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
  607. at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
  608. at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
  609. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
  610. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
  611. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
  612. at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
  613. at com.sun.proxy.$Proxy81.getVdsCapabilities(Unknown Source)
  614. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
  615. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
  616. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  617. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  618. at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:263) [vdsbroker.jar:]
  619. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  620. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_79]
  621. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  622. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]
  623. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]
  624. at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
  625.  
  626. 2015-09-21 09:21:38,593 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-30) [5e166dae] Lock Acquired to object EngineLock [exclusiveLocks= key: 16bd9688-cb24-40be-9ab7-f8643711afb1 value: VDS_FENCE
  627. , sharedLocks= ]
  628. 2015-09-21 09:21:38,595 ERROR [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-30) Failed to run Fence script on vds:hosted_engine_2.
  629. 2015-09-21 09:21:38,693 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-30) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host hosted_engine_2 became non responsive. It has no power management configured. Please check the host status, manually reboot it, and click "Confirm Host Has Been Rebooted"
  630. 2015-09-21 09:21:38,694 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-30) CanDoAction of action VdsNotRespondingTreatment failed for user null@N/A. Reasons: VAR__ACTION__RESTART,VDS_FENCE_DISABLED_AT_SYSTEM_STARTUP_INTERVAL
  631. 2015-09-21 09:21:38,694 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-30) Lock freed to object EngineLock [exclusiveLocks= key: 16bd9688-cb24-40be-9ab7-f8643711afb1 value: VDS_FENCE
  632. , sharedLocks= ]
  633. 2015-09-21 09:21:40,826 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) RefreshVmList vm id 37bb9732-544c-4c2a-b0e4-cdb0c54088a3 status = PoweringUp on vds hosted_engine_1 ignoring it in the refresh until migration is done
  634. 2015-09-21 09:21:43,609 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = hosted_engine_2, HostId = 16bd9688-cb24-40be-9ab7-f8643711afb1, vds=Host[hosted_engine_2,16bd9688-cb24-40be-9ab7-f8643711afb1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
  635.  
  636. 2015-09-21 09:21:43,610 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host hosted_engine_2 is not responding. It will stay in Connecting state for a grace period of 124 seconds and after that an at
  637. tempt to fence the host will be issued.
  638. 2015-09-21 09:21:43,641 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTi
  639. meoutException: connect timed out
  640. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
  641. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
  642. at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
  643. at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
  644. at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:]
  645. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:649) [vdsbroker.jar:]
  646. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:495) [vdsbroker.jar:]
  647. at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
  648. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
  649. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
  650. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  651. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  652. at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
  653. at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
  654. at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
  655. Caused by: java.net.SocketTimeoutException: connect timed out
  656. at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_79]
  657. at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_79]
  658. at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_79]
  659. at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_79]
  660. at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_79]
  661. at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_79]
  662. at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625) [jsse.jar:1.7.0_79]
  663. at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:293) [utils.jar:]
  664. at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
  665. at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
  666. at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
  667. at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
  668. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
  669. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
  670. at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
  671. at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
  672. at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
  673. at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
  674. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
  675. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
  676. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
  677. at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
  678. at com.sun.proxy.$Proxy81.getVdsCapabilities(Unknown Source)
  679. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
  680. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
  681. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  682. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  683. at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:263) [vdsbroker.jar:]
  684. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  685. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_79]
  686. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  687. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]
  688. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]
  689. at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
  690.  
  691. 2015-09-21 09:21:43,871 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) RefreshVmList vm id 37bb9732-544c-4c2a-b0e4-cdb0c54088a3 status = PoweringUp on vds hosted_engine_1 ignoring it in the refresh until migration is done
  692. 2015-09-21 09:21:46,912 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) RefreshVmList vm id 37bb9732-544c-4c2a-b0e4-cdb0c54088a3 status = PoweringUp on vds hosted_engine_1 ignoring it in the refresh until migration is done
  693. 2015-09-21 09:21:48,686 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = hosted_engine_2, HostId = 16bd9688-cb24-40be-9ab7-f8643711afb1, vds=Host[hosted_engine_2,16bd9688-cb24-40be-9ab7-f8643711afb1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
  694. 2015-09-21 09:21:48,687 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host hosted_engine_2 is not responding. It will stay in Connecting state for a grace period of 124 seconds and after that an attempt to fence the host will be issued.
  695. 2015-09-21 09:21:48,719 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out
  696. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
  697. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
  698. at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
  699. at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
  700. at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:]
  701. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:649) [vdsbroker.jar:]
  702. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:495) [vdsbroker.jar:]
  703. at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
  704. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
  705. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
  706. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  707. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  708. at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
  709. at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
  710. at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
  711. Caused by: java.net.SocketTimeoutException: connect timed out
  712. at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_79]
  713. at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_79]
  714. at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_79]
  715. at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_79]
  716. at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_79]
  717. at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_79]
  718. at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625) [jsse.jar:1.7.0_79]
  719. at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:293) [utils.jar:]
  720. at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
  721. at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
  722. at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
  723. at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
  724. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
  725. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
  726. at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
  727. at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
  728. at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
  729. at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
  730. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
  731. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
  732. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
  733. at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
  734. at com.sun.proxy.$Proxy81.getVdsCapabilities(Unknown Source)
  735. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
  736. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
  737. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  738. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  739. at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:263) [vdsbroker.jar:]
  740. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  741. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_79]
  742. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  743. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]
  744. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]
  745. at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
  746.  
  747. 2015-09-21 09:21:49,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-73) START, FullListVdsCommand(HostName = hosted_engine_1, HostId = fe41f4d5-bab9-45e6-a8fa-df02e3c93e42, vds=Host[hosted_engine_1,fe41f4d5-bab9-45e6-a8fa-df02e3c93e42], vmIds=[37bb9732-544c-4c2a-b0e4-cdb0c54088a3]), log id: 51e2909c
  748. 2015-09-21 09:21:49,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-73) FINISH, FullListVdsCommand, return: [{displaySecurePort=-1, memSize=8192, nicModel=rtl8139,pv, displayPort=5905, smp=2, status=Up, emulatedMachine=pc, vmType=kvm, guestDiskMapping={}, displayIp=0, pauseCode=NOERR, display=vnc, pid=28279, spiceSecureChannels=smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir, vmId=37bb9732-544c-4c2a-b0e4-cdb0c54088a3, clientIp=, devices=[Ljava.lang.Object;@6dbbc416, vmName=HostedEngine, cpuType=SandyBridge}], log id: 51e2909c
  749. 2015-09-21 09:21:50,004 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Received a console Device without an address when processing VM 37bb9732-544c-4c2a-b0e4-cdb0c54088a3 devices, skipping device: {alias=console0, specParams={}, device=console, type=console, deviceId=6125ee2e-8a9a-43cb-a71f-dcfb4ebb011c}
  750. 2015-09-21 09:21:50,005 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Received a memballoon Device without an address when processing VM 37bb9732-544c-4c2a-b0e4-cdb0c54088a3 devices, skipping device: {alias=balloon0, specParams={model=none}, device=memballoon, type=balloon}
  751. 2015-09-21 09:21:50,006 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Received a vnc Device without an address when processing VM 37bb9732-544c-4c2a-b0e4-cdb0c54088a3 devices, skipping device: {port=5905, specParams={spiceSecureChannels=smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir, displayIp=0}, device=vnc, type=graphics}
  752. 2015-09-21 09:21:50,009 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Empty or NULL values were passed for a VM 37bb9732-544c-4c2a-b0e4-cdb0c54088a3 device, Device is skipped
  753. 2015-09-21 09:21:53,760 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = hosted_engine_2, HostId = 16bd9688-cb24-40be-9ab7-f8643711afb1, vds=Host[hosted_engine_2,16bd9688-cb24-40be-9ab7-f8643711afb1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
  754. 2015-09-21 09:21:53,761 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host hosted_engine_2 is not responding. It will stay in Connecting state for a grace period of 124 seconds and after that an attempt to fence the host will be issued.
  755. 2015-09-21 09:21:53,789 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out
  756. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
  757. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
  758. at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
  759. at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
  760. at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:]
  761. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:649) [vdsbroker.jar:]
  762. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:495) [vdsbroker.jar:]
  763. at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
  764. at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) [:1.7.0_79]
  765. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  766. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  767. at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
  768. at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
  769. at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
  770. Caused by: java.net.SocketTimeoutException: connect timed out
  771. at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_79]
  772. at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_79]
  773. at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_79]
  774. at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_79]
  775. at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_79]
  776. at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_79]
  777. at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625) [jsse.jar:1.7.0_79]
  778. at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:293) [utils.jar:]
  779. at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
  780. at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
  781. at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
  782. at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
  783. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
  784. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
  785. at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
  786. at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
  787. at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
  788. at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
  789. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
  790. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
  791. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
  792. at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
  793. at com.sun.proxy.$Proxy81.getVdsCapabilities(Unknown Source)
  794. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
  795. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
  796. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  797. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  798. at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:263) [vdsbroker.jar:]
  799. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  800. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_79]
  801. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  802. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]
  803. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]
  804. at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
  805.  
  806. 2015-09-21 09:21:58,840 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) Command GetCapabilitiesVDSCommand(HostName = hosted_engine_2, HostId = 16bd9688-cb24-40be-9ab7-f8643711afb1, vds=Host[hosted_engine_2,16bd9688-cb24-40be-9ab7-f8643711afb1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
  807. 2015-09-21 09:21:58,841 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) Host hosted_engine_2 is not responding. It will stay in Connecting state for a grace period of 124 seconds and after that an attempt to fence the host will be issued.
  808. 2015-09-21 09:21:58,870 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out
  809. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
  810. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
  811. at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
  812. at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
  813. at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:]
  814. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:649) [vdsbroker.jar:]
  815. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:495) [vdsbroker.jar:]
  816. at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
  817. at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) [:1.7.0_79]
  818. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  819. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  820. at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
  821. at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
  822. at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
  823. Caused by: java.net.SocketTimeoutException: connect timed out
  824. at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_79]
  825. at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_79]
  826. at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_79]
  827. at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_79]
  828. at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_79]
  829. at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_79]
  830. at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625) [jsse.jar:1.7.0_79]
  831. at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:293) [utils.jar:]
  832. at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
  833. at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
  834. at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
  835. at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
  836. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
  837. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
  838. at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
  839. at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
  840. at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
  841. at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
  842. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
  843. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
  844. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
  845. at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
  846. at com.sun.proxy.$Proxy81.getVdsCapabilities(Unknown Source)
  847. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
  848. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
  849. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  850. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  851. at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:263) [vdsbroker.jar:]
  852. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  853. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_79]
  854. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  855. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]
  856. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]
  857. at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
  858.  
  859. 2015-09-21 09:22:03,909 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = hosted_engine_2, HostId = 16bd9688-cb24-40be-9ab7-f8643711afb1, vds=Host[hosted_engine_2,16bd9688-cb24-40be-9ab7-f8643711afb1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
  860. 2015-09-21 09:22:03,910 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host hosted_engine_2 is not responding. It will stay in Connecting state for a grace period of 124 seconds and after that an attempt to fence the host will be issued.
  861. 2015-09-21 09:22:03,946 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out
  862. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
  863. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
  864. at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
  865. at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
  866. at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:]
  867. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:649) [vdsbroker.jar:]
  868. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:495) [vdsbroker.jar:]
  869. at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
  870. at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) [:1.7.0_79]
  871. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  872. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  873. at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
  874. at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
  875. at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
  876. Caused by: java.net.SocketTimeoutException: connect timed out
  877. at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_79]
  878. at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_79]
  879. at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_79]
  880. at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_79]
  881. at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_79]
  882. at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_79]
  883. at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625) [jsse.jar:1.7.0_79]
  884. at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:293) [utils.jar:]
  885. at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
  886. at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
  887. at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
  888. at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
  889. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
  890. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
  891. at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
  892. at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
  893. at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
  894. at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
  895. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
  896. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
  897. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
  898. at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
  899. at com.sun.proxy.$Proxy81.getVdsCapabilities(Unknown Source)
  900. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
  901. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
  902. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  903. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  904. at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:263) [vdsbroker.jar:]
  905. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  906. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_79]
  907. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  908. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]
  909. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]
  910. at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
  911.  
  912. 2015-09-21 09:22:08,991 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = hosted_engine_2, HostId = 16bd9688-cb24-40be-9ab7-f8643711afb1, vds=Host[hosted_engine_2,16bd9688-cb24-40be-9ab7-f8643711afb1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
  913. 2015-09-21 09:22:08,992 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-10) Host hosted_engine_2 is not responding. It will stay in Connecting state for a grace period of 124 seconds and after that an attempt to fence the host will be issued.
  914. 2015-09-21 09:22:09,022 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out
  915. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
  916. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
  917. at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
  918. at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
  919. at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:]
  920. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:649) [vdsbroker.jar:]
  921. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:495) [vdsbroker.jar:]
  922. at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
  923. at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) [:1.7.0_79]
  924. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  925. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  926. at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
  927. at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
  928. at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
  929. Caused by: java.net.SocketTimeoutException: connect timed out
  930. at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_79]
  931. at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_79]
  932. at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_79]
  933. at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_79]
  934. at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_79]
  935. at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_79]
  936. at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625) [jsse.jar:1.7.0_79]
  937. at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:293) [utils.jar:]
  938. at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
  939. at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
  940. at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
  941. at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
  942. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
  943. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
  944. at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
  945. at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
  946. at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
  947. at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
  948. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
  949. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
  950. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
  951. at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
  952. at com.sun.proxy.$Proxy81.getVdsCapabilities(Unknown Source)
  953. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
  954. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
  955. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  956. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  957. at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:263) [vdsbroker.jar:]
  958. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  959. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_79]
  960. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  961. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]
  962. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]
  963. at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
  964.  
  965. 2015-09-21 09:22:14,077 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = hosted_engine_2, HostId = 16bd9688-cb24-40be-9ab7-f8643711afb1, vds=Host[hosted_engine_2,16bd9688-cb24-40be-9ab7-f8643711afb1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
  966. 2015-09-21 09:22:14,078 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host hosted_engine_2 is not responding. It will stay in Connecting state for a grace period of 124 seconds and after that an attempt to fence the host will be issued.
  967. 2015-09-21 09:22:14,108 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out
  968. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
  969. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
  970. at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
  971. at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
  972. at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:]
  973. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:649) [vdsbroker.jar:]
  974. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:495) [vdsbroker.jar:]
  975. at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
  976. at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) [:1.7.0_79]
  977. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  978. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  979. at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
  980. at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
  981. at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
  982. Caused by: java.net.SocketTimeoutException: connect timed out
  983. at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_79]
  984. at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_79]
  985. at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_79]
  986. at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_79]
  987. at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_79]
  988. at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_79]
  989. at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625) [jsse.jar:1.7.0_79]
  990. at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:293) [utils.jar:]
  991. at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
  992. at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
  993. at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
  994. at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
  995. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
  996. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
  997. at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
  998. at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
  999. at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
  1000. at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
  1001. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
  1002. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
  1003. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
  1004. at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
  1005. at com.sun.proxy.$Proxy81.getVdsCapabilities(Unknown Source)
  1006. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
  1007. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
  1008. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  1009. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  1010. at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:263) [vdsbroker.jar:]
  1011. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  1012. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_79]
  1013. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  1014. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]
  1015. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]
  1016. at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
  1017.  
  1018. 2015-09-21 09:22:19,150 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) Command GetCapabilitiesVDSCommand(HostName = hosted_engine_2, HostId = 16bd9688-cb24-40be-9ab7-f8643711afb1, vds=Host[hosted_engine_2,16bd9688-cb24-40be-9ab7-f8643711afb1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
  1019. 2015-09-21 09:22:19,151 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) Host hosted_engine_2 is not responding. It will stay in Connecting state for a grace period of 124 seconds and after that an attempt to fence the host will be issued.
  1020. 2015-09-21 09:22:19,179 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out
  1021. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
  1022. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
  1023. at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
  1024. at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
  1025. at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:]
  1026. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:649) [vdsbroker.jar:]
  1027. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:495) [vdsbroker.jar:]
  1028. at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
  1029. at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) [:1.7.0_79]
  1030. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  1031. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  1032. at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
  1033. at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
  1034. at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
  1035. Caused by: java.net.SocketTimeoutException: connect timed out
  1036. at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_79]
  1037. at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_79]
  1038. at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_79]
  1039. at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_79]
  1040. at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_79]
  1041. at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_79]
  1042. at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625) [jsse.jar:1.7.0_79]
  1043. at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:293) [utils.jar:]
  1044. at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
  1045. at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
  1046. at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
  1047. at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
  1048. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
  1049. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
  1050. at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
  1051. at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
  1052. at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
  1053. at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
  1054. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
  1055. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
  1056. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
  1057. at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
  1058. at com.sun.proxy.$Proxy81.getVdsCapabilities(Unknown Source)
  1059. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
  1060. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
  1061. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  1062. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  1063. at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:263) [vdsbroker.jar:]
  1064. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  1065. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_79]
  1066. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  1067. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]
  1068. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]
  1069. at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
  1070.  
  1071. 2015-09-21 09:22:24,225 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetCapabilitiesVDSCommand(HostName = hosted_engine_2, HostId = 16bd9688-cb24-40be-9ab7-f8643711afb1, vds=Host[hosted_engine_2,16bd9688-cb24-40be-9ab7-f8643711afb1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
  1072. 2015-09-21 09:22:24,226 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) Host hosted_engine_2 is not responding. It will stay in Connecting state for a grace period of 124 seconds and after that an attempt to fence the host will be issued.
  1073. 2015-09-21 09:22:24,254 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTi
  1074. meoutException: connect timed out
  1075. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
  1076. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
  1077. at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
  1078. at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
  1079. at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:]
  1080. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:649) [vdsbroker.jar:]
  1081. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:495) [vdsbroker.jar:]
  1082. at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
  1083. at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) [:1.7.0_79]
  1084. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  1085. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  1086. at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
  1087. at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
  1088. at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
  1089. Caused by: java.net.SocketTimeoutException: connect timed out
  1090. at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_79]
  1091. at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_79]
  1092. at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_79]
  1093. at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_79]
  1094. at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_79]
  1095. at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_79]
  1096. at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625) [jsse.jar:1.7.0_79]
  1097. at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:293) [utils.jar:]
  1098. at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
  1099. at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
  1100. at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
  1101. at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
  1102. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
  1103. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
  1104. at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
  1105. at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
  1106. at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
  1107. at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
  1108. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
  1109. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
  1110. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
  1111. at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
  1112. at com.sun.proxy.$Proxy81.getVdsCapabilities(Unknown Source)
  1113. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
  1114. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
  1115. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  1116. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  1117. at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:263) [vdsbroker.jar:]
  1118. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  1119. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_79]
  1120. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  1121. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]
  1122. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]
  1123. at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
  1124.  
  1125. 2015-09-21 09:22:29,294 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = hosted_engine_2, HostId = 16bd9688-cb24-40be-9ab7-f8643711afb1, vds=Host[hosted_engine_2,16bd9688-cb24-40be-9ab7-f8643711afb1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
  1126. 2015-09-21 09:22:29,295 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host hosted_engine_2 is not responding. It will stay in Connecting state for a grace period of 124 seconds and after that an attempt to fence the host will be issued.
  1127. 2015-09-21 09:22:29,325 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out
  1128. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
  1129. at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
  1130. at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
  1131. at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
  1132. at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:]
  1133. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:649) [vdsbroker.jar:]
  1134. at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:495) [vdsbroker.jar:]
  1135. at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
  1136. at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) [:1.7.0_79]
  1137. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  1138. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  1139. at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
  1140. at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
  1141. at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
  1142. Caused by: java.net.SocketTimeoutException: connect timed out
  1143. at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_79]
  1144. at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_79]
  1145. at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_79]
  1146. at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_79]
  1147. at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_79]
  1148. at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_79]
  1149. at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625) [jsse.jar:1.7.0_79]
  1150. at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:293) [utils.jar:]
  1151. at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:]
  1152. at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:]
  1153. at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:]
  1154. at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:]
  1155. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:]
  1156. at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:]
  1157. at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]
  1158. at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3]
  1159. at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3]
  1160. at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3]
  1161. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3]
  1162. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3]
  1163. at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3]
  1164. at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3]
  1165. at com.sun.proxy.$Proxy81.getVdsCapabilities(Unknown Source)
  1166. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
  1167. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
  1168. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
  1169. at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
  1170. at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:263) [vdsbroker.jar:]
  1171. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  1172. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_79]
  1173. at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]
  1174. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]
  1175. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]
  1176. at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
  1177.  
  1178. 2015-09-21 09:22:34,366 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) Command GetCapabilitiesVDSCommand(HostName = hosted_engine_2, HostId = 16bd9688-cb24-40be-9ab7-f8643711afb1, vds=Host[hosted_engine_2,16bd9688-cb24-40be-9ab7-f8643711afb1]) execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out
  1179. 2015-09-21 09:22:34,367 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) Host hosted_engine_2 is not responding. It will stay in Connecting state for a grace period of 124 seconds and after that an attempt to fence the host will be issued.
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!