API tools faq
paste
Advertisement
Guest User

Solr-ZK Connection hiccup

a guest
Feb 21st, 2014
99
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 17.34 KB | None | 0 0
raw download clone embed print report
  1. 2014-02-20 09:20:44,512 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 WrapperSimpleAppMain-SendThread(zookeeper1.aws.corp:2181) - common N
  2. ? Client session timed out, have not heard from server in 19397ms for sessionid 0x244080312d207be, closing socket connection and attempting reconnect
  3. 2014-02-20 09:20:44,571 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 qtp665563146-1539 - common N ? [si_shard10_replica1] webapp=/solr path=/update params={wt=javabin&version=2} status=0 QTime=58
  4. 2014-02-20 09:20:44,576 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 qtp665563146-1532 - common N ? [si_shard10_replica1] webapp=/solr path=/update params={wt=javabin&version=2} status=0 QTime=63
  5. 2014-02-20 09:20:44,612 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 WrapperSimpleAppMain-EventThread - common N ? Watcher org.apache.solr.common.cloud.ConnectionManager@2645bf2d name:ZooKeeperConnection Watcher:zookeeper0.aws.corp:2181,zookeeper1.aws.corp:2181,zookeeper2.aws.corp:2181/si got event WatchedEvent state:Disconnected type:None path:null path:null type:None
  6. 2014-02-20 09:20:44,612 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 WrapperSimpleAppMain-EventThread - common N ? zkClient has disconnected
  7. 2014-02-20 09:20:44,784 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 qtp665563146-1537 - common N ? [si_shard10_replica1] webapp=/solr path=/update params={wt=javabin&version=2} status=0 QTime=43
  8. 2014-02-20 09:20:44,894 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 qtp665563146-1541 - common N ? [si_shard10_replica1] webapp=/solr path=/update params={wt=javabin&version=2} status=0 QTime=62
  9. 2014-02-20 09:20:44,925 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 searcherExecutor-5-thread-1 - common N ? QuerySenderListener sending requests to Searcher@41f1c8 main{StandardDirectoryReader(segments_1i:14579:nrt _1m0(4.6):C4311707 _3pp(4.6):C5997015 _4t6(4.6):C2240007/13563:delGen=1 _3u5(4.6):C733558 _5cm(4.6):C1946783/349432:delGen=4 _427(4.6):C792999 _5eu(4.6):C262359/98550:delGen=1 _59u(4.6):C496950/368574:delGen=3 _5a4(4.6):C11527 _5b7(4.6):C37736 _5ea(4.6):C56172 _5f5(4.6):C60336 _5g9(4.6):C59246 _5ff(4.6):C48341 _5fz(4.6):C38688 _5fp(4.6):C57695 _5gj(4.6):C34751 _5gt(4.6):C59461 _5h3(4.6):C57948 _5hn(4.6):C50015 _5hd(4.6):C56193 _5h8(4.6):C6831 _5hx(4.6):C43183 _5hw(4.6):C1320 _5ii(4.6):C59047 _5i0(4.6):C4916 _5i1(4.6):C4919 _5i2(4.6):C2924 _5i3(4.6):C1689 _5i4(4.6):C522 _5ic(4.6):C4202 _5ie(4.6):C1 _5ig(4.6):C23214 _5ih(4.6):C5693 _5ij(4.6):C5898)}
  10. 2014-02-20 09:20:44,925 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 searcherExecutor-5-thread-1 - common N ? QuerySenderListener done.
  11. 2014-02-20 09:20:44,927 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 searcherExecutor-5-thread-1 - common N ? [si_shard10_replica1] Registered new searcher Searcher@41f1c8 main{StandardDirectoryReader(segments_1i:14579:nrt _1m0(4.6):C4311707 _3pp(4.6):C5997015 _4t6(4.6):C2240007/13563:delGen=1 _3u5(4.6):C733558 _5cm(4.6):C1946783/349432:delGen=4 _427(4.6):C792999 _5eu(4.6):C262359/98550:delGen=1 _59u(4.6):C496950/368574:delGen=3 _5a4(4.6):C11527 _5b7(4.6):C37736 _5ea(4.6):C56172 _5f5(4.6):C60336 _5g9(4.6):C59246 _5ff(4.6):C48341 _5fz(4.6):C38688 _5fp(4.6):C57695 _5gj(4.6):C34751 _5gt(4.6):C59461 _5h3(4.6):C57948 _5hn(4.6):C50015 _5hd(4.6):C56193 _5h8(4.6):C6831 _5hx(4.6):C43183 _5hw(4.6):C1320 _5ii(4.6):C59047 _5i0(4.6):C4916 _5i1(4.6):C4919 _5i2(4.6):C2924 _5i3(4.6):C1689 _5i4(4.6):C522 _5ic(4.6):C4202 _5ie(4.6):C1 _5ig(4.6):C23214 _5ih(4.6):C5693 _5ij(4.6):C5898)}
  12. 2014-02-20 09:20:45,197 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 qtp665563146-1532 - common N ? [si_shard10_replica1] webapp=/solr path=/update params={wt=javabin&version=2} status=0 QTime=65
  13. 2014-02-20 09:20:45,208 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 qtp665563146-1541 - common N ? [si_shard10_replica1] webapp=/solr path=/update params={wt=javabin&version=2} status=0 QTime=58
  14. 2014-02-20 09:20:45,334 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 WrapperSimpleAppMain-SendThread(zookeeper2.aws.corp:2181) - common N
  15. ? Opening socket connection to server zookeeper2.aws.corp/101.13.0.106:2181. Will not attempt to authenticate using SASL (unknown error)
  16. 2014-02-20 09:20:45,335 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 WrapperSimpleAppMain-SendThread(zookeeper2.aws.corp:2181) - common N
  17. ? Socket connection established to zookeeper2.aws.corp/101.13.0.106:2181, initiating session
  18. 2014-02-20 09:20:45,337 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 WrapperSimpleAppMain-EventThread - common N ? Watcher org.apache.solr.common.cloud.ConnectionManager@2645bf2d name:ZooKeeperConnection Watcher:zookeeper0.aws.corp:2181,zookeeper1.aws.corp:2181,zookeeper2.aws.corp:2181/si got event WatchedEvent state:Expired type:None path:null path:null type:None
  19. 2014-02-20 09:20:45,337 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 WrapperSimpleAppMain-EventThread - common N ? Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
  20. 2014-02-20 09:20:45,344 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 WrapperSimpleAppMain-EventThread - common N ? Connection expired - starting a new one...
  21. 2014-02-20 09:20:45,344 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 WrapperSimpleAppMain-SendThread(zookeeper2.aws.corp:2181) - common N
  22. ? Unable to reconnect to ZooKeeper service, session 0x244080312d207be has expired, closing socket connection
  23. 2014-02-20 09:20:45,344 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 WrapperSimpleAppMain-EventThread - common N ? Initiating client connection, connectString=zookeeper0.aws.corp:2181,zookeeper1.aws.corp:2181,zookeeper2.aws.corp:2181/si sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@2645bf2d
  24. 2014-02-20 09:20:45,380 ERROR COMMON 1 si ip-101-13-20-21.aws.corp 8980 qtp665563146-1537 - common N ? org.apache.solr.common.SolrException: Cannot talk to ZooKeeper - Updates are disabled.
  25. at org.apache.solr.update.processor.DistributedUpdateProcessor.zkCheck(DistributedUpdateProcessor.java:1132)
  26. at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:429)
  27. at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:96)
  28. at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:166)
  29. at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
  30. at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:225)
  31. at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121)
  32. at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:190)
  33. at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:116)
  34. at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:173)
  35. at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:106)
  36. at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:58)
  37. at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
  38. at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
  39. at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
  40. at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
  41. at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:721)
  42. at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)
  43. at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)
  44. at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1624)
  45. at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
  46. at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
  47. at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:568)
  48. at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
  49. at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1112)
  50. at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:479)
  51. at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
  52. at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1046)
  53. at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
  54. at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
  55. at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
  56. at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
  57. at org.eclipse.jetty.server.Server.handle(Server.java:459)
  58. at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:281)
  59. at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
  60. at org.eclipse.jetty.io.AbstractConnection$1.run(AbstractConnection.java:505)
  61. at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
  62. at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
  63. at java.lang.Thread.run(Thread.java:744)
  64. 2014-02-20 09:20:45,380 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 qtp665563146-1537 - common N ? [si_shard10_replica1] webapp=/solr
  65. path=/update params={wt=javabin&version=2} status=503 QTime=0
  66. 2014-02-20 09:20:45,384 ERROR COMMON 1 si ip-101-13-20-21.aws.corp 8980 qtp665563146-1532 - common N ? org.apache.solr.common.SolrExceptio
  67. n: Cannot talk to ZooKeeper - Updates are disabled.
  68. at org.apache.solr.update.processor.DistributedUpdateProcessor.zkCheck(DistributedUpdateProcessor.java:1132)
  69. at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:429)
  70. at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:96)
  71. at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:166)
  72. at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
  73. at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:225)
  74. at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121)
  75. at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:190)
  76. at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:116)
  77. at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:173)
  78. at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:106)
  79. at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:58)
  80. at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
  81. at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
  82. at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
  83. at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
  84. at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:721)
  85. at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)
  86. at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)
  87. at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1624)
  88. at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
  89. at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
  90. at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:568)
  91. at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
  92. at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1112)
  93. at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:479)
  94. at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
  95. at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1046)
  96. at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
  97. at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
  98. at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
  99. at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
  100. at org.eclipse.jetty.server.Server.handle(Server.java:459)
  101. at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:281)
  102. at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
  103. at org.eclipse.jetty.io.AbstractConnection$1.run(AbstractConnection.java:505)
  104. at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
  105. at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
  106. at java.lang.Thread.run(Thread.java:744)
  107. 2014-02-20 09:20:45,445 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 WrapperSimpleAppMain-EventThread - common N ? Waiting for client to connect to ZooKeeper
  108. 2014-02-20 09:20:45,466 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 WrapperSimpleAppMain-SendThread(zookeeper1.aws.corp:2181) - common N
  109. ? Opening socket connection to server zookeeper1.aws.corp/101.13.0.105:2181. Will not attempt to authenticate using SASL (unknown error)
  110. 2014-02-20 09:20:45,467 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 WrapperSimpleAppMain-SendThread(zookeeper1.aws.corp:2181) - common N
  111. ? Socket connection established to zookeeper1.aws.corp/101.13.0.105:2181, initiating session
  112. 2014-02-20 09:20:45,472 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 WrapperSimpleAppMain-SendThread(zookeeper1.aws.corp:2181) - common N
  113. ? Session establishment complete on server zookeeper1.aws.corp/101.13.0.105:2181, sessionid = 0x14408034fe80320, negotiated timeout = 10000
  114. 2014-02-20 09:20:45,472 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 WrapperSimpleAppMain-EventThread - common N ? Watcher org.apache.solr.common.cloud.ConnectionManager@2645bf2d name:ZooKeeperConnection Watcher:zookeeper0.aws.corp:2181,zookeeper1.aws.corp:2181,zookeeper2.aws.corp:2181/si got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  115. 2014-02-20 09:20:45,472 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 WrapperSimpleAppMain-EventThread - common N ? Client is connected to ZooKeeper
  116. 2014-02-20 09:20:45,472 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 WrapperSimpleAppMain-EventThread - common N ? Connection with ZooKeeper reestablished.
  117. 2014-02-20 09:20:45,472 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 WrapperSimpleAppMain-EventThread - common N ? Reconnected to ZooKeeper
  118. 2014-02-20 09:20:45,472 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 WrapperSimpleAppMain-EventThread - common N ? Connected:true
  119. 2014-02-20 09:20:45,472 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 WrapperSimpleAppMain-EventThread - common N ? EventThread shut down
  120. 2014-02-20 09:20:45,473 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 Thread-1509 - common N ? publishing core=si_shard10_replica1 state=down
  121. 2014-02-20 09:20:45,495 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 Thread-1509 - common N ? Updating cluster state from ZooKeeper...
  122. 2014-02-20 09:20:45,503 INFO COMMON 1 si ip-101-13-20-21.aws.corp 8980 Thread-1509 - common N ? Register node as live in ZooKeeper:/live_nodes/101.13.20.21:8980_solr
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!