API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Dec 10th, 2018
73
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 108.82 KB | None | 0 0
raw download clone embed print report
  1. 2018-12-10 17:01:04.893 INFO (main) [ ] o.e.j.u.log Logging initialized @1289ms to org.eclipse.jetty.util.log.Slf4jLog
  2. 2018-12-10 17:01:05.146 INFO (main) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_191-b12
  3. 2018-12-10 17:01:05.187 INFO (main) [ ] o.e.j.d.p.ScanningAppProvider Deployment monitor [file:///opt/solr/server/contexts/] at interval 0
  4. 2018-12-10 17:01:05.487 INFO (main) [ ] o.e.j.w.StandardDescriptorProcessor NO JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet
  5. 2018-12-10 17:01:05.499 INFO (main) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
  6. 2018-12-10 17:01:05.499 INFO (main) [ ] o.e.j.s.session No SessionScavenger set, using defaults
  7. 2018-12-10 17:01:05.502 INFO (main) [ ] o.e.j.s.session node0 Scavenging every 660000ms
  8. 2018-12-10 17:01:05.647 INFO (main) [ ] o.a.s.u.c.SSLCredentialProviderFactory Processing SSL Credential Provider chain: env;sysprop
  9. 2018-12-10 17:01:05.699 INFO (main) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
  10. 2018-12-10 17:01:05.711 INFO (main) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.5.0
  11. 2018-12-10 17:01:05.795 INFO (main) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port 8983
  12. 2018-12-10 17:01:05.795 INFO (main) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: /opt/solr
  13. 2018-12-10 17:01:05.797 INFO (main) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-12-10T17:01:05.797Z
  14. 2018-12-10 17:01:05.917 INFO (main) [ ] o.a.s.c.SolrResourceLoader Using system property solr.solr.home: /opt/solr/server/solr
  15. 2018-12-10 17:01:06.045 INFO (zkConnectionManagerCallback-2-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
  16. 2018-12-10 17:01:06.068 INFO (main) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
  17. 2018-12-10 17:01:06.073 INFO (main) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /opt/solr/server/solr/solr.xml
  18. 2018-12-10 17:01:06.223 INFO (main) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@453da22c, but no JMX reporters were configured - adding default JMX reporter.
  19. 2018-12-10 17:01:07.109 INFO (main) [ ] o.a.s.c.ZkContainer Zookeeper client=zookeeper.node1.metaonly01.analyticstest.internal.net:2181,zookeeper.node2.metaonly01.analyticstest.internal.net:2181,zookeeper.node3.metaonly01.analyticstest.internal.net:2181/solr
  20. 2018-12-10 17:01:07.138 INFO (zkConnectionManagerCallback-6-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
  21. 2018-12-10 17:01:07.168 INFO (zkConnectionManagerCallback-8-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
  22. 2018-12-10 17:01:07.251 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
  23. 2018-12-10 17:01:07.333 INFO (main) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
  24. 2018-12-10 17:01:07.335 INFO (main) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/solr.node4.metaonly01.analyticstest.internal.net:8983_solr
  25. 2018-12-10 17:01:07.353 INFO (zkCallback-7-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
  26. 2018-12-10 17:01:07.524 INFO (zkConnectionManagerCallback-15-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
  27. 2018-12-10 17:01:07.529 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
  28. 2018-12-10 17:01:07.533 INFO (main) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at zookeeper.node1.metaonly01.analyticstest.internal.net:2181,zookeeper.node2.metaonly01.analyticstest.internal.net:2181,zookeeper.node3.metaonly01.analyticstest.internal.net:2181/solr ready
  29. 2018-12-10 17:01:07.578 INFO (main) [ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
  30. 2018-12-10 17:01:07.679 INFO (main) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@453da22c
  31. 2018-12-10 17:01:07.683 INFO (main) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@453da22c
  32. 2018-12-10 17:01:07.697 INFO (main) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@453da22c
  33. 2018-12-10 17:01:07.812 INFO (main) [ ] o.a.s.c.CorePropertiesLocator Found 4 core definitions underneath /opt/solr/server/solr
  34. 2018-12-10 17:01:07.814 INFO (main) [ ] o.a.s.c.CorePropertiesLocator Cores are: [main_shard1_replica_n2, main_shard3_replica_n8, history_shard2_replica_n6, history_shard1_replica_n1]
  35. 2018-12-10 17:01:08.014 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
  36. 2018-12-10 17:01:08.015 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
  37. 2018-12-10 17:01:08.015 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
  38. 2018-12-10 17:01:08.017 INFO (main) [ ] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@58a9760d{/solr,file:///opt/solr/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr/server/solr-webapp/webapp}
  39. 2018-12-10 17:01:08.018 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.RequestParams request params refreshed to version 0
  40. 2018-12-10 17:01:08.018 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.RequestParams request params refreshed to version 0
  41. 2018-12-10 17:01:08.018 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.RequestParams request params refreshed to version 0
  42. 2018-12-10 17:01:08.042 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.SolrResourceLoader [history_shard1_replica_n1] Added 56 libs to classloader, from paths: [/opt/solr/contrib/clustering/lib, /opt/solr/contrib/extraction/lib, /opt/solr/contrib/langid/lib, /opt/solr/contrib/velocity/lib, /opt/solr/dist]
  43. 2018-12-10 17:01:08.052 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.SolrResourceLoader [main_shard1_replica_n2] Added 56 libs to classloader, from paths: [/opt/solr/contrib/clustering/lib, /opt/solr/contrib/extraction/lib, /opt/solr/contrib/langid/lib, /opt/solr/contrib/velocity/lib, /opt/solr/dist]
  44. 2018-12-10 17:01:08.055 INFO (main) [ ] o.e.j.s.AbstractConnector Started ServerConnector@186978a6{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
  45. 2018-12-10 17:01:08.055 INFO (main) [ ] o.e.j.s.Server Started @4455ms
  46. 2018-12-10 17:01:08.056 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.SolrResourceLoader [main_shard3_replica_n8] Added 56 libs to classloader, from paths: [/opt/solr/contrib/clustering/lib, /opt/solr/contrib/extraction/lib, /opt/solr/contrib/langid/lib, /opt/solr/contrib/velocity/lib, /opt/solr/dist]
  47. 2018-12-10 17:01:08.453 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.5.0
  48. 2018-12-10 17:01:08.469 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.5.0
  49. 2018-12-10 17:01:08.479 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.5.0
  50. 2018-12-10 17:01:08.692 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.s.IndexSchema [main_shard1_replica_n2] Schema name=main-config
  51. 2018-12-10 17:01:08.701 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.s.IndexSchema [main_shard3_replica_n8] Schema name=main-config
  52. 2018-12-10 17:01:08.743 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.s.IndexSchema [history_shard1_replica_n1] Schema name=default-config
  53. 2018-12-10 17:01:09.000 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.s.IndexSchema Loaded schema main-config/1.6 with uniqueid field id
  54. 2018-12-10 17:01:09.000 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.s.IndexSchema Loaded schema main-config/1.6 with uniqueid field id
  55. 2018-12-10 17:01:09.053 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://hdfstarget.internal.net:8020/user01v2/drop5
  56. 2018-12-10 17:01:09.053 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
  57. 2018-12-10 17:01:09.053 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://hdfstarget.internal.net:8020/user01v2/drop5
  58. 2018-12-10 17:01:09.054 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
  59. 2018-12-10 17:01:09.168 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
  60. 2018-12-10 17:01:09.168 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.c.RequestParams request params refreshed to version 0
  61. 2018-12-10 17:01:09.177 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.c.SolrResourceLoader [main_shard3_replica_n8] Added 56 libs to classloader, from paths: [/opt/solr/contrib/clustering/lib, /opt/solr/contrib/extraction/lib, /opt/solr/contrib/langid/lib, /opt/solr/contrib/velocity/lib, /opt/solr/dist]
  62. 2018-12-10 17:01:09.199 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
  63. 2018-12-10 17:01:09.200 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.c.RequestParams request params refreshed to version 0
  64. 2018-12-10 17:01:09.204 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.c.SolrResourceLoader [main_shard1_replica_n2] Added 56 libs to classloader, from paths: [/opt/solr/contrib/clustering/lib, /opt/solr/contrib/extraction/lib, /opt/solr/contrib/langid/lib, /opt/solr/contrib/velocity/lib, /opt/solr/dist]
  65. 2018-12-10 17:01:09.250 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.5.0
  66. 2018-12-10 17:01:09.272 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.5.0
  67. 2018-12-10 17:01:09.311 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.s.IndexSchema [main_shard3_replica_n8] Schema name=main-config
  68. 2018-12-10 17:01:09.319 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.s.IndexSchema [main_shard1_replica_n2] Schema name=main-config
  69. 2018-12-10 17:01:09.350 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.s.IndexSchema Loaded schema main-config/1.6 with uniqueid field id
  70. 2018-12-10 17:01:09.373 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.c.CoreContainer Creating SolrCore 'main_shard3_replica_n8' using configuration from collection main, trusted=false
  71. 2018-12-10 17:01:09.408 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema main-config/1.6 with uniqueid field id
  72. 2018-12-10 17:01:09.416 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'main_shard1_replica_n2' using configuration from collection main, trusted=false
  73. 2018-12-10 17:01:09.466 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.main.shard3.replica_n8' (registry 'solr.core.main.shard3.replica_n8') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@453da22c
  74. 2018-12-10 17:01:09.467 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://hdfstarget.internal.net:8020/user01v2/drop5
  75. 2018-12-10 17:01:09.467 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
  76. 2018-12-10 17:01:09.474 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.main.shard1.replica_n2' (registry 'solr.core.main.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@453da22c
  77. 2018-12-10 17:01:09.475 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://hdfstarget.internal.net:8020/user01v2/drop5
  78. 2018-12-10 17:01:09.475 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
  79. 2018-12-10 17:01:09.482 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.c.SolrCore [[main_shard1_replica_n2] ] Opening new SolrCore at [/opt/solr/server/solr/main_shard1_replica_n2], dataDir=[hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node5/data/]
  80. 2018-12-10 17:01:09.482 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.c.SolrCore [[main_shard3_replica_n8] ] Opening new SolrCore at [/opt/solr/server/solr/main_shard3_replica_n8], dataDir=[hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node11/data/]
  81. 2018-12-10 17:01:09.507 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node11/data/snapshot_metadata
  82. 2018-12-10 17:01:09.510 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node5/data/snapshot_metadata
  83. 2018-12-10 17:01:09.602 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
  84. 2018-12-10 17:01:09.602 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [134217728] will allocate [1] slabs and use ~[134217728] bytes
  85. 2018-12-10 17:01:09.603 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.c.HdfsDirectoryFactory Creating new global HDFS BlockCache
  86. 2018-12-10 17:01:09.604 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
  87. 2018-12-10 17:01:09.604 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [134217728] will allocate [1] slabs and use ~[134217728] bytes
  88. 2018-12-10 17:01:09.726 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
  89. 2018-12-10 17:01:09.839 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://hdfstarget.internal.net:8020/user01v2/drop5
  90. 2018-12-10 17:01:09.840 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
  91. 2018-12-10 17:01:09.882 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
  92. 2018-12-10 17:01:09.882 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.RequestParams request params refreshed to version 0
  93. 2018-12-10 17:01:09.885 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.SolrResourceLoader [history_shard1_replica_n1] Added 56 libs to classloader, from paths: [/opt/solr/contrib/clustering/lib, /opt/solr/contrib/extraction/lib, /opt/solr/contrib/langid/lib, /opt/solr/contrib/velocity/lib, /opt/solr/dist]
  94. 2018-12-10 17:01:09.938 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.5.0
  95. 2018-12-10 17:01:10.007 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.s.IndexSchema [history_shard1_replica_n1] Schema name=default-config
  96. 2018-12-10 17:01:10.340 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
  97. 2018-12-10 17:01:10.423 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'history_shard1_replica_n1' using configuration from collection history, trusted=true
  98. 2018-12-10 17:01:10.427 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.history.shard1.replica_n1' (registry 'solr.core.history.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@453da22c
  99. 2018-12-10 17:01:10.428 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://hdfstarget.internal.net:8020/user01v2/drop5
  100. 2018-12-10 17:01:10.428 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
  101. 2018-12-10 17:01:10.429 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.SolrCore [[history_shard1_replica_n1] ] Opening new SolrCore at [/opt/solr/server/solr/history_shard1_replica_n1], dataDir=[hdfs://hdfstarget.internal.net:8020/user01v2/drop5/history/core_node3/data/]
  102. 2018-12-10 17:01:10.442 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://hdfstarget.internal.net:8020/user01v2/drop5/history/core_node3/data/snapshot_metadata
  103. 2018-12-10 17:01:10.470 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
  104. 2018-12-10 17:01:10.470 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [134217728] will allocate [1] slabs and use ~[134217728] bytes
  105. 2018-12-10 17:01:11.088 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.s.b.BlockDirectory Block cache on write is disabled
  106. 2018-12-10 17:01:11.088 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.s.b.BlockDirectory Block cache on write is disabled
  107. 2018-12-10 17:01:11.092 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
  108. 2018-12-10 17:01:11.111 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node5/data
  109. 2018-12-10 17:01:11.111 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node11/data
  110. 2018-12-10 17:01:11.114 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://hdfstarget.internal.net:8020/user01v2/drop5/history/core_node3/data
  111. 2018-12-10 17:01:11.176 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://hdfstarget.internal.net:8020/user01v2/drop5/history/core_node3/data/index
  112. 2018-12-10 17:01:11.194 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
  113. 2018-12-10 17:01:11.194 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [134217728] will allocate [1] slabs and use ~[134217728] bytes
  114. 2018-12-10 17:01:11.204 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
  115. 2018-12-10 17:01:11.244 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node5/data/index.20181210022931338
  116. 2018-12-10 17:01:11.251 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node11/data/index.20181208153548551
  117. 2018-12-10 17:01:11.261 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
  118. 2018-12-10 17:01:11.261 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [134217728] will allocate [1] slabs and use ~[134217728] bytes
  119. 2018-12-10 17:01:11.266 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
  120. 2018-12-10 17:01:11.272 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [134217728] will allocate [1] slabs and use ~[134217728] bytes
  121. 2018-12-10 17:01:11.278 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.s.b.BlockDirectory Block cache on write is disabled
  122. 2018-12-10 17:01:11.282 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.s.b.BlockDirectory Block cache on write is disabled
  123. 2018-12-10 17:01:11.305 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
  124. 2018-12-10 17:01:11.325 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
  125. 2018-12-10 17:01:11.330 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
  126. 2018-12-10 17:01:12.200 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
  127. 2018-12-10 17:01:12.200 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
  128. 2018-12-10 17:01:12.200 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
  129. 2018-12-10 17:01:12.208 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
  130. 2018-12-10 17:01:12.209 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
  131. 2018-12-10 17:01:12.209 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
  132. 2018-12-10 17:01:12.245 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.FSHDFSUtils Recovering lease on dfs file hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node11/data/tlog/tlog.0000000000000040275
  133. 2018-12-10 17:01:12.256 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
  134. 2018-12-10 17:01:12.256 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
  135. 2018-12-10 17:01:12.277 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
  136. 2018-12-10 17:01:12.278 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
  137. 2018-12-10 17:01:12.279 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
  138. 2018-12-10 17:01:12.311 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.FSHDFSUtils Recovering lease on dfs file hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node11/data/tlog/tlog.0000000000000040276
  139. 2018-12-10 17:01:12.315 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.u.FSHDFSUtils Recovering lease on dfs file hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node5/data/tlog/tlog.0000000000000040239
  140. 2018-12-10 17:01:12.328 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.FSHDFSUtils Recovering lease on dfs file hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node11/data/tlog/tlog.0000000000000040277
  141. 2018-12-10 17:01:12.332 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.u.FSHDFSUtils Recovering lease on dfs file hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node5/data/tlog/tlog.0000000000000040240
  142. 2018-12-10 17:01:12.350 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.FSHDFSUtils Recovering lease on dfs file hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node11/data/tlog/tlog.0000000000000040278
  143. 2018-12-10 17:01:12.353 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.u.FSHDFSUtils Recovering lease on dfs file hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node5/data/tlog/tlog.0000000000000040241
  144. 2018-12-10 17:01:12.364 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.FSHDFSUtils Recovering lease on dfs file hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node11/data/tlog/tlog.0000000000000040279
  145. 2018-12-10 17:01:12.370 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.u.HdfsUpdateLog Closing output for old non-recovery log hdfs tlog{file=hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node5/data/tlog/tlog.0000000000000040239 refcount=1}
  146. 2018-12-10 17:01:12.388 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.FSHDFSUtils Recovering lease on dfs file hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node11/data/tlog/tlog.0000000000000040280
  147. 2018-12-10 17:01:12.410 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.FSHDFSUtils Recovering lease on dfs file hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node11/data/tlog/tlog.0000000000000040281
  148. 2018-12-10 17:01:12.421 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.FSHDFSUtils Recovering lease on dfs file hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node11/data/tlog/tlog.0000000000000040282
  149. 2018-12-10 17:01:12.424 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 300000ms; if 20000 uncommitted docs;
  150. 2018-12-10 17:01:12.424 INFO (coreLoadExecutor-9-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard1 r:core_node5 x:main_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 1800000ms; if 10000000 uncommitted docs;
  151. 2018-12-10 17:01:12.438 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.FSHDFSUtils Recovering lease on dfs file hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node11/data/tlog/tlog.0000000000000040283
  152. 2018-12-10 17:01:12.456 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.FSHDFSUtils Recovering lease on dfs file hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node11/data/tlog/tlog.0000000000000040284
  153. 2018-12-10 17:01:12.461 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6f8e5972[history_shard1_replica_n1] main]
  154. 2018-12-10 17:01:12.470 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.HdfsUpdateLog Closing output for old non-recovery log hdfs tlog{file=hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node11/data/tlog/tlog.0000000000000040282 refcount=1}
  155. 2018-12-10 17:01:12.472 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.HdfsUpdateLog Closing output for old non-recovery log hdfs tlog{file=hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node11/data/tlog/tlog.0000000000000040281 refcount=1}
  156. 2018-12-10 17:01:12.474 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.HdfsUpdateLog Closing output for old non-recovery log hdfs tlog{file=hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node11/data/tlog/tlog.0000000000000040280 refcount=1}
  157. 2018-12-10 17:01:12.476 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.HdfsUpdateLog Closing output for old non-recovery log hdfs tlog{file=hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node11/data/tlog/tlog.0000000000000040279 refcount=1}
  158. 2018-12-10 17:01:12.491 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.HdfsUpdateLog Closing output for old non-recovery log hdfs tlog{file=hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node11/data/tlog/tlog.0000000000000040278 refcount=1}
  159. 2018-12-10 17:01:12.494 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.HdfsUpdateLog Closing output for old non-recovery log hdfs tlog{file=hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node11/data/tlog/tlog.0000000000000040277 refcount=1}
  160. 2018-12-10 17:01:12.497 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.HdfsUpdateLog Closing output for old non-recovery log hdfs tlog{file=hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node11/data/tlog/tlog.0000000000000040276 refcount=1}
  161. 2018-12-10 17:01:12.499 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.HdfsUpdateLog Closing output for old non-recovery log hdfs tlog{file=hdfs://hdfstarget.internal.net:8020/user01v2/drop5/main/core_node11/data/tlog/tlog.0000000000000040275 refcount=1}
  162. 2018-12-10 17:01:12.501 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/history
  163. 2018-12-10 17:01:12.546 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/history
  164. 2018-12-10 17:01:12.549 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/history/managed-schema
  165. 2018-12-10 17:01:12.553 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
  166. 2018-12-10 17:01:12.564 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
  167. 2018-12-10 17:01:12.575 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 300000ms; if 20000 uncommitted docs;
  168. 2018-12-10 17:01:12.575 INFO (coreLoadExecutor-9-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:main s:shard3 r:core_node11 x:main_shard3_replica_n8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 1800000ms; if 10000000 uncommitted docs;
  169. 2018-12-10 17:01:12.579 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
  170. 2018-12-10 17:01:12.598 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
  171. 2018-12-10 17:01:12.600 INFO (searcherExecutor-12-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@6f8e5972[history_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
  172. 2018-12-10 17:01:12.600 INFO (searcherExecutor-12-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener done.
  173. 2018-12-10 17:01:12.600 INFO (searcherExecutor-12-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
  174. 2018-12-10 17:01:12.617 INFO (searcherExecutor-12-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.SolrCore [history_shard1_replica_n1] Registered new searcher Searcher@6f8e5972[history_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
  175. 2018-12-10 17:01:12.640 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1619485023419760640
  176. 2018-12-10 17:01:12.699 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
  177. 2018-12-10 17:01:12.700 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.RequestParams request params refreshed to version 0
  178. 2018-12-10 17:01:12.702 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.SolrResourceLoader [history_shard2_replica_n6] Added 56 libs to classloader, from paths: [/opt/solr/contrib/clustering/lib, /opt/solr/contrib/extraction/lib, /opt/solr/contrib/langid/lib, /opt/solr/contrib/velocity/lib, /opt/solr/dist]
  179. 2018-12-10 17:01:12.718 INFO (coreZkRegister-1-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.ZkController Core needs to recover:history_shard1_replica_n1
  180. 2018-12-10 17:01:12.739 INFO (updateExecutor-3-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.u.DefaultSolrCoreState Running recovery
  181. 2018-12-10 17:01:12.761 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.5.0
  182. 2018-12-10 17:01:12.793 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
  183. 2018-12-10 17:01:12.795 ERROR (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.RecoveryStrategy Error getting recent versions.:java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
  184. at java.util.ArrayList.rangeCheck(ArrayList.java:657)
  185. at java.util.ArrayList.get(ArrayList.java:433)
  186. at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:491)
  187. at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:310)
  188. at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:294)
  189. at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
  190. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  191. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  192. at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
  193. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  194. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  195. at java.lang.Thread.run(Thread.java:748)
  196.  
  197. 2018-12-10 17:01:12.830 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.s.IndexSchema [history_shard2_replica_n6] Schema name=default-config
  198. 2018-12-10 17:01:13.019 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[history_shard1_replica_n1]
  199. 2018-12-10 17:01:13.034 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.u.UpdateLog Starting to buffer updates. HDFSUpdateLog{state=ACTIVE, tlog=null}
  200. 2018-12-10 17:01:13.034 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.RecoveryStrategy Publishing state of core [history_shard1_replica_n1] as recovering, leader is [http://solr.node3.metaonly01.analyticstest.internal.net:8983/solr/history_shard1_replica_n2/] and I am [http://solr.node4.metaonly01.analyticstest.internal.net:8983/solr/history_shard1_replica_n1/]
  201. 2018-12-10 17:01:13.054 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/history/terms/shard1 to Terms{values={core_node3=0, core_node3_recovering=0, core_node5=0}, version=16}
  202. 2018-12-10 17:01:13.066 INFO (zkCallback-7-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/history/state.json] for collection [history] has occurred - updating... (live nodes size: [3])
  203. 2018-12-10 17:01:13.070 INFO (zkCallback-7-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/history/state.json] for collection [history] has occurred - updating... (live nodes size: [3])
  204. 2018-12-10 17:01:13.071 INFO (zkCallback-7-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/history/state.json] for collection [history] has occurred - updating... (live nodes size: [3])
  205. 2018-12-10 17:01:13.091 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://solr.node3.metaonly01.analyticstest.internal.net:8983/solr]; [WaitForState: action=PREPRECOVERY&core=history_shard1_replica_n2&nodeName=solr.node4.metaonly01.analyticstest.internal.net:8983_solr&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
  206. 2018-12-10 17:01:13.131 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
  207. 2018-12-10 17:01:13.229 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://hdfstarget.internal.net:8020/user01v2/drop5
  208. 2018-12-10 17:01:13.230 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
  209. 2018-12-10 17:01:13.252 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
  210. 2018-12-10 17:01:13.253 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.RequestParams request params refreshed to version 0
  211. 2018-12-10 17:01:13.260 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.SolrResourceLoader [history_shard2_replica_n6] Added 56 libs to classloader, from paths: [/opt/solr/contrib/clustering/lib, /opt/solr/contrib/extraction/lib, /opt/solr/contrib/langid/lib, /opt/solr/contrib/velocity/lib, /opt/solr/dist]
  212. 2018-12-10 17:01:13.301 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.5.0
  213. 2018-12-10 17:01:13.413 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.s.IndexSchema [history_shard2_replica_n6] Schema name=default-config
  214. 2018-12-10 17:01:13.779 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
  215. 2018-12-10 17:01:13.927 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.CoreContainer Creating SolrCore 'history_shard2_replica_n6' using configuration from collection history, trusted=true
  216. 2018-12-10 17:01:13.929 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.history.shard2.replica_n6' (registry 'solr.core.history.shard2.replica_n6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@453da22c
  217. 2018-12-10 17:01:13.929 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://hdfstarget.internal.net:8020/user01v2/drop5
  218. 2018-12-10 17:01:13.930 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
  219. 2018-12-10 17:01:13.930 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.SolrCore [[history_shard2_replica_n6] ] Opening new SolrCore at [/opt/solr/server/solr/history_shard2_replica_n6], dataDir=[hdfs://hdfstarget.internal.net:8020/user01v2/drop5/history/core_node9/data/]
  220. 2018-12-10 17:01:13.933 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://hdfstarget.internal.net:8020/user01v2/drop5/history/core_node9/data/snapshot_metadata
  221. 2018-12-10 17:01:13.942 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
  222. 2018-12-10 17:01:13.943 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [134217728] will allocate [1] slabs and use ~[134217728] bytes
  223. 2018-12-10 17:01:13.954 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.s.b.BlockDirectory Block cache on write is disabled
  224. 2018-12-10 17:01:13.959 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://hdfstarget.internal.net:8020/user01v2/drop5/history/core_node9/data
  225. 2018-12-10 17:01:13.986 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://hdfstarget.internal.net:8020/user01v2/drop5/history/core_node9/data/index
  226. 2018-12-10 17:01:13.997 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
  227. 2018-12-10 17:01:13.997 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [134217728] will allocate [1] slabs and use ~[134217728] bytes
  228. 2018-12-10 17:01:14.002 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.s.b.BlockDirectory Block cache on write is disabled
  229. 2018-12-10 17:01:14.026 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
  230. 2018-12-10 17:01:14.090 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
  231. 2018-12-10 17:01:14.091 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
  232. 2018-12-10 17:01:14.091 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
  233. 2018-12-10 17:01:14.104 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
  234. 2018-12-10 17:01:14.105 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
  235. 2018-12-10 17:01:14.128 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@3e09319[history_shard2_replica_n6] main]
  236. 2018-12-10 17:01:14.132 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/history
  237. 2018-12-10 17:01:14.132 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/history
  238. 2018-12-10 17:01:14.133 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/history/managed-schema
  239. 2018-12-10 17:01:14.135 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
  240. 2018-12-10 17:01:14.136 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
  241. 2018-12-10 17:01:14.136 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
  242. 2018-12-10 17:01:14.138 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
  243. 2018-12-10 17:01:14.140 INFO (coreLoadExecutor-9-thread-3-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1619485024992624640
  244. 2018-12-10 17:01:14.140 INFO (searcherExecutor-25-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@3e09319[history_shard2_replica_n6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
  245. 2018-12-10 17:01:14.140 INFO (searcherExecutor-25-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.QuerySenderListener QuerySenderListener done.
  246. 2018-12-10 17:01:14.140 INFO (searcherExecutor-25-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
  247. 2018-12-10 17:01:14.144 INFO (searcherExecutor-25-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [ ] o.a.s.c.SolrCore [history_shard2_replica_n6] Registered new searcher Searcher@3e09319[history_shard2_replica_n6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
  248. 2018-12-10 17:01:14.153 INFO (coreZkRegister-1-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.c.ZkController Core needs to recover:history_shard2_replica_n6
  249. 2018-12-10 17:01:14.154 INFO (updateExecutor-3-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.u.DefaultSolrCoreState Running recovery
  250. 2018-12-10 17:01:14.156 INFO (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
  251. 2018-12-10 17:01:14.157 ERROR (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.c.RecoveryStrategy Error getting recent versions.:java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
  252. at java.util.ArrayList.rangeCheck(ArrayList.java:657)
  253. at java.util.ArrayList.get(ArrayList.java:433)
  254. at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:491)
  255. at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:310)
  256. at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:294)
  257. at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
  258. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  259. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  260. at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
  261. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  262. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  263. at java.lang.Thread.run(Thread.java:748)
  264.  
  265. 2018-12-10 17:01:14.190 INFO (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[history_shard2_replica_n6]
  266. 2018-12-10 17:01:14.194 INFO (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.u.UpdateLog Starting to buffer updates. HDFSUpdateLog{state=ACTIVE, tlog=null}
  267. 2018-12-10 17:01:14.194 INFO (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.c.RecoveryStrategy Publishing state of core [history_shard2_replica_n6] as recovering, leader is [http://solr.node2.metaonly01.analyticstest.internal.net:8983/solr/history_shard2_replica_n4/] and I am [http://solr.node4.metaonly01.analyticstest.internal.net:8983/solr/history_shard2_replica_n6/]
  268. 2018-12-10 17:01:14.197 INFO (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/history/terms/shard2 to Terms{values={core_node7=0, core_node9_recovering=0, core_node9=0}, version=18}
  269. 2018-12-10 17:01:14.200 INFO (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://solr.node2.metaonly01.analyticstest.internal.net:8983/solr]; [WaitForState: action=PREPRECOVERY&core=history_shard2_replica_n4&nodeName=solr.node4.metaonly01.analyticstest.internal.net:8983_solr&coreNodeName=core_node9&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
  270. 2018-12-10 17:01:14.302 INFO (zkCallback-7-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/history/state.json] for collection [history] has occurred - updating... (live nodes size: [3])
  271. 2018-12-10 17:01:14.302 INFO (zkCallback-7-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/history/state.json] for collection [history] has occurred - updating... (live nodes size: [3])
  272. 2018-12-10 17:01:14.303 INFO (zkCallback-7-thread-3) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/history/state.json] for collection [history] has occurred - updating... (live nodes size: [3])
  273. 2018-12-10 17:01:14.571 INFO (qtp1571967156-18) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.main.shard1.replica_n2:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:QUERY./select.requests&key=solr.core.history.shard2.replica_n6:QUERY./select.requests&key=solr.core.history.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.main.shard3.replica_n8:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:INDEX.sizeInBytes} status=0 QTime=749
  274. 2018-12-10 17:01:14.631 INFO (qtp1571967156-18) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=14
  275. 2018-12-10 17:01:15.674 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://solr.node3.metaonly01.analyticstest.internal.net:8983/solr/history_shard1_replica_n2/] - recoveringAfterStartup=[true]
  276. 2018-12-10 17:01:15.679 WARN (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.u.PeerSyncWithLeader no frame of reference to tell if we've missed updates
  277. 2018-12-10 17:01:15.680 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.RecoveryStrategy PeerSync Recovery was not successful - trying replication.
  278. 2018-12-10 17:01:15.680 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.RecoveryStrategy Starting Replication Recovery.
  279. 2018-12-10 17:01:15.680 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.RecoveryStrategy Attempting to replicate from [http://solr.node3.metaonly01.analyticstest.internal.net:8983/solr/history_shard1_replica_n2/].
  280. 2018-12-10 17:01:15.801 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.h.IndexFetcher Master's generation: 1
  281. 2018-12-10 17:01:15.801 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.h.IndexFetcher Master's version: 0
  282. 2018-12-10 17:01:15.801 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.h.IndexFetcher Slave's generation: 4
  283. 2018-12-10 17:01:15.802 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.h.IndexFetcher Slave's version: 0
  284. 2018-12-10 17:01:15.802 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
  285. 2018-12-10 17:01:15.817 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@76a56693[history_shard1_replica_n1] main]
  286. 2018-12-10 17:01:15.820 INFO (searcherExecutor-12-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@76a56693[history_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
  287. 2018-12-10 17:01:15.821 INFO (searcherExecutor-12-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener done.
  288. 2018-12-10 17:01:15.834 INFO (searcherExecutor-12-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.SolrCore [history_shard1_replica_n1] Registered new searcher Searcher@76a56693[history_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
  289. 2018-12-10 17:01:15.853 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.RecoveryStrategy No replay needed.
  290. 2018-12-10 17:01:15.855 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.RecoveryStrategy Replication Recovery was successful.
  291. 2018-12-10 17:01:15.855 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
  292. 2018-12-10 17:01:15.860 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/history/terms/shard1 to Terms{values={core_node3=0, core_node5=0}, version=17}
  293. 2018-12-10 17:01:15.863 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.RecoveryStrategy Updating version bucket highest from index after successful recovery.
  294. 2018-12-10 17:01:15.863 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1619485026799321088
  295. 2018-12-10 17:01:15.873 INFO (recoveryExecutor-4-thread-1-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard1_replica_n1 c:history s:shard1 r:core_node3) [c:history s:shard1 r:core_node3 x:history_shard1_replica_n1] o.a.s.c.RecoveryStrategy Finished recovery process, successful=[true]
  296. 2018-12-10 17:01:15.966 INFO (zkCallback-7-thread-3) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/history/state.json] for collection [history] has occurred - updating... (live nodes size: [3])
  297. 2018-12-10 17:01:15.966 INFO (zkCallback-7-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/history/state.json] for collection [history] has occurred - updating... (live nodes size: [3])
  298. 2018-12-10 17:01:15.966 INFO (zkCallback-7-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/history/state.json] for collection [history] has occurred - updating... (live nodes size: [3])
  299. 2018-12-10 17:01:17.708 INFO (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://solr.node2.metaonly01.analyticstest.internal.net:8983/solr/history_shard2_replica_n4/] - recoveringAfterStartup=[true]
  300. 2018-12-10 17:01:17.708 WARN (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.u.PeerSyncWithLeader no frame of reference to tell if we've missed updates
  301. 2018-12-10 17:01:17.708 INFO (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.c.RecoveryStrategy PeerSync Recovery was not successful - trying replication.
  302. 2018-12-10 17:01:17.708 INFO (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.c.RecoveryStrategy Starting Replication Recovery.
  303. 2018-12-10 17:01:17.709 INFO (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.c.RecoveryStrategy Attempting to replicate from [http://solr.node2.metaonly01.analyticstest.internal.net:8983/solr/history_shard2_replica_n4/].
  304. 2018-12-10 17:01:17.830 INFO (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.h.IndexFetcher Master's generation: 2
  305. 2018-12-10 17:01:17.831 INFO (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.h.IndexFetcher Master's version: 1544044264809
  306. 2018-12-10 17:01:17.831 INFO (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.h.IndexFetcher Slave's generation: 2
  307. 2018-12-10 17:01:17.831 INFO (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.h.IndexFetcher Slave's version: 1544044264809
  308. 2018-12-10 17:01:17.831 INFO (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.h.IndexFetcher Slave in sync with master.
  309. 2018-12-10 17:01:17.831 INFO (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.c.RecoveryStrategy No replay needed.
  310. 2018-12-10 17:01:17.836 INFO (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.c.RecoveryStrategy Replication Recovery was successful.
  311. 2018-12-10 17:01:17.836 INFO (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
  312. 2018-12-10 17:01:17.843 INFO (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/history/terms/shard2 to Terms{values={core_node7=0, core_node9=0}, version=19}
  313. 2018-12-10 17:01:17.848 INFO (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.c.RecoveryStrategy Updating version bucket highest from index after successful recovery.
  314. 2018-12-10 17:01:17.849 INFO (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1619485028881793024
  315. 2018-12-10 17:01:17.854 INFO (recoveryExecutor-4-thread-2-processing-n:solr.node4.metaonly01.analyticstest.internal.net:8983_solr x:history_shard2_replica_n6 c:history s:shard2 r:core_node9) [c:history s:shard2 r:core_node9 x:history_shard2_replica_n6] o.a.s.c.RecoveryStrategy Finished recovery process, successful=[true]
  316. 2018-12-10 17:01:17.950 INFO (zkCallback-7-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/history/state.json] for collection [history] has occurred - updating... (live nodes size: [3])
  317. 2018-12-10 17:01:17.950 INFO (zkCallback-7-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/history/state.json] for collection [history] has occurred - updating... (live nodes size: [3])
  318. 2018-12-10 17:01:17.950 INFO (zkCallback-7-thread-3) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/history/state.json] for collection [history] has occurred - updating... (live nodes size: [3])
  319. 2018-12-10 17:02:16.452 INFO (qtp1571967156-20) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.main.shard1.replica_n2:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:QUERY./select.requests&key=solr.core.history.shard2.replica_n6:QUERY./select.requests&key=solr.core.history.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.main.shard3.replica_n8:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:INDEX.sizeInBytes} status=0 QTime=696
  320. 2018-12-10 17:02:16.455 INFO (qtp1571967156-14) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=0
  321. 2018-12-10 17:03:18.311 INFO (qtp1571967156-13) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.main.shard1.replica_n2:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:QUERY./select.requests&key=solr.core.history.shard2.replica_n6:QUERY./select.requests&key=solr.core.history.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.main.shard3.replica_n8:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:INDEX.sizeInBytes} status=0 QTime=680
  322. 2018-12-10 17:03:18.315 INFO (qtp1571967156-21) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=0
  323. 2018-12-10 17:04:20.115 INFO (qtp1571967156-22) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.main.shard1.replica_n2:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:QUERY./select.requests&key=solr.core.history.shard2.replica_n6:QUERY./select.requests&key=solr.core.history.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.main.shard3.replica_n8:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:INDEX.sizeInBytes} status=0 QTime=585
  324. 2018-12-10 17:04:20.118 INFO (qtp1571967156-16) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=0
  325. 2018-12-10 17:05:21.840 INFO (qtp1571967156-22) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.main.shard1.replica_n2:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:QUERY./select.requests&key=solr.core.history.shard2.replica_n6:QUERY./select.requests&key=solr.core.history.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.main.shard3.replica_n8:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:INDEX.sizeInBytes} status=0 QTime=492
  326. 2018-12-10 17:05:21.844 INFO (qtp1571967156-18) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=1
  327. 2018-12-10 17:05:25.005 INFO (qtp1571967156-21) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={indexInfo=false&wt=json&_=1544461524784} status=0 QTime=1
  328. 2018-12-10 17:05:25.024 INFO (qtp1571967156-13) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json&_=1544461524785} status=0 QTime=19
  329. 2018-12-10 17:05:25.048 INFO (qtp1571967156-21) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&wt=json&_=1544461524785 and sendToOCPQueue=true
  330. 2018-12-10 17:05:25.049 INFO (qtp1571967156-18) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json&_=1544461524785} status=0 QTime=17
  331. 2018-12-10 17:05:25.054 INFO (qtp1571967156-21) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=json&_=1544461524785} status=0 QTime=8
  332. 2018-12-10 17:05:28.136 INFO (qtp1571967156-21) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={indexInfo=false&wt=json&_=1544461524784} status=0 QTime=0
  333. 2018-12-10 17:05:28.149 INFO (qtp1571967156-18) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/zookeeper params={path=/live_nodes&wt=json&_=1544461528109} status=0 QTime=11
  334. 2018-12-10 17:05:28.151 INFO (qtp1571967156-19) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json&_=1544461524785} status=0 QTime=14
  335. 2018-12-10 17:05:28.166 INFO (qtp1571967156-19) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&wt=json&_=1544461524785 and sendToOCPQueue=true
  336. 2018-12-10 17:05:28.166 INFO (qtp1571967156-16) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/zookeeper params={path=/clusterstate.json&view=graph&detail=true&wt=json&_=1544461528109} status=0 QTime=8
  337. 2018-12-10 17:05:28.167 INFO (qtp1571967156-19) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=json&_=1544461524785} status=0 QTime=3
  338. 2018-12-10 17:05:30.265 INFO (qtp1571967156-16) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={indexInfo=false&wt=json&_=1544461524784} status=0 QTime=0
  339. 2018-12-10 17:05:30.272 INFO (qtp1571967156-17) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging params={wt=json&_=1544461530250&since=0} status=0 QTime=4
  340. 2018-12-10 17:05:30.285 INFO (qtp1571967156-18) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json&_=1544461524785} status=0 QTime=19
  341. 2018-12-10 17:05:30.314 INFO (qtp1571967156-17) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&wt=json&_=1544461524785 and sendToOCPQueue=true
  342. 2018-12-10 17:05:30.315 INFO (qtp1571967156-17) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=json&_=1544461524785} status=0 QTime=1
  343. 2018-12-10 17:05:40.269 INFO (qtp1571967156-16) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging params={wt=json&_=1544461530250&since=0} status=0 QTime=0
  344. 2018-12-10 17:05:50.267 INFO (qtp1571967156-17) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging params={wt=json&_=1544461530250&since=0} status=0 QTime=0
  345. 2018-12-10 17:06:00.269 INFO (qtp1571967156-16) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging params={wt=json&_=1544461530250&since=0} status=0 QTime=0
  346. 2018-12-10 17:06:10.276 INFO (qtp1571967156-17) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging params={wt=json&_=1544461530250&since=0} status=0 QTime=0
  347. 2018-12-10 17:06:18.673 INFO (qtp1571967156-16) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={indexInfo=false&wt=json&_=1544461524784} status=0 QTime=0
  348. 2018-12-10 17:06:18.677 INFO (qtp1571967156-17) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/zookeeper params={path=/live_nodes&wt=json&_=1544461528109} status=0 QTime=4
  349. 2018-12-10 17:06:18.685 INFO (qtp1571967156-14) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json&_=1544461524785} status=0 QTime=11
  350. 2018-12-10 17:06:18.697 INFO (qtp1571967156-19) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/zookeeper params={path=/clusterstate.json&view=graph&detail=true&wt=json&_=1544461528109} status=0 QTime=7
  351. 2018-12-10 17:06:18.698 INFO (qtp1571967156-21) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&wt=json&_=1544461524785 and sendToOCPQueue=true
  352. 2018-12-10 17:06:18.699 INFO (qtp1571967156-21) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=json&_=1544461524785} status=0 QTime=1
  353. 2018-12-10 17:06:21.902 INFO (qtp1571967156-21) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={indexInfo=false&wt=json&_=1544461581767} status=0 QTime=0
  354. 2018-12-10 17:06:21.912 INFO (qtp1571967156-19) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/zookeeper params={path=/live_nodes&wt=json&_=1544461581872} status=0 QTime=6
  355. 2018-12-10 17:06:21.917 INFO (qtp1571967156-14) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json&_=1544461581768} status=0 QTime=14
  356. 2018-12-10 17:06:21.929 INFO (qtp1571967156-17) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&wt=json&_=1544461581767 and sendToOCPQueue=true
  357. 2018-12-10 17:06:21.930 INFO (qtp1571967156-16) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/zookeeper params={path=/clusterstate.json&view=graph&detail=true&wt=json&_=1544461581872} status=0 QTime=6
  358. 2018-12-10 17:06:21.930 INFO (qtp1571967156-17) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=json&_=1544461581767} status=0 QTime=1
  359. 2018-12-10 17:06:23.526 INFO (qtp1571967156-14) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.main.shard1.replica_n2:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:QUERY./select.requests&key=solr.core.history.shard2.replica_n6:QUERY./select.requests&key=solr.core.history.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.main.shard3.replica_n8:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:INDEX.sizeInBytes} status=0 QTime=560
  360. 2018-12-10 17:06:23.528 INFO (qtp1571967156-17) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=0
  361. 2018-12-10 17:07:25.158 INFO (qtp1571967156-14) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.main.shard1.replica_n2:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:QUERY./select.requests&key=solr.core.history.shard2.replica_n6:QUERY./select.requests&key=solr.core.history.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.main.shard3.replica_n8:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:INDEX.sizeInBytes} status=0 QTime=454
  362. 2018-12-10 17:07:25.161 INFO (qtp1571967156-21) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=0
  363. 2018-12-10 17:08:27.087 INFO (qtp1571967156-17) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.main.shard1.replica_n2:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:QUERY./select.requests&key=solr.core.history.shard2.replica_n6:QUERY./select.requests&key=solr.core.history.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.main.shard3.replica_n8:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:INDEX.sizeInBytes} status=0 QTime=678
  364. 2018-12-10 17:08:27.100 INFO (qtp1571967156-19) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=1
  365. 2018-12-10 17:09:28.816 INFO (qtp1571967156-19) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.main.shard1.replica_n2:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:QUERY./select.requests&key=solr.core.history.shard2.replica_n6:QUERY./select.requests&key=solr.core.history.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.main.shard3.replica_n8:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:INDEX.sizeInBytes} status=0 QTime=463
  366. 2018-12-10 17:09:28.819 INFO (qtp1571967156-20) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=0
  367. 2018-12-10 17:10:30.576 INFO (qtp1571967156-19) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.main.shard1.replica_n2:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:QUERY./select.requests&key=solr.core.history.shard2.replica_n6:QUERY./select.requests&key=solr.core.history.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.main.shard3.replica_n8:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:INDEX.sizeInBytes} status=0 QTime=447
  368. 2018-12-10 17:10:30.579 INFO (qtp1571967156-21) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=0
  369. 2018-12-10 17:11:32.155 INFO (qtp1571967156-19) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.main.shard1.replica_n2:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:QUERY./select.requests&key=solr.core.history.shard2.replica_n6:QUERY./select.requests&key=solr.core.history.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.main.shard3.replica_n8:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:INDEX.sizeInBytes} status=0 QTime=421
  370. 2018-12-10 17:11:32.158 INFO (qtp1571967156-18) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=0
  371. 2018-12-10 17:11:48.281 INFO (qtp1571967156-13) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={indexInfo=false&wt=json&_=1544461905891} status=0 QTime=0
  372. 2018-12-10 17:11:48.284 INFO (qtp1571967156-21) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging params={wt=json&_=1544461908275&since=0} status=0 QTime=0
  373. 2018-12-10 17:11:48.293 INFO (qtp1571967156-16) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json&_=1544461905892} status=0 QTime=10
  374. 2018-12-10 17:11:48.375 INFO (qtp1571967156-13) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&wt=json&_=1544461905891 and sendToOCPQueue=true
  375. 2018-12-10 17:11:48.376 INFO (qtp1571967156-13) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=json&_=1544461905891} status=0 QTime=1
  376. 2018-12-10 17:11:50.428 INFO (qtp1571967156-16) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={indexInfo=false&wt=json&_=1544461905891} status=0 QTime=0
  377. 2018-12-10 17:11:50.440 INFO (qtp1571967156-16) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/zookeeper params={path=/live_nodes&wt=json&_=1544461910407} status=0 QTime=4
  378. 2018-12-10 17:11:50.457 INFO (qtp1571967156-13) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json&_=1544461905892} status=0 QTime=26
  379. 2018-12-10 17:11:50.492 INFO (qtp1571967156-13) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&wt=json&_=1544461905891 and sendToOCPQueue=true
  380. 2018-12-10 17:11:50.493 INFO (qtp1571967156-13) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=json&_=1544461905891} status=0 QTime=0
  381. 2018-12-10 17:11:50.498 INFO (qtp1571967156-21) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/zookeeper params={path=/clusterstate.json&view=graph&detail=true&wt=json&_=1544461910407} status=0 QTime=8
  382. 2018-12-10 17:11:53.779 INFO (qtp1571967156-21) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={indexInfo=false&wt=json&_=1544461913637} status=0 QTime=0
  383. 2018-12-10 17:11:53.788 INFO (qtp1571967156-19) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/zookeeper params={path=/live_nodes&wt=json&_=1544461913743} status=0 QTime=4
  384. 2018-12-10 17:11:53.795 INFO (qtp1571967156-16) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json&_=1544461913637} status=0 QTime=13
  385. 2018-12-10 17:11:53.805 INFO (qtp1571967156-19) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&wt=json&_=1544461913637 and sendToOCPQueue=true
  386. 2018-12-10 17:11:53.805 INFO (qtp1571967156-19) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=json&_=1544461913637} status=0 QTime=0
  387. 2018-12-10 17:11:53.806 INFO (qtp1571967156-21) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/zookeeper params={path=/clusterstate.json&view=graph&detail=true&wt=json&_=1544461913743} status=0 QTime=4
  388. 2018-12-10 17:11:55.132 INFO (qtp1571967156-21) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={indexInfo=false&wt=json&_=1544461915018} status=0 QTime=0
  389. 2018-12-10 17:11:55.137 INFO (qtp1571967156-19) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/zookeeper params={path=/live_nodes&wt=json&_=1544461915097} status=0 QTime=3
  390. 2018-12-10 17:11:55.145 INFO (qtp1571967156-13) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json&_=1544461915019} status=0 QTime=12
  391. 2018-12-10 17:11:55.157 INFO (qtp1571967156-13) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&wt=json&_=1544461915019 and sendToOCPQueue=true
  392. 2018-12-10 17:11:55.158 INFO (qtp1571967156-13) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=json&_=1544461915019} status=0 QTime=1
  393. 2018-12-10 17:11:55.158 INFO (qtp1571967156-19) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/zookeeper params={path=/clusterstate.json&view=graph&detail=true&wt=json&_=1544461915097} status=0 QTime=5
  394. 2018-12-10 17:12:34.187 INFO (qtp1571967156-13) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.main.shard1.replica_n2:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:QUERY./select.requests&key=solr.core.history.shard2.replica_n6:QUERY./select.requests&key=solr.core.history.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.main.shard3.replica_n8:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:INDEX.sizeInBytes} status=0 QTime=858
  395. 2018-12-10 17:12:34.190 INFO (qtp1571967156-16) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=0
  396. 2018-12-10 17:13:36.839 INFO (qtp1571967156-13) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.main.shard1.replica_n2:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:QUERY./select.requests&key=solr.core.history.shard2.replica_n6:QUERY./select.requests&key=solr.core.history.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.main.shard3.replica_n8:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:INDEX.sizeInBytes} status=0 QTime=1502
  397. 2018-12-10 17:13:36.841 INFO (qtp1571967156-21) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=0
  398. 2018-12-10 17:14:39.696 INFO (qtp1571967156-20) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.main.shard1.replica_n2:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:QUERY./select.requests&key=solr.core.history.shard2.replica_n6:QUERY./select.requests&key=solr.core.history.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.main.shard3.replica_n8:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:INDEX.sizeInBytes} status=0 QTime=1648
  399. 2018-12-10 17:14:39.699 INFO (qtp1571967156-13) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=1
  400. 2018-12-10 17:15:42.226 INFO (qtp1571967156-19) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.main.shard1.replica_n2:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:QUERY./select.requests&key=solr.core.history.shard2.replica_n6:QUERY./select.requests&key=solr.core.history.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.main.shard3.replica_n8:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:QUERY./select.requests&key=solr.core.main.shard3.replica_n8:UPDATE./update.requests&key=solr.core.history.shard1.replica_n1:UPDATE./update.requests&key=solr.core.main.shard1.replica_n2:INDEX.sizeInBytes&key=solr.core.history.shard2.replica_n6:INDEX.sizeInBytes} status=0 QTime=1381
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!