API tools faq
paste
sohil

ElasticSearch Exception log file

sohil
Sep 7th, 2015
291
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 66.04 KB | None | 0 0
raw download clone embed print report
  1. [2015-09-07 00:00:01,396][INFO ][index.shard.service ] [Sohil11003] [blks][0] updating refresh_interval from [-1] to [1s]
  2. [2015-09-07 00:00:01,397][INFO ][index.shard.service ] [Sohil11003] [blks][3] updating refresh_interval from [-1] to [1s]
  3. [2015-09-07 00:00:01,397][INFO ][index.shard.service ] [Sohil11003] [blks][1] updating refresh_interval from [-1] to [1s]
  4. [2015-09-07 00:00:01,397][INFO ][index.shard.service ] [Sohil11003] [blks][2] updating refresh_interval from [-1] to [1s]
  5. [2015-09-07 00:00:01,397][INFO ][index.shard.service ] [Sohil11003] [blks][4] updating refresh_interval from [-1] to [1s]
  6. [2015-09-07 00:00:01,429][INFO ][river.jdbc.RiverMetrics ] pipeline org.xbib.elasticsearch.plugin.jdbc.RiverPipeline@4149d514 complete: river jdbc/update_blk_jdbc_river metrics: 109002 rows, 0.6819849669847429 mean, (0.0 0.0 0.0), ingest metrics: elapsed 1 second, 2.66 MB bytes, 25.0 bytes avg, 1.702 MB/s
  7. [2015-09-07 00:44:16,269][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172519][20195] duration [764ms], collections [1]/[1.4s], total [764ms]/[34.5m], memory [2.2gb]->[2.2gb]/[15.9gb], all_pools {[young] [87.6mb]->[7.4kb]/[266.2mb]}{[survivor] [32.7mb]->[33.2mb]/[33.2mb]}{[old] [2.1gb]->[2.2gb]/[15.6gb]}
  8. [2015-09-07 00:44:17,734][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172520][20196] duration [765ms], collections [1]/[1.4s], total [765ms]/[34.5m], memory [2.2gb]->[2.4gb]/[15.9gb], all_pools {[young] [7.4kb]->[3.3mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [2.2gb]->[2.4gb]/[15.6gb]}
  9. [2015-09-07 00:44:19,414][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172521][20197] duration [818ms], collections [1]/[1.6s], total [818ms]/[34.6m], memory [2.4gb]->[2.6gb]/[15.9gb], all_pools {[young] [3.3mb]->[939.1kb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [2.4gb]->[2.5gb]/[15.6gb]}
  10. [2015-09-07 00:44:33,678][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172535][20206] duration [871ms], collections [1]/[1s], total [871ms]/[34.6m], memory [3.1gb]->[3.1gb]/[15.9gb], all_pools {[young] [250.1mb]->[450.7kb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [2.8gb]->[3gb]/[15.6gb]}
  11. [2015-09-07 00:44:35,074][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172536][20207] duration [980ms], collections [1]/[1.3s], total [980ms]/[34.6m], memory [3.1gb]->[3.3gb]/[15.9gb], all_pools {[young] [450.7kb]->[559.3kb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [3gb]->[3.3gb]/[15.6gb]}
  12. [2015-09-07 00:44:40,196][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172541][20211] duration [750ms], collections [1]/[1s], total [750ms]/[34.7m], memory [3.7gb]->[3.8gb]/[15.9gb], all_pools {[young] [57.2mb]->[6.1kb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [3.6gb]->[3.8gb]/[15.6gb]}
  13. [2015-09-07 00:44:42,687][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172543][20213] duration [955ms], collections [1]/[1.4s], total [955ms]/[34.7m], memory [3.9gb]->[4.1gb]/[15.9gb], all_pools {[young] [280kb]->[1mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [3.8gb]->[4.1gb]/[15.6gb]}
  14. [2015-09-07 00:44:43,775][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172544][20214] duration [729ms], collections [1]/[1s], total [729ms]/[34.7m], memory [4.1gb]->[4.3gb]/[15.9gb], all_pools {[young] [1mb]->[912.9kb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [4.1gb]->[4.2gb]/[15.6gb]}
  15. [2015-09-07 00:44:48,010][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172548][20217] duration [849ms], collections [1]/[1.2s], total [849ms]/[34.7m], memory [4.7gb]->[4.7gb]/[15.9gb], all_pools {[young] [182.1mb]->[2.8mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [4.5gb]->[4.7gb]/[15.6gb]}
  16. [2015-09-07 00:44:49,346][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172549][20218] duration [989ms], collections [1]/[1.3s], total [989ms]/[34.7m], memory [4.7gb]->[4.9gb]/[15.9gb], all_pools {[young] [2.8mb]->[1.3mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [4.7gb]->[4.9gb]/[15.6gb]}
  17. [2015-09-07 00:44:55,860][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172555][20224] duration [776ms], collections [1]/[1.4s], total [776ms]/[34.8m], memory [5.4gb]->[5.5gb]/[15.9gb], all_pools {[young] [59.1mb]->[676.2kb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [5.3gb]->[5.5gb]/[15.6gb]}
  18. [2015-09-07 00:44:57,863][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172557][20226] duration [709ms], collections [1]/[1s], total [709ms]/[34.8m], memory [5.8gb]->[5.8gb]/[15.9gb], all_pools {[young] [189.5mb]->[44.2mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [5.6gb]->[5.7gb]/[15.6gb]}
  19. [2015-09-07 00:45:00,930][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172560][20229] duration [984ms], collections [1]/[1s], total [984ms]/[34.8m], memory [6.1gb]->[6.1gb]/[15.9gb], all_pools {[young] [230.6mb]->[76.1kb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [5.8gb]->[6.1gb]/[15.6gb]}
  20. [2015-09-07 00:45:03,399][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172562][20232] duration [839ms], collections [1]/[1.3s], total [839ms]/[34.8m], memory [6.2gb]->[6.4gb]/[15.9gb], all_pools {[young] [39mb]->[516.9kb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [6.1gb]->[6.3gb]/[15.6gb]}
  21. [2015-09-07 00:45:04,952][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172563][20233] duration [917ms], collections [1]/[1.5s], total [917ms]/[34.9m], memory [6.4gb]->[6.6gb]/[15.9gb], all_pools {[young] [516.9kb]->[2.1mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [6.3gb]->[6.5gb]/[15.6gb]}
  22. [2015-09-07 00:45:11,769][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172569][20239] duration [851ms], collections [1]/[1s], total [851ms]/[34.9m], memory [7.2gb]->[7.2gb]/[15.9gb], all_pools {[young] [188.5mb]->[2.2mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [6.9gb]->[7.1gb]/[15.6gb]}
  23. [2015-09-07 00:45:13,485][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172570][20240] duration [733ms], collections [1]/[1.7s], total [733ms]/[34.9m], memory [7.2gb]->[7.3gb]/[15.9gb], all_pools {[young] [2.2mb]->[530.1kb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [7.1gb]->[7.3gb]/[15.6gb]}
  24. [2015-09-07 00:45:15,149][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172571][20241] duration [755ms], collections [1]/[1.6s], total [755ms]/[34.9m], memory [7.3gb]->[7.5gb]/[15.9gb], all_pools {[young] [530.1kb]->[1.3mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [7.3gb]->[7.4gb]/[15.6gb]}
  25. [2015-09-07 00:45:16,593][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172572][20242] duration [896ms], collections [1]/[1.4s], total [896ms]/[34.9m], memory [7.5gb]->[7.7gb]/[15.9gb], all_pools {[young] [1.3mb]->[6.1kb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [7.4gb]->[7.6gb]/[15.6gb]}
  26. [2015-09-07 00:45:18,147][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172573][20243] duration [879ms], collections [1]/[1.5s], total [879ms]/[35m], memory [7.7gb]->[7.8gb]/[15.9gb], all_pools {[young] [6.1kb]->[10.1kb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [7.6gb]->[7.8gb]/[15.6gb]}
  27. [2015-09-07 00:45:19,705][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172574][20244] duration [943ms], collections [1]/[1.5s], total [943ms]/[35m], memory [7.8gb]->[8gb]/[15.9gb], all_pools {[young] [10.1kb]->[22.3kb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [7.8gb]->[8gb]/[15.6gb]}
  28. [2015-09-07 00:45:21,410][WARN ][monitor.jvm ] [Sohil11003] [gc][young][172575][20245] duration [1s], collections [1]/[1.7s], total [1s]/[35m], memory [8gb]->[8.3gb]/[15.9gb], all_pools {[young] [22.3kb]->[10.1kb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [8gb]->[8.2gb]/[15.6gb]}
  29. [2015-09-07 00:45:26,082][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172579][20249] duration [925ms], collections [1]/[1.4s], total [925ms]/[35m], memory [8.6gb]->[8.7gb]/[15.9gb], all_pools {[young] [115.6mb]->[2.1mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [8.5gb]->[8.7gb]/[15.6gb]}
  30. [2015-09-07 00:45:34,786][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172586][20256] duration [703ms], collections [1]/[1.4s], total [703ms]/[35.1m], memory [9.4gb]->[9.5gb]/[15.9gb], all_pools {[young] [1.3mb]->[19.1mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [9.3gb]->[9.5gb]/[15.6gb]}
  31. [2015-09-07 00:45:40,788][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172591][20261] duration [852ms], collections [1]/[1.3s], total [852ms]/[35.1m], memory [10gb]->[10.1gb]/[15.9gb], all_pools {[young] [47.1mb]->[393.8kb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [9.9gb]->[10.1gb]/[15.6gb]}
  32. [2015-09-07 00:45:45,193][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172595][20265] duration [744ms], collections [1]/[1.4s], total [744ms]/[35.2m], memory [10.5gb]->[10.6gb]/[15.9gb], all_pools {[young] [59.4mb]->[764.9kb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [10.4gb]->[10.5gb]/[15.6gb]}
  33. [2015-09-07 00:45:46,743][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172596][20266] duration [775ms], collections [1]/[1.5s], total [775ms]/[35.2m], memory [10.6gb]->[10.7gb]/[15.9gb], all_pools {[young] [764.9kb]->[2.1mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [10.5gb]->[10.7gb]/[15.6gb]}
  34. [2015-09-07 00:46:14,499][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172620][20291] duration [789ms], collections [1]/[1.5s], total [789ms]/[35.4m], memory [13.1gb]->[13.3gb]/[15.9gb], all_pools {[young] [1.1mb]->[8.4mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [13.1gb]->[13.3gb]/[15.6gb]}
  35. [2015-09-07 00:46:18,182][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172623][20294] duration [778ms], collections [1]/[1.4s], total [778ms]/[35.4m], memory [13.5gb]->[13.7gb]/[15.9gb], all_pools {[young] [4kb]->[32.8mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [13.4gb]->[13.6gb]/[15.6gb]}
  36. [2015-09-07 00:46:19,722][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172624][20295] duration [836ms], collections [1]/[1.5s], total [836ms]/[35.4m], memory [13.7gb]->[13.9gb]/[15.9gb], all_pools {[young] [32.8mb]->[51mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [13.6gb]->[13.8gb]/[15.6gb]}
  37. [2015-09-07 00:46:21,086][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172625][20296] duration [732ms], collections [1]/[1.3s], total [732ms]/[35.4m], memory [13.9gb]->[14gb]/[15.9gb], all_pools {[young] [51mb]->[4kb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [13.8gb]->[13.9gb]/[15.6gb]}
  38. [2015-09-07 00:46:22,688][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172626][20297] duration [875ms], collections [1]/[1.6s], total [875ms]/[35.4m], memory [14gb]->[14.2gb]/[15.9gb], all_pools {[young] [4kb]->[4kb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [13.9gb]->[14.1gb]/[15.6gb]}
  39. [2015-09-07 00:46:24,334][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172627][20298] duration [828ms], collections [1]/[1.6s], total [828ms]/[35.5m], memory [14.2gb]->[14.4gb]/[15.9gb], all_pools {[young] [4kb]->[14.4mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [14.1gb]->[14.3gb]/[15.6gb]}
  40. [2015-09-07 00:46:28,127][INFO ][monitor.jvm ] [Sohil11003] [gc][young][172630][20301] duration [806ms], collections [1]/[1.5s], total [806ms]/[35.5m], memory [14.6gb]->[14.8gb]/[15.9gb], all_pools {[young] [2kb]->[2.1mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old] [14.6gb]->[14.8gb]/[15.6gb]}
  41. [2015-09-07 00:46:56,704][WARN ][monitor.jvm ] [Sohil11003] [gc][young][172633][20304] duration [10.9s], collections [1]/[26.2s], total [10.9s]/[35.7m], memory [15.1gb]->[687mb]/[15.9gb], all_pools {[young] [27.6mb]->[39.1mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old] [15.1gb]->[647.9mb]/[15.6gb]}
  42. [2015-09-07 00:46:56,731][WARN ][monitor.jvm ] [Sohil11003] [gc][old][172633][26] duration [14.7s], collections [1]/[26.2s], total [14.7s]/[39.5s], memory [15.1gb]->[687mb]/[15.9gb], all_pools {[young] [27.6mb]->[39.1mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old] [15.1gb]->[647.9mb]/[15.6gb]}
  43. [2015-09-07 00:46:56,827][DEBUG][action.search.type ] [Sohil11003] [89471] Failed to execute fetch phase
  44. org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@76ef4308
  45. at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
  46. at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
  47. at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
  48. at org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:551)
  49. at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:449)
  50. at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:431)
  51. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeFetch(TransportSearchDfsQueryThenFetchAction.java:167)
  52. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.innerExecuteFetchPhase(TransportSearchDfsQueryThenFetchAction.java:162)
  53. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeFetchPhase(TransportSearchDfsQueryThenFetchAction.java:138)
  54. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$1.onResult(TransportSearchDfsQueryThenFetchAction.java:110)
  55. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$1.onResult(TransportSearchDfsQueryThenFetchAction.java:104)
  56. at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:568)
  57. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  58. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  59. at java.lang.Thread.run(Thread.java:745)
  60. [2015-09-07 00:46:56,828][DEBUG][action.search.type ] [Sohil11003] [blks][2], node[-y2z8tICS_6mywTPw77x2A], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@259a0352] lastShard [true]
  61. org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@61529db2
  62. at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
  63. at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
  64. at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
  65. at org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:551)
  66. at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteDfs(SearchServiceTransportAction.java:194)
  67. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchDfsQueryThenFetchAction.java:88)
  68. at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:171)
  69. at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:153)
  70. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction.doExecute(TransportSearchDfsQueryThenFetchAction.java:65)
  71. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction.doExecute(TransportSearchDfsQueryThenFetchAction.java:55)
  72. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
  73. at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101)
  74. at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:45)
  75. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
  76. at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:63)
  77. at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:51)
  78. at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:217)
  79. at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)
  80. at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  81. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  82. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  83. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
  84. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
  85. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
  86. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
  87. at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  88. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  89. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  90. at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
  91. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  92. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
  93. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
  94. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
  95. at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
  96. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
  97. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
  98. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
  99. at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
  100. at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  101. at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  102. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  103. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  104. at java.lang.Thread.run(Thread.java:745)
  105. [2015-09-07 00:46:56,829][DEBUG][action.search.type ] [Sohil11003] [blks][3], node[-y2z8tICS_6mywTPw77x2A], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@259a0352] lastShard [true]
  106. org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@15cf5ff7
  107. at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
  108. at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
  109. at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
  110. at org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:551)
  111. at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteDfs(SearchServiceTransportAction.java:194)
  112. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchDfsQueryThenFetchAction.java:88)
  113. at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:171)
  114. at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:153)
  115. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction.doExecute(TransportSearchDfsQueryThenFetchAction.java:65)
  116. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction.doExecute(TransportSearchDfsQueryThenFetchAction.java:55)
  117. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
  118. at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101)
  119. at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:45)
  120. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
  121. at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:63)
  122. at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:51)
  123. at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:217)
  124. at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)
  125. at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  126. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  127. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  128. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
  129. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
  130. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
  131. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
  132. at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  133. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  134. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  135. at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
  136. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  137. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
  138. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
  139. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
  140. at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
  141. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
  142. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
  143. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
  144. at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
  145. at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  146. at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  147. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  148. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  149. at java.lang.Thread.run(Thread.java:745)
  150. [2015-09-07 00:46:56,830][DEBUG][action.search.type ] [Sohil11003] [blks][4], node[-y2z8tICS_6mywTPw77x2A], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@259a0352] lastShard [true]
  151. org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@2f4fba45
  152. at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
  153. at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
  154. at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
  155. at org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:551)
  156. at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteDfs(SearchServiceTransportAction.java:194)
  157. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchDfsQueryThenFetchAction.java:88)
  158. at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:171)
  159. at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:153)
  160. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction.doExecute(TransportSearchDfsQueryThenFetchAction.java:65)
  161. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction.doExecute(TransportSearchDfsQueryThenFetchAction.java:55)
  162. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
  163. at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101)
  164. at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:45)
  165. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
  166. at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:63)
  167. at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:51)
  168. at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:217)
  169. at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)
  170. at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  171. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  172. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  173. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
  174. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
  175. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
  176. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
  177. at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  178. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  179. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  180. at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
  181. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  182. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
  183. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
  184. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
  185. at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
  186. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
  187. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
  188. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
  189. at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
  190. at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  191. at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  192. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  193. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  194. at java.lang.Thread.run(Thread.java:745)
  195. [2015-09-07 00:46:56,832][DEBUG][action.search.type ] [Sohil11003] [blks][0], node[-y2z8tICS_6mywTPw77x2A], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@6a4bd6f5] lastShard [true]
  196. org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@56cdf010
  197. at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
  198. at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
  199. at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
  200. at org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:551)
  201. at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteDfs(SearchServiceTransportAction.java:194)
  202. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchDfsQueryThenFetchAction.java:88)
  203. at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:171)
  204. at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:153)
  205. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction.doExecute(TransportSearchDfsQueryThenFetchAction.java:65)
  206. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction.doExecute(TransportSearchDfsQueryThenFetchAction.java:55)
  207. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
  208. at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101)
  209. at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:45)
  210. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
  211. at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:63)
  212. at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:51)
  213. at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:217)
  214. at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)
  215. at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  216. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  217. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  218. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
  219. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
  220. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
  221. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
  222. at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  223. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  224. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  225. at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
  226. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  227. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
  228. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
  229. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
  230. at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
  231. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
  232. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
  233. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
  234. at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
  235. at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  236. at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  237. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  238. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  239. at java.lang.Thread.run(Thread.java:745)
  240. [2015-09-07 00:46:56,833][DEBUG][action.search.type ] [Sohil11003] [blks][1], node[-y2z8tICS_6mywTPw77x2A], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@6a4bd6f5] lastShard [true]
  241. org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@682a3adc
  242. at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
  243. at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
  244. at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
  245. at org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:551)
  246. at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteDfs(SearchServiceTransportAction.java:194)
  247. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchDfsQueryThenFetchAction.java:88)
  248. at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:171)
  249. at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:153)
  250. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction.doExecute(TransportSearchDfsQueryThenFetchAction.java:65)
  251. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction.doExecute(TransportSearchDfsQueryThenFetchAction.java:55)
  252. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
  253. at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101)
  254. at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:45)
  255. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
  256. at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:63)
  257. at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:51)
  258. at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:217)
  259. at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)
  260. at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  261. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  262. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  263. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
  264. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
  265. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
  266. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
  267. at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  268. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  269. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  270. at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
  271. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  272. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
  273. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
  274. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
  275. at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
  276. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
  277. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
  278. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
  279. at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
  280. at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  281. at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  282. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  283. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  284. at java.lang.Thread.run(Thread.java:745)
  285. [2015-09-07 00:46:56,834][DEBUG][action.search.type ] [Sohil11003] [blks][2], node[-y2z8tICS_6mywTPw77x2A], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@6a4bd6f5] lastShard [true]
  286. org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@22b2df09
  287. at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
  288. at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
  289. at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
  290. at org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:551)
  291. at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteDfs(SearchServiceTransportAction.java:194)
  292. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchDfsQueryThenFetchAction.java:88)
  293. at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:171)
  294. at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:153)
  295. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction.doExecute(TransportSearchDfsQueryThenFetchAction.java:65)
  296. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction.doExecute(TransportSearchDfsQueryThenFetchAction.java:55)
  297. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
  298. at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101)
  299. at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:45)
  300. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
  301. at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:63)
  302. at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:51)
  303. at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:217)
  304. at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)
  305. at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  306. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  307. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  308. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
  309. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
  310. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
  311. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
  312. at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  313. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  314. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  315. at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
  316. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  317. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
  318. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
  319. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
  320. at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
  321. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
  322. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
  323. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
  324. at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
  325. at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  326. at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  327. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  328. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  329. at java.lang.Thread.run(Thread.java:745)
  330. [2015-09-07 00:46:56,840][DEBUG][action.search.type ] [Sohil11003] [blks][3], node[-y2z8tICS_6mywTPw77x2A], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@6a4bd6f5] lastShard [true]
  331. org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@5f1424c5
  332. at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
  333. at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
  334. at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
  335. at org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:551)
  336. at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteDfs(SearchServiceTransportAction.java:194)
  337. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchDfsQueryThenFetchAction.java:88)
  338. at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:171)
  339. at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:153)
  340. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction.doExecute(TransportSearchDfsQueryThenFetchAction.java:65)
  341. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction.doExecute(TransportSearchDfsQueryThenFetchAction.java:55)
  342. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
  343. at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101)
  344. at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:45)
  345. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
  346. at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:63)
  347. at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:51)
  348. at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:217)
  349. at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)
  350. at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  351. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  352. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  353. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
  354. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
  355. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
  356. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
  357. at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  358. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  359. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  360. at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
  361. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  362. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
  363. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
  364. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
  365. at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
  366. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
  367. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
  368. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
  369. at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
  370. at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  371. at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  372. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  373. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  374. at java.lang.Thread.run(Thread.java:745)
  375. [2015-09-07 00:46:56,841][DEBUG][action.search.type ] [Sohil11003] [blks][4], node[-y2z8tICS_6mywTPw77x2A], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@6a4bd6f5]
  376. org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@2b7ae583
  377. at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
  378. at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
  379. at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
  380. at org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:551)
  381. at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteDfs(SearchServiceTransportAction.java:194)
  382. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchDfsQueryThenFetchAction.java:88)
  383. at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:171)
  384. at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:153)
  385. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction.doExecute(TransportSearchDfsQueryThenFetchAction.java:65)
  386. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction.doExecute(TransportSearchDfsQueryThenFetchAction.java:55)
  387. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
  388. at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101)
  389. at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:45)
  390. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
  391. at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:63)
  392. at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:51)
  393. at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:217)
  394. at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)
  395. at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  396. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  397. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  398. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
  399. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
  400. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
  401. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
  402. at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  403. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  404. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  405. at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
  406. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  407. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
  408. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
  409. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
  410. at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
  411. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
  412. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
  413. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
  414. at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
  415. at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  416. at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  417. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  418. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  419. at java.lang.Thread.run(Thread.java:745)
  420. [2015-09-07 00:46:56,842][DEBUG][action.search.type ] [Sohil11003] All shards failed for phase: [dfs]
  421. [2015-09-07 00:46:56,828][DEBUG][action.search.type ] [Sohil11003] [89472] Failed to execute fetch phase
  422. org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@22500fc7
  423. at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
  424. at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
  425. at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
  426. at org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:551)
  427. at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:449)
  428. at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:431)
  429. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeFetch(TransportSearchDfsQueryThenFetchAction.java:167)
  430. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.innerExecuteFetchPhase(TransportSearchDfsQueryThenFetchAction.java:162)
  431. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeFetchPhase(TransportSearchDfsQueryThenFetchAction.java:138)
  432. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$1.onResult(TransportSearchDfsQueryThenFetchAction.java:110)
  433. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$1.onResult(TransportSearchDfsQueryThenFetchAction.java:104)
  434. at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:568)
  435. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  436. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  437. at java.lang.Thread.run(Thread.java:745)
  438. [2015-09-07 00:46:56,827][DEBUG][action.search.type ] [Sohil11003] [blks][0], node[-y2z8tICS_6mywTPw77x2A], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@566f4ebb] lastShard [true]
  439. org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@5619ff38
  440. at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
  441. at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
  442. at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
  443. at org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:551)
  444. at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteDfs(SearchServiceTransportAction.java:194)
  445. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchDfsQueryThenFetchAction.java:88)
  446. at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:171)
  447. at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:153)
  448. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction.doExecute(TransportSearchDfsQueryThenFetchAction.java:65)
  449. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction.doExecute(TransportSearchDfsQueryThenFetchAction.java:55)
  450. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
  451. at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101)
  452. at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:45)
  453. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
  454. at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:63)
  455. at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:51)
  456. at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:217)
  457. at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)
  458. at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  459. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  460. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  461. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
  462. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
  463. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
  464. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
  465. at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  466. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  467. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  468. at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
  469. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  470. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
  471. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
  472. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
  473. at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
  474. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
  475. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
  476. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
  477. at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
  478. at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  479. at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  480. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  481. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  482. at java.lang.Thread.run(Thread.java:745)
  483. [2015-09-07 00:46:56,846][DEBUG][action.search.type ] [Sohil11003] [blks][1], node[-y2z8tICS_6mywTPw77x2A], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@566f4ebb] lastShard [true]
  484. org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@3fb65ed5
  485. at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
  486. at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
  487. at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
  488. at org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:551)
  489. at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteDfs(SearchServiceTransportAction.java:194)
  490. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchDfsQueryThenFetchAction.java:88)
  491. at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:171)
  492. at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:153)
  493. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction.doExecute(TransportSearchDfsQueryThenFetchAction.java:65)
  494. at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction.doExecute(TransportSearchDfsQueryThenFetchAction.java:55)
  495. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
  496. at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101)
  497. at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:45)
  498. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
  499. at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:63)
  500. at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:51)
  501. at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:217)
  502. at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)
  503. at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  504. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  505. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  506. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
  507. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
  508. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
  509. at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
  510. at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  511. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  512. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  513. at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
  514. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  515. at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
  516. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
  517. at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
  518. at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
  519. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
  520. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
  521. at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
  522. at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
  523. at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  524. at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  525. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  526. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  527. at java.lang.Thread.run(Thread.java:745)
  528. [2015-09-07 00:46:56,847][DEBUG][action.search.type ] [Sohil11003] [blks][2], node[-y2z8tICS_6mywTPw77x2A], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@566f4ebb] lastShard [true]
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!