API tools faq
paste
spuder

kafka-crash

spuder
Mar 19th, 2015
401
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 7.88 KB | None | 0 0
raw download clone embed print report
  1. tail of kafka.log file. I have 3 kafka nodes, and two of them crashed overnight. Smells like a dns problem since I renamed the nodes from `kafka01.example.com` to `swat-kafka01.example.com`
  2.  
  3. [2015-03-19 03:29:32,785] INFO 1664195520 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
  4. [2015-03-19 03:29:33,278] INFO New leader is 1664195520 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
  5. [2015-03-19 03:29:33,315] INFO Completed load of log elasticsearch-1 with log end offset 0 (kafka.log.Log)
  6. [2015-03-19 03:29:33,337] INFO Created log for partition [elasticsearch,1] in /tmp/kafka-logs with properties {segment.index.bytes -> 10485760, file.delete.delay.ms -> 60000, segment.bytes -> 1073741824, flush.ms -> 9223372036854775807, delete.retention.ms -> 86400000, index.interval.bytes -> 4096, retention.bytes -> -1, cleanup.policy -> delete, segment.ms -> 604800000, max.message.bytes -> 1000012, flush.messages -> 9223372036854775807, min.cleanable.dirty.ratio -> 0.5, retention.ms -> 604800000}. (kafka.log.LogManager)
  7. [2015-03-19 03:29:33,343] WARN Partition [elasticsearch,1] on broker 1664195520: No checkpointed highwatermark is found for partition [elasticsearch,1] (kafka.cluster.Partition)
  8. [2015-03-19 03:29:33,354] INFO [ReplicaFetcherManager on broker 1664195520] Removed fetcher for partitions [elasticsearch,1],[elasticsearch,0],[elasticsearch,2] (kafka.server.ReplicaFetcherManager)
  9. [2015-03-19 03:29:33,361] INFO Completed load of log elasticsearch-0 with log end offset 0 (kafka.log.Log)
  10. [2015-03-19 03:29:33,361] INFO Created log for partition [elasticsearch,0] in /tmp/kafka-logs with properties {segment.index.bytes -> 10485760, file.delete.delay.ms -> 60000, segment.bytes -> 1073741824, flush.ms -> 9223372036854775807, delete.retention.ms -> 86400000, index.interval.bytes -> 4096, retention.bytes -> -1, cleanup.policy -> delete, segment.ms -> 604800000, max.message.bytes -> 1000012, flush.messages -> 9223372036854775807, min.cleanable.dirty.ratio -> 0.5, retention.ms -> 604800000}. (kafka.log.LogManager)
  11. [2015-03-19 03:29:33,362] WARN Partition [elasticsearch,0] on broker 1664195520: No checkpointed highwatermark is found for partition [elasticsearch,0] (kafka.cluster.Partition)
  12. [2015-03-19 03:29:33,373] INFO Completed load of log elasticsearch-2 with log end offset 0 (kafka.log.Log)
  13. [2015-03-19 03:29:33,374] INFO Created log for partition [elasticsearch,2] in /tmp/kafka-logs with properties {segment.index.bytes -> 10485760, file.delete.delay.ms -> 60000, segment.bytes -> 1073741824, flush.ms -> 9223372036854775807, delete.retention.ms -> 86400000, index.interval.bytes -> 4096, retention.bytes -> -1, cleanup.policy -> delete, segment.ms -> 604800000, max.message.bytes -> 1000012, flush.messages -> 9223372036854775807, min.cleanable.dirty.ratio -> 0.5, retention.ms -> 604800000}. (kafka.log.LogManager)
  14. [2015-03-19 03:29:33,374] WARN Partition [elasticsearch,2] on broker 1664195520: No checkpointed highwatermark is found for partition [elasticsearch,2] (kafka.cluster.Partition)
  15. [2015-03-19 03:29:33,375] INFO Truncating log elasticsearch-1 to offset 0. (kafka.log.Log)
  16. [2015-03-19 03:29:33,377] INFO Truncating log elasticsearch-0 to offset 0. (kafka.log.Log)
  17. [2015-03-19 03:29:33,377] INFO Truncating log elasticsearch-2 to offset 0. (kafka.log.Log)
  18. [2015-03-19 03:29:33,435] INFO [ReplicaFetcherThread-0-1664195521], Starting (kafka.server.ReplicaFetcherThread)
  19. [2015-03-19 03:29:33,437] INFO [ReplicaFetcherManager on broker 1664195520] Added fetcher for partitions ArrayBuffer([[elasticsearch,1], initOffset 0 to broker id:1664195521,host:kafka03.example.com,port:6667] ) (kafka.server.ReplicaFetcherManager)
  20. [2015-03-19 03:29:33,483] ERROR [ReplicaFetcherThread-0-1664195521], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 0; ClientId: ReplicaFetcherThread-0-1664195521; ReplicaId: 1664195520; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo: [elasticsearch,1] -> PartitionFetchInfo(0,1048576) (kafka.server.ReplicaFetcherThread)
  21. java.nio.channels.UnresolvedAddressException
  22. at sun.nio.ch.Net.checkAddress(Net.java:127)
  23. at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:644)
  24. at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
  25. at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:44)
  26. at kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:142)
  27. at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:69)
  28. at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:109)
  29. at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:109)
  30. at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:109)
  31. at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
  32. at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:108)
  33. at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:108)
  34. at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:108)
  35. at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
  36. at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:107)
  37. at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
  38. at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
  39. at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
  40. [2015-03-19 03:29:33,512] WARN Reconnect due to socket error: null (kafka.consumer.SimpleConsumer)
  41. [2015-03-19 03:29:33,513] ERROR [ReplicaFetcherThread-0-1664195521], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 1; ClientId: ReplicaFetcherThread-0-1664195521; ReplicaId: 1664195520; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo: [elasticsearch,1] -> PartitionFetchInfo(0,1048576) (kafka.server.ReplicaFetcherThread)
  42. java.nio.channels.UnresolvedAddressException
  43. at sun.nio.ch.Net.checkAddress(Net.java:127)
  44. at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:644)
  45. at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
  46. at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:44)
  47. at kafka.consumer.SimpleConsumer.reconnect(SimpleConsumer.scala:57)
  48. at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:79)
  49. at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
  50. at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:109)
  51. at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:109)
  52. at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:109)
  53. at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
  54. at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:108)
  55. at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:108)
  56. at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:108)
  57. at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
  58. at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:107)
  59. at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
  60. at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
  61. at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
  62. [2015-03-19 03:29:33,514] WARN Reconnect due to socket error: null (kafka.consumer.SimpleConsumer)
  63. [2015-03-19 03:29:33,514] INFO [ReplicaFetcherManager on broker 1664195520] Removed fetcher for partitions [elasticsearch,0] (kafka.server.ReplicaFe/var/log/kafka/kafka.log
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!