spuder

logstash kafka looses messages

Apr 14th, 2015
291
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 1.53 KB | None | 0 0
  1. A product called aspire pushes to a kafka message queue. Notice how there are 2 types of messages. "index" "msgDoc".
  2.  
  3.  
  4. {"index":{"_type":"","_index":"1429048344266Dispatcher"}}
  5. {"msgDoc":{"message":"[ASPIRE01] Dispatcher request for batch ID:DE-7621 is completed.","aspireApplication":"Dispatcher","msgType":""}}
  6.  
  7. {"index":{"_type":"","_index":"1429048353764Dispatcher"}}
  8. {"msgDoc":{"message":"[ASPIRE01] An add/update request is added to Solr batch for batch ID:DE-7627 and for DocID:556923086 and Solr Instance:http://192.0.2.0:9999/solr/core02","aspireApplication":"Dispatcher","msgType":""}}
  9.  
  10. Logstash only ever receives the 'index' messages
  11.  
  12. {
  13. "index" => {
  14. "_type" => "ERROR",
  15. "_index" => "1429048501894ERRORDispatcher"
  16. },
  17. "@version" => "1",
  18. "@timestamp" => "2015-04-14T21:55:26.721Z"
  19. }
  20. {
  21. "index" => {
  22. "_type" => "ERROR",
  23. "_index" => "1429048547347ERRORDispatcher"
  24. },
  25. "@version" => "1",
  26. "@timestamp" => "2015-04-14T21:56:12.170Z"
  27. }
  28. {
  29. "index" => {
  30. "_type" => "ERROR",
  31. "_index" => "1429048547388ERRORDispatcher"
  32. },
  33. "@version" => "1",
  34. "@timestamp" => "2015-04-14T21:56:12.214Z"
  35. }
  36.  
  37.  
  38. logstash config
  39.  
  40. input {
  41. kafka {
  42. zk_connect => 'zoo01.example.com:2181/kafka'
  43. topic_id => 'foobar'
  44. }
  45. }
  46. output {
  47. stdout {
  48. codec => rubydebug
  49. }
  50. }
  51. output {
  52. elasticsearch {
  53. host => 'elasticsearchpool.example.com'
  54. cluster => 'foobar'
  55. embedded => false
  56. protocol => 'http'
  57. }
  58. }
Advertisement
Add Comment
Please, Sign In to add comment