Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- A product called aspire pushes to a kafka message queue. Notice how there are 2 types of messages. "index" "msgDoc".
- {"index":{"_type":"","_index":"1429048344266Dispatcher"}}
- {"msgDoc":{"message":"[ASPIRE01] Dispatcher request for batch ID:DE-7621 is completed.","aspireApplication":"Dispatcher","msgType":""}}
- {"index":{"_type":"","_index":"1429048353764Dispatcher"}}
- {"msgDoc":{"message":"[ASPIRE01] An add/update request is added to Solr batch for batch ID:DE-7627 and for DocID:556923086 and Solr Instance:http://192.0.2.0:9999/solr/core02","aspireApplication":"Dispatcher","msgType":""}}
- Logstash only ever receives the 'index' messages
- {
- "index" => {
- "_type" => "ERROR",
- "_index" => "1429048501894ERRORDispatcher"
- },
- "@version" => "1",
- "@timestamp" => "2015-04-14T21:55:26.721Z"
- }
- {
- "index" => {
- "_type" => "ERROR",
- "_index" => "1429048547347ERRORDispatcher"
- },
- "@version" => "1",
- "@timestamp" => "2015-04-14T21:56:12.170Z"
- }
- {
- "index" => {
- "_type" => "ERROR",
- "_index" => "1429048547388ERRORDispatcher"
- },
- "@version" => "1",
- "@timestamp" => "2015-04-14T21:56:12.214Z"
- }
- logstash config
- input {
- kafka {
- zk_connect => 'zoo01.example.com:2181/kafka'
- topic_id => 'foobar'
- }
- }
- output {
- stdout {
- codec => rubydebug
- }
- }
- output {
- elasticsearch {
- host => 'elasticsearchpool.example.com'
- cluster => 'foobar'
- embedded => false
- protocol => 'http'
- }
- }
Advertisement
Add Comment
Please, Sign In to add comment