Advertisement
Guest User

es_rollup

a guest
May 5th, 2022
29
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. # This paste shows that Elasticsearch (at least versions 7.17 and 8.1) ignore minimum_should_match clause in aggregations against rollup indices (using the rollup_search endpoint), despite the official documentation stating otherwise.
  2. # The paste creates all necessary components and has a clean up section too.
  3.  
  4. # Create sample index with data
  5. PUT cardata
  6. PUT cardata/_mapping
  7. {"dynamic":"true","properties":{"amount":{"type":"long"},"vendor":{"type":"keyword"},"currency":{"type":"keyword"},"date":{"type":"date"},"car.original_state":{"type":"keyword"}}}
  8.  
  9. ## Put data into the sample index
  10. PUT cardata/_doc/1
  11. {"amount":1000,"currency":"USD","vendor":"Ford","date":"2022-04-01T12:10:00Z","car.original_state":"CLEAN"}
  12. PUT cardata/_doc/2
  13. {"amount":5000,"name":"John Smith","currency":"USD","vendor":"Chevrolet","date":"2022-04-02T12:10:00Z","car.original_state":"WRECKED"}
  14. PUT cardata/_doc/3
  15. {"amount":7500,"currency":"HUF","vendor":"Chrysler","date":"2022-04-03T12:10:00Z","car.original_state":"CLEAN"}
  16. PUT cardata/_doc/4
  17. {"amount":65,"currency":"GBP","vendor":"Vauxhall","date":"2022-04-03T12:10:00Z","car.original_state":"WRECKED"}
  18. PUT cardata/_doc/5
  19. {"amount":17,"currency":"GBP","vendor":"Rover","date":"2022-04-03T12:10:00Z","car.original_state":"CLEAN"}
  20.  
  21. # Create and start rollup jobs, run every minute
  22. PUT _rollup/job/car_states
  23. {"index_pattern":"cardata*","rollup_index":"rollup-cardata","cron":"*/1 * * * * ?","page_size":1000,"groups":{"date_histogram":{"field":"date","fixed_interval":"1d"},"terms":{"fields":["currency","amount","car.original_state"]}},"metrics":[{"field":"amount","metrics":["min","max","sum"]}]}
  24. POST _rollup/job/car_states/_start
  25.  
  26. # Query data
  27. ## Query live index data, should return 1 doc with key GBP and sum value 17.0
  28. GET cardata/_search?size=0
  29. {"query":{"bool":{"filter":[{"term":{"currency":"GBP"}}],"should":[{"term":{"car.original_state":"CLEAN"}},{"term":{"car.original_state":"UPGRADED"}}],"minimum_should_match":1}},"aggs":{"currency_agg":{"terms":{"field":"currency"},"aggs":{"amount_agg":{"sum":{"field":"amount"}}}}}}
  30. ## Query rollup data with minimum_should_match (identical query to the live data query), but somehow returns 2 docs with key GBP and sum value 82.0
  31. ## *************** THIS IS WHERE YOU CAN SEE THAT minimum_should_match is ignored *****
  32. ## Query rollup data without minimum_should_match
  33. GET rollup-cardata/_rollup_search?size=0
  34. {"query":{"term":{"currency":"GBP"}},"aggs":{"currency_agg":{"terms":{"field":"currency"},"aggs":{"amount_agg":{"sum":{"field":"amount"}}}}}}
  35. ## Query rollup data with minimum_should_match returns the same result, even though it filters for CLEAN cars paid in GBP. (So should return what the 'live index data' search does.)
  36. ## It's IDENTICAL to the 'live index data' query.
  37. GET rollup-cardata/_rollup_search?size=0
  38. {"query":{"bool":{"filter":[{"term":{"currency":"GBP"}}],"should":[{"term":{"car.original_state":"CLEAN"}},{"term":{"car.original_state":"UPGRADED"}}],"minimum_should_match":1}},"aggs":{"currency_agg":{"terms":{"field":"currency"},"aggs":{"amount_agg":{"sum":{"field":"amount"}}}}}}
  39. ## *************** ^^^ THIS IS WHERE YOU CAN SEE THAT minimum_should_match is ignored *****
  40.  
  41. # Clean up
  42. ## Clean up rollup job
  43. POST _rollup/job/car_states/_stop
  44. DELETE _rollup/job/car_states
  45. DELETE rollup-cardata
Advertisement
RAW Paste Data Copied
Advertisement