Tracking Delayed Metrics Senders

If the collectionTime of a metric payload is older than 5 minutes, we term it as a delayed metric. We would want to track who is the tenant responsible for sending delayed metrics to us with the JMX tracking in Blueflood.

There is a script in the blueflood-chef repo that enables tracking of delayed metrics.
After cloning the blueflood-chef, cd into the contrib directory and run the script. The usage is as follows.
./blueflood_tenant_tracker.sh myusername track_delayed_start app00.iad.prod.bf.k1k.me

You can also do prod_all (but generally only one node is enough to display who are the delayed metrics senders)
./blueflood_tenant_tracker.sh myusername track_delayed_start prod_all

In Kibana logs you will now be able to see something like this:
[TRACKER][DELAYED METRICS] Tenant sending delayed metrics is 706456

To stop tracking
./blueflood_tenant_tracker.sh myusername track_delayed_stop app00.iad.prod.bf.k1k.me