Hi all, I have a spark application with Arbitrary Stateful Aggregation implemented with FlatMapGroupsWithStateFunction.
I want to make some statistics about incoming events inside FlatMapGroupsWithStateFunction. The statistics are made from some event property which on the one hand has dynamic values but on the other hand - small finite set (thought unknown) of values (e.g. country name). So I thought to register dynamic metrics inside FlatMapGroupsWithStateFunction but as far as I understand, this requires accessing MetricsSystem via SparkEnv.get() which is unavailable from executors. Any thoughts/suggestions? With best regards, Yurii
