Hey Yurii, > which is unavailable from executors.
Register it on the driver and use accumulators on executors to update the values (on the driver)? Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski "The Internals Of" Online Books <https://books.japila.pl/> Follow me on https://twitter.com/jaceklaskowski <https://twitter.com/jaceklaskowski> On Sat, Jan 16, 2021 at 2:21 PM Yuri Oleynikov (יורי אולייניקוב < yur...@gmail.com> wrote: > Hi all, > I have a spark application with Arbitrary Stateful Aggregation implemented > with FlatMapGroupsWithStateFunction. > > I want to make some statistics about incoming events inside > FlatMapGroupsWithStateFunction. > The statistics are made from some event property which on the one hand has > dynamic values but on the other hand - small finite set (thought unknown) > of values (e.g. country name). > > So I thought to register dynamic metrics inside > FlatMapGroupsWithStateFunction but as far as I understand, this requires > accessing MetricsSystem via SparkEnv.get() which is unavailable from > executors. > > Any thoughts/suggestions? > > With best regards, > Yurii > >
